Semantic alignment method and device based on text graph model, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a semantic alignment method, device, equipment and medium based on a text generation graph model.The semantic alignment method comprises the steps that a text training set is obtained, semantic enhancement is conducted on a text, and enhanced text content is obtained; extracting a global semantic feature vector corresponding to the enhanced text content; converting the enhanced text content into a mark sequence by using a word segmentation device; performing index mapping on the mark sequence to obtain a target input identifier, and generating a corresponding attention mask; encoding the target input identifier by using an encoder to obtain a mark-level vector matrix; performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector; and generating guidance information according to the fine-grained semantic feature vector and the global semantic feature vector, and obtaining aligned semantics by using the guidance information. According to the method, the accuracy of semantic alignment of the text and the image in the text generation graph model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a semantic alignment method, device, equipment and medium based on a text graph model. Background Art

[0002] Text-to-image models (TIMs) are a type of AI model that generates images from natural language text input. Current mainstream models include DALL-E, Stable Diffusion, and MidJourney. Their core architecture typically consists of two parts: a text encoder (such as the CLIP text branch and BERT variants) that converts text into semantic vectors, and an image generator (such as diffusion models and generative adversarial networks) that generates images based on these semantic vectors. These models are widely used in creative design, content creation, and visually assisted expression, generating diverse visual content based on text descriptions.

[0003] In the field of healthcare, the low accuracy of semantic alignment between text and image in the text-based graph model may cause practical problems. For example, when a doctor inputs professional descriptions such as "ground-glass nodules in the lungs with vascular clustering signs", the text-based graph model may incorrectly map "ground-glass" into a combination of multiple fuzzy pixels due to the mismatch between the text semantic granularity and the image features. The generated nodule morphology deviates from the actual clinical manifestation, resulting in inaccurate medical results.

[0004] In the fintech business field, when a user inputs "low-risk and stable financial product return fluctuation trend chart", the traditional text-to-graph model may incorrectly associate the abstract concept of "low risk" with a flat curve shape due to the low accuracy of semantic alignment between text and image.

[0005] The pre-trained text encoders relied upon by current mainstream models face multiple challenges in text understanding and cross-modal alignment, resulting in low accuracy in semantic alignment between text and images. These challenges include insufficient handling of semantic ambiguity and polysemy, a mismatch in the granularity of text and image semantics, and the fact that text semantic units are discrete tokens, while image semantics are continuous pixel features. This creates a "semantic granularity mismatch" during encoder alignment, which can easily lead to generated content deviating from expectations. Abstract concepts in text lack clear visual anchors, and the reliance on weakly supervised alignment can easily lead to ambiguity, resulting in low accuracy in semantic alignment between text and image in text-to-image models. Summary of the Invention

[0006] The present invention provides a semantic alignment method, apparatus, device and medium based on a text-graph model to solve the problem of low accuracy of semantic alignment between text and image in the text-graph model.

[0007] First, a semantic alignment method based on a text graph model is provided, including:

[0008] Obtaining a text data training set, performing visual semantic enhancement on original text content in the text data training set to obtain enhanced text content, and extracting a global semantic feature vector corresponding to the enhanced text content;

[0009] Converting the enhanced text content into a token sequence using a word segmenter in a preset image-text feature joint encoding model;

[0010] Performing index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and generating a corresponding attention mask according to the target input identifier;

[0011] Using the text encoder in the image-text feature joint encoding model to embed the target input identifier to obtain a tag-level vector matrix;

[0012] performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content;

[0013] Guidance information of a preset text-graph model is generated according to the fine-grained semantic feature vector and the global semantic feature vector, and the text and the image in the text-graph model are aligned using the guidance information to obtain alignment semantics.

[0014] In a second aspect, a semantic alignment device based on a cultural graph model is provided, comprising:

[0015] A global semantic feature vector extraction module is used to obtain a text data training set, perform visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content, and extract a global semantic feature vector corresponding to the enhanced text content;

[0016] A token sequence conversion module, configured to convert the enhanced text content into a token sequence using a word segmenter in a preset graphic-text feature joint encoding model;

[0017] An attention mask generation module is used to perform index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and to generate an attention mask corresponding to the target input identifier;

[0018] A tag-level vector matrix acquisition module is used to embed the target input identifier using the text encoder in the image-text feature joint encoding model to obtain a tag-level vector matrix;

[0019] A fine-grained semantic feature vector analysis module, configured to perform self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content;

[0020] The semantic alignment module is used to generate guidance information of a preset text-graph model based on the fine-grained semantic feature vector and the global semantic feature vector, and use the guidance information to align the text and image in the text-graph model to obtain aligned semantics.

[0021] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned semantic alignment method based on the literary graph model are implemented.

[0022] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the semantic alignment method based on the cultural graph model are implemented.

[0023] In the scheme implemented by the semantic alignment method, device, equipment and medium based on the above-mentioned text-graph model, a text data training set can be obtained through the client, the original text content in the text data training set is visually semantically enhanced to obtain enhanced text content, and the global semantic feature vector corresponding to the enhanced text content is extracted; the enhanced text content is converted into a tag sequence by using a word segmenter in a preset text-graph feature joint coding model; the tag sequence is indexed and mapped to obtain a target input identifier of the text-graph feature joint coding model, and a corresponding attention mask is generated according to the target input identifier; the target input identifier is embedded by using a text encoder in the text-graph feature joint coding model. The present invention uses a method for encoding to obtain a tag-level vector matrix; performs self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content; generates guidance information of a preset text-graph model according to the fine-grained semantic feature vector and the global semantic feature vector, and uses the guidance information to align the text and image in the text-graph model to obtain alignment semantics. In the present invention, there is no need to make structural changes to the existing text-graph framework. Only a slight adjustment is made to the implementation method of the text encoder to reuse the original training process and training loss, etc., which reduces the threshold for technical application and transformation costs. With the powerful image-text matching capability of CLIP and the excellent reasoning capability of Gemma 3, the semantic understanding and cross-modal alignment effect of the text encoder can be effectively improved. By introducing fine-grained semantic encoding and reasoning models to enhance text, problems such as semantic ambiguity and granularity misalignment can be accurately handled, the semantic alignment capability of the text-graph model can be significantly improved, and images that are more in line with the text intent can be generated. The problem of low accuracy of semantic alignment between text and image in the text-graph model can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 1 is a schematic diagram of an application environment of a semantic alignment method based on a cultural graph model according to an embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of a semantic alignment method based on a text graph model according to an embodiment of the present invention;

[0027] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S2;

[0028] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S3;

[0029] Figure 5 1 is a structural diagram of a semantic alignment device based on a cultural graph model according to an embodiment of the present invention;

[0030] Figure 6 is a structural diagram of a computer device in one embodiment of the present invention;

[0031] Figure 7 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] The semantic alignment method based on the text graph model provided by the embodiment of the present invention can be applied in the following situations: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain a text data training set through the client, perform visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content, and extract the global semantic feature vector corresponding to the enhanced text content; use the word segmenter in the preset graphic-text feature joint coding model to convert the enhanced text content into a tag sequence; perform index mapping on the tag sequence to obtain the target input identifier of the graphic-text feature joint coding model, and generate a corresponding attention mask according to the target input identifier; use the text encoder in the graphic-text feature joint coding model to embed the target input identifier to obtain a tag-level vector matrix; according to the attention The mask performs self-attention processing on the tag-level vector matrix to obtain a fine-grained semantic feature vector corresponding to the enhanced text content; guidance information for a preset text graph model is generated based on the fine-grained semantic feature vector and the global semantic feature vector; the guidance information is used to align the text and image in the text graph model to obtain alignment semantics, and the alignment semantics are fed back to the client. In this invention, no structural changes are required to the existing text graph framework; only minor adjustments are made to the implementation of the text encoder to reuse the original training process and training loss, reducing the technical application threshold and transformation costs. Leveraging the powerful image-text matching capabilities of CLIP and the excellent reasoning capabilities of Gemma 3, the semantic understanding and cross-modal alignment of the text encoder can be effectively improved. By introducing fine-grained semantic encoding and reasoning models to enhance text, problems such as semantic ambiguity and granularity misalignment can be accurately handled, significantly improving the semantic alignment capabilities of the text graph model and generating images that better align with the text intent. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0034] See also Figure 2 As shown, Figure 2 A flowchart of a semantic alignment method based on a text graph model provided in an embodiment of the present invention includes the following steps:

[0035] S1. Obtain a text data training set, perform visual semantic enhancement on original text content in the text data training set to obtain enhanced text content, and extract a global semantic feature vector corresponding to the enhanced text content.

[0036] In an embodiment of the present invention, the text data training set is a text collection used to train the text-graph model, which contains a large number of text descriptions corresponding to image annotations (such as natural language sentences, phrases, etc.), and is the basic input for the model to learn the mapping relationship between text semantics and image vision.

[0037] In detail, obtaining a text data training set usually involves collecting natural language descriptions corresponding to image data to form a set for text-based image model training. These text descriptions cover a variety of semantic scenarios, object features, and visual attributes.

[0038] In an embodiment of the present invention, the original text content is the unprocessed initial text information in the text data training set; the enhanced text content refers to the text content with richer semantics and more specific visual details obtained after the original text in the text data training set is rewritten by an inference model (such as Gemma 3) based on a preset prompt template (such as "enhance the description of design sense, color richness, and entity richness, and output detailed visual copy").

[0039] In an embodiment of the present invention, performing visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content includes:

[0040] Generating a prompt word template corresponding to the original text content according to preset visual semantic enhancement requirements;

[0041] Analyzing the enhanced content between the prompt word template and the original text content using a preset reasoning model;

[0042] The original text content is visually and semantically rewritten according to the enhanced content to generate enhanced text content.

[0043] In detail, the preset visual semantic enhancement requirements refer to the pre-set visual semantic directions and goals that need to be enhanced for the original text content before the text-to-graph model is trained. For example, it explicitly requires improving the specific visual attributes such as the sense of design, color richness, richness of entity details, and scene hierarchy in the text description; the prompt word template is a text framework or structure used to guide the inference model to rewrite the original text content so that it contains richer visual semantic information.

[0044] Specifically, a prompt word template corresponding to the original text is generated according to the preset visual semantic enhancement requirements. Usually, the visual attributes that need to be enhanced in the requirements are first clarified, and then these attributes are converted into specific guiding instruction sentences to form a structured prompt framework, so that the reasoning model can rewrite the original text in a targeted manner based on the template, thereby generating text content that meets the enhancement requirements.

[0045] Furthermore, the preset reasoning model refers to a pre-trained model (such as Gemma 3) for performing text semantic enhancement tasks, which has the ability to understand text semantics and generate text that conforms to specific rules. Enhanced content refers to the visual semantic information that needs to be supplemented or strengthened after the reasoning model analyzes the original text based on the prompt word template.

[0046] Furthermore, when using the preset reasoning model to analyze the prompt word template and the original text, the model first identifies the preset visual semantic enhancement direction in the template (such as color, entity details, scene hierarchy, etc.), and then compares it with the original text content to locate the missing or weak visual semantic elements, and generate a structured analysis result containing specific enhancement points. Subsequently, based on the analysis results, the model semantically expands and fills in details of the original text according to the instruction framework of the prompt word template, completes the visual semantic rewriting, and finally outputs enhanced text content containing rich visual information.

[0047] For example, in a medical scenario, the preset visual semantic enhancement requirement is to supplement the detailed features of medical images. For the original text "Lung CT shows shadows", a prompt word template is generated according to the demand, "Please supplement the description of medical imaging features such as the shape, boundary clarity, density, and relationship with surrounding tissues of the shadows." Using the preset medical field reasoning model, the difference between the template and the original text is analyzed, and it is determined that the content to be enhanced is details such as shape and boundaries. Finally, the model rewrites the original text according to the enhanced content to "Lung CT shows a circular, unclear boundary, medium-density shadow, and adhesion to the pulmonary artery branch" to obtain the enhanced text content.

[0048] For example, in a financial scenario, the preset visual semantic enhancement requirement is to supplement the financial data with trend and risk details. For the original text "A company's quarterly revenue growth," a prompt template is generated: "Please supplement the revenue growth rate, quarter-on-quarter or year-on-year data, growth drivers, and potential risk analysis." Using a preset financial domain reasoning model, the template and original text are analyzed to identify the content that needs to be enhanced, such as the growth rate value and comparative data. The model rewrites the original text to "A company's quarterly revenue increased by 15%, an increase of 8% quarter-on-quarter, mainly due to the expansion of new businesses, but there is a risk of subsequent weak growth due to intensified market competition," creating enhanced text content.

[0049] In the embodiment of the present invention, the global semantic feature vector refers to a vector that can represent the overall semantic information of the enhanced text, which is extracted after encoding the text content enhanced by the Prompt template through the inference model using a text encoder.

[0050] Specifically, after obtaining semantically enriched text (i.e., enhanced text content), it is input into the inference model. The model generates an embedding that represents the overall semantics of the text through processing. This embedding is the global semantic feature vector, which is used to guide the semantic alignment of the text graph.

[0051] S2. Convert the enhanced text content into a tag sequence using a word segmenter in a preset graphic-text feature joint encoding model.

[0052] In an embodiment of the present invention, the preset joint encoding model of image and text features is the CLIP model (which has image and text matching capabilities and can achieve cross-modal alignment); the word segmenter is the Byte-Level BPE word segmenter built into CLIP (used to split text into tokens such as subword units); the tag sequence is a sequence containing special tokens (such as CLS and SEP) converted from the text after being processed by the word segmenter.

[0053] In the embodiment of the present invention, referring to Figure 3 As shown, the method of converting the enhanced text content into a tag sequence by using a word segmenter in a preset graphic-text feature joint coding model includes:

[0054] S31, performing standardization processing on the enhanced text content to obtain a standardized text sequence;

[0055] S32, using the word segmenter in the preset graphic-text feature joint coding model to split the standardized text sequence into semantic units to obtain a subword sequence;

[0056] S33 , inserting preset target tags at the beginning and end of the subword sequence respectively, and using the subword sequence after the target tags are inserted as the tag sequence corresponding to the enhanced text content.

[0057] In detail, the standardized text sequence refers to the text sequence obtained after preprocessing the enhanced text content, such as removing special symbols and unifying the case; the subword sequence refers to the semantic unit splitting of the standardized text sequence through the Byte-LevelBPE word segmenter built into CLIP (such as splitting "design sense" into "design" and "sense"); the preset target tag refers to the special token in the CLIP model used to identify the beginning and end of the text sequence (such as CLS and SEP, which may vary in different versions), which is used to help the model identify the starting and ending positions of the text sequence.

[0058] Specifically, the enhanced text content is first standardized by removing special symbols, unifying upper and lower case, etc. to obtain a text sequence that meets the requirements. The Byte-Level BPE word segmenter built into the CLIP model is then used to split the standardized text sequence into subword sequences according to semantic units. Special tokens preset by the CLIP model (such as CLS and SEP, which may differ in different versions) are inserted at the beginning and end of the subword sequence as target tags to form a tag sequence containing special tags and subword units.

[0059] For example, in a medical scenario, the text "Differentiation of benign and malignant ground-glass nodules in the lungs" is first standardized to "ground-glass nodules in the lungs, with clear boundaries, uniform density, 8mm in diameter, and a high probability of benignity." The CLIP tokenizer is then used to break the text into sub-word sequences such as "lung," "ground," "glass," and "nodule." The CLS and SEP tokens are then inserted at the beginning and end to form a token sequence.

[0060] For example, in a financial scenario, the text of a "Small and Micro Enterprise Credit Assessment Report" is first standardized to read "Enterprise established 3 years, annual revenue of 5 million, no overdue records, fixed assets accounting for 40%, and strong debt repayment ability." This is then split into sub-word sequences such as "enterprise," "established," "3 years," and "revenue" using the CLIP tokenizer. CLS and SEP tags are then inserted at the beginning and end to form a token sequence.

[0061] S3. Perform index mapping on the tag sequence to obtain the target input identifier of the image-text feature joint encoding model, and generate a corresponding attention mask according to the target input identifier.

[0062] In an embodiment of the present invention, the target input identifier maps each token in the token sequence (including subword units and special tokens) to a unique integer ID recognizable by the model (such as the index value of the corresponding token in CLIP) to form a numerical sequence of the input model.

[0063] In the embodiment of the present invention, referring to Figure 4 As shown, the index mapping of the tag sequence to obtain the target input identifier of the image-text feature joint encoding model includes:

[0064] S41, identifying the index position corresponding to each tag in the tag sequence according to a preset vocabulary;

[0065] S42. Determine an index value corresponding to each tag in the tag sequence according to the index position, and replace the tag sequence with an initial integer identification sequence according to the index value;

[0066] S43, using a preset position coding strategy to add a position offset to each index value in the initial integer identifier sequence to obtain a target integer identifier sequence;

[0067] S44: Perform dimension alignment processing on the target integer identifier sequence to obtain a target input identifier of the image-text feature joint encoding model.

[0068] In detail, the preset vocabulary is a set of all valid tags (such as subword units, special tokens) pre-constructed by the joint encoding model of image and text features, and each tag corresponds to a unique identifier; the index position refers to the serial number of each tag in the preset vocabulary after being arranged in a fixed order; the index value is the unique integer number corresponding to the tag in the preset vocabulary; the initial integer identification sequence is a continuous integer sequence formed by replacing each tag in the tag sequence with its index value.

[0069] Specifically, based on a preset vocabulary (tokens have been arranged in a fixed order and assigned index positions), each token in the token sequence is traversed, and its corresponding index position (that is, the arrangement sequence of the token in the vocabulary) is found through the vocabulary. The corresponding index value is determined according to the index position (usually consistent with the index position value, depending on the model numbering rule), and each token is replaced with its index value to form an initial integer identification sequence composed of consecutive integers, completing the conversion of text tokens to numerical sequences.

[0070] Furthermore, the preset position encoding strategy refers to a pre-set strategy for adding a position offset to the index value (such as a fixed formula, a learnable parameter, etc., used to reflect the relative or absolute position information of the tag in the sequence); the target integer identifier sequence refers to a new integer sequence obtained by adding a position offset to each index value in the initial integer identifier sequence according to this rule, which not only contains the index information of the tag itself, but also incorporates the position information of the tag in the sequence.

[0071] Furthermore, based on the preset position encoding strategy, an offset corresponding to the position in the sequence is added to each index value in the initial integer identifier sequence one by one to generate a target integer identifier sequence containing position information; then the sequence is adjusted to the input dimension required by the joint encoding model of image and text features (such as specifying batch_size and sequence_length), and the dimension alignment processing is completed, and finally the target input identifier that can be directly received by the model is obtained.

[0072] In an embodiment of the present invention, the attention mask is a binary mask of the same length as the target input identifier, which is used to mark the positions of valid tags in the sequence (valid positions are 1 and padding positions are 0), so that the model ignores the padding tags when calculating attention and focuses on the real input content.

[0073] In detail, when generating the attention mask corresponding to the target input identifier, first identify the position of the valid token (non-filling token) in the sequence, and then create a binary sequence of the same length as the target input identifier, set the valid token position to 1 and the filling position to 0 to form an attention mask. At the same time, it needs to be organized into the format required by the model, usually converted into a tensor with a shape of (batch_size, sequence_length) to ensure that the model can correctly identify the valid position and filling position when calculating the attention, and realize differentiated processing of different parts of the sequence.

[0074] S4. Use the text encoder in the image-text feature joint encoding model to embed the target input identifier to obtain a tag-level vector matrix.

[0075] In an embodiment of the present invention, the text encoder is a module (such as a Transformer structure) used to process text input in a joint encoding model of image and text features, and its function is to semantically encode the target input identifier (such as words and characters in the text); the token-level vector matrix is ​​a matrix generated by the text encoder after independently encoding each input token (token), in which each row corresponds to a semantic vector of a token, containing the contextual semantic information and position information of the token.

[0076] In the embodiment of the present invention, the embedding encoding of the target input identifier using the text encoder in the image-text feature joint encoding model to obtain a tag-level vector matrix includes:

[0077] Performing identity mapping on the target input identity using the embedding layer in the image-text feature joint encoding model to obtain an identity vector sequence;

[0078] Performing position encoding on each identification vector in the identification vector sequence to obtain an encoding vector;

[0079] Semantic features are extracted from the encoding vector to obtain a tag-level vector matrix.

[0080] In detail, the embedding layer is the basic layer in the joint encoding model of image and text features, which is used to map discrete target input identifiers into continuous vectors. The mapping from identifiers to vectors is achieved through a pre-trained or randomly initialized embedding matrix. The identifier vector sequence is a vector sequence generated after the embedding layer maps the target input identifiers one by one. Each vector corresponds to the initial semantic representation of an identifier.

[0081] Specifically, when using the embedding layer of the joint image-text feature encoding model for identity mapping, the target input identity is first converted into a discrete identity that can be recognized by the model. Through the embedding matrix predefined or trained in the embedding layer, each discrete identity is mapped into a continuous vector of fixed dimension (such as a vector of dimension d), and combined into an identity vector sequence in the identity order.

[0082] Furthermore, the encoding vector is a result obtained by superimposing a position information vector generated by a position encoding mechanism on each identification vector in the identification vector sequence.

[0083] Furthermore, when position encoding is performed on the sequence of identification vectors, a corresponding position information vector is generated for each identification vector through a position encoding algorithm (such as sine-cosine function or learnable parameters), which is then added element-by-element to the original identification vector to obtain an encoding vector, so that each vector contains semantic and position information; the encoding vector is processed in multiple layers using a semantic feature extraction module, while capturing the contextual dependencies between tags, and the features are further refined through a feedforward neural network to output a tag-level vector matrix, in which each row vector is a deep semantic representation of the corresponding tag.

[0084] S5. Perform self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content.

[0085] In an embodiment of the present invention, the fine-grained semantic feature vector is a vector obtained by performing self-attention processing on the tag-level vector matrix through an attention mask. It focuses on the semantic information of key tags in the text, suppresses irrelevant or noise information by dynamically calculating the attention weights between each tag, enhances the semantic association between tags and the extraction of detailed features, thereby forming a more accurate and detailed text semantic representation.

[0086] In an embodiment of the present invention, performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content includes:

[0087] Performing a linear transformation on the tag-level vector matrix to obtain a three-element matrix;

[0088] Using the attention mask, position-marking is performed on the dot product results of the query matrix and the key matrix in the three-element matrix to obtain a marking weight matrix;

[0089] Performing weighted aggregation on the value matrices in the three-element matrix according to the label weight matrix to obtain a semantic matrix;

[0090] Performing self-attention feature association mapping on the semantic matrix to obtain relational features;

[0091] The relationship features are weightedly fused with the semantic matrix to obtain a fused feature matrix, and the fused feature matrix is ​​globally pooled to obtain a fine-grained semantic feature vector.

[0092] In detail, the three-element matrix is ​​the three matrices obtained by linearly transforming the tag-level vector matrix (by multiplying the weight matrix), corresponding to the query matrix (Query), key matrix (Key), and value matrix (Value) in the self-attention mechanism; the dot product result is a numerical matrix obtained by multiplying the corresponding elements of the query matrix and the key matrix and summing them up, reflecting the degree of semantic association between the query vector and the key vector; the tag weight matrix is ​​a matrix generated by using the attention mask to positionally mark the above dot product results (such as masking the association scores of invalid positions, retaining or enhancing the scores of valid positions).

[0093] Specifically, when performing a linear transformation on the tag-level vector matrix, three independent weight matrices are multiplied with the matrix to generate a three-element matrix consisting of a query matrix, a key matrix, and a value matrix. The dot product of the query matrix and the key matrix is ​​calculated to obtain the original score matrix. The attention mask is used to mask the scores of invalid positions in the matrix, and the scores of valid positions are retained to form a tag weight matrix.

[0094] Furthermore, the semantic matrix is ​​a matrix obtained by weighted aggregation of the value matrix in the three-element matrix according to the tag weight matrix (by multiplying the weight matrix by the value matrix and summing them up), which represents the semantic information of the text tags after being filtered by the attention weights; the relationship feature is a feature obtained by performing self-attention feature association mapping on the semantic matrix. Specifically, in the self-attention mechanism, the attention weights between each tag vector in the semantic matrix are calculated, and the tag information is re-aggregated based on the weights to generate a vector or matrix representing the semantic dependency relationship, long-distance association and other features between tags within the text; the fusion feature matrix is ​​a matrix obtained by fusing the relationship feature with the semantic matrix according to the weights (such as weighted addition or linear transformation after splicing), which integrates semantic information and semantic relationships.

[0095] Furthermore, the value matrix is ​​weighted and aggregated according to the tag weight matrix, and the weighted summation of the semantic information of each tag is achieved through matrix multiplication of the weight matrix and the value matrix to obtain the semantic matrix; self-attention feature association mapping is performed on the semantic matrix, and the attention scores between the tags are calculated and the association information is aggregated to generate relational features that represent the semantic relationship of the tags; the relational features are fused with the semantic matrix according to the set weights (such as learnable parameters) (such as element-by-element addition or linear transformation after splicing) to form a fused feature matrix, and then the spatial dimensions of the fused feature matrix are compressed through global pooling operations (such as average pooling and maximum pooling) to extract fine-grained semantic feature vectors.

[0096] S6. Generate guidance information of a preset text-graph model according to the fine-grained semantic feature vector and the global semantic feature vector, and use the guidance information to align text and images in the text-graph model to obtain alignment semantics.

[0097] In an embodiment of the present invention, the preset text-based graph model refers to a model based on an existing framework (a mainstream model without changing the structure) that can receive external input guidance information to generate an image; the guidance information is a feature representation formed by the fusion or combination of fine-grained semantic feature vectors and global semantic feature vectors, and is used to guide the model to align the semantics of text and image.

[0098] In an embodiment of the present invention, the step of generating guidance information of a preset text graph model based on the fine-grained semantic feature vector and the global semantic feature vector includes:

[0099] Performing feature normalization processing on the fine-grained semantic feature vector and the global semantic feature vector to obtain normalized fine-grained features and normalized global features;

[0100] Identifying local semantic units corresponding to the standardized fine-grained features according to preset visual elements, and extracting abstract semantic information from the standardized global features;

[0101] Performing weight association on the local semantic unit and the abstract semantic information to obtain a semantic association tensor;

[0102] Performing weighted fusion on the standardized fine-grained features and the standardized global features according to the semantic association tensor to obtain a multi-dimensional fused feature vector;

[0103] The multi-dimensional fusion feature vector is feature mapped using the embedding space in the preset text graph model to obtain semantic space mapping features, and the semantic space mapping features are used as guidance information.

[0104] In detail, standardized fine-grained features are feature vectors obtained after normalization and other standardization processing on fine-grained semantic feature vectors, so that they meet the model input requirements or optimize the feature distribution; standardized global features are feature vectors obtained after the same standardization processing on global semantic feature vectors; preset visual elements refer to pre-defined basic image components used to map text semantics (such as color, shape, texture, object category, etc.); local semantic units are specific semantic units corresponding to standardized fine-grained features identified by visual elements (such as "dark blue", "round object" and other concrete local semantics); abstract semantic information is abstract concepts extracted from standardized global features that are difficult to directly correspond to visual anchor points in the text (such as emotions or abstract states such as "happiness" and "loneliness").

[0105] Specifically, the fine-grained semantic feature vectors and the global semantic feature vectors are numerically transformed through normalization, regularization and other methods, and their distribution, scale and mean variance are adjusted to obtain standardized fine-grained features and standardized global features; using defined visual elements (such as color, shape, texture, etc.) as a reference, the corresponding specific semantic units (such as concrete descriptions such as "red cube") are matched and located in the standardized fine-grained features, and at the same time, abstract semantic information such as emotions and abstract concepts that are difficult to directly visualize (such as "warm atmosphere" and "science fiction feeling") are extracted from the standardized global features.

[0106] Furthermore, the semantic association tensor is a multidimensional array that establishes an association between local semantic units and abstract semantic information by assigning weights to the two; the multidimensional fused feature vector is a feature vector containing local concrete and global abstract semantics, which is formed by weighted fusion of standardized fine-grained features and standardized global features based on the weight allocation of the semantic association tensor; the embedding space is a mathematical space used in the preset text-graph model to map input features into low-dimensional dense vectors, thereby realizing a unified representation of cross-modal semantics; the semantic space mapping feature is the feature representation obtained after the multidimensional fused feature vector is mapped into the embedding space, which serves as the key information to guide the model to align the semantics of text and image.

[0107] Furthermore, by setting weight coefficients, the association relationship between local semantic units and abstract semantic information is established, and the semantic association tensor is stored in tensor form. The standardized fine-grained features and the standardized global features are weightedly summed according to the weights in the tensor to achieve fusion, and a feature vector containing multi-dimensional semantic information is obtained; the multi-dimensional fused feature vector is input into the embedding space of the preset text-graph model, and the mapping rules of the embedding space (such as linear transformation, nonlinear mapping) are used to convert it into a specific semantic space to generate semantic space mapping features. This feature contains text semantic information and serves as key information to guide the text-graph model to align text and image.

[0108] In the embodiment of the present invention, the alignment semantics refers to the correspondence between text semantics (such as discrete token meanings, abstract concepts) and image semantics (such as pixel features, object layout) established by the model through guidance information.

[0109] In detail, the guidance information composed of the fine-grained semantic feature vector corresponding to the enhanced text content and the global semantic feature vector is used. Through the word segmentation, Token mapping and context modeling of the Transformer structure of the CLIP text encoder, a token-level embedding matrix containing position information is output, and the embedding vectors of valid text tokens are extracted. Combined with the multimodal reasoning capability of the Gemma 3 text encoder, the semantics of discrete tokens in the text (such as words and phrases) and the continuous pixel features of the image (such as color and object layout) are mapped probabilistically through model learning, so that the text embedding space and the image semantic space are established in correspondence, and the semantic alignment of text and image is achieved and the aligned semantics is obtained.

[0110] Specifically, the input text is processed using CLIP's ViT-B / 32 text encoder, which outputs an embedding of dimension 512. The embedding vector corresponding to each token is obtained, and the influence of special tokens (such as CLS) is removed through slicing, retaining the embedding of valid text tokens. This process is implemented based on the transformer library. The processed text features are combined with guidance information, and the semantic information of the text is matched with the pixel features of the image by establishing an association between text features and image features in the text-based graph model. The guidance information is used to constrain and guide the generation process of the text-based graph model, so that when the model generates an image, it can adjust the generation strategy according to the semantic information of the text, achieve semantic alignment between text and image, and obtain aligned semantics.

[0111] It can be seen that in the above scheme, there is no need to make structural changes to the existing text-graph framework. Only a slight adjustment is made to the implementation method of the text encoder, and the original training process and training loss can be reused, which reduces the threshold for technical application and the cost of transformation. With the help of CLIP's powerful image-text matching capabilities and Gemma3's excellent reasoning capabilities, the semantic understanding and cross-modal alignment effects of the text encoder can be effectively improved. By introducing fine-grained semantic encoding and reasoning models to enhance text, it can accurately handle problems such as semantic ambiguity and granularity dislocation, significantly improve the semantic alignment capability of the text-graph model, and generate images that are more in line with the text intent.

[0112] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] In one embodiment, a semantic alignment device based on a text graph model is provided. The semantic alignment device based on a text graph model corresponds one-to-one to the semantic alignment method based on a text graph model in the above embodiment. Figure 5 As shown, the semantic alignment device 100 based on the text graph model includes a global semantic feature vector extraction module 101, a tag sequence conversion module 102, an attention mask generation module 103, a tag-level vector matrix acquisition module 104, a fine-grained semantic feature vector analysis module 105, and a semantic alignment module 106. The functional modules are described in detail as follows:

[0114] A global semantic feature vector extraction module 101 is used to obtain a text data training set, perform visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content, and extract a global semantic feature vector corresponding to the enhanced text content;

[0115] A token sequence conversion module 102 is configured to convert the enhanced text content into a token sequence using a word segmenter in a preset graphic-text feature joint coding model;

[0116] An attention mask generation module 103 is configured to perform index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and generate a corresponding attention mask according to the target input identifier;

[0117] The tag-level vector matrix acquisition module 104 is configured to embed the target input identifier using the text encoder in the image-text feature joint encoding model to obtain a tag-level vector matrix;

[0118] A fine-grained semantic feature vector analysis module 105 is configured to perform self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content;

[0119] The semantic alignment module 105 is configured to generate guidance information of a preset text-graph model based on the fine-grained semantic feature vector and the global semantic feature vector, and align text and images in the text-graph model using the guidance information to obtain aligned semantics.

[0120] In one embodiment, the global semantic feature vector extraction module 101, when performing visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content, is configured to:

[0121] Generating a prompt word template corresponding to the original text content according to preset visual semantic enhancement requirements;

[0122] Analyzing the enhanced content between the prompt word template and the original text content using a preset reasoning model;

[0123] The original text content is visually and semantically rewritten according to the enhanced content to generate enhanced text content.

[0124] In one embodiment, the token sequence conversion module 102, when converting the enhanced text content into a token sequence using a word segmenter in a preset graphic-text feature joint encoding model, is configured to:

[0125] Standardizing the enhanced text content to obtain a standardized text sequence;

[0126] Using the word segmenter in the preset graphic-text feature joint encoding model to split the standardized text sequence into semantic units to obtain a subword sequence;

[0127] Preset target tags are inserted at the beginning and end of the subword sequence respectively, and the subword sequence after the target tags are inserted is used as the tag sequence corresponding to the enhanced text content.

[0128] In one embodiment, the attention mask generation module 103, when performing index mapping on the tag sequence to obtain the target input identifier of the image-text feature joint encoding model, is configured to:

[0129] Identifying an index position corresponding to each token in the token sequence according to a preset vocabulary;

[0130] Determine an index value corresponding to each tag in the tag sequence according to the index position, and replace the tag sequence with an initial integer identification sequence according to the index value;

[0131] Using a preset position coding strategy, adding a position offset to each index value in the initial integer identifier sequence to obtain a target integer identifier sequence;

[0132] The target integer identifier sequence is dimensionally aligned to obtain the target input identifier of the image-text feature joint encoding model.

[0133] In one embodiment, the token-level vector matrix acquisition module 104, when performing embedding encoding on the target input identifier using the text encoder in the image-text feature joint encoding model to obtain the token-level vector matrix, is configured to:

[0134] Performing identity mapping on the target input identity using the embedding layer in the image-text feature joint encoding model to obtain an identity vector sequence;

[0135] Performing position encoding on each identification vector in the identification vector sequence to obtain an encoding vector;

[0136] Semantic features are extracted from the encoding vector to obtain a tag-level vector matrix.

[0137] In one embodiment, the fine-grained semantic feature vector analysis module 105, when performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain the fine-grained semantic feature vector corresponding to the enhanced text content, is configured to:

[0138] Performing a linear transformation on the tag-level vector matrix to obtain a three-element matrix;

[0139] Using the attention mask, position-marking is performed on the dot product results of the query matrix and the key matrix in the three-element matrix to obtain a marking weight matrix;

[0140] Performing weighted aggregation on the value matrices in the three-element matrix according to the label weight matrix to obtain a semantic matrix;

[0141] Performing self-attention feature association mapping on the semantic matrix to obtain relational features;

[0142] The relationship features are weightedly fused with the semantic matrix to obtain a fused feature matrix, and the fused feature matrix is ​​globally pooled to obtain a fine-grained semantic feature vector.

[0143] In one embodiment, the semantic alignment module 106 , when executing the guidance information for generating a preset text graph model based on the fine-grained semantic feature vector and the global semantic feature vector, is configured to:

[0144] Performing feature normalization processing on the fine-grained semantic feature vector and the global semantic feature vector to obtain normalized fine-grained features and normalized global features;

[0145] Identifying local semantic units corresponding to the standardized fine-grained features according to preset visual elements, and extracting abstract semantic information from the standardized global features;

[0146] Performing weight association on the local semantic unit and the abstract semantic information to obtain a semantic association tensor;

[0147] Performing weighted fusion on the standardized fine-grained features and the standardized global features according to the semantic association tensor to obtain a multi-dimensional fused feature vector;

[0148] The multi-dimensional fusion feature vector is feature mapped using the embedding space in the preset text graph model to obtain semantic space mapping features, and the semantic space mapping features are used as guidance information.

[0149] The present invention provides a semantic alignment device based on a text-graph model. It does not require structural changes to the existing text-graph framework. Only slight adjustments are made to the implementation method of the text encoder to reuse the original training process and training loss, thereby lowering the threshold for technical application and the cost of transformation. Leveraging the powerful image-text matching capabilities of CLIP and the excellent reasoning capabilities of Gemma 3, the semantic understanding and cross-modal alignment effects of the text encoder can be effectively improved. By introducing fine-grained semantic encoding and reasoning models to enhance text, the device can accurately handle problems such as semantic ambiguity and granularity misalignment, significantly improving the semantic alignment capability of the text-graph model and generating images that better fit the text intent.

[0150] For the specific definition of the semantic alignment device based on the cultural graph model, please refer to the definition of the semantic alignment method based on the cultural graph model above, which will not be repeated here. The various modules in the above-mentioned semantic alignment device based on the cultural graph model can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a semantic alignment method based on a text graph model.

[0152] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a semantic alignment method based on a text graph model.

[0153] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0154] Obtaining a text data training set, performing visual semantic enhancement on original text content in the text data training set to obtain enhanced text content, and extracting a global semantic feature vector corresponding to the enhanced text content;

[0155] Converting the enhanced text content into a token sequence using a word segmenter in a preset image-text feature joint encoding model;

[0156] Performing index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and generating a corresponding attention mask according to the target input identifier;

[0157] Using the text encoder in the image-text feature joint encoding model to embed the target input identifier to obtain a tag-level vector matrix;

[0158] performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content;

[0159] Guidance information of a preset text-graph model is generated according to the fine-grained semantic feature vector and the global semantic feature vector, and the text and the image in the text-graph model are aligned using the guidance information to obtain alignment semantics.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0161] Obtaining a text data training set, performing visual semantic enhancement on original text content in the text data training set to obtain enhanced text content, and extracting a global semantic feature vector corresponding to the enhanced text content;

[0162] Converting the enhanced text content into a token sequence using a word segmenter in a preset image-text feature joint encoding model;

[0163] Performing index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and generating a corresponding attention mask based on the generated target input identifier;

[0164] Using the text encoder in the image-text feature joint encoding model to embed the target input identifier to obtain a tag-level vector matrix;

[0165] performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content;

[0166] Guidance information of a preset text-graph model is generated according to the fine-grained semantic feature vector and the global semantic feature vector, and the text and the image in the text-graph model are aligned using the guidance information to obtain alignment semantics.

[0167] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0168] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0169] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0170] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

[0171] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A semantic alignment method based on a text graph model, characterized in that: include: Obtaining a text data training set, performing visual semantic enhancement on original text content in the text data training set to obtain enhanced text content, and extracting a global semantic feature vector corresponding to the enhanced text content; Converting the enhanced text content into a token sequence using a word segmenter in a preset image-text feature joint encoding model; Performing index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and generating a corresponding attention mask according to the target input identifier; Using the text encoder in the image-text feature joint encoding model to embed the target input identifier to obtain a tag-level vector matrix; performing self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content; Guidance information of a preset text-graph model is generated according to the fine-grained semantic feature vector and the global semantic feature vector, and the text and the image in the text-graph model are aligned using the guidance information to obtain alignment semantics.

2. The semantic alignment method based on the cultural graph model according to claim 1, characterized in that: The performing visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content includes: Generating a prompt word template corresponding to the original text content according to preset visual semantic enhancement requirements; Analyzing the enhanced content between the prompt word template and the original text content using a preset reasoning model; The original text content is visually and semantically rewritten according to the enhanced content to generate enhanced text content.

3. The semantic alignment method based on the cultural graph model according to claim 1, characterized in that: The method of converting the enhanced text content into a tag sequence by using a word segmenter in a preset graphic-text feature joint coding model includes: Standardizing the enhanced text content to obtain a standardized text sequence; Using the word segmenter in the preset graphic-text feature joint encoding model to split the standardized text sequence into semantic units to obtain a subword sequence; Preset target tags are inserted at the beginning and end of the subword sequence respectively, and the subword sequence after the target tags are inserted is used as the tag sequence corresponding to the enhanced text content.

4. The semantic alignment method based on the cultural graph model according to claim 1, wherein: The index mapping of the tag sequence to obtain a target input identifier of the image-text feature joint encoding model includes: Identifying an index position corresponding to each token in the token sequence according to a preset vocabulary; Determine an index value corresponding to each tag in the tag sequence according to the index position, and replace the tag sequence with an initial integer identification sequence according to the index value; Using a preset position coding strategy, adding a position offset to each index value in the initial integer identifier sequence to obtain a target integer identifier sequence; The target integer identifier sequence is dimensionally aligned to obtain the target input identifier of the image-text feature joint encoding model.

5. The semantic alignment method based on the cultural graph model according to claim 1, wherein: The method of embedding the target input identifier using the text encoder in the image-text feature joint encoding model to obtain a tag-level vector matrix includes: Performing identity mapping on the target input identity using the embedding layer in the image-text feature joint encoding model to obtain an identity vector sequence; Performing position encoding on each identification vector in the identification vector sequence to obtain an encoding vector; Semantic features are extracted from the encoding vector to obtain a tag-level vector matrix.

6. The semantic alignment method based on the cultural graph model according to claim 1, wherein: The self-attention processing is performed on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content, including: Performing a linear transformation on the tag-level vector matrix to obtain a three-element matrix; Using the attention mask, position-marking is performed on the dot product results of the query matrix and the key matrix in the three-element matrix to obtain a marking weight matrix; Performing weighted aggregation on the value matrices in the three-element matrix according to the label weight matrix to obtain a semantic matrix; Performing self-attention feature association mapping on the semantic matrix to obtain relational features; The relationship features are weightedly fused with the semantic matrix to obtain a fused feature matrix, and the fused feature matrix is ​​globally pooled to obtain a fine-grained semantic feature vector.

7. The semantic alignment method based on the cultural graph model according to claim 1, characterized in that: The generating of the guidance information of the preset text graph model according to the fine-grained semantic feature vector and the global semantic feature vector includes: Performing feature normalization processing on the fine-grained semantic feature vector and the global semantic feature vector to obtain normalized fine-grained features and normalized global features; Identifying local semantic units corresponding to the standardized fine-grained features according to preset visual elements, and extracting abstract semantic information from the standardized global features; Performing weight association on the local semantic unit and the abstract semantic information to obtain a semantic association tensor; Performing weighted fusion on the standardized fine-grained features and the standardized global features according to the semantic association tensor to obtain a multi-dimensional fused feature vector; The multi-dimensional fusion feature vector is feature mapped using the embedding space in the preset text graph model to obtain semantic space mapping features, and the semantic space mapping features are used as guidance information.

8. A semantic alignment device based on a cultural graph model, characterized in that: include: A global semantic feature vector extraction module is used to obtain a text data training set, perform visual semantic enhancement on the original text content in the text data training set to obtain enhanced text content, and extract a global semantic feature vector corresponding to the enhanced text content; A token sequence conversion module, configured to convert the enhanced text content into a token sequence using a word segmenter in a preset graphic-text feature joint encoding model; An attention mask generation module is used to perform index mapping on the tag sequence to obtain a target input identifier of the image-text feature joint encoding model, and to generate an attention mask corresponding to the target input identifier; The tag-level vector matrix acquisition module is used to embed the target input identifier using the text encoder in the image-text feature joint encoding model to obtain the tag-level vector matrix A fine-grained semantic feature vector analysis module, configured to perform self-attention processing on the tag-level vector matrix according to the attention mask to obtain a fine-grained semantic feature vector corresponding to the enhanced text content; The semantic alignment module is used to generate guidance information of a preset text-graph model based on the fine-grained semantic feature vector and the global semantic feature vector, and use the guidance information to align the text and image in the text-graph model to obtain aligned semantics.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the semantic alignment method based on the culture graph model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semantic alignment method based on the culture graph model according to any one of claims 1 to 7 is implemented.

Images