Multimodal model semantic enhancement and comparative learning method based on colored lamp knowledge graph

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a multimodal model semantic enhancement and comparative learning method based on a colored lamp knowledge graph, which comprises the following steps: associating an original text with a knowledge graph to generate a structured sentence tree, and carrying out embedded coding to obtain an embedded matrix; inputting the embedded matrix and the visibility matrix into a stacked Mask-Transform encoder to obtain a structured semantic feature, and carrying out modeling through a stacked self-attention block to obtain structured knowledge; respectively inputting the original text into a text encoder and a visual encoder of the multi-modal model, obtaining a reference text feature and a reference image feature, and carrying out dynamic gating weighted fusion to obtain a fusion vector; and obtaining a positive sample text, obtaining a high-quality negative set according to the positive sample text, inputting the fusion vector, the reference image features and difficult negative samples in the corresponding high-quality negative set into a contrast learning module, and obtaining symmetric contrast learning loss for training a multi-modal model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a multi-modal model semantic enhancement and contrast learning method based on a colored lamp knowledge graph. BACKGROUND

[0002] Colored lamps are an important part of China's intangible cultural heritage, with deep historical and cultural roots and distinct regional characteristics, such as Zigong colored lamps, Beijing palace lamps, Xianju boneless lanterns, and Qinhuai lanterns. Colored lamps have evolved from primarily serving as practical lighting to becoming artistic objects for decoration, appreciation, and entertainment. Despite the continuous progress of modern technology and the integration of emerging technologies into the colored lamp industry, the innovation of their display forms and creation methods remains relatively limited. In recent years, generative artificial intelligence (AIGC) has rapidly developed, and text-to-image generation technology based on large models has made significant breakthroughs in visual content creation. Models such as DALL·E, Stable Diffusion, and Midjourney have demonstrated strong capabilities in generating high-quality images, bringing new opportunities for colored lamp creative design.

[0003] However, current generative artificial intelligence (AIGC) technologies represented by DALL·E, Stable Diffusion, and Midjourney, although they have made significant breakthroughs in visual content creation, their core multi-modal framework, particularly models relying on CLIP text-image alignment mechanisms, still have significant shortcomings when applied to the specific cultural field of colored lamps. Specifically, the training corpus of existing CLIP models lacks colored lamp industry terminology and process knowledge such as "dragon lamp skeleton," "horse palace lamp," and "lamp core bamboo," making it difficult for text encoding vectors to accurately recall corresponding image content. At the same time, the default bag-of-words representation of the CLIP model also fails to effectively distinguish between "dragon dance lamp - Spring Festival" and "Spring Festival - dragon dance lamp" with their sequential and hierarchical relationships. In addition, the model lacks effective understanding of cultural contexts such as "Lantern Festival" and "Zigong Lantern Festival," making it difficult to fully reflect the unique atmosphere and creative details of colored lamp culture. Therefore, it is urgent to effectively enhance the text encoding stage of the CLIP model to compensate for its semantic deficiencies in the colored lamp professional field, combining inheritance and innovation and development. Overall, improve the understanding and accurate expression of multi-modal generation models of colored lamp cultural elements and creative design, thereby promoting the digital development of intangible cultural heritage. SUMMARY

[0004] In order to solve the problems existing in the prior art, the purpose of the present application is to provide a multi-modal model semantic enhancement and contrast learning method based on a colored lamp knowledge graph, which uses the colored lamp cultural knowledge graph to enhance the knowledge and attention constraint of the text encoding link of the CLIP-type multi-modal model, adaptively fuses professional semantics and original semantics through a learnable gate, and introduces knowledge-driven high-difficulty negative sample contrast training, thereby significantly improving the accuracy of image-text alignment and generation of the model on fine-grained elements such as colored lamp materials, processes and festival contexts, and providing an efficient and reliable new way for colored lamp non-heritage digital creativity.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] The multi-modal model semantic enhancement and contrast learning method based on the colored lamp knowledge graph comprises:

[0007] An original text is obtained, the original text is associated with a knowledge graph, a structured sentence tree is generated, embedding coding is performed, and an embedding matrix is obtained;

[0008] The visibility matrix of each pair of word elements in the structured sentence tree is obtained, the embedding matrix and the visibility matrix are input into a stacked Mask-Transformer encoder, structured semantic features are obtained, and the structured knowledge is modeled through a stacked self-attention block to obtain structured knowledge;

[0009] The original text is input into the text encoder and the visual encoder of the multi-modal model respectively, reference text features and reference image features are obtained, and the reference text features are dynamically gate-weighted and fused with the structured knowledge to obtain a fusion vector;

[0010] A positive sample text is obtained, a high-quality negative set is obtained according to the positive sample text, the fusion vector, the reference image features and the difficult negative samples in the corresponding high-quality negative set are input into a contrast learning module, a symmetric contrast learning loss is obtained, and the multi-modal model is trained.

[0011] Optionally, the structured sentence tree is generated by:

[0012] Entity recognition is performed on the original text, a set of entities involved in the text is determined, each entity in the set of entities is searched in the knowledge graph, and a set of triples is obtained:

[0013]

[0014] wherein, the set of triples is, the knowledge query is, the input sentence in the original text is, For the pre-constructed color light culture field knowledge graph;

[0015] The triples in the triple set are injected into the sentences in the text according to the corresponding positions of the entities to form the structured sentence tree.

[0016] Optionally, obtaining the embedding matrix comprises:

[0017] The structured sentence tree is linearly expanded into a token sequence of a target length, and the token sequence after linear expansion is embedded and coded:

[0018] The token sequence after linear expansion is dictionary embedded and coded: the token sequence after linear expansion is input into a trainable dictionary matrix to obtain a semantic vector;

[0019] The token sequence after linear expansion is soft position embedded and coded: according to the soft position number, a position index is obtained and mapped to a target dimension through a learnable projection matrix;

[0020] The token sequence after linear expansion is paragraph embedded and coded: according to the paragraph identifier, a paragraph embedding is retrieved from a paragraph vector table;

[0021] The semantic vector, the position index in the target dimension, and the paragraph embedding are added to obtain the embedding matrix.

[0022] Optionally, obtaining the structured semantic feature comprises:

[0023] The embedding matrix and the visibility matrix are input into a stacked Mask-Transformer encoder, each layer in turn adopts multi-head mask self-attention, feedforward network and residual normalization, and introduces visibility constraint to obtain the structured semantic feature:

[0024]

[0025] wherein, is a trainable linear projection matrix used to project the input embedding E into the query (Q), key (K), and value (V) space, is the element of the visibility matrix in the i-th row and the j-th column, is the attention weight, is the query vector, is the input embedding, is the key vector, is the value vector, the function is used to normalize the attention score into a probability distribution, so that the weight sum of each row is 1, is the attention scoring matrix, denotes the match score computed by the query vector of the i-th input position and the key vector of the j-th input position under the visibility matrix constraint, is the attention output, is the updated feature representation, is the position-wise feed-forward network, is the layer normalization, is the i-th row vector in the query matrix Q, is the j-th row vector in the key matrix K, is the dimension of the key / query vector, is the transpose.

[0026] Optionally, the visibility constraint comprises:

[0027]

[0028] wherein, is the element of the visibility matrix at the i-th row and the j-th column, is the baseline decay coefficient, is the distance adjustment factor, is the i-th row wordpiece , j-th column wordpiece branch distance in the tree.

[0029] Optionally, obtaining the structured knowledge comprises:

[0030] obtaining an enhanced triple sequence:

[0031]

[0032] wherein, is the enhanced triple sequence, is the head entity of the i-th triple, is the relation of the i-th triple, is the tail entity of the i-th triple, is the number of triples;

[0033] inserting a mark at the head of each triple in the enhanced triple sequence , and linearly arranging according to a target condition , further mapping to a word vector to obtain an input tensor;

[0034] modeling the input tensor by a stacked self-attention block:

[0035] ;

[0036] wherein, is the visibility matrix, is the standard encoding block, input / output hidden states of the

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] ​​​​​​​​​​​​​​​​​​​​​​​​ for the replaced head entity, for the replaced relation, for the replaced tail entity;

[0048] Pruning the text of the candidate negative sample set that still matches the corresponding image, to obtain the high-quality negative set.

[0049] Optionally, obtaining the symmetric contrastive learning loss includes:

[0050]

[0051] wherein, is the symmetric contrastive learning loss, is the B-group positive sample, is the InfoNCE loss in the image-text direction, is the InfoNCE loss in the text-image direction.

[0052] Optionally, the InfoNCE loss in the image-text direction includes:

[0053]

[0054] The InfoNCE loss in the text-image direction includes:

[0055]

[0056] wherein, is the similarity of the i-th pair of positive samples, i.e., the image embedding is the similarity of the i-th text embedding in the batch, is the hard negative sample weight, is the natural logarithm function, is the natural exponential function, is the image embedding of the i-th sample, is the text embedding of the i-th sample, is the temperature coefficient, is the B-group positive sample, is the number of hard negative samples for each anchor point, is the text embedding is the similarity of the i-th image embedding in the batch, is the similarity of the text embedding to the negative sample image embedding, is the similarity of the image embedding to the negative sample text embedding, is the k-th negative sample image embedding relative to the i-th text embedding , is the i-th image embedding relative to the i-th text embedding the kth negative sample text embedding of the kth negative sample text.

[0057] The beneficial effects of the present application are:

[0058] (1) The present application injects authoritative entity-property-relation information into the text with the aid of the color lamp culture field knowledge graph, enabling the multi-modal model to have original professional understanding ability for traditional color lamp themes, and no longer being limited by general data sets.

[0059] (2) The present application precisely restricts the attention flow direction with a three-grade visibility matrix (completely visible / distance attenuation / shield), which is controllable and information sufficient, which not only suppresses semantic drift caused by knowledge injection, but also retains necessary contact across branches.

[0060] (3) The dynamic gating fusion of the present application prevents overfitting, and the learnable gating coefficient α automatically adjusts the proportion of "original text semantics" and "structural knowledge semantics" according to the sample context, avoiding excessive interference of knowledge caused by fixed weighting.

[0061] (4) The present application is driven by high-difficulty negative samples, and generates "surface reasonable but not consistent with images" HardNegatives based on key element replacement (entity, attribute, relationship) of the knowledge graph, and amplifies the weight in the loss, significantly improving the recognition ability of the model to fine-grained differences.

[0062] (5) The present application is fully compatible with CLIP, has low implementation cost, does not change the structure of CLIP text / visual encoder, only makes light expansion on the input side and the loss side, can directly reuse pre-training weights, and reduces the difficulty of engineering landing.

[0063] (6) The present application synchronously enhances the robustness and explainability of training, and the difficult negative sample curriculum learning makes the training more stable and converges faster; the gating coefficient and the visibility matrix can be visualized, which is convenient for analyzing knowledge contribution and attention path. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0065] Figure 1 The flow chart of the multi-modal model semantic enhancement and contrastive learning method based on color lamp knowledge graph of the present application;

[0066] Figure 2 The branch tree embedding representation schematic diagram of the present application;

[0067] Figure 3 A comparison effect diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0070] As Figure 1 shown, the embodiment discloses a multi-modal model semantic enhancement and contrast learning method based on a color lamp knowledge graph, which comprises the following steps: obtaining an original text, associating the original text with a knowledge graph, generating a structured sentence tree, and performing embedding coding to obtain an embedding matrix; obtaining the visibility matrix of each pair of word elements in the structured sentence tree, inputting the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder, obtaining structured semantic features, and modeling through a stacked self-attention block to obtain structured knowledge; inputting the original text into a text encoder and a visual encoder of a multi-modal model respectively, obtaining baseline text features and baseline image features, and dynamically gate-weighted fusing the baseline text features and the structured knowledge to obtain a fusion vector; obtaining a positive sample text, obtaining a high-quality negative set according to the positive sample text, inputting the fusion vector, the baseline image features and the difficult negative samples in the corresponding high-quality negative set into a contrast learning module to obtain a symmetric contrast learning loss, which is used for training the multi-modal model.

[0071] Specifically, the embodiment discloses a multi-modal model semantic enhancement and contrast learning method based on a color lamp knowledge graph, which comprises the following steps:

[0072] Step 1, knowledge retrieval and branch tree generation (knowledge layer), receiving an input text, calling a knowledge graph through a knowledge layer, matching and retrieving entities in the text, outputting corresponding triples, and injecting them into the text to generate a branch tree structure.

[0073] Step 2, embedding layer, performing soft position word embedding and paragraph embedding on the branch tree to generate an embedding representation containing sequential information and paragraph identifiers.

[0074] Step 3, visibility layer, constructing a visibility matrix according to the branch tree structure to limit the attention range between word elements, defining that word elements in the same trunk or branch are completely visible, word elements across branches are partially visible, and unrelated word elements are invisible.

[0075] Step 4, Mask-Transformer encoding, input embedding representation and visible matrix, feature extraction through Mask-Transformer encoder.

[0076] Step 5, triple Transformer encoding, input the encoded triple into Transformer for further modeling to obtain structured knowledge embedding.

[0077] Step 6, clip text encoding, input the original text, extract text embedding through the text encoder.

[0078] Step 7, clip image encoding, input the corresponding image, extract image embedding through the image encoder.

[0079] Step 8, dynamic gate fusion, dynamically gate weighted fusion of text embedding and triple embedding, automatically adjust the weights of the two according to the content.

[0080] Step 9, negative sample generation, key element replacement and data enhancement of text by knowledge graph, generate high-quality negative sample set.

[0081] Step 10, contrastive learning training, input the fusion embedding, image embedding and negative sample into the contrastive learning module for multi-modal alignment optimization.

[0082] The application takes the color lamp field knowledge graph as the core driving force, and enhances the semantic of the text encoding link of the CLIP framework: first, retrieve and inject entities, attributes and relationships corresponding to the input text from the knowledge graph, so that the CLIP text branch can capture detailed semantics such as "lamp type, material, process and festival" in the feature learning stage; then use the inherent hierarchical and dependent information of the graph to model these fine-grained semantics, achieving dual enhancement of semantic and structural dimensions; finally, with the help of learnable gate and contrastive learning mechanism, the enhanced text representation and original image representation are mapped to the same vector space, realizing seamless alignment of knowledge semantics and visual semantics. Through the above process, the text encoder of CLIP is no longer limited to general corpus, and can fully and accurately utilize the professional knowledge of color lamps and their internal relations, significantly improving the understanding and performance in multi-modal tasks such as color lamp creative generation and image-text retrieval.

[0083] Further, the generating the structured sentence tree comprises: performing entity recognition on the original text, determining an entity set involved in the text, retrieving each entity in the entity set in a knowledge graph to obtain a triple set; and injecting the triples in the triple set into the sentence in the text according to the corresponding positions of the entities to form the structured sentence tree.

[0084] Specifically, step 1: this step is used for associating the input text with the knowledge graph, and generating a structured sentence tree containing knowledge injection, and the specific process includes the following contents:

[0085] (1) input receiving:

[0086] receiving an input sentence:

[0087]

[0088] wherein, represents the nth word unit in the text.

[0089] Knowledge query (K-Query): performing entity recognition on the input sentence to determine an entity set involved in the sentence, retrieving each entity in the knowledge graph to obtain a triple set related to the entity:

[0090]

[0091] wherein, is a pre-constructed knowledge graph in the field of colored light culture, represents all the triples obtained by retrieval:

[0092]

[0093] wherein, is the ith recognized entity in the sentence, is the 0th relationship type of the entity in the graph, is the 0th recognized entity, is the jth recognized entity, is the 3rd relationship type of the entity in the graph, is the target entity connected through the relationship and .

[0094] (3) Knowledge injection (K-Inject): injecting the retrieved triples into the input sentence according to the corresponding positions of the entities to form a sentence tree structure. The structure takes the original sentence as the main trunk, and the information corresponding to the triples is attached to the entity nodes in the form of branches, with a fixed depth of 1 and no recursive expansion.

[0095] The process is represented by the formula:

[0096]

[0097] wherein: t is a sentence tree representation;

[0098] The structure satisfies: .

[0099] (4) Output: The sentence tree structure t output in this step includes: a trunk token sequence, a branch triple corresponding to each matched entity, and is used for subsequent embedding coding and multi-modal alignment.

[0100] Further, obtaining the embedding matrix comprises: linearly expanding the structured sentence tree into a token sequence of a target length, embedding coding the linearly expanded token sequence; dictionary embedding coding the linearly expanded token sequence: inputting the linearly expanded token sequence into a trainable dictionary matrix to obtain a semantic vector; soft position embedding coding the linearly expanded token sequence: obtaining a position index according to a soft position number, and mapping to a target dimension through a learnable projection matrix; paragraph embedding coding the linearly expanded token sequence: retrieving a paragraph embedding from a paragraph vector table according to a paragraph identifier; adding the semantic vector, the position index in the target dimension, and the paragraph embedding to obtain the embedding matrix.

[0101] Specifically, step 2: This step is used to embed code all tokens in the branch tree, including soft position token embedding and paragraph embedding, to retain order information and paragraph identifiers. Each token is assigned a position number, so that the inserted branch node inherits the order information of its trunk node in semantics, as shown in Figure 2 .

[0102] This layer linearly expands the branch tree t into a token sequence of length L and maps it into a vector matrix of a uniform dimension . Each token embedding is the sum of the dictionary, soft position, and paragraph:

[0103]

[0104] wherein, is TokenEmbedding, i.e., the semantic vector obtained after inputting the linearly expanded token sequence T into the trainable dictionary matrix , and Voc is the size of the vocabulary (vocabulary);

[0105] Soft position embedding is obtained by first obtaining a position index according to the soft position number (the trunk takes the hard sequence number, and the branch adds a fine-grained offset to the parent node hard sequence number ), and then mapping to a trainable mapping of through a learnable projection matrix;

[0106] For paragraph embedding, according to the paragraph identifier From the paragraph vector table , to distinguish different sentence segments; wherein, The segment identifier of the word element, The trainable paragraph vector corresponding to the two types of segment identifiers.

[0107] Three element-wise additions form the output feature matrix ; wherein, The length of the expanded sequence, d is the dimension of the hidden vector. This embedding not only preserves word meaning, but also encodes fine-grained relative order and sentence segment information, providing structured input for subsequent visible layers and Mask-Transformers.

[0108] Further, obtaining structured semantic features includes: inputting the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder, each layer in turn adopting multi-head mask self-attention, feedforward network and residual normalization, and introducing visibility constraints to obtain structured semantic features.

[0109] Specifically, step 3: According to the topological relationship of the branch tree, a visibility matrix is generated for each pair of word elements , so as to accurately limit the information flow in the subsequent mask Transformer self-attention. Let denote the branch distance of the word element in the tree (the number of layers to the nearest common parent node), then the visibility weight is defined as:

[0110]

[0111] wherein, is the reference decay coefficient, is the distance adjustment factor. The first two cases give 0 or limited negative offset according to the structural proximity (the decay increases linearly with the branch distance); the last case is directly set to , completely shielding the attention between unrelated word elements, is the number of triplets.

[0112] Step 4: This step accepts the embedding matrix generated in step 2 and the visibility matrix constructed in step 3, and extracts structured semantic features through a stacked Mask-Transformer encoder. Each layer adopts a multi-head mask self-attention (MSA) + feedforward network (FFN) + residual normalization framework, and explicitly injects visibility constraints. Its core calculation is as follows:

[0113]

[0114] where, is a trainable linear projection matrix used to project the input embeddings E into the query (Q), key (K), value (V) space, is the element of the visibility matrix at the i-th row and j-th column, is the attention weight, is the query vector, is the input embedding, is the key vector, is the value vector, is a function used to normalize the attention scores into a probability distribution, making the sum of each row of weights equal to 1, is the attention score matrix, is the matching score computed by the query vector at the i-th input position and the key vector at the j-th input position under the constraint of the visibility matrix, is the attention output, is the updated feature representation, is the position-wise feed-forward network, is the layer normalization, is the i-th row vector in the query matrix Q, is the j-th row vector in the key matrix K, is the dimension of the key / query vector, is the transpose, from step 3, ensuring full visibility within the same trunk / same branch, weak visibility across branches, and full shielding of unrelated information, i.e., the element of the visibility matrix at the i-th row and j-th column. The softmaxed attention weight A is only normalized within the permissible range, blocking illegal information flow. After stacking N times, the final feature is output as the unified semantic representation for subsequent tasks such as contrastive learning, classification, or generation. The entire encoding not only preserves the original sentence semantics but also integrates branch information, and avoids semantic drift through the visibility matrix.

[0115] Further, obtaining the structured knowledge comprises: obtaining an enhanced triple sequence, inserting a mark [CLS] at the head of each triple in the enhanced triple sequence, and arranging the triples in the enhanced triple sequence in a linear order according to a target condition further mapping to a word vector, obtaining an input tensor, modeling the input tensor through a stacked self-attention block, extracting a mark vector of each triple in the modeled input tensor through linear mapping, obtaining a triple embedding, and using average pooling to converge all triple embeddings to obtain the structured knowledge.

[0116] Specifically, step 5: first, the updated feature representation extracts the sentence context vector:

[0117] ;

[0118] wherein, is a pooling function (sequence→vector) that compresses the updated feature representation into a sentence-level vector.

[0119] After step 1, the initial candidate triple set is obtained by knowledge retrieval and expansion. The relevance score of is calculated, and the Top-K / threshold cut-off and conflict resolution are performed to obtain the final enhanced triple set , and the enhanced triple sequence is obtained:

[0120]

[0121] wherein, is the enhanced triple sequence (set), is the head entity of the i-th triple, is the relation of the i-th triple, is the tail entity of the i-th triple, is the number of triples, and each has integrated the attribute or hierarchical information supplemented by the knowledge graph. To capture the deep dependency relationship between triples, the head of each triple is marked and linearly arranged according to the position to obtain the input tensor:

[0122] wherein,

[0123] is the input tensor, is the hidden vector dimension, is the vector representation of the k-th triple at position 3, is the matrix space in the real field, is the number of enhanced triples involved in the encoding. Then, the

[0124] layer self-attention block is stacked for modeling:

[0125]

[0126] wherein, is the visible matrix generated in step 3. The visible matrix is sliced according to the position subset and subjected to local constraints to obtain is the visible matrix, ​For the ℓth Transformer coded block, the input / output hidden states are... It is a standard coding block (multi-head self-attention + residual + layer normalization + feedforward network). For the coding layer index, The length of the linearized sequence for a single triplet is given, ensuring that the attention follows the constraints of "fully visible within the same domain / attenuated across domains / irrelevant masking". For each triplet... Vectors, after linear mapping, yield triplet embeddings:

[0127]

[0128] in, To map the [CLS] vector to linear mapping parameters of triple embeddings, For the first The triplet after this step After stacked encoding, the final hidden vector is located at position [CLS].

[0129] Finally, average pooling is used to aggregate all triplet embeddings to obtain a structured knowledge representation:

[0130]

[0131] in, The number of enhanced triples participating in the encoding, It also includes the semantics of the original sentence, knowledge extension information, and triple-level interdependencies, providing high-quality knowledge embedding for subsequent gating fusion with the text backbone vector and cross-modal comparative learning.

[0132] Furthermore, obtaining benchmark text features and benchmark image features includes:

[0133] Step 6: This step directly uses the text encoder provided by the CLIP (Commonly Targeted Learning Platform) model. The input is the original text s, which is first segmented using CLIP's built-in BPE vocabulary and [SOS] and [EOS] tags are inserted at the beginning and end to obtain the index sequence t. The model maps words using a fixed, learnable word vector table Wclip, plus CLIP's predefined absolute position embeddings. This forms the input matrix:

[0134]

[0135] in, For the input matrix, as a word element Embedded vector, For absolute position embedding table for position The position vector;

[0136] Subsequently Send in Layer TransformerEncoder (multi-head self-attention + feed-forward + residual layer normalization). Take the hidden state vector of the [SOS] position in the final layer as the semantic representation of the whole sentence, and do Normalization to get:

[0137]

[0138] where, is the L-layer Transformer Encoder stack of the CLIP text branch;

[0139] The vector is in the same alignment space as the CLIP image encoder output, serving as the baseline text feature for subsequent "text backbone-knowledge vector" fusion and cross-modal contrastive learning.

[0140] Step 7: This step uses the visual encoder of the CLIP model. The input image I is preprocessed first: the long side is scaled to 224px, center cropping, RGB is normalized to [0, 1] and the channel mean is subtracted. The global feature vector of the last block is obtained via the pre-trained Vision-Transformer . Then multiply by the CLIP fixed projection matrix and do Normalization to get the image embedding aligned with the text branch in the same space:

[0141]

[0142] where, is the image embedding aligned with the text branch in the same space, is the global feature vector.

[0143] The vector serves as the baseline image feature for similarity calculation with the text / knowledge fusion representation in subsequent cross-modal contrastive learning.

[0144] Further, obtaining the fusion vector includes: splicing the baseline text feature and the structured knowledge through a layer of linear mapping, activation, and linear mapping to obtain a scalar gating coefficient; using the scalar gating coefficient to perform residual weighted fusion on the baseline text feature and the structured knowledge to obtain the fusion vector.

[0145] Specifically, step 8: To fully utilize the text native semantics and structured knowledge semantics , this step introduces a learnable gating function that automatically determines the fusion ratio of the two signals according to the sample content. The specific approach is: first splice the two into , through a linear mapping-activation-linear mapping to get the scalar gating coefficient:

[0146]

[0147] wherein, and the bias b1, b2 are trainable parameters, and σ is a Sigmoid activation function, is a Rectified Linear Unit activation function, is the input vector of the gating network. Then, the final fusion vector is obtained by residual weighting:

[0148]

[0149] When the input sentence has high knowledge dependency (for example, contains a large number of professional entities or relationships), the network learns , so that the knowledge semantics dominates; if the text itself can fully express the meaning, , mainly retaining the original semantics. This dynamic gating fusion not only avoids excessive invasion of knowledge noise, but also ensures that structured supplementary information is fully utilized when needed, providing content-adaptive unified text representation for subsequent cross-modal contrast learning.

[0150] Further, obtaining the high-quality negative set comprises: retrieving a corresponding triple set from the knowledge graph for the positive sample text, performing controllable perturbation on key elements in the triple set corresponding to the positive sample text; the controllable perturbation comprises: same-class entity replacement, attribute value replacement and relationship replacement; rewriting the replaced triple to the original sentence template to obtain a candidate negative sample set, and removing the text that still matches the corresponding image from the candidate negative sample set to obtain the high-quality negative set.

[0151] Specifically, step 9: for each positive sample text s, first retrieve its matching triple set K from the knowledge graph K:

[0152]

[0153] wherein, K is the knowledge graph in the field of colored light culture, is a query operator / process based on entity recognition and graph retrieval, is the number of retrieved triples;

[0154] Under the premise of maintaining syntactic readability, controllable perturbation is performed on the key elements in the triple set

[0155] Same-class entity replacement:

[0156] Attribute value replacement: ​

[0157] Relation replacement:

[0158] wherein, is the head entity of the i-th replaced triple, is the tail entity of the i-th replaced triple, is the relation of the i-th replaced triple;

[0159] The replaced triples are written back to the original sentence template to obtain a candidate negative sample set:

[0160]

[0161] wherein, is a candidate negative sample set, is the k-th candidate negative sample text, is a text rewriting / replacement operator, is an attention scoring matrix, is the head entity of the i-th triple, is the relation of the i-th triple, is the tail entity of the i-th triple, is the replaced head entity, is the replaced relation, is the replaced tail entity;

[0162] Subsequently, the image-text coarse sorter or rule is used to remove sentences that can still match the corresponding image, forming a final high-quality negative set:

[0163]

[0164] According to the replacement depth, Easy / Medium / Hard difficulty labels are annotated and sampled according to the curriculum learning strategy, so that the model can see both easy negative samples with obvious semantic differences and difficult negative samples that "seem reasonable but are wrong" during contrast training, thereby obtaining stronger discriminative ability in fine-grained cross-modal alignment.

[0165] Further, the symmetric contrast learning loss includes:

[0166] Step 10: Complete the text-knowledge fusion vector and the extraction of the image vector After that, they are input into the contrast learning module together with the difficult negative samples generated by the knowledge graph. Let a batch contain B groups of positive samples , and the i-th text is equipped with m negative samples The normalized dot product similarity is adopted:

[0167]

[0168] The InfoNCE loss of the image-text direction is:

[0169]

[0170] where γ ≥ 1 is the hard-negative sample weight; the text-image direction is the same:

[0171]

[0172] where, is the similarity of the i-th pair of positive samples, i.e., the similarity of the image embedding and the i-th text embedding in the batch, is the hard-negative sample weight, is the natural logarithm function, is the natural exponential function, is the image embedding of the i-th sample, is the text embedding of the i-th sample, is the temperature coefficient, is the B group of positive samples, is the number of hard-negative samples for each anchor point, is the similarity of the text embedding and the i-th image embedding in the batch, is the similarity of the text embedding and the negative sample image embedding, is the similarity of the image embedding and the negative sample text embedding, is the k-th negative sample image embedding relative to the i-th text embedding , is the k-th negative sample text embedding relative to the i-th image embedding .

[0173] The final symmetric contrastive learning loss is:

[0174]

[0175] where, is the symmetric contrastive learning loss, is the B group of positive samples, is the InfoNCE loss of the image-text direction, is the InfoNCE loss of the text-image direction.

[0176] This loss not only narrows the distance between positive samples in the same group, but also strengthens the punishment of "semantically similar but not matching" negative samples constructed by the knowledge graph through γ, thereby significantly improving the discriminability and robustness of cross-modal representation in the fine-grained alignment task.

[0177] The CLIP model fine-tuned by the knowledge enhancement method shows obvious effect improvement in image generation tasks such as Flux and Stable Diffusion model. Traditional image-text pair training methods usually focus on simple matching and comparison, often ignoring the fine-grained semantic relationship between different modalities. This leads to the existing CLIP model performing poorly when dealing with complex semantic relationships, especially in tasks that require reasoning, often failing to fully understand and utilize the deep connections between modalities.

[0178] The knowledge enhancement framework significantly enhances the semantic alignment ability between images and language by introducing a structured knowledge graph. Through this enhancement, the model can better understand and integrate fine-grained semantic information between different modalities, thereby improving the model's perception and reasoning ability for text details in generative model tasks. This method effectively compensates for the shortcomings of traditional models in handling complex semantic reasoning, promoting progress in semantic understanding in image generation tasks. Figure 3 as shown.

[0179] The above-described embodiments are only descriptions of the preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A multi-modal model semantic enhancement and contrast learning method based on a colored lamp knowledge graph, characterized in that, The method comprises the following steps: obtaining an original text, associating the original text with a knowledge graph, generating a structured sentence tree, and obtaining an embedding matrix through embedding coding; obtaining a visibility matrix of each pair of word units in the structured sentence tree, inputting the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder, obtaining structured semantic features, and modeling through a stacked self-attention block to obtain structured knowledge; inputting the original text into a text encoder and a visual encoder of a multi-modal model respectively, obtaining baseline text features and baseline image features, and dynamically gate-weighting fusion the baseline text features and the structured knowledge to obtain a fusion vector; obtaining a positive sample text, obtaining a high-quality negative set according to the positive sample text, inputting the fusion vector, the baseline image features and the difficult negative samples in the corresponding high-quality negative set into a contrast learning module to obtain a symmetric contrast learning loss for training the multi-modal model.

2. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, Generating the structured sentence tree comprises: performing entity recognition on the original text to determine a set of entities involved in the text, retrieving each entity in the set of entities in the knowledge graph to obtain a set of triples: wherein, is a set of triples, is a knowledge query, is an input sentence in raw text, is a pre-constructed knowledge graph in the lantern culture field. injecting triples in the set of triples into sentences in the text according to the corresponding positions of the entities to form the structured sentence tree.

3. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, Obtaining the embedding matrix comprises: linearly expanding the structured sentence tree into a word unit sequence of a target length, and embedding coding the linearly expanded word unit sequence: dictionary embedding coding the linearly expanded word unit sequence: inputting the linearly expanded word unit sequence into a trainable dictionary matrix to obtain a semantic vector; soft position embedding coding the linearly expanded word unit sequence: obtaining a position index according to a soft position number and mapping it to a target dimension through a learnable projection matrix; paragraph embedding coding the linearly expanded word unit sequence: retrieving a paragraph embedding from a paragraph vector table according to a paragraph identifier; adding the semantic vector, the position index in the target dimension and the paragraph embedding to obtain the embedding matrix.

4. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, Obtaining the structured semantic features comprises: inputting the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder, sequentially adopting multi-head mask self-attention, feedforward network and residual normalization at each layer, and introducing visibility constraints to obtain the structured semantic features: wherein, is a trainable linear projection matrix used to project the input embeddings E into the query (Q), key (K), value (V) space, is the element of the visibility matrix at the ith row, jth column, is the attention weight, is the query vector, is the input embedding, is the key vector, is the value vector, is a function to normalize the attention scores into a probability distribution, making the sum of each row weight 1, is the attention score matrix, represents the match score computed from the query vector at the ith input position and the key vector at the jth input position under the constraint of the visibility matrix, is the attention output, is the updated feature representation, is the position-wise feed forward network, is the layer normalization, is the ith row vector in the query matrix Q, is the jth row vector in the key matrix K, is the dimension of the key / query vector, is the transpose.

5. The multi-modal model semantic enhancement and contrastive learning method based on the color lamp knowledge graph according to claim 4, characterized in that, The visibility constraint comprises: wherein, is the element of the visibility matrix in the i-th row, j-th column, is the reference attenuation coefficient, is the distance adjustment factor, is the i-th row of the word, , j-th column of the word, branch distance in the tree.

6. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, Obtaining the structured knowledge comprises: obtaining an enhanced triple sequence: wherein, is the enhanced sequence of triples, is the head entity of the ith triple, is the relation of the ith triple, is the tail entity of the ith triple, is the number of triples; inserting a marker at the head of each triple in the enhanced sequence of triples and in accordance with target conditions linear arrangement, further mapped to word vectors, obtaining an input tensor; modeling the input tensor through a stacked self-attention block: ； wherein, is a visible matrix, is a standard encoding block, is an input / output hidden state of the th Transformer encoding block, is an encoding layer index, is a sequence length after linearization of a single triple. extracting a label vector of each triple in the modeled input tensor, obtaining a triple embedding through linear mapping, and obtaining the structured knowledge by averaging all triple embeddings through average pooling.

7. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, Obtaining the fusion vector comprises: concatenating the baseline text features and the structured knowledge, performing one-layer linear mapping, activation and linear mapping to obtain a scalar gating coefficient: wherein, is a scalar gating coefficient, is a trainable parameter, is a Sigmoid activation function, is a Rectified Linear Unit activation function, is an input vector to the gating network; using the scalar gating coefficient to perform residual weighted fusion on the baseline text features and the structured knowledge to obtain the fusion vector.

8. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, The obtaining of the high-quality negative set comprises: Retrieving a corresponding triple set from the knowledge graph for the positive sample text, and performing controllable perturbation on key elements in the triple set corresponding to the positive sample text; the controllable perturbation comprises: same-class entity replacement, attribute value replacement and relationship replacement; Rewriting the replaced triple to the original sentence template to obtain a candidate negative sample set: wherein, is a candidate negative sample set, is the kth candidate negative sample text, is a text rewriting / replacement operator, is an attention scoring matrix, is the head entity of the ith triple, is the relation of the ith triple, is the tail entity of the ith triple, is the replaced head entity, is the replaced relation, is the replaced tail entity. Eliminating the text that still matches the corresponding image in the candidate negative sample set to obtain the high-quality negative set.

9. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 1, characterized in that, The obtaining of the symmetric contrast learning loss comprises: wherein, is a symmetric contrastive learning loss, is a positive sample for group B, is an InfoNCE loss for image-text direction, is an InfoNCE loss for text-image direction.

10. The multi-modal model semantic enhancement and contrastive learning method based on the colored lamp knowledge graph according to claim 9, characterized in that, The InfoNCE loss in the image-text direction comprises: The InfoNCE loss in the text-image direction comprises: where, is the similarity of the i-th pair of positive samples, i.e., image embeddings is the similarity of the i-th text embedding in the batch, is the hard negative sample weight, is the natural logarithm function, is the natural exponential function, is the image embedding of the i-th sample, is the text embedding of the i-th sample, is the temperature coefficient, is the B set of positive samples, is the number of hard negative samples for each anchor point, is the text embedding is the similarity of the i-th image embedding in the batch, is the text embedding is the similarity of the negative sample image embedding, is the image embedding is the similarity of the negative sample text embedding, is the k-th negative sample image embedding with respect to the i-th text embedding is the k-th negative sample text embedding with respect to the i-th image embedding.

Images