Multi-modal model semantic enhancement and contrast learning method based on color light knowledge graph

Abstract

The application belongs to the technical field of artificial intelligence, and relates to a multi-modal model semantic enhancement and contrast learning method based on a colored lamp knowledge graph, which comprises the following steps: associating original text with a knowledge graph, generating a structured sentence tree, embedding coding, and obtaining an embedding matrix; inputting the embedding matrix and a visibility matrix into a stacked Mask-Transformer encoder, obtaining structured semantic features, modeling through a stacked self-attention block, and obtaining structured knowledge; inputting the original text into a text encoder and a visual encoder of a multi-modal model respectively, obtaining benchmark text features and benchmark image features, and performing dynamic gate weighted fusion to obtain a fusion vector; obtaining positive sample texts, obtaining a high-quality negative set according to the positive sample texts, inputting the fusion vector, benchmark image features and difficult negative samples in the corresponding high-quality negative set into a contrast learning module, obtaining a symmetric contrast learning loss, and using the symmetric contrast learning loss to train the multi-modal model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for semantic enhancement and contrastive learning of multimodal models based on a color light knowledge graph. Background Technology

[0002] Lanterns are an important part of the intangible cultural heritage of the Chinese nation, possessing profound historical and cultural connotations and distinct regional characteristics, such as Zigong lanterns, Beijing palace lanterns, Xianju boneless lanterns, and Qinhuai lanterns. Initially primarily for practical lighting, lanterns have gradually evolved into art forms mainly for decoration, appreciation, and entertainment. Despite continuous advancements in modern technology and the integration of emerging technologies into the lantern industry, innovation in their presentation and creative methods remains relatively limited. In recent years, generative artificial intelligence (AIGC) has developed rapidly. Text-to-Image Generation technology based on large models has achieved significant breakthroughs in the field of visual content creation. Models such as DALL·E, StableDiffusion, and Midjourney have demonstrated powerful capabilities in generating high-quality images, bringing new opportunities for lantern creative design.

[0003] However, while generative artificial intelligence (AIGC) technologies such as DALL·E, StableDiffusion, and Midjourney have achieved significant breakthroughs in visual content creation, their core multimodal frameworks—especially those relying on the CLIP text-image alignment mechanism—still have significant shortcomings when applied to the specific cultural domain of lanterns. Specifically, the training corpus of existing CLIP models lacks lantern industry terminology and craft knowledge such as "dragon lantern skeleton," "revolving palace lantern," and "lantern wick bamboo strips," making it difficult for text encoding vectors to accurately recall precisely corresponding image content. Furthermore, the default bag-of-words representation of CLIP models struggles to effectively distinguish semantic differences such as "dragon lantern dance - Spring Festival" versus "Spring Festival - dragon lantern dance," which have sequential and hierarchical relationships. In addition, the models lack an effective understanding of cultural contexts such as festival customs and regional characteristics, including "Lantern Festival appreciation" and "Zigong Lantern Festival," making it difficult to fully reflect the unique atmosphere and creative details of lantern culture. Therefore, there is an urgent need to effectively enhance the text encoding stage of the CLIP model to compensate for its semantic deficiencies in the field of lantern making, thereby promoting its inheritance and innovative development. This will comprehensively improve the multimodal generative model's ability to understand and accurately express lantern cultural elements and creative designs, thus driving the digital development of intangible cultural heritage. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a semantic enhancement and contrastive learning method for multimodal models based on lantern knowledge graphs. Utilizing a knowledge graph in the lantern culture domain, the method enhances knowledge and imposes attention constraints on the text encoding stage of CLIP-type multimodal models. Through learnable gating adaptive fusion of professional and native semantics, and by introducing knowledge-driven, high-difficulty negative sample contrastive training, the method significantly improves the accuracy of image-text alignment and generation on fine-grained elements such as lantern materials, craftsmanship, and festival context. This provides an efficient and reliable new approach for the digital creation of lantern intangible cultural heritage.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] Multimodal model semantic enhancement and contrastive learning methods based on lantern knowledge graphs include:

[0007] Obtain the original text, associate the original text with the knowledge graph, generate a structured sentence tree, perform embedding encoding, and obtain the embedding matrix;

[0008] Obtain the visibility matrix of each pair of lexical units in the structured sentence tree, input the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder to obtain structured semantic features, and model them using stacked self-attention blocks to obtain structured knowledge;

[0009] The original text is input into the text encoder and visual encoder of the multimodal model respectively to obtain the baseline text features and baseline image features. The baseline text features are then dynamically gated and weighted to fuse with the structured knowledge to obtain the fusion vector.

[0010] Obtain positive sample text, and based on the positive sample text, obtain a high-quality negative set. Input the fusion vector, the baseline image features, and the corresponding difficult negative samples from the high-quality negative set into the contrastive learning module to obtain the symmetric contrastive learning loss, which is used to train the multimodal model.

[0011] Optionally, generating the structured sentence tree includes:

[0012] Entity recognition is performed on the original text to determine the set of entities involved in the text. Each entity in the set is then retrieved from the knowledge graph to obtain a set of triples.

[0013]

[0014] in, For a set of triples, For knowledge retrieval, The input sentence is from the original text. A pre-constructed knowledge graph for the field of lantern culture;

[0015] The triples in the triple set are injected into the sentences in the text according to the corresponding positions of the entities to form the structured sentence tree.

[0016] Optionally, obtaining the embedding matrix includes:

[0017] The structured sentence tree is linearly expanded into a word sequence of the target length, and the linearly expanded word sequence is then embedded and encoded.

[0018] Dictionary embedding encoding is performed on the linearly expanded word sequence: the linearly expanded word sequence is input into a trainable dictionary matrix to obtain a semantic vector;

[0019] The linearly expanded word sequence is encoded using soft position embedding: the position index is obtained based on the soft position number and mapped to the target dimension via a learnable projection matrix;

[0020] The linearly expanded word sequence is encoded by paragraph embedding: the paragraph embedding is retrieved from the paragraph vector table based on the paragraph identifier;

[0021] The semantic vector, the position index under the target dimension, and the paragraph embedding are added together to obtain the embedding matrix.

[0022] Optionally, obtaining the structured semantic features includes:

[0023] The embedding matrix and the visibility matrix are input into a stacked Mask-Transformer encoder. Each layer sequentially employs multi-head mask self-attention, a feedforward network, and residual normalization, and introduces visibility constraints to obtain the structured semantic features.

[0024]

[0025] in, A trainable linear projection matrix is ​​used to project the input embedding E onto the query (Q), key (K), and value (V) space. Let i be the element in the i-th row and j-th column of the visibility matrix. For attention weights, For query vector, For input embedding, For key vectors, For value vectors, The function is used to normalize the attention scores into a probability distribution, such that the weights of each row sum to 1. For attention scoring matrix, This represents the matching score calculated from the query vector at the i-th input position and the key vector at the j-th input position, under the visibility matrix constraint. For attention output, For the updated feature representation, For position-by-position feedforward networks, For layer normalization, To query the vector in the i-th row of matrix Q, Let J be the vector in the j-th row of the key matrix K. The dimension of the key / query vector. This is a transpose.

[0026] Optionally, the visibility constraint includes:

[0027]

[0028] in, Let i be the element in the i-th row and j-th column of the visibility matrix. As the reference attenuation coefficient, As a distance adjustment factor, For the word element in the i-th row j column of words Branch distance in the tree.

[0029] Optionally, obtaining the structured knowledge includes:

[0030] Obtain the enhanced triple sequence:

[0031]

[0032] in, For the enhanced triplet sequence, Let i be the head entity of the i-th triple. For the i-th triple, For the tail entity of the i-th triple, The number of triples;

[0033] Insert a marker at the beginning of each triple in the enhanced triple sequence. and in accordance with the target conditions Linear arrangement, further mapped to word vectors, to obtain the input tensor;

[0034] The input tensor is modeled using stacked self-attention blocks:

[0035] ;

[0036] in, The visible matrix For standard coded blocks, For the first Input / output hidden states of a Transformer coded block For the coding layer index, The length of the linearized sequence of a single triplet;

[0037] The label vector of each triplet in the input tensor after modeling is extracted and linearly mapped to obtain the triplet embedding. Then, average pooling is used to aggregate all triplet embeddings to obtain the structured knowledge.

[0038] Optionally, obtaining the fusion vector includes:

[0039] The baseline text features and the structured knowledge are concatenated, and then subjected to a linear mapping, activation, and another linear mapping to obtain scalar gating coefficients.

[0040]

[0041] in, The scalar gating coefficient. For trainable parameters, It is the Sigmoid activation function. The activation function for the Rectified Linear Unit. is the input vector of the gated network;

[0042] The scalar gating coefficients are used to perform residual weighted fusion of the baseline text features and the structured knowledge to obtain the fusion vector.

[0043] Optionally, obtaining the high-quality negative set includes:

[0044] The positive sample text is retrieved from the knowledge graph to obtain the corresponding set of triples. The key elements in the set of triples corresponding to the positive sample text are subjected to controlled perturbation. The controlled perturbation includes: replacement of similar entities, replacement of attribute values, and replacement of relations.

[0045] Write the replaced triples back to the original sentence template to obtain the candidate negative sample set:

[0046]

[0047] in, For the candidate negative sample set, For the k-th candidate negative sample text, For text rewriting / replacement operators, For attention scoring matrix, Let i be the head entity of the i-th triple. For the i-th triple, For the tail entity of the i-th triple, For the replaced head entity, For the replaced relationship, The replaced tail entity;

[0048] Remove the text that still matches the corresponding image from the candidate negative sample set to obtain the high-quality negative set.

[0049] Optionally, obtaining the symmetric contrastive learning loss includes:

[0050]

[0051] in, For symmetric contrastive learning loss, This is a positive sample from group B. For the InfoNCE loss in the image-to-text direction, For the InfoNCE loss in the text-image direction.

[0052] Optionally, the InfoNCE loss in the image-text direction includes:

[0053]

[0054] The InfoNCE loss in the text-image orientation includes:

[0055]

[0056] in, Let be the similarity of the i-th pair of positive samples, i.e., the image embedding. Similarity to the i-th text embedding in the batch, To avoid negative sample weights, It is the natural logarithm function. It is a natural exponential function. Image embedding for the i-th sample, For the text embedding of the i-th sample, For temperature coefficient, This is a positive sample from group B. The number of hard-to-bear samples for each anchor point Embedding text Similarity to the embedding of the i-th image in the batch, Embedding text Similarity with negative sample image embeddings, Image embedding Similarity with negative sample text embeddings For embedding relative to the i-th text The embedding of the k-th negative sample image, Embedding relative to the i-th image The k-th negative sample text embedding.

[0057] The beneficial effects of this invention are as follows:

[0058] (1) This invention uses a knowledge graph in the field of lantern culture to inject authoritative entity-attribute-relationship information into the text, so that the multimodal model has a native professional understanding of traditional lantern themes and is no longer limited by general datasets.

[0059] (2) This invention is aimed at noise controllability, sufficient information, and three-level visibility matrix (fully visible / distance attenuation / masking) to accurately constrain the flow of attention, which not only suppresses semantic drift caused by knowledge injection, but also preserves necessary cross-branch connections.

[0060] (3) The dynamic gating fusion of the present invention prevents overfitting. The learnable gating coefficient α can automatically adjust the ratio of "original text semantics" and "structural knowledge semantics" according to the sample context, avoiding excessive interference of knowledge caused by fixed weighting.

[0061] (4) The present invention uses a high-difficulty negative sample driving method to generate HardNegatives that are “superficially reasonable but inconsistent with the image” based on the replacement of key elements (entities, attributes, and relationships) in the knowledge graph, and amplifies the weights in the loss, which significantly improves the model’s ability to identify fine-grained differences.

[0062] (5) This invention is fully compatible with CLIP, has low implementation cost, does not change the CLIP text / visual encoder structure, only makes lightweight extensions on the input side and loss side, and can directly reuse pre-trained weights, reducing the difficulty of engineering implementation.

[0063] (6) The present invention simultaneously enhances training robustness and interpretability, and the learning of difficult sample courses makes training more stable and converges faster; both the gating coefficients and the visible matrix can be visualized, which facilitates the analysis of knowledge contribution and attention path. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of the semantic enhancement and contrastive learning method for multimodal models based on lantern knowledge graphs according to an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the branch tree embedding representation according to an embodiment of the present invention;

[0067] Figure 3 These are comparative effect diagrams of embodiments of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] like Figure 1 As shown, this embodiment discloses a multimodal model semantic enhancement and contrastive learning method based on a color light knowledge graph, including: acquiring the original text, associating the original text with the knowledge graph to generate a structured sentence tree, and performing embedding encoding to obtain an embedding matrix; acquiring the visibility matrix of each pair of words in the structured sentence tree, inputting the embedding matrix and visibility matrix into a stacked Mask-Transformer encoder to obtain structured semantic features, and modeling them using stacked self-attention blocks to obtain structured knowledge; inputting the original text into the text encoder and visual encoder of the multimodal model respectively to obtain baseline text features and baseline image features, and performing dynamic gating weighted fusion of the baseline text features and structured knowledge to obtain a fusion vector; acquiring positive sample text, acquiring a high-quality negative set based on the positive sample text, and inputting the fusion vector, baseline image features, and the corresponding difficult negative samples from the high-quality negative set into a contrastive learning module to obtain a symmetric contrastive learning loss for training the multimodal model.

[0071] Specifically, this embodiment discloses a multimodal model semantic enhancement and contrastive learning method based on a colored lantern knowledge graph, which includes the following steps:

[0072] Step 1, Knowledge Retrieval and Branch Tree Generation (Knowledge Layer): Receive input text, call the knowledge graph through the knowledge layer, match and retrieve entities in the text, output the corresponding triples, and inject them into the text to generate a branch tree structure.

[0073] Step 2, Embedding layer: Soft positional lexical embedding and paragraph embedding are performed on the branch tree to generate an embedded representation containing sequence information and segment identifiers.

[0074] Step 3, Visibility Layer: Based on the branch tree structure, construct a visibility matrix to limit the attention range between lexical units. Define lexical units within the same trunk or branch as fully visible, lexical units across branches as partially visible, and unrelated lexical units as invisible.

[0075] Step 4, Mask-Transformer encoding: Input the embedding representation and the visibility matrix, and extract features using the Mask-Transformer encoder.

[0076] Step 5: Triple Transformer Encoding. The encoded triples are input into the Transformer for further modeling, resulting in structured knowledge embedding.

[0077] Step 6, Clip text encoding: Input the original text and extract the text embedding using a text encoder.

[0078] Step 7, image clip encoding: input the corresponding image, and extract the image embedding through the image encoder.

[0079] Step 8, Dynamic Gated Fusion: Dynamically gated weighted fusion is performed on the text embedding and triplet embedding, and the weights of the two are automatically adjusted according to the content.

[0080] Step 9: Negative sample generation. Use knowledge graphs to replace key elements and augment data in the text to generate a high-quality negative sample set.

[0081] Step 10, Contrastive learning training: The fused embedding, image embedding, and negative samples are input into the contrastive learning module for multimodal alignment optimization.

[0082] This invention uses a knowledge graph in the field of lanterns as its core driving force to semantically enhance the text encoding stage of the CLIP framework. First, it retrieves and injects entities, attributes, and relationships closely corresponding to the input text from the knowledge graph, enabling the CLIP text branch to capture detailed semantics such as "lantern type, material, craftsmanship, and festival" during the feature learning stage. Then, utilizing the inherent hierarchical and dependency information of the knowledge graph, it performs structured modeling of these fine-grained semantics, achieving dual enhancement of semantic and structural dimensions. Finally, through learnable gating and contrastive learning mechanisms, the enhanced text representation and the original image representation are mapped to the same vector space, achieving seamless alignment between knowledge semantics and visual semantics. Through this process, the CLIP text encoder is no longer limited to general corpora, but can comprehensively and accurately utilize lantern-related professional knowledge and its inherent relationships, significantly improving its understanding and performance capabilities in multimodal tasks such as lantern creative generation and image-text retrieval.

[0083] Furthermore, generating a structured sentence tree includes: performing entity recognition on the original text to determine the set of entities involved in the text; retrieving each entity in the entity set from the knowledge graph to obtain a set of triples; and injecting the triples in the triple set into the sentences in the text according to the corresponding positions of the entities to form a structured sentence tree.

[0084] Specifically, Step 1: This step is used to associate the input text with the knowledge graph and generate a structured sentence tree containing knowledge injection. The specific process includes the following:

[0085] (1) Input reception:

[0086] Receive input sentence:

[0087]

[0088] in, This represents the nth word in the text.

[0089] Knowledge Query (K-Query): Entity identification is performed on the input sentence to determine the set of entities involved in the sentence. Each entity is then retrieved from the knowledge graph to obtain a set of associated triples.

[0090]

[0091] in, For a pre-constructed knowledge graph in the field of lantern culture, This represents all the triples retrieved:

[0092]

[0093] in, Let i be the i-th entity identified in the sentence. This is the 0th relation type for this entity in the graph. For the 0th identified entity, For the j-th identified entity, This is the third relation type for this entity in the graph. To pass through relationships and Connected target entities.

[0094] (3) Knowledge Injection (K-Inject): The retrieved triples are injected into the input sentence according to the corresponding positions of the entities, forming a sentence tree structure. The structure has the original sentence as the main body, and the information corresponding to the triples is attached to the entity nodes in the form of branches. The depth is fixed at 1 and it is not recursively expanded.

[0095] The formula for this process is expressed as follows:

[0096]

[0097] Where: t represents the sentence tree representation;

[0098] The structure satisfies: .

[0099] (4) Output: The sentence tree structure t output in this step includes: the main word sequence and the branch triplet corresponding to each matched entity, which are used for subsequent embedding encoding and multimodal alignment.

[0100] Further, obtaining the embedding matrix includes: linearly expanding the structured sentence tree into a word sequence of the target length; performing embedding encoding on the linearly expanded word sequence; performing dictionary embedding encoding on the linearly expanded word sequence; inputting the linearly expanded word sequence into a trainable dictionary matrix to obtain a semantic vector; performing soft position embedding encoding on the linearly expanded word sequence; obtaining the position index based on the soft position number and mapping it to the target dimension via a learnable projection matrix; performing paragraph embedding encoding on the linearly expanded word sequence; retrieving the paragraph embedding from the paragraph vector table based on the paragraph identifier; and adding the semantic vector, the position index in the target dimension, and the paragraph embedding to obtain the embedding matrix.

[0101] Specifically, step 2: This step is used to embed and encode all lexical units in the branch tree, including soft-position lexical embedding and paragraph embedding, to preserve sequence information and segment identifiers. A position number is assigned to each lexical unit so that the inserted branch node semantically inherits the sequence information of its main node, such as... Figure 2 As shown.

[0102] This layer linearly expands the branch tree t into a sequence of words of length L and maps it into a vector matrix of uniform dimension. Each word embedding is the sum of three components: dictionary, soft position, and paragraph.

[0103]

[0104] in, For TokenEmbedding, the linearly expanded word sequence T is input into the trainable dictionary matrix. The resulting semantic vector, where Voc represents the size of the vocabulary;

[0105] Soft position embedding First, number the soft positions. (The main branch uses a hard index, and the branches are offset by adding a fine-grained offset to the hard index of the parent node.) The location index is obtained and then mapped to a learnable projection matrix. Trainable mappings;

[0106] For paragraph embedding, based on paragraph identifier From the segment vector table The data was retrieved from the database and used to distinguish different sentence segments; among them, For segment markers of word elements, These are the trainable segment vectors corresponding to the two types of segment labels.

[0107] The output feature matrix is ​​formed by adding the three terms element by element. ;in, d is the length of the unfolded sequence and d is the dimension of the latent vector. This embedding preserves word meaning and encodes fine-grained relative order and sentence information, providing structured input for subsequent visible layers and Mask-Transformer.

[0108] Furthermore, obtaining structured semantic features includes: inputting the embedding matrix and visibility matrix into a stacked Mask-Transformer encoder, with each layer sequentially employing multi-head mask self-attention, a feedforward network, and residual normalization, and introducing visibility constraints to obtain structured semantic features.

[0109] Specifically, step 3: This step, based on the topological relationship of the branch tree, performs a process for each pair of tokens. Generate visibility matrix This allows for precise control over the information flow in the self-attention of the subsequent masked Transformer. Let... Indicates word elements The visibility weight is defined as follows: (Based on the branch distance in the tree, which is the number of levels to the nearest common parent node)

[0110]

[0111] in, As the reference attenuation coefficient, This is the distance adjustment factor. The first two cases are given 0 or a finite negative offset based on structural proximity (attenuation increases linearly with branch distance); the last case is directly set to 0. Completely eliminate attention between unrelated word groups. The number of triples.

[0112] Step 4: This step accepts the embedding matrix generated in Step 2. and the visibility matrix constructed in step 3 Structured semantic features are extracted using a stacked Mask-Transformer encoder. Each layer employs a multi-head masked self-attention (MSA) + feedforward network (FFN) + residual normalization framework, explicitly injecting visibility constraints. Its core computation is as follows:

[0113]

[0114] in, A trainable linear projection matrix is ​​used to project the input embedding E onto the query (Q), key (K), and value (V) space. Let i be the element in the i-th row and j-th column of the visibility matrix. For attention weights, For query vector, For input embedding, For key vectors, For value vectors, The function is used to normalize the attention scores into a probability distribution, such that the weights of each row sum to 1. For attention scoring matrix, This represents the matching score calculated from the query vector at the i-th input position and the key vector at the j-th input position, under the visibility matrix constraint. For attention output, For the updated feature representation, For position-by-position feedforward networks, For layer normalization, To query the vector in the i-th row of matrix Q, Let J be the vector in the j-th row of the key matrix K. The dimension of the key / query vector. For transpose, From step 3, ensure that elements within the same trunk / branch are fully visible, those across branches are weakly visible, and those unrelated are completely masked. This is represented by the element in the i-th row and j-th column of the visibility matrix. The attention weights A after softmax are normalized only within permissible limits to block illegal information flow. After stacking layers N times, the final feature is output. This serves as a unified semantic representation for subsequent tasks such as contrastive learning, classification, or generation. The entire encoding preserves the semantics of the original sentence while integrating knowledge branch information, and avoids semantic drift through a visibility matrix.

[0115] Furthermore, acquiring structured knowledge includes: obtaining the enhanced triple sequence, inserting a marker [CLS] at the beginning of each triple in the enhanced triple sequence, and applying it according to the target conditions. Linear arrangement is further mapped to word vectors to obtain the input tensor. The input tensor is modeled by stacked self-attention blocks. The label vector of each triple in the modeled input tensor is extracted and linearly mapped to obtain the triple embedding. Average pooling is used to aggregate all triple embeddings to obtain structured knowledge.

[0116] Specifically, step 5: First, the updated feature representation from step 4... Extracting sentence context vectors:

[0117] ;

[0118] in, It is to represent the updated features Pooling functions that compress into sentence-level vectors (sequence → vector).

[0119] After knowledge retrieval and expansion in step 1, an initial set of candidate triples is obtained. ,use right Relevance scoring, Top-K / threshold truncation, and conflict resolution are performed to obtain the final set of enhanced triples. To obtain the enhanced triplet sequence:

[0120]

[0121] in, This is the enhanced sequence (set) of triples. Let i be the head entity of the i-th triple. Let i be the relation of the i-th triple. Let i be the tail entity of the i-th triple. The number of triples, each All have been integrated with the supplementary attributes or hierarchical information of the knowledge graph. To capture deep dependencies between triples, the beginning of each triple is inserted with... Mark and press Linear arrangement, mapped to word vectors to obtain the input tensor:

[0122]

[0123] in, For the input tensor, For the latent vector dimension, Let be the vector representation of the k-th triple at position 3. Let be a matrix space over the real number field. The number of enhanced triples involved in the encoding;

[0124] Then stacked Modeling using layer self-attention blocks:

[0125]

[0126] in, For step 3 generated As can be seen, the matrix is ​​obtained by slicing it into subsets of positions and applying local constraints. The visible matrix For the first Input / output hidden states of a Transformer coded block It is a standard coding block (multi-head self-attention + residual + layer normalization + feedforward network). For the coding layer index, The length of the linearized sequence for a single triplet is given, ensuring that the attention follows the constraints of "fully visible within the same domain / attenuated across domains / irrelevant masking". For each triplet... Vectors, after linear mapping, yield triplet embeddings:

[0127]

[0128] in, To map the [CLS] vector to linear mapping parameters of triple embeddings, For the first The triplet after this step After stacked encoding, the final hidden vector is located at position [CLS].

[0129] Finally, average pooling is used to aggregate all triplet embeddings to obtain a structured knowledge representation:

[0130]

[0131] in, The number of enhanced triples participating in the encoding, It also includes the semantics of the original sentence, knowledge extension information, and triple-level interdependencies, providing high-quality knowledge embedding for subsequent gating fusion with the text backbone vector and cross-modal comparative learning.

[0132] Furthermore, obtaining benchmark text features and benchmark image features includes:

[0133] Step 6: This step directly uses the text encoder provided by the CLIP (Commonly Targeted Learning Platform) model. The input is the original text s, which is first segmented using CLIP's built-in BPE vocabulary and [SOS] and [EOS] tags are inserted at the beginning and end to obtain the index sequence t. The model maps words using a fixed, learnable word vector table Wclip, plus CLIP's predefined absolute position embeddings. This forms the input matrix:

[0134]

[0135] in, For the input matrix, as a word element Embedded vector, Embedding a table for absolute position as position The position vector;

[0136] Then Send in A multi-layer TransformerEncoder (multi-head self-attention + feedforward + residual layer normalization). The hidden state vector at the [SOS] position in the final layer is taken as the semantic representation of the entire sentence, and then normalized by the layer and... Normalization yields:

[0137]

[0138] in, For CLIP text branches, stack the L-level Transformer Encoder;

[0139] This vector is in the same alignment space as the CLIP image encoder output, serving as the benchmark text feature for subsequent "text backbone-knowledge vector" fusion and cross-modal comparative learning.

[0140] Step 7: This step utilizes the CLIP model's visual encoder. The input image I is preprocessed: the longer side is scaled to 224px, the center is cropped, RGB is normalized to [0,1], and the channel mean is subtracted. The global feature vector of the last block is obtained through a pre-trained Vision-Transformer. Then multiply by the projection matrix fixed by CLIP. and carry out Normalization yields image embeddings spatially aligned with the text branches:

[0141]

[0142] in, Image embedding that is spatially aligned with the text branches, This is the global feature vector.

[0143] This vector serves as the baseline image feature for similarity calculation with text / knowledge fusion representation in subsequent cross-modal contrastive learning.

[0144] Furthermore, obtaining the fusion vector includes: concatenating the baseline text features and structured knowledge through a linear mapping, activation, and linear mapping layer to obtain scalar gating coefficients; and using the scalar gating coefficients to perform residual weighted fusion of the baseline text features and structured knowledge to obtain the fusion vector.

[0145] Specifically, step 8: to fully utilize the native semantics of the text With structured knowledge semantics This step introduces a learnable gating function. It automatically determines the fusion ratio of the two signals based on the sample content. Specifically, it first splices the two signals together. The scalar gating coefficients are obtained by passing through a linear mapping-activation-linear mapping layer:

[0146]

[0147] in, Both biases b1 and b2 are trainable parameters, and σ is the sigmoid activation function. The activation function for the Rectified Linear Unit. This is the input vector to the gated network. Then, the final fused vector is obtained by weighting the vectors using residuals.

[0148]

[0149] When the input sentence has a high degree of knowledge dependence (e.g., contains a large number of specialized entities or relationships), the network learns... This allows semantic knowledge to dominate; if the text itself can fully express the meaning, then... It primarily preserves the original semantics. This dynamic gating fusion avoids excessive intrusion of knowledge noise while ensuring full utilization of structured supplementary information when needed, providing a content-adaptive unified text representation for subsequent cross-modal contrastive learning.

[0150] Furthermore, obtaining a high-quality negative set includes: retrieving the corresponding set of triples from the knowledge graph for the positive sample text, performing controlled perturbations on the key elements in the set of triples corresponding to the positive sample text; the controlled perturbations include: replacing similar entities, replacing attribute values, and replacing relations; writing the replaced triples back to the original sentence template to obtain a candidate negative sample set, and removing texts that still match the corresponding image from the candidate negative sample set to obtain a high-quality negative set.

[0151] Specifically, step 9: For each positive sample text s, first retrieve its matching set of triples from the knowledge graph K:

[0152]

[0153] Among them, K A knowledge graph for the field of lantern culture. This is a query operator / process based on entity recognition and graph retrieval. The number of triplet records retrieved;

[0154] While maintaining syntactic readability, the set of triples The key element in this process is the execution of controlled disturbances:

[0155] Replacement of similar entities:

[0156] Attribute value replacement:

[0157] Relationship replacement:

[0158] in, For the head entity of the i-th triple after replacement, For the tail entity of the i-th triple after replacement, The relation of the i-th triple after replacement;

[0159] Replaced triplet Write back to the original sentence template to obtain the candidate negative sample set:

[0160]

[0161] in, This is the set of candidate negative samples. For the k-th candidate negative sample text, For text rewriting / replacement operators, For attention scoring matrix, Let i be the head entity of the i-th triple. For the i-th triple, For the tail entity of the i-th triple, For the replaced head entity, For the replaced relationship, The replaced tail entity;

[0162] Then use an image-text coarse finisher or rules. Sentences that may still match the corresponding image are removed to form the final high-quality negative set:

[0163]

[0164] According to the replacement depth By labeling the model with Easy / Medium / Hard difficulty levels and sampling according to the course learning strategy, the model can see both semantically distinct easy negative samples and seemingly reasonable but incorrect difficult negative samples during comparative training, thereby gaining stronger discriminative ability in fine-grained cross-modal alignment.

[0165] Furthermore, obtaining the symmetric contrastive learning loss includes:

[0166] Step 10: Complete the text-knowledge fusion vector With image vectors After extraction, these samples, along with the difficult negative samples generated from the knowledge graph, are input into the contrastive learning module. Let a batch contain group B positive samples. And the i-th text is equipped with m negative samples. Normalized dot product similarity is used:

[0167]

[0168] The InfoNCE loss in the image-to-text direction is:

[0169]

[0170] Where γ≥1 represents the weight of the hardest-to-negative sample; the same applies to the text-image direction:

[0171]

[0172] in, Let be the similarity of the i-th pair of positive samples, i.e., the image embedding. Similarity to the i-th text embedding in the batch, To avoid negative sample weights, It is the natural logarithm function. It is a natural exponential function. Image embedding for the i-th sample, For the text embedding of the i-th sample, For temperature coefficient, This is a positive sample from group B. The number of hard-to-bear samples for each anchor point Embedding text Similarity to the embedding of the i-th image in the batch, Embedding text Similarity with negative sample image embeddings, Image embedding Similarity with negative sample text embeddings For embedding relative to the i-th text The embedding of the k-th negative sample image, Embedding relative to the i-th image The k-th negative sample text embedding.

[0173] The final symmetric contrastive learning loss is:

[0174]

[0175] in, For symmetric contrastive learning loss, This is a positive sample from group B. For the InfoNCE loss in the image-to-text direction, For the InfoNCE loss in the text-image direction.

[0176] This loss not only narrows the distance between positive samples in the same group, but also strengthens the penalty for negative samples that are "semantically similar but not matched" by the knowledge graph through γ, thereby significantly improving the discriminative power and robustness of cross-modal representations in fine-grained alignment tasks.

[0177] The CLIP model, fine-tuned using knowledge enhancement methods, demonstrates significant performance improvements in image generation tasks (such as Flux and StableDiffusion models). Traditional image-text pairing training methods typically focus on simple matching and comparison, often neglecting fine-grained semantic relationships between different modalities. This results in the existing CLIP model performing poorly when handling complex semantic relationships, especially in tasks requiring reasoning, where it often fails to fully understand and utilize the deep connections between modalities.

[0178] The knowledge-enhanced framework significantly improves the semantic alignment between images and language by introducing a structured knowledge graph. This enhancement allows the model to better understand and fuse fine-grained semantic information across different modalities, thereby improving its ability to perceive and reason about textual details in generative modeling tasks. This approach effectively compensates for the shortcomings of traditional models in handling complex semantic reasoning, driving progress in semantic understanding within image generation tasks. Figure 3 As shown.

[0179] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal model semantic enhancement and contrastive learning method based on a lantern knowledge graph, characterized in that, include: Obtain the original text, associate the original text with the knowledge graph, generate a structured sentence tree, perform embedding encoding, and obtain the embedding matrix; Obtaining the embedding matrix includes: The structured sentence tree is linearly expanded into a word sequence of the target length, and the linearly expanded word sequence is then embedded and encoded. Dictionary embedding encoding is performed on the linearly expanded word sequence: the linearly expanded word sequence is input into a trainable dictionary matrix to obtain a semantic vector; The linearly expanded word sequence is encoded using soft position embedding: the position index is obtained based on the soft position number and mapped to the target dimension via a learnable projection matrix; The linearly expanded word sequence is encoded by paragraph embedding: the paragraph embedding is retrieved from the paragraph vector table based on the paragraph identifier; The semantic vector, the position index under the target dimension, and the paragraph embedding are added together to obtain the embedding matrix; Obtain the visibility matrix of each pair of lexical units in the structured sentence tree, input the embedding matrix and the visibility matrix into a stacked Mask-Transformer encoder to obtain structured semantic features, and model them using stacked self-attention blocks to obtain structured knowledge; The original text is input into the text encoder and visual encoder of the multimodal model respectively to obtain the baseline text features and baseline image features. The baseline text features are then dynamically gated and weighted to fuse with the structured knowledge to obtain the fusion vector. Obtain positive sample text, and based on the positive sample text, obtain a high-quality negative set. Input the fusion vector, the baseline image features, and the corresponding difficult negative samples from the high-quality negative set into the contrastive learning module to obtain the symmetric contrastive learning loss, which is used to train the multimodal model.

2. The multimodal model semantic enhancement and contrastive learning method based on colored lantern knowledge graph according to claim 1, characterized in that, Generating the structured sentence tree includes: Entity recognition is performed on the original text to determine the set of entities involved in the text. Each entity in the set is then retrieved from the knowledge graph to obtain a set of triples. in, For a set of triples, For knowledge retrieval, The input sentence is from the original text. A pre-constructed knowledge graph for the field of lantern culture; The triples in the triple set are injected into the sentences in the text according to the corresponding positions of the entities to form the structured sentence tree.

3. The multimodal model semantic enhancement and contrastive learning method based on colored lantern knowledge graph according to claim 1, characterized in that, Obtaining the structured semantic features includes: The embedding matrix and the visibility matrix are input into a stacked Mask-Transformer encoder. Each layer sequentially employs multi-head mask self-attention, a feedforward network, and residual normalization, and introduces visibility constraints to obtain the structured semantic features. in, A trainable linear projection matrix is ​​used to project the input embedding E onto the query (Q), key (K), and value (V) space. Let i be the element in the i-th row and j-th column of the visibility matrix. For attention weights, For query vector, For input embedding, For key vectors, For value vectors, The function is used to normalize the attention scores into a probability distribution, such that the weights of each row sum to 1. For attention scoring matrix, This represents the matching score calculated from the query vector at the i-th input position and the key vector at the j-th input position, under the visibility matrix constraint. For attention output, For the updated feature representation, For position-by-position feedforward networks, For layer normalization, To query the vector in the i-th row of matrix Q, Let J be the vector in the j-th row of the key matrix K. The dimension of the key / query vector. This is a transpose.

4. The multimodal model semantic enhancement and contrastive learning method based on the knowledge graph of colored lanterns according to claim 3, characterized in that, The visibility constraints include: in, Let i be the element in the i-th row and j-th column of the visibility matrix. As the reference attenuation coefficient, As a distance adjustment factor, For the word element in the i-th row j column of words Branch distance in the tree.

5. The multimodal model semantic enhancement and contrastive learning method based on colored lantern knowledge graph according to claim 1, characterized in that, Obtaining the structured knowledge includes: Obtain the enhanced triple sequence: in, For the enhanced triplet sequence, Let i be the head entity of the i-th triple. For the i-th triple, For the tail entity of the i-th triple, The number of triples; Insert a marker at the beginning of each triple in the enhanced triple sequence. and in accordance with the target conditions Linear arrangement, further mapped to word vectors, to obtain the input tensor; The input tensor is modeled using stacked self-attention blocks: ； in, The visible matrix, For standard coded blocks, For the first Input / output hidden states of a Transformer coded block For the coding layer index, The length of the linearized sequence of a single triplet; The label vector of each triplet in the input tensor after modeling is extracted and linearly mapped to obtain the triplet embedding. Then, average pooling is used to aggregate all triplet embeddings to obtain the structured knowledge.

6. The multimodal model semantic enhancement and contrastive learning method based on lantern knowledge graph according to claim 1, characterized in that, Obtaining the fusion vector includes: The baseline text features and the structured knowledge are concatenated, and then subjected to a linear mapping, activation, and another linear mapping to obtain scalar gating coefficients. in, For scalar gating coefficients, For trainable parameters, It is the Sigmoid activation function. The activation function for the Rectified Linear Unit. is the input vector of the gated network; The scalar gating coefficients are used to perform residual weighted fusion of the baseline text features and the structured knowledge to obtain the fusion vector.

7. The multimodal model semantic enhancement and contrastive learning method based on lantern knowledge graph according to claim 1, characterized in that, Obtaining the high-quality negative set includes: The positive sample text is retrieved from the knowledge graph to obtain the corresponding set of triples. The key elements in the set of triples corresponding to the positive sample text are subjected to controlled perturbation. The controlled perturbation includes: replacement of similar entities, replacement of attribute values, and replacement of relations. Write the replaced triples back to the original sentence template to obtain the candidate negative sample set: in, For the candidate negative sample set, For the k-th candidate negative sample text, For text rewriting / replacement operators, For attention scoring matrix, Let i be the head entity of the i-th triple. For the i-th triple, For the tail entity of the i-th triple, For the replaced head entity, For the replaced relationship, The replaced tail entity; Remove the text that still matches the corresponding image from the candidate negative sample set to obtain the high-quality negative set.

8. The multimodal model semantic enhancement and contrastive learning method based on lantern knowledge graph according to claim 1, characterized in that, Obtaining the symmetric contrastive learning loss includes: in, For symmetric contrastive learning loss, This is a positive sample from group B. For the InfoNCE loss in the image-to-text direction, For the InfoNCE loss in the text-image direction.

9. The multimodal model semantic enhancement and contrastive learning method based on colored lantern knowledge graph according to claim 8, characterized in that, The InfoNCE loss in the image-text direction includes: The InfoNCE loss in the text-image orientation includes: in, Let be the similarity of the i-th pair of positive samples, i.e., the image embedding. Similarity to the i-th text embedding in the batch, To avoid negative sample weights, It is the natural logarithm function. It is a natural exponential function. Image embedding for the i-th sample, For the text embedding of the i-th sample, For temperature coefficient, This is a positive sample from group B. The number of hard-to-bear samples for each anchor point Embedding text Similarity to the embedding of the i-th image in the batch, Embedding text Similarity with negative sample image embeddings, Image embedding Similarity with negative sample text embeddings For embedding relative to the i-th text The embedding of the k-th negative sample image, Embedding relative to the i-th image The k-th negative sample text embedding.

Images