Conflict mitigation cross-modal search method

Abstract

The invention discloses a conflict mitigation cross-modal search method. The method comprises the following steps: feature extraction: constructing feature representation for each modal and a corresponding Hash tag; modal decoupling: separating modal features to form universal and unique Hash codes; label optimization: refining label information by using universal hash codes; semantic mapping, wherein semantic alignment between modes is achieved through the optimized tag hash codes. According to the method, through an automatic feature extraction and modal decoupling mechanism, images and texts can be converted into deep feature representation, and modal features are separated into common and specific Hash codes, so that the inconsistency between modals is reduced, and the retrieval accuracy is improved; according to the modal decoupling part, a negative sample contrast learning method is adopted, and the semantic definition of modal public hash codes is ensured by enhancing exclusiveness between modal unique hash codes. The comparative learning strategy is beneficial to improving the discrimination degree of information between modals, so that the model can identify and distinguish unique features of different modal data more accurately.

Description

Technical Field

[0001] The present invention relates to a search method, and in particular to a conflict mitigation cross-modal search method. Background Art

[0002] With the advent of the digital age, multimodal data (including images, text, etc.) has become a key asset for the development of enterprises and society. There are complex relationships between these different modal data, and traditional retrieval methods often focus on a single modality, ignoring the mutual complementation and enhancement between modalities. The development of deep learning technology has provided new possibilities for the effective alignment and retrieval of cross-modal data. Through feature extraction and modal decoupling mechanisms, multimodal data can be better processed and understood. In addition, cross-modal retrieval faces the challenge of how to effectively align data of different modalities, especially in the presence of data inconsistency. This inconsistency may be caused by differences in data collection, processing or annotation. Therefore, deep learning technology is used to improve the accuracy and efficiency of cross-modal data retrieval, and innovative methods such as modal decoupling and label optimization are used to enhance the robustness of the model to data inconsistency.

[0003] Traditional retrieval methods have obvious limitations when dealing with multimodal data. They usually only focus on a single modality and fail to fully utilize the interrelationships and complementarities between different modal data, which limits the accuracy and efficiency of retrieval. In terms of multimodal label optimization, some studies have proposed different methods and techniques, but these methods still have shortcomings in dealing with inter-modal inconsistencies and modality-label inconsistencies. For example, the current multimodal retrieval enhanced generation technology (mRAG) (Siyue Z, Yuxiang X, Yiming Z, et al. MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering [J]. arXiv preprint arXiv: 2412.15540, 2024.) is a technology in the field of multimodal retrieval, which enhances the generation model by retrieving multimodal information. However, mRAG has a trade-off between attribution and fluency, and the output quality will be affected due to the additional restrictions of retrieval knowledge. At the same time, there are large differences between the data expressions of different modalities, such as the huge differences between text and images. How to measure the correlation between different modal data has become one of the challenges. Summary of the invention

[0004] In order to solve the defects in the prior art, the present invention discloses a conflict mitigation cross-modal search method, which is characterized by:

[0005] Step 1: Construct feature representation for each modality and its corresponding hash tag to implement the feature extraction process of the data;

[0006] Step 2: Separate modal features to form common and unique hash codes to achieve modal decoupling process;

[0007] Step 3: Use universal hash coding to refine label information and implement label optimization process;

[0008] Step 4: Use the optimized tag hash code to achieve semantic alignment between modalities and realize the search result matching process.

[0009] The present invention also discloses a conflict mitigation cross-modal search device, which is characterized by:

[0010] Feature extraction module: construct feature representation for each modality and its corresponding hash tag;

[0011] Modal decoupling module: separates modal features to form common and unique hash codes;

[0012] Label optimization module: refine label information using universal hash coding;

[0013] Semantic mapping module: uses optimized tag hash codes to achieve semantic alignment between modalities and match search results.

[0014] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0015] The present invention also discloses a terminal device, characterized in that the terminal device includes: a processor, a memory, a communication interface and a bus; the processor, the memory and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the above method.

[0016] Beneficial Effects

[0017] (1) A feature extraction method is designed to extract the original image and text Data feature extraction It provides high-quality input features for deep modal decoupling and semantic mapping.

[0018] (2) In the modal decoupling step, negative sample contrast learning is used to separate the common and unique features between modalities, reduce the inconsistency between modalities, and enhance the consistency of information within the modality.

[0019] (3) By analyzing and understanding the label data, common semantic information is extracted and used to optimize the label hash code to improve the accuracy of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the overall framework of the conflict mitigation cross-modal search method of the present invention;

[0021] Figure 2 This is the modal decoupling structure diagram of the present invention. DETAILED DESCRIPTION

[0022] The present invention includes four key steps: (1) Feature extraction: constructing feature representation for each modality and its corresponding hash tag; (2) Modality decoupling: separating modality features to form common and unique hash codes; (3) Label optimization: refining label information using common hash codes; (4) Semantic mapping: using optimized label hash codes to achieve semantic alignment between modalities. This process builds an efficient multimodal decoupling network. This facilitates the learning of accurate label hash codes (unified hash codes) to maintain the consistency of instances in the dataset.

[0023] The data of the present invention is composed of image-text pairs, which are used as the image modality and text modality of the experiment respectively. Each image-text pair is annotated by one or more tags to mark the image and text. The present invention randomly selects 10,500 pairs of multi-label image-text pairs and keeps them evenly distributed on the label categories.

[0024] Step 1: Construct feature representation for each modality and its corresponding hash tag to implement the feature extraction process of the data.

[0025] The data of the present invention is composed of image-text pairs, which are used as the image modality and text modality of the experiment respectively. Each image-text pair is annotated by one or more tags to mark the image and text. The present invention randomly selects 10,500 pairs of multi-label image-text pairs and keeps them evenly distributed on the label categories.

[0026] For the image modality, the fc7 layer of the CNN-F network is used to extract features from the image sample to obtain a 4096-dimensional feature vector. This feature vector contains the visual information of the image sample and provides the input data of the image modality for modality decoupling. For the text modality, the Bag-of-Words (BoW) model is used to convert the text into a vector form. For each text sample, the BoW model calculates the frequency of occurrence of words and generates a high-dimensional sparse vector, where each dimension corresponds to a word in the vocabulary. Then the features are further extracted through the fully connected layer to obtain the text feature vector. The first fully connected layer usually has a large dimension (e.g., 1024) and can capture the complex patterns and semantic information in the BoW vector. The second fully connected layer has a larger dimension (e.g., 4096), which further integrates and refines the features to generate a dense feature vector. This feature vector captures the semantic information of the text sample and provides the input data of the text modality for modality decoupling. This 4096-dimensional feature vector is the representation of the image sample in the feature space, which provides the input data of the image modality for modality decoupling. The resulting text feature vector captures the semantic information of the text sample, including the topic, sentiment, and context. This feature vector provides input data of the text modality for modal decoupling, enabling the model to identify and distinguish between modal common information and modal-specific information in the text. The quality of the feature vector directly affects the effect of modal decoupling. Therefore, using a powerful CNN-F network can improve the accuracy and robustness of feature extraction.

[0027] For the image modality, the fc7 layer of the CNN-F network is used to extract features from the image sample to obtain a 4096-dimensional feature vector. This feature vector contains the visual information of the image sample and provides the input data of the image modality for modality decoupling. For the text modality, the Bag-of-Words (BoW) model is used to convert the text into a vector form. For each text sample, the BoW model calculates the frequency of occurrence of words and generates a high-dimensional sparse vector, where each dimension corresponds to a word in the vocabulary. Then the features are further extracted through the fully connected layer to obtain the text feature vector. The first fully connected layer usually has a large dimension (e.g., 1024) and can capture the complex patterns and semantic information in the BoW vector. The second fully connected layer has a larger dimension (e.g., 4096), which further integrates and refines the features to generate a dense feature vector. This feature vector captures the semantic information of the text sample and provides the input data of the text modality for modality decoupling. This 4096-dimensional feature vector is the representation of the image sample in the feature space, which provides the input data of the image modality for modality decoupling. The resulting text feature vector captures the semantic information of the text sample, including the topic, sentiment, and context. This feature vector provides input data of the text modality for modal decoupling, enabling the model to identify and distinguish between modal common information and modal-specific information in the text. The quality of the feature vector directly affects the effect of modal decoupling. Therefore, using a powerful CNN-F network can improve the accuracy and robustness of feature extraction.

[0028] The present invention converts the original samples of the image modality and text modality in the dataset into deep features, k j and u j Respectively represent instance o j The original samples of the image modality and the original samples of the text modality are used as the input for feature extraction; the features extracted from each modality are expressed as follows:

[0029]

[0030] and Respectively represent the backbone networks for image modality and text modality feature extraction, and t represents the dimension of modality features. j represents the index features of image and text modalities, which are used to distinguish or reference specific data points or feature vectors; k represents the original sample parameters of image modality, which are used for classification tasks or model training; u represents the original sample parameters of text modality; R represents the association mapping between image modality and text modality. For multi-label, the present invention does not operate on the intermediate feature vector, but directly maps it to the hash space, and obtains the multi-label hash code through several fully connected layers, as shown below:

[0031] p j = lp (x j )∈-1,1 n (3)

[0032] Among them l p (x j ) is a label hash encoder, p represents a label hash encoder, which is a function that maps multimodal features to Hamming space; x j The feature vector representing the image or text, n is the length of the learned hash code, and the final output is the multi-label hash code p j ;

[0033] Step 2: Separate the modal features to form common and unique hash codes to achieve the modal decoupling process.

[0034] Two forms of contrastive learning loss are used, one is the modality common contrastive learning loss, and the other is the modality specific contrastive learning loss. And a mutually exclusive constraint is used to effectively decouple the modality information into modality common hash codes and modality specific hash codes. Two hash encoders are designed for each modality: a common hash encoder (used to generate modality common hash codes) and a unique hash encoder (used to generate modality specific hash codes). These encoders can separate the feature information of the modality into common and unique parts.

[0035] With multimodal samples j =(k j ,u j ,x j ) as an example, k j represents the image modality, u j Indicates text mode, x j Represents a label annotation vector. and They represent the images generated by formula (1) and text The feature vector of the image and text The public hash code of each modality is determined by the public hash encoder of each modality. and The learning is expressed as:

[0036]

[0037] and They represent the images generated by formula (1) and text The eigenvector of j and u j Unique hash code and By the corresponding unique hash function and Generates as follows:

[0038]

[0039] In the modality decoupling part, the present invention designs a special structure to decouple modality common information and modality specific information. The correlation between modality common hash codes of related instances is ensured by negative sample contrast learning loss, while the information redundancy between modality common and modality specific hash codes is reduced by using modality specific hash codes as negative samples.

[0040] Negative sample contrast learning is carried out as follows:

[0041] For each positive image-text pair in the mini-batch The public hash code of Different negative text and Different negative images, then, by comparing the sampled negative text or image with the positive image or text Pairing to create negative sets Based on the modality common information, each positive image-text pair The contrast loss function Expressed as:

[0042]

[0043] τ Represents image-text pairs Scaling parameters for data processing.

[0044] In practice, information tends to flow between generic and specific hash codes. The information of many instances can be fully encoded by the modality-unique hash code, allowing the decoder to easily reconstruct the input, rendering the public hash code useless. To avoid the redundancy of the public hash code, it is necessary to ensure that the unique hash codes of different modalities in the same instance contain mutually exclusive information. To this end, we use the L2 norm (i.e., Euclidean norm) to enhance the differences between modality-unique hash codes.

[0045] To enhance feature decoupling, a mutual exclusion constraint is introduced to ensure that the modality-specific hash code has completely different information in each instance, avoiding information leakage between common and unique hash codes. Subsequently, the semantic information of common labels is optimized through label refinement loss, and effective alignment is performed using a cross-modal common semantic alignment module.

[0046] Step 3: Use universal hash coding to refine label information and implement label optimization process.

[0047] Label optimization uses the common hash codes shared by image and text modalities to extract and optimize the shared semantic information in labels. When the common hash codes in the same modality are highly similar, the similarity of the label hash codes is enhanced accordingly. Given that semantically similar instances should have consistent common labels, the present invention first extends it to n-dimensional label hash codes. When the common hash codes in the same modality are highly similar, the similarity of the label hash codes is enhanced accordingly. The cosine similarity cos(a, b) between the common hash codes of the two modalities is used to measure the similarity between the two common hash codes, and the exponential function e is used to calculate the similarity between the two common hash codes. cos(a,b) Map the similarity to a positive value to obtain the refined label hash code The multi-label refinement loss function is as follows:

[0048]

[0049] represents the multi-label refinement loss function, which is used to adjust the label hash code to reflect the similarity between the modal public hash codes; j, r represents the index of the label, which is used to traverse all labels; and are the public hash codes for image and text modalities respectively; and is a unique hash code for image and text modal; p j and p r Represents the hash code of the multi-label image and text modality output by formula (3); Indicates that the cosine similarity between two label hash codes is mapped to a positive value through an exponential function; Indicates that the cosine similarity is mapped to a positive value through an exponential function to enhance the similarity of the label hash code; bs represents the total number of labels.

[0050] Step 4: Use the optimized tag hash code to achieve semantic alignment between modalities and realize the search result matching process.

[0051] Through semantic mapping, the optimized label information is used to achieve effective alignment between different modalities. For any two instances o j and r Construct two types of triplets: (1) positive triplets between instances (2) Negative triples in each instance Based on these two types of triplets, different constraint loss functions are constructed; the positive triplet constraint uses the improved multi-label hash code as a bridge to establish correlations between the public hash codes of different instances; the positive triplet constraint loss The function can be defined as:

[0052]

[0053] represents the positive triple constraint loss function, which is used to enhance the similarity of hash codes in positive triples; bs represents the total number of positive triples; j, r represents the index of positive triples, which is used to traverse all positive triples; p′ j and p′ r represents the normalized version of the modal hash code of the jth and rth instances; m represents a scaling factor in the cosine similarity function, which is used to adjust the weight of the similarity; cos(p′ j ,p′ r ) represents the cosine similarity between the modal hash codes of the j-th and i-th instances; is the public hash code of the image modality of the bth label of the jth instance; A unique hash code representing the text modality of the b-th label of the i-th instance.

[0054] In addition, the present invention designs a negative triple constraint to enhance the distinction between the modality-unique hash codes in each instance. can be written as:

[0055]

[0056] represents the negative triple constraint loss function, which is used to enhance the difference of hash codes in negative triples; bs represents the total number of negative triples; j, r represents the index of negative triples; used to traverse all negative triples; p′ j and p′ r represents the normalized version of the modal hash code of the jth and rth instances; m represents a scaling factor in the cosine similarity function, which is used to adjust the weight of the similarity; cos(p′ j ,p′ r ) represents the cosine similarity between the modal hash codes of the j-th and i-th instances; represents the difference between the modal hash code and the negative modal hash code of the jth instance; Represents the difference between the modal hash code and the negative label hash code of the jth instance.

[0057] The positive and negative triple constraint losses are combined as follows:

[0058]

[0059] L Triple - represents the negative triple constraint loss function, which is used to enhance the difference of hash codes in negative triples; α represents a balancing factor, which is used to adjust the weight between positive and negative triple constraint loss functions; in the combined loss function L Triple-, α determines the influence of the negative triplet loss function on the total loss. Finally, the optimal result matching is obtained by comparing the loss function results.

[0060] Through feature extraction and modality decoupling mechanisms, the present invention can convert images and texts into deep feature representations, and separate modal features into common and unique hash codes, thereby reducing inconsistencies between modalities and improving retrieval accuracy.

[0061] A label optimization strategy is adopted, which helps improve the semantic alignment between different modalities by refining and enhancing the shared semantic information in the labels. This helps ensure that semantically related instances can be accurately retrieved even when there are inconsistencies between modalities.

[0062] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A conflict mitigation cross-modal search method, characterized by: Step 1: Construct feature representation for each modality and its corresponding hash tag to implement the feature extraction process of the data; Step 2: Separate modal features to form common and unique hash codes to achieve modal decoupling process; Step 3: Use universal hash coding to refine label information and implement label optimization process; Step 4: Use the optimized tag hash code to achieve semantic alignment between modalities and realize the search result matching process.

2. The conflict mitigation cross-modal search method according to claim 1 is characterized in that: the feature extraction process of the data in step 1 includes the following contents: The data consists of image-text pairs, which are used as the image modality and text modality of the experiment respectively; each image-text pair is annotated by one or more labels to mark the image and text, and multiple pairs of multi-label image-text pairs are randomly selected and kept evenly distributed on the label categories; the original samples of the image modality and the text modality are converted into deep features, where k j and u j Respectively represent the original samples of the image modality and the original samples of the text modality of the instance, as the input of feature extraction, j represents the index features of the image and text modalities, which are used to distinguish or reference specific data points or feature vectors; k represents the original sample parameters of the image modality, which are used for classification tasks or model training; u represents the original sample parameters of the text modality; R represents the association mapping between the image modality and the text modality; the features extracted from each modality are expressed as follows: and Respectively represent the backbone networks for image modality and text modality feature extraction, k represents, t represents the dimension of modality features; for multi-label, no operation is performed on the intermediate feature vector, it is directly mapped to the hash space, and the multi-label hash code is obtained through several fully connected layers, as shown below: p j =l p (x j )∈-1,1 n (3) Among them l p (x j ) is a label hash encoder, p represents a label hash encoder, which is a function that maps multimodal features to Hamming space; x j The feature vector representing the image or text; n is the length of the learned hash code, and the final output is the multi-label hash code p j ; p parameter definition.

3. The conflict mitigation cross-modal search method according to claim 2, wherein step 2 comprises the following contents: If the modal decoupling process is based on multimodal samples o j =(k j ,u j ,x j ),image and text The public hash code of each modality is determined by the public hash encoder of each modality. and The learning is expressed as: in, and They represent the images generated by formula (1) and text The eigenvector of j and u j Unique hash code and By the corresponding unique hash function and Generates as follows: The negative sample contrastive learning is carried out as follows: For each positive image-text pair in a mini-batch The public hash code of Different negative text and Different negative images, then, by comparing the sampled negative text or image with the positive image or text Pairing to create negative sets Based on the modality common information, each positive image-text pair The contrast loss function Expressed as: Among them, τ represents the image-text pair Scaling parameters for data processing.

4. The conflict mitigation cross-modal search method according to claim 1 is characterized in that: the process of implementing label optimization by refining label information using universal hash coding includes the following contents: The label optimization process first expands the label to an n-dimensional label hash code. When the public hash codes in the same modality are highly similar, the similarity of the label hash codes is correspondingly enhanced. The cosine similarity cos(a, b) between the public hash codes of the two modalities is used to measure the similarity between the two public hash codes, and the exponential function e is used to calculate the similarity between the two public hash codes. cos(a,b) Map the similarity to a positive value to obtain the refined label hash code The multi-label refinement loss function is as follows: in, represents the multi-label refinement loss function, which is used to adjust the label hash code to reflect the similarity between the modal public hash codes; j, r represents the index of the label, which is used to traverse all labels; and are the public hash codes for image and text modalities respectively; and is a unique hash code for image and text modal; p j and p r Represents the hash code of the multi-label image and text modality output by formula (3); Indicates that the cosine similarity between two label hash codes is mapped to a positive value through an exponential function; Indicates that the cosine similarity is mapped to a positive value through an exponential function to enhance the similarity of the label hash code; bs represents the total number of labels.

5. The conflict mitigation cross-modal search method according to claim 1 is characterized in that: the semantic alignment between modalities is achieved by using the optimized tag hash code, and the search result matching process includes the following contents: For any two instances o j and r Construct two types of triplets: (1) positive triplets between instances (2) Negative triples in each instance Based on these two types of triplets, different constraint loss functions are constructed; the positive triplet constraint uses the improved multi-label hash code as a bridge to establish correlations between the public hash codes of different instances; the positive triplet constraint loss The function can be defined as: in, represents the positive triple constraint loss function, which is used to enhance the similarity of hash codes in positive triples; bs represents the total number of positive triples; j, r represents the index of positive triples, which is used to traverse all positive triples; p′ j and p′ r represents the normalized version of the modal hash code of the jth and rth instances; m represents a scaling factor in the cosine similarity function, which is used to adjust the weight of the similarity; cos(p′ j ,p′ r ) represents the cosine similarity between the modal hash codes of the j-th and i-th instances; is the public hash code of the image modality of the bth label of the jth instance; A unique hash code representing the text modality of the bth label of the i-th instance A negative triple constraint is adopted to enhance the distinction between the modal unique hash codes in each instance; the loss function ripple can be written as: in, represents the negative triple constraint loss function, which is used to enhance the difference of hash codes in negative triples; bs represents the total number of negative triples; j, r represents the index of negative triples, which is used to traverse all negative triples; p′ j and p′ r represents the normalized version of the modal hash code of the jth and rth instances; m represents a scaling factor in the cosine similarity function, which is used to adjust the weight of the similarity; cos(p′ j ,p′ r ) represents the cosine similarity between the modal hash codes of the j-th and i-th instances; represents the difference between the modal hash code and the negative modal hash code of the jth instance; represents the difference between the modal hash code and the negative label hash code of the jth instance; The positive and negative triple constraint losses are combined as follows: Among them, L Triple - represents the negative triple constraint loss function, which is used to enhance the difference of hash codes in negative triples; α represents a balancing factor, which is used to adjust the weight between the positive and negative triple constraint loss functions in the combined loss function L Triple -, α determines the influence of the negative triplet loss function on the total loss. Finally, the optimal result matching is obtained by comparing the loss function results.

6. A conflict mitigation cross-modal search device, characterized by: Feature extraction module: construct feature representation for each modality and its corresponding hash tag; Modal decoupling module: separates modal features to form common and unique hash codes; Label optimization module: refine label information using universal hash coding; Semantic mapping module: uses optimized tag hash codes to achieve semantic alignment between modalities and match search results.

7. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute the method according to any one of claims 1 to 5 when the program is executed.

8. A terminal device, characterized in that: The terminal device includes: a processor, a memory, a communication interface and a bus; the processor, the memory and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of claims 1 to 5.

Images