A multimodal content retrieval method and system of text content combined with image analysis

Abstract

The application discloses a kind of multi-modal content retrieval methods of text content combined with image analysis.It includes: data set is preprocessed, and text image information pair is obtained;Image and text features are extracted, and multi-modal attention calculation is carried out on image features and text features, to obtain multi-modal features;Image, text and multi-modal features are encoded to form corresponding hash code;Target loss function is constructed, and multi-modal hash code generation model is trained;Using multi-modal hash code generation model, the multi-modal hash code database of the database is constructed from the database to be retrieved;According to the text information input by user, generate multi-modal hash code, and match with multi-modal hash code database, to obtain retrieval result.The application also discloses a kind of multi-modal content retrieval systems of text content combined with image analysis.The application uses multi-modal hash code, fundamentally captures the commonality between modalities, makes up the heterogeneous gap between modalities, and significantly improves the extraction efficiency of effective features.

Description

Technical Field

[0001] This invention relates to retrieval technology, specifically to a multimodal content retrieval method and system that combines text content with image analysis. Background Technology

[0002] Over the past decade, various types of multimedia data have exploded on the internet through text descriptions, images, and videos. Web pages, when conveying information to their audiences, use language, images, and videos to narrate the same event or topic. This diverse range of data presented in this way is called multimodal data. Therefore, how to achieve rapid and accurate multimodal retrieval in the internet age has attracted widespread attention from researchers.

[0003] Today, mobile communication devices and emerging social networking sites (such as Facebook, Flickr, YouTube, and Twitter) are changing the way people interact with the world and search for information of interest. It would be incredibly convenient if users could use any media content as input for their queries. For example, if we were visiting the Great Wall, we might want to retrieve related textual materials based on the photos we took, thus learning about the history and interesting facts of the place we visited. Therefore, multimodal retrieval is becoming increasingly important as a common search method. Multimodal retrieval aims to retrieve related data of another type using one type of data as a query. Furthermore, when users submit data of any media type to search for information, they can obtain search results from various media types. Because different forms of data can complement each other, the search results obtained in this way are more comprehensive. The initial research direction in the field of multimodal research was multimodal retrieval. Multimodal retrieval technology involves multiple fields such as natural language processing, image processing, speech recognition, computer vision, and machine learning, and is closely related to many different disciplines such as computer science, statistics, and mathematics. On the one hand, research on multimodal retrieval methods will inspire the further development of many machine learning theories (such as multi-view learning, hash learning, subspace learning, metric learning, deep learning, etc.). On the other hand, research on multimodal retrieval methods is an essential stage in the emergence and development of new retrieval technologies, which can promote the use of information technology to benefit socio-economic development. Therefore, research on multimodal retrieval methods is of great significance. With the increasing prevalence of complementary explanations between multimodal data, multimodal data on various search engines and social media is experiencing explosive growth. Researchers are focusing on how to quickly and accurately search for data representing the same event topic from different modalities within large-scale multimodal datasets. Since data from different modalities often have incomparable feature representations and distributions, it is necessary to map them to a common feature space. To meet the requirements of low storage cost and high query speed in practical applications, researchers have proposed hash-based multimodal retrieval methods. This method maps high-dimensional multimodal data to a common Hamming space, and after obtaining hash codes, the similarity between multimodal data can be calculated simply through XOR operations. Most existing hash-based multimodal retrieval methods use manually crafted features for hash learning. These methods learn hash codes quickly and achieve good retrieval results. However, a common drawback of these hash-based multimodal retrieval algorithms is that the manual feature creation process and the hash learning process are completely independent. Consequently, manually created features may not be fully compatible with the hash learning process, leading to poor retrieval performance when using hash-based multimodal retrieval methods with manually created features. Researchers have demonstrated through experiments that on several commonly used multimodal datasets, continuing to use manually created features for hash learning is unlikely to improve multimodal retrieval performance.To address the incompatibility between manually generated features and the hash learning process, it is necessary to develop feature learning methods that are compatible with hash learning. This paper aims to explore hash-based multimodal retrieval technology using deep learning to achieve feature learning that aligns with hash learning, focusing on reducing encoding errors, mining semantic information from multimodal data, and minimizing the differences between multimodal data.

[0004] The basic idea of ​​hash-based multimodal retrieval is to map data from the original feature space to a binary encoding space for similarity retrieval. Hash-based multimodal retrieval methods map multimodal data to binary codes, which have the advantages of small space requirements and fast computation speed. Generally, hash-based multimodal retrieval methods consist of two steps: first, mapping data from different modalities or manually created features to a common feature space through linear or nonlinear transformations; second, encoding the features in the common feature space. Most hash-based multimodal retrieval methods use binary partitioning functions for encoding. The key challenge in multimodal retrieval lies in how to correlate the semantic relevance between multimodal data. This is usually addressed by learning a unified hash code for different modalities of the same sample or by reducing the Hamming distance between semantically related multimodal data. Hash-based multimodal retrieval methods can be categorized into supervised and unsupervised methods based on whether they use label information, and into linear and nonlinear methods based on the projection method.

[0005] One current technique is Cross-Modal Similarity Sensitive Hashing (CMSSH), as described in the paper "Large-scale supervised multimodal hashing with semantic correlation maximization." This method first maps the original data into hash codes, then uses these hash codes and a defined multimodal data fusion method to obtain a similarity matrix. The similarity matrix is ​​then used to determine the weights of the next hash function. This method treats the learning of each hash function as a binary classification process, i.e., a weak classifier. Finally, a standard boosting algorithm is used to combine the weak classifiers into a strong classifier. The drawback of this method is that CMSSH maintains consistency between multimodal data using a point-pair-based approach, but it does not consider the similarity between data within a modality.

[0006] The second existing technology is Supervised Matrix Factorization Hashing (SMFH), proposed in the paper "Kenel-based supervised hashing for cross-view similarity search." This method borrows from matrix factorization in hash multimodal retrieval and effectively utilizes label information and local geometric structure. It considers both the consistency of labels across different modalities and the local geometric consistency within each modality. These two elements are formulated as graph Laplacian operator terms in the objective function, greatly improving the discriminative ability of latent semantic features obtained through joint matrix factorization. The paper "Discrete Cross-modal Hashing (DCH)" also proposes this method. DCH preserves discrete constraints, uses labels as supervisory information, and constructs a linear classifier to directly learn discriminative hash codes. The drawback of this method is that, compared to nonlinear structures, the semantic information learned by linear structures is limited, and linear methods cannot be divided into point-pair-based and label-based methods.

[0007] The third existing technology is the Multimodal Latent Binary Embedding (MLBE) model proposed in the paper "Deep cross-modal hashing". MLBE uses a generative model to encode intra-modal and inter-modal similarities in multimodal data. Based on maximum a posteriori estimation, it effectively obtains binary latent factors that preserve both intra-modal and inter-modal similarities, and then uses them as learned hash codes. The disadvantage of this method is that during the learning process, especially when the code length is large, there are many parameters, resulting in high computational complexity, and optimization is prone to getting trapped in local minima. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing methods and propose a multimodal content retrieval method and system that combines text content with image analysis. The main problems addressed by this invention are: first, CMSSH uses a point-to-point method to maintain consistency between multimodal data, but neglects the similarity of data within a modality; second, current multimodal hashing methods suffer from insufficient hash code feature expression capabilities and low efficiency in effective feature extraction; and third, when the code length is large, the number of parameters is numerous, and the computational complexity is high, the MLBE model optimization is prone to getting trapped in local minima.

[0009] To address the aforementioned problems, this invention proposes a multimodal content retrieval method that combines text content with image analysis, the method comprising:

[0010] Annotate the image set in the ImageNet dataset to obtain text-image information pairs;

[0011] Input the text-image information pair, construct a feature extraction network to extract image and text features, and output image features and text features;

[0012] Input the image features and the text features, perform multimodal attention calculation, and obtain the multimodal features after weighted attention;

[0013] The image features, text features, and multimodal features are hashed respectively to generate image hash codes, text hash codes, and multimodal hash codes;

[0014] Input the image hash code, the text hash code, and the multimodal hash code, construct a target loss function, train the model using the loss function, and finally obtain the multimodal hash code generation model;

[0015] Using the trained multimodal hash code generation model, a multimodal hash code database of the database to be retrieved is constructed.

[0016] Based on the text information input by the user, a multimodal hash code is generated using the multimodal hash code generation model, and then matched with the constructed multimodal hash code database to obtain the retrieval result.

[0017] Preferably, the image set in the ImageNet dataset is labeled to obtain text-image information pairs, specifically:

[0018] Information about images and their corresponding text descriptions is collected to form the ImageNet dataset. The image sets in the ImageNet dataset are labeled to obtain text-image information pairs.

[0019] Preferably, the input text-image information pair is used to construct a feature extraction network to extract image and text features, and output image and text features, specifically as follows:

[0020] For image features, a convolutional neural network is used, combined with a 512-node fully connected layer, and then connected to a K-node fully connected layer with a softmax activation function to construct an image feature extraction network. The output is used as the learned image features. This is then combined with a fully connected layer with a sigmoid activation function that determines the number of label categories. The last layer of the image network outputs the predicted label to preserve the instance's label features. For text features, each text is represented by a bag-of-words vector. To address the issue of feature sparsity caused by bag-of-words vectors, a multi-scale fusion model (MS) is used to extract text data features. The MS includes five levels of pooling layers (1x1, 2x2, 3x3, 5x5, 10x10). After the MS, a 4096-node fully connected layer is connected, followed by a 512-node fully connected layer, then a K-node fully connected layer with a softmax activation function, and finally a fully connected layer with a sigmoid activation function that determines the number of label categories. The output of the penultimate layer of the text network is used as the learned text features, and the last layer outputs the predicted label.

[0021] Preferably, the input image features and text features are subjected to multimodal attention calculation to obtain weighted attention-weighted multimodal features, specifically as follows:

[0022] A multimodal cross-attention mechanism based on self-attention is adopted to capture the correlation between text and images. The text is used as the query subQ and the images are used as the key values ​​K and V. Then, multimodal attention calculation is performed, and the calculations of different heads are fused and normalized to obtain the final multimodal features.

[0023] Preferably, the step of hashing the image features, text features, and multimodal features respectively, and outputting image hash codes, text hash codes, and multimodal hash codes, specifically involves:

[0024] The hash codes for text, images, and multimodal features are obtained using the `sign` function, as shown in the following formula:

[0025]

[0026] Multimodal features include text features and image features. Multimodal features are used as an intermediate bridge to link the hash code learning processes between different modalities. In the image hash code learning process, the sign function is used to first encode image features to form image hash codes; then the sign function is used to encode multimodal features to form multimodal hash codes. In the text hash code learning process, the sign function is used to encode text features to form text hash codes. After modeling the hash codes of the above different modalities, a target loss function is constructed.

[0027] Preferably, the input image hash code, text hash code, and multimodal hash code are used to construct a target loss function, and the model is trained using the loss function to finally obtain a multimodal hash code generation model, specifically as follows:

[0028] Negative log-likelihood loss is used to model the similarity relationship between image features and multimodal features, while cosine similarity function is used to model the similarity between the image and multimodal features. As shown below:

[0029]

[0030] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. F = f(x) represents the similarity relationship between samples from two different modalities in the real space. i ;θ x )∈R C With L=g(y i ;θ y )∈R C Let represent image features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is the length of the subsequent binary hash representation, also known as bits. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0031] Subsequently, negative log-likelihood loss is used to model the similarity relationship between text and multimodal features, and a cosine similarity function is used to model the similarity between text and multimodal features. As shown below:

[0032]

[0033] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. L represents the similarity relationship between samples of two different modalities in the real space, where L = f(x) i ;θ x )∈R C With G = g(y i ;θ y )∈R CLet represent text features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is also the length of the subsequent binary hash representation, also known as bit. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0034] Meanwhile, to compensate for the information lost during the quantization process of different modes, a quantization loss is constructed as follows:

[0035]

[0036]

[0037] Where B is an n*c matrix used to continuously store and update the binary hash representation of each sample during training. The generation of B is actually a post-processing process. sign(·) is a function that converts numbers greater than 0 to 1 and numbers less than 0 to -1.

[0038] Subsequently, in order to balance the relationship between different modes, a balancing loss is introduced, as shown below:

[0039]

[0040] Finally, the total target loss is:

[0041] L all =L1+L2+L3+L4+L5

[0042] By learning the above loss function, similar data from different modalities are mapped to similar Hamming spaces, so that the hash codes of dissimilar data between different modalities have a greater Hamming distance in the Hamming space, and a multimodal hash code generation model is trained.

[0043] The multimodal hash code generation model allows users to perform searches simply by uploading data. The data is processed by the trained model to obtain a hash code, which is then compared with the hash codes in the database to achieve the search function, thus improving search speed and accuracy.

[0044] Accordingly, the present invention also provides a multimodal content retrieval system that combines text content with image analysis, comprising:

[0045] The data preprocessing unit is used to annotate the image collection in the ImageNet dataset to obtain text-image information pairs;

[0046] The feature extraction unit is used to input the text-image information pair, construct a feature extraction network to extract image and text features, and output image and text features;

[0047] A multimodal attention unit is used to input the image features and the text features, perform multimodal attention calculation, and obtain multimodal features after weighted attention.

[0048] The hash code generation unit is used to generate hashes for the image features, the text features, and the multimodal features respectively, and output image hash codes, text hash codes, and multimodal hash codes;

[0049] The model training unit is used to take the image hash code, the text hash code, and the multimodal hash code as input, construct a target loss function, train the model using the loss function, and finally obtain a multimodal hash code generation model.

[0050] The database construction unit uses the trained multimodal hash code generation model to construct a multimodal hash code database of the database to be retrieved.

[0051] The matching unit generates a multimodal hash code based on the text information input by the user using the multimodal hash code generation model, and then matches it with the constructed multimodal hash code database to obtain the retrieval result.

[0052] Implementing this invention has the following beneficial effects:

[0053] This invention utilizes convolutional neural networks to extract image features and a bag-of-words model to extract text features. It employs an advanced multimodal attention method to fuse these two modalities, avoiding the need for complex networks and thus reducing computational complexity. Furthermore, this scheme incorporates multimodal hash codes. Due to the correlations between different modalities, multimodal attention aims to capture commonalities across modalities. These common features are distributed across different single modalities, building a bridge for similarity calculation between modalities. This approach is more effective at bridging the heterogeneity gap between modalities than traditional methods that directly use hash codes from different modalities for similarity calculation, and it fundamentally uncovers commonalities between modalities. This innovative approach constructs multimodal hash codes, thereby enhancing the feature representation capabilities of current hash codes. By linking the multimodal feature extraction process through backpropagation of hash codes, it significantly improves the efficiency of effective feature extraction. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the overall process of a multimodal content retrieval method combining text content and image analysis according to an embodiment of the present invention.

[0055] Figure 2 This is a flowchart illustrating the training process of the multimodal hash generation model according to an embodiment of the present invention.

[0056] Figure 3 This is a flowchart of the multimodal attention calculation according to an embodiment of the present invention;

[0057] Figure 4 This is a flowchart illustrating the process of generating a multimodal hash code database according to an embodiment of the present invention.

[0058] Figure 5 This is a flowchart of the retrieval and matching process according to an embodiment of the present invention;

[0059] Figure 6 This is a structural diagram of a multimodal content retrieval system that combines text content with image analysis according to an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Figure 1 This is a flowchart illustrating the overall process of a multimodal content retrieval method combining text content and image analysis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] S1, annotate the image set in the ImageNet dataset to obtain text-image information pairs;

[0063] S2, Input the text-image information pair, construct a feature extraction network to extract image and text features, and output image features and text features;

[0064] S3, Input the image features and the text features, perform multimodal attention calculation, and obtain the multimodal features after weighted attention;

[0065] S4, perform hash generation on the image features, text features and multimodal features respectively, and output image hash code, text hash code and multimodal hash code;

[0066] S5, Input the image hash code, the text hash code, and the multimodal hash code, construct the target loss function, train the model using the loss function, and finally obtain the multimodal hash code generation model;

[0067] S6, such as Figure 4 As shown, the multimodal hash code generation model obtained through training is used to construct a multimodal hash code database from the database to be retrieved.

[0068] S7, such as Figure 5As shown, based on the text information input by the user, a multimodal hash code is generated using the multimodal hash code generation model, and then matched with the constructed multimodal hash code database to obtain the retrieval result.

[0069] Step S1 is as follows:

[0070] S1-1: Collect information about images and their corresponding text descriptions to form the Imagenet dataset. Label the image set in the Imagenet dataset to obtain text-image information pairs.

[0071] Step S2, as follows Figure 2 As shown, the details are as follows:

[0072] S2-1, for image features, a convolutional neural network is used, combined with a 512-node fully connected layer, and then connected to a K-node fully connected layer with the softmax activation function to construct an image feature extraction network. The output is used as the learned image features. This is combined with a fully connected layer with the sigmoid activation function based on the number of label categories. The last layer of the image network outputs the predicted label to preserve the label features of the instance. For text features, each text is represented by a bag-of-words vector. To address the issue of feature sparsity caused by bag-of-words vectors, a multi-scale fusion model (MS) is used to extract text data features. The MS includes five levels of pooling layers (1x1, 2x2, 3x3, 5x5, 10x10). After the MS, a 4096-node fully connected layer is connected, followed by a 512-node fully connected layer, then a K-node fully connected layer with the softmax activation function, and finally a fully connected layer with the sigmoid activation function based on the number of label categories. The output of the penultimate layer of the text network is used as the learned text features, and the last layer outputs the predicted label.

[0073] Step S3, as follows Figure 3 As shown, the details are as follows:

[0074] S3-1 employs a multimodal cross-attention mechanism based on self-attention to capture the correlation between text and images. Text is used as the query subQ, and images are used as key values ​​K and V. Multimodal attention calculation is then performed, and the calculations of different heads are fused and normalized to obtain the final multimodal features.

[0075] Step S4 is as follows:

[0076] S4-1 uses the sign function to obtain the hash codes of text, images, and multimodal features respectively, as shown in the following formula:

[0077]

[0078] S4-2, multimodal features include text features and image features. Multimodal features are used as an intermediate bridge to link the hash code learning processes between different modalities. In the image hash code learning process, the sign function is used to first encode image features to form image hash codes; then the sign function is used to encode multimodal features to form multimodal hash codes. In the text hash code learning process, the sign function is used to encode text features to form text hash codes. After modeling the hash codes of the above different modalities, the target loss function is constructed.

[0079] Step S5 is as follows:

[0080] Negative log-likelihood loss is used to model the similarity relationship between image features and multimodal features, while cosine similarity function is used to model the similarity between the image and multimodal features. As shown below:

[0081]

[0082] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. F = f(x) represents the similarity relationship between samples from two different modalities in the real space. i ;θ x )∈R C With L=g(y i ;θ y )∈R C Let represent image features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is the length of the subsequent binary hash representation, also known as bits. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0083] Subsequently, negative log-likelihood loss is used to model the similarity relationship between text and multimodal features, and a cosine similarity function is used to model the similarity between text and multimodal features. As shown below:

[0084]

[0085] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. L represents the similarity relationship between samples of two different modalities in the real space, where L = f(x)i ;θ x )∈R C With G = g(y i ;θ y )∈R C Let represent text features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is also the length of the subsequent binary hash representation, also known as bit. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0086] Meanwhile, to compensate for the information lost during the quantization process of different modes, a quantization loss is constructed as follows:

[0087]

[0088]

[0089] Where B is an n*c matrix used to continuously store and update the binary hash representation of each sample during training. The generation of B is actually a post-processing process. sign(·) is a function that converts numbers greater than 0 to 1 and numbers less than 0 to -1.

[0090] Subsequently, in order to balance the relationship between different modes, a balancing loss is introduced, as shown below:

[0091]

[0092] Finally, the total target loss is:

[0093] L all =L1+L2+L3+L4+L5

[0094] By learning the above loss function, similar data from different modalities are mapped to similar Hamming spaces, so that the hash codes of dissimilar data between different modalities have a greater Hamming distance in the Hamming space, and a multimodal hash code generation model is trained.

[0095] The multimodal hash code generation model allows users to perform searches simply by uploading data. The data is processed by the trained model to obtain a hash code, which is then compared with the hash codes in the database to achieve the search function, thus improving search speed and accuracy.

[0096] Accordingly, the present invention also provides a multimodal content retrieval system that combines text content with image analysis, such as... Figure 6 As shown, it includes:

[0097] Data preprocessing unit 1 is used to annotate the image set in the Imagenet dataset to obtain text-image information pairs.

[0098] Specifically, information about images and their corresponding text descriptions is collected to form the ImageNet dataset. The image sets in the ImageNet dataset are labeled to obtain text-image information pairs.

[0099] Feature extraction unit 2 is used to input the text image information pair, construct a feature extraction network to extract image and text features, and output image features and text features.

[0100] Specifically, for image features, a convolutional neural network is used, combined with a 512-node fully connected layer, and then connected to a K-node fully connected layer with a softmax activation function to construct an image feature extraction network. The output is used as the learned image features. This is then combined with a fully connected layer with a sigmoid activation function that determines the number of label categories. The last layer of the image network outputs the predicted label to preserve the label features of the instance. For text features, each text is represented by a bag-of-words vector. To address the issue of feature sparsity caused by bag-of-words vectors, a multi-scale fusion model (MS) is used to extract text data features. The MS includes five pooling layers (1x1, 2x2, 3x3, 5x5, 10x10). After the MS, a 4096-node fully connected layer is connected, followed by a 512-node fully connected layer, then a K-node fully connected layer with a softmax activation function, and finally a fully connected layer with a sigmoid activation function that determines the number of label categories. The output of the penultimate layer of the text network is used as the learned text features, and the last layer outputs the predicted label.

[0101] The multimodal attention unit 3 is used to input the image features and the text features, perform multimodal attention calculation, and obtain multimodal features after weighted attention.

[0102] Specifically, a multimodal cross-attention mechanism based on self-attention is used to capture the correlation between text and images. Text is used as query subQ and images are used as key values ​​K and V. Then, multimodal attention calculation is performed, and the calculations of different heads are fused and normalized to obtain the final multimodal features.

[0103] The hash code generation unit 4 is used to generate hashes for the image features, the text features and the multimodal features respectively, and output image hash codes, text hash codes and multimodal hash codes.

[0104] Specifically, the `sign` function is used to obtain the hash codes of text, images, and multimodal features, as shown in the following formula:

[0105]

[0106] Multimodal features include text features and image features. Multimodal features are used as an intermediate bridge to link the hash code learning processes between different modalities. In the image hash code learning process, the sign function is used to first encode image features to form image hash codes; then the sign function is used to encode multimodal features to form multimodal hash codes. In the text hash code learning process, the sign function is used to encode text features to form text hash codes. After modeling the hash codes of the above different modalities, a target loss function is constructed.

[0107] Model training unit 5 is used to input the image hash code, the text hash code, and the multimodal hash code, construct a target loss function, train the model using the loss function, and finally obtain a multimodal hash code generation model.

[0108] Specifically, negative log-likelihood loss is used to model the similarity relationship between image features and multimodal features, and cosine similarity function is used to model the similarity between images and multimodal features. As shown below:

[0109]

[0110] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. F = f(x) represents the similarity relationship between samples from two different modalities in the real space. i ;θ x )∈R C With L=g(y i ;θ y )∈R C Let represent image features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is the length of the subsequent binary hash representation, also known as bits. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0111] Subsequently, negative log-likelihood loss is used to model the similarity relationship between text and multimodal features, and a cosine similarity function is used to model the similarity between text and multimodal features. As shown below:

[0112]

[0113] Where S is an n*n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. L represents the similarity relationship between samples of two different modalities in the real space, where L = f(x) i ;θ x )∈R C With G = g(y i ;θ y )∈R C Let represent text features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is also the length of the subsequent binary hash representation, also known as bit. Optimizing the above negative log-likelihood loss can preserve similarity relationships in the feature space.

[0114] Meanwhile, to compensate for the information lost during the quantization process of different modes, a quantization loss is constructed as follows:

[0115]

[0116]

[0117] Where B is an n*c matrix used to continuously store and update the binary hash representation of each sample during training. The generation of B is actually a post-processing process. sign(·) is a function that converts numbers greater than 0 to 1 and numbers less than 0 to -1.

[0118] Subsequently, in order to balance the relationship between different modes, a balancing loss is introduced, as shown below:

[0119]

[0120] Finally, the total target loss is:

[0121] L all =L1+L2+L3+L4+L5

[0122] By learning the above loss function, similar data from different modalities are mapped to similar Hamming spaces, so that the hash codes of dissimilar data between different modalities have a greater Hamming distance in the Hamming space, and a multimodal hash code generation model is trained.

[0123] Database construction unit 6 uses the trained multimodal hash code generation model to construct a multimodal hash code database from the database to be retrieved.

[0124] Matching unit 7 generates a multimodal hash code based on the text information input by the user using the multimodal hash code generation model, and then matches it with the constructed multimodal hash code database to obtain the retrieval result.

[0125] Therefore, this invention utilizes convolutional neural networks to extract image features and a bag-of-words model to extract text features. It employs an advanced multimodal attention method to fuse these two modalities, rather than using a complex network to fuse multimodal features, thus reducing computational complexity. Simultaneously, this scheme adds multimodal hash codes. Due to the correlation between different modalities, the purpose of multimodal attention is to capture the commonalities between modalities. These common features are distributed across different single modalities, constructing a bridge for similarity calculation between modalities. This approach is more effective at bridging the heterogeneity gap between modalities than the traditional method of directly using hash codes from different modalities for similarity calculation, and it fundamentally uncovers the commonalities between modalities. This innovative approach constructs multimodal hash codes, thereby improving the feature representation capability of current hash codes. Furthermore, by linking the multimodal feature extraction process through backpropagation of hash codes, it significantly improves the efficiency of effective feature extraction.

[0126] The foregoing has provided a detailed description of a multimodal content retrieval method and system combining text content and image analysis provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal content retrieval method combining text content and image analysis, characterized in that, The method includes: Annotate the image set in the ImageNet dataset to obtain text-image information pairs; Input the text-image information pair, construct a feature extraction network to extract image and text features, and output image features and text features; Input the image features and the text features, perform multimodal attention calculation, and obtain the multimodal features after weighted attention; The image features, text features, and multimodal features are hashed respectively to generate image hash codes, text hash codes, and multimodal hash codes; Input the image hash code, the text hash code, and the multimodal hash code, construct a target loss function, train the model using the loss function, and finally obtain the multimodal hash code generation model; Using the trained multimodal hash code generation model, a multimodal hash code database of the database to be retrieved is constructed. Based on the text information input by the user, a multimodal hash code is generated using the multimodal hash code generation model, and then matched with the constructed multimodal hash code database to obtain the retrieval result.

2. The multimodal content retrieval method combining text content and image analysis as described in claim 1, characterized in that, The process of annotating the image set in the ImageNet dataset to obtain text-image information pairs specifically involves: Information about images and their corresponding text descriptions is collected to form the ImageNet dataset. The image sets in the ImageNet dataset are labeled to obtain text-image information pairs.

3. The multimodal content retrieval method combining text content and image analysis as described in claim 1, characterized in that, The input text-image information pair is used to construct a feature extraction network to extract image and text features, and output image and text features, specifically as follows: For image features, a convolutional neural network is used, combined with a 512-node fully connected layer and a K-node softmax activation layer to construct an image feature extraction network. The output is used as the learned image features. This is then combined with a fully connected layer that uses a sigmoid activation function based on the number of label categories. The last layer of the image network outputs the predicted label to preserve the instance's label features. For text features, each text is represented using a bag-of-words vector. To address the issue of feature sparsity caused by bag-of-words vectors, a multi-scale fusion model (MS) is used to extract text data features. The MS consists of five pooling layers: 1×1, 2×2, 3×3, 5×5, and 10×10. After the MS, a 4096-node fully connected layer is connected, followed by a 512-node fully connected layer, then a K-node softmax activation layer, and finally a sigmoid activation layer based on the number of label categories. The output of the penultimate layer of the text network is used as the learned text features, and the last layer outputs the predicted label.

4. The multimodal content retrieval method combining text content and image analysis as described in claim 1, characterized in that, The input image features and text features are subjected to multimodal attention calculation to obtain weighted attention-weighted multimodal features, specifically: A multimodal cross-attention mechanism based on self-attention is adopted to capture the correlation between text and images. The text is used as the query subQ and the images are used as the key values ​​K and V. Then, multimodal attention calculation is performed, and the calculations of different heads are fused and normalized to obtain the final multimodal features.

5. The multimodal content retrieval method combining text content and image analysis as described in claim 1, characterized in that, The process of hashing the image features, text features, and multimodal features respectively, and outputting image hash codes, text hash codes, and multimodal hash codes, specifically involves: The hash codes for text, images, and multimodal features are obtained using the `sign` function, as shown in the following formula: ； Multimodal features include text features and image features. Multimodal features are used as an intermediate bridge to link the hash code learning processes between different modalities. In the image hash code learning process, the sign function is used to first encode image features to form image hash codes; then the sign function is used to encode multimodal features to form multimodal hash codes. In the text hash code learning process, the sign function is used to encode text features to form text hash codes. After modeling the hash codes of the above different modalities, a target loss function is constructed.

6. The multimodal content retrieval method combining text content and image analysis as described in claim 1, characterized in that, The input image hash code, text hash code, and multimodal hash code are used to construct a target loss function. The model is then trained using this loss function to finally obtain a multimodal hash code generation model. Specifically: Negative log-likelihood loss is used to model the similarity relationship between image features and multimodal features, while cosine similarity function is used to model the similarity between the image and multimodal features. As shown below: ； Where S is an n×n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. This represents the similarity relationship between samples from two different modalities in the real-valued space. and Let represent image features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is the length of the subsequent binary hash representation, also known as bits. Optimizing the above negative log-likelihood loss allows the similarity relationship to be preserved in the feature space. Subsequently, negative log-likelihood loss is used to model the similarity relationship between text and multimodal features, and a cosine similarity function is used to model the similarity between text and multimodal features. As shown below: ； Where S is an n×n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. This represents the similarity relationship between samples from two different modalities in the real-valued space. and Let represent text features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is also the length of the subsequent binary hash representation, also known as bit. Optimizing the above negative log-likelihood loss allows the similarity relationship to be preserved in the feature space. Meanwhile, to compensate for the information lost during the quantization process of different modes, a quantization loss is constructed as follows: ； ； Where B is an n×c matrix used to continuously store and update the binary hash representation of each sample during training. The generation of B is actually a post-processing process, obtained through sign(F+L) and sign(F+G). sign(·) is a function that converts numbers greater than 0 to 1 and numbers less than 0 to -1. Subsequently, in order to balance the relationship between different modes, a balancing loss is introduced, as shown below: )； Finally, the total target loss is: ； By learning the above loss function, similar data from different modalities are mapped to similar Hamming spaces, so that the hash codes of dissimilar data between different modalities have a greater Hamming distance in the Hamming space, and a multimodal hash code generation model is trained. The multimodal hash code generation model allows users to perform searches simply by uploading data. The data is processed by the trained model to obtain a hash code, which is then compared with the hash codes in the database to achieve the search function, thus improving search speed and accuracy.

7. A multimodal content retrieval system that combines text content with image analysis, characterized in that, The system includes: The data preprocessing unit is used to annotate the image collection in the ImageNet dataset to obtain text-image information pairs; The feature extraction unit is used to input the text-image information pair, construct a feature extraction network to extract image and text features, and output image and text features; The multimodal attention unit takes the image features and text features as input, performs multimodal attention calculation, and obtains multimodal features after weighted attention. The hash code generation unit performs hash generation on the image features, the text features, and the multimodal features respectively, and outputs the image hash code, the text hash code, and the multimodal hash code; The model training unit takes the image hash code, the text hash code, and the multimodal hash code as input, constructs a target loss function, trains the model using the loss function, and finally obtains the multimodal hash code generation model. The database construction unit uses the trained multimodal hash code generation model to construct a multimodal hash code database of the database to be retrieved. The matching unit generates a multimodal hash code based on the text information input by the user using the multimodal hash code generation model, and then matches it with the constructed multimodal hash code database to obtain the retrieval result.

8. A multimodal content retrieval system combining text content and image analysis as described in claim 7, characterized in that, The data preprocessing unit needs to collect information about images and their corresponding text descriptions to form the ImageNet dataset, and then annotate the image sets in the ImageNet dataset to obtain text-image information pairs.

9. A multimodal content retrieval system combining text content and image analysis as described in claim 7, characterized in that, The feature extraction unit, for image features, employs a convolutional neural network combined with a 512-node fully connected layer, and simultaneously connects to a K-node fully connected layer with a softmax activation function to construct an image feature extraction network. The output serves as the learned image features. This is then combined with a fully connected layer with a sigmoid activation function that determines the number of label categories. The final layer of the image network outputs the predicted label to preserve the instance's label features. For text features, each text is represented using a bag-of-words vector. To address the issue of feature sparsity caused by bag-of-words vectors, a multi-scale fusion model (MS) is used to extract text data features. The MS includes five levels of pooling layers: 1×1, 2×2, 3×3, 5×5, and 10×10. After the MS, a 4096-node fully connected layer is connected, followed by a 512-node fully connected layer, then a K-node fully connected layer with a softmax activation function, and finally a fully connected layer with a sigmoid activation function that determines the number of label categories. The output of the penultimate layer of the text network serves as the learned text features, and the final layer outputs the predicted label.

10. A multimodal content retrieval system combining text content and image analysis as described in claim 7, characterized in that, The multimodal attention unit needs to use a multimodal cross-attention mechanism based on self-attention to capture the correlation between text and image. The text is used as the query subQ and the image is used as the key values ​​K and V. Then, multimodal attention calculation is performed, and the calculations of different heads are fused and normalized to obtain the final multimodal features.

11. A multimodal content retrieval system combining text content and image analysis as described in claim 7, characterized in that, The hash code generation unit needs to use the sign function to obtain the hash codes of text, images, and multimodal features respectively, as shown in the following formula: ； Multimodal features include text features and image features. Multimodal features are used as an intermediate bridge to link the hash code learning processes between different modalities. In the image hash code learning process, the sign function is used to first encode image features to form image hash codes; then the sign function is used to encode multimodal features to form multimodal hash codes. In the text hash code learning process, the sign function is used to encode text features to form text hash codes. After modeling the hash codes of the above different modalities, a target loss function is constructed.

12. A multimodal content retrieval system combining text content and image analysis as described in claim 7, characterized in that, The model training unit needs to use negative log-likelihood loss to model the similarity relationship between image features and multimodal features, and cosine similarity function to model the similarity between images and multimodal features. As shown below: ； Where S is an n×n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. This represents the similarity relationship between samples from two different modalities in the real-valued space. and Let represent image features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is the length of the subsequent binary hash representation, also known as bits. Optimizing the above negative log-likelihood loss allows the similarity relationship to be preserved in the feature space. Subsequently, negative log-likelihood loss is used to model the similarity relationship between text and multimodal features, and a cosine similarity function is used to model the similarity between text and multimodal features. As shown below: ； Where S is an n×n matrix, This represents the similarity relationship between the i-th sample of the current modality and the j-th sample of another modality. -1 indicates similarity, and -1 indicates dissimilarity. This represents the similarity relationship between samples from two different modalities in the real-valued space. and Let represent text features and multimodal features respectively. Both are real-valued feature representations. c represents the length of the feature vector, which is also the length of the subsequent binary hash representation, also known as bit. Optimizing the above negative log-likelihood loss allows the similarity relationship to be preserved in the feature space. Meanwhile, to compensate for the information lost during the quantization process of different modes, a quantization loss is constructed as follows: ； ； Where B is an n×c matrix used to continuously store and update the binary hash representation of each sample during training. The generation of B is actually a post-processing process, obtained through sign(F+L) and sign(F+G). sign(·) is a function that converts numbers greater than 0 to 1 and numbers less than 0 to -1. Subsequently, in order to balance the relationship between different modes, a balancing loss is introduced, as shown below: )； Finally, the total target loss is: ； By learning the above loss function, similar data from different modalities are mapped to similar Hamming spaces, so that the hash codes of dissimilar data between different modalities have a greater Hamming distance in the Hamming space, and a multimodal hash code generation model is trained. The multimodal hash code generation model allows users to perform searches simply by uploading data. The data is processed by the trained model to obtain a hash code, which is then compared with the hash codes in the database to achieve the search function, thus improving search speed and accuracy.

Images