Partial multi-modal Hash method based on fine-grained feature fusion

Through fine-grained feature fusion and deep semantic information capture technology, the problem of insufficient representation of multimodal hashing method in the case of missing samples is solved, the efficiency and robustness of multimodal data are improved, and more accurate hashing is achieved Code generation.

CN118981507AInactive Publication Date: 2024-11-19NANJING UNIV OF AERONAUTICS & ASTRONAUTICS

Patent Information

Application Number
CN202411046264.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing multimodal hashing method is not effective when dealing with the situation where partial modality is missing in the sample. Most methods are based on shallow learning models and cannot effectively capture deep semantic information, resulting in insufficient hash code representation ability, which limits retrieval. efficiency.

Method used

Using a partial multimodal hashing method based on fine-grained feature fusion, features are extracted through VGGNet and bag-of-word vectors, a homomodal complement module and a cross-modal generation module are constructed to complete missing modal complements, combining the self-attention mechanism and The Transformer encoder captures deep semantic information and performs feature mapping through multi-layer perceptrons and fully connected layers, and finally generates a binary hash code.

Benefits of technology

It improves the representation ability of hash code, enhances the efficiency of multimodal data retrieval, can more effectively deal with the situation where partial modality of samples are missing, shortens database retrieval time, and improves the robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981507A_ABST
    Figure CN118981507A_ABST
Patent Text Reader

Abstract

The invention discloses a partial multi-modal hashing method based on fine-grained feature fusion, which comprises the following steps: preparing a multi-modal retrieval data set which comprises information such as image modal data, text modal data, sample categories and the like, performing feature extraction on original data by using a VGGNet network and a bag-of-word (bag-of-word, BoW) vector respectively, and performing feature extraction on the original data to obtain a feature extraction result; the data set is divided into a part with a complete modal of a sample and a part with a missing partial modal of the sample; constructing a missing modal supplementing module, randomly sampling an anchor point set from the part of the sample with the complete modal, and supplementing the missing modal features of the sample by using the modal features of anchor points; then, taking the supplemented modal features as true values to supervise training, and directly generating another missing modal feature in a cross-modal manner by depending on the existing modal features of the sample; a deep semantic information module is constructed, the complete modal features of the sample are decomposed into K pieces of shallow semantic information corresponding to each hash bit, a Transform encoder is introduced, the internal relation between the shallow semantic information is adaptively captured in a self-attention mode, and encoded deep semantic information is obtained; and performing fine-grained feature fusion by using the obtained deep semantic information, and finally obtaining a K-bit binary hash code. According to the method, all the samples are fully utilized, noise introduced when missing modal features are supplemented is reduced, and a multi-modal data retrieval scene of a sample missing partial modal can be effectively dealt with.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of novel databases for multimodal data, and discloses a multimodal data retrieval method for samples with missing partial modal features, which is implemented by using a self-attention mechanism and a binary hash code. Background Art

[0002] The rapid development of big data and artificial intelligence is setting off a new round of information revolution. Traditional information systems are also undergoing digital transformation through knowledge empowerment, which not only leads to an explosive growth in the amount of data in various industries, but also makes data formats and types more diverse. Compared with traditional unimodal data, multimodal data can provide richer information representation, and multimodal data representation also has a wider range of applications, such as visual question answering, intelligent medical care, and sentiment analysis. At the same time, massive multimodal data brings retrieval challenges to traditional databases. Compared with the retrieval method based on true values, the hash-based retrieval method can greatly reduce the storage overhead of large-scale data and improve the retrieval efficiency of the database by mapping high-dimensional data into compact binary hash codes. Therefore, how to effectively use hash codes is an urgent research direction in the field of new database technology for multimodal data.

[0003] Currently, multimodal data retrieval has become a key research direction of multimodal databases and has been widely studied by scholars at home and abroad. Hash-based retrieval methods can map high-dimensional data into compact binary hash codes and have achieved good results in data retrieval tasks. How to improve the representation ability of hash codes is the research focus of hash-based retrieval methods.

[0004] In recent years, some methods for multimodal hashing have emerged, but most of them tend to consider that all samples are modally complete during the training and query stages, while the situation of missing some modalities is also common for multimodal samples stored in the database. For example, in real social network scenarios, data is uploaded by various users. Some users upload image data but lack corresponding text information, while some users upload descriptive text information but lack corresponding image data. As a result, a large amount of partial multimodal data is generated, which limits the practical application of these methods to a certain extent. At the same time, most existing methods for multimodal hashing are based on shallow learning models. These methods are limited by the learning ability of shallow models and cannot effectively capture the deep semantic information implicit in multimodal data. Although some methods have used pre-trained deep networks to extract the original feature information of each modality and adopted specific strategies to fuse the extracted multimodal features to generate hash codes, they ignore the modeling of deep semantic information, resulting in the inability to give full play to the powerful learning ability of deep neural networks. Summary of the invention

[0005] Existing multimodal hashing methods tend to consider the situation where all samples are modally complete. However, in actual retrieval scenarios, the situation where samples are missing some modalities still exists. In order to effectively deal with this situation, we proposed missing modality completion to solve this problem. At the same time, existing studies mostly capture shallow semantic information and only perform feature fusion at a coarse-grained level. However, rich deep semantic information is fused between shallow semantic information. The use of deep semantic information can more effectively enhance the representation ability of hash codes and ultimately improve the retrieval effect. In order to solve the above problems, the purpose of the present invention is to propose a partial multimodal hashing method based on fine-grained feature fusion.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A partial multimodal hashing method based on fine-grained feature fusion includes the following steps:

[0008] Step 1: Prepare a multimodal retrieval dataset. The multimodal retrieval dataset contains data in two completely different modalities: images and texts, and contains the category information to which each multimodal sample belongs. Use the VGGNet network and bag-of-words vector to extract features from the original data to obtain a high-dimensional feature vector.

[0009] In step 1, we used two public datasets commonly used in the field of multimodal retrieval, namely the MIR Flickr dataset and the MS COCO dataset. Among them, the MIR Flickr dataset contains 25,000 image-text pairs collected from the Flickr website, with a total of 24 different categories. 17,772 samples to be retrieved, 5,000 training samples and 2,243 query samples were selected. The image modality features of the samples in the MIR Flickr dataset are 4,096 dimensions, and the text modality features are 1,386 dimensions. The MS COCO dataset contains samples of 80 different categories, and 82,783 samples to be retrieved, 18,000 training samples and 5,981 query samples were selected. The image modality features of the samples in the MS COCO dataset are 4,096 dimensions, and the text modality features are 2,000 dimensions.

[0010] Step 2: Construct the same-modality completion module in the missing modality completion module Its input is the existing modal features of the sample and the complete modal features of the pre-sampled anchor points. By measuring the semantic similarity between the sample and the anchor point, the modal features of the anchor point with similar semantics are used to fill in the missing modal features and serve as the "true value" of the subsequent supervision cross-modal generation module.

[0011] In step 2, the same modality completion module The overall structure is as follows:

[0012]

[0013] in, is the text modality feature of the sample. In order to reduce the search range, it is necessary to first select N samples from the sample set with complete modality. a The anchor point set is composed of samples. The sampling of anchor points is random, and the number of anchor points should be greater than the number of types of samples in the training set. is the image modality feature composed of all anchor points in the anchor point set, is the text modality feature composed of all anchor points in the anchor point set, Represents the image modality features completed by this module.

[0014] There are two different methods to implement this module: one based on K-nearest neighbor (KNN) and the other based on self-attention.

[0015] The KNN-based method calculates the distance between the text modality features of the sample and the text modality features of all anchors in the anchor set, and sorts them based on the distance. The smaller the distance, the closer the sample is to the corresponding anchor in the anchor set in terms of text modality, which indicates that the sample has a similar semantic relationship with the anchor. Then the image modality features of the first K most semantically similar anchors are averaged, or the inverse of the distance is normalized and used as a weighted sum. The KNN-based method is expressed as follows:

[0016]

[0017] The method based on the self-attention mechanism measures the similarity between the query and the key as the attention score and takes the weighted sum of the values ​​as the encoding result to model the correlation between the input sequences. Specifically, the text modality features of the sample are As query, the text modality feature Y of the anchor set is a As the key, the image modality feature X of the anchor set a As value, the calculation formula is as follows:

[0018]

[0019] in, V=X a W V , ⊙ is the Hadamard product, is the result of encoding using the cross-modal self-attention mechanism, a i The vector representing the similarity between the sample and all anchor points in the anchor point set is used Perform calculations.

[0020] After obtaining the result of the cross-modal self-attention mechanism encoding, a simple fully connected layer is used as a decoder to produce the final output:

[0021]

[0022] in It is the output of the decoder and serves as the image modality feature completed by this module.

[0023] Step 3: Construct the cross-modal generation module in the missing modality completion module Its input is the existing modal features of the sample. The result generated by the same modality completion module in step 2 is used as the "true value" for training this module. Multiple fully connected layers are used to directly map the existing modal features of the sample to the missing modal features through simple nonlinear transformations, so as to accurately model the similar semantic relationships between different modalities of the same sample.

[0024] In step 3, the cross-modal generation module The overall structure is as follows:

[0025]

[0026] in, Indicates the missing image modality features of the sample generated by this module. Since the selection of anchor points in step 2 above is random, there will be a situation where the new multimodal sample in the query phase does not belong to the same category as all anchor points. In this case, step 2 will introduce a lot of noise. At the same time, there are similar semantic relationships between different modalities of the same sample, so the missing modality features can be directly generated across modalities using the existing modality features of the sample.

[0027] When implementing this module, multiple fully connected layers are directly stacked, that is, the text modality features of the sample are directly mapped to the image modality features through nonlinear transformation. Under the supervision of the "true value" generated in step 2, such a relatively simple structure can also accurately model the similar semantic relationships between different modalities of the same sample. In addition, such a module setting can reduce the time spent on filling in the missing modalities of the query sample in the online query stage, thereby effectively shortening the retrieval time of the database.

[0028] Step 4: Build a deep semantic information module, whose input is the complete modal features of the sample after completion. Decompose each modal feature of the sample into K vectors, representing K shallow semantic information. Then, introduce a modality-specific Transformer encoder for each modality, model the relationship between shallow semantic information through self-attention, and adaptively capture deep semantic information.

[0029] In step 4, the overall structure of the deep semantic information module is as follows:

[0030]

[0031] in, is the missing modal feature generated in step 3, is the existing modal feature of the sample. First, use the Multi-Layer Perceptron (MLP) to map different modal features into the same dimension:

[0032]

[0033] in, Represents the feature vector after mapping image and text features into the same dimension.

[0034] Afterwards Decompose into shallow semantic information corresponding to K hash bits: First, Mapping is (K×d common )-dimensional vector, and then transform it back into a K×d common dimensional matrix and is denoted by

[0035] Next, a modality-specific Transformer encoder is introduced for each modality to capture deep semantic information:

[0036]

[0037] in, Represents deep semantic information. Specifically,

[0038] Step 5: Construct a fusion and mapping module, whose input is the K deep semantic information obtained by the Transformer encoder. Add the K deep semantic information of each modality, use K different mapping functions to map them bit by bit to the corresponding hash bit, and obtain a K-bit relaxed hash code. Finally, use the sign function sign to convert the relaxed hash code into the final binary representation.

[0039] In step 5, the overall structure of the fusion and mapping module is as follows:

[0040]

[0041] The input is the deep semantic information obtained in step 5. After capturing the deep semantic information, the two modal features can be fused in a fine-grained manner. The fusion method can be defined as follows:

[0042]

[0043] in, It represents the result of multimodal feature fusion of the i-th sample. Then, K different mapping functions are used to map it bit by bit to the corresponding hash bit:

[0044]

[0045] in, Used to combine the k-th fused features Mapped to the relaxation value on the corresponding hash bit

[0046] Finally, the sign function sign is used to convert the relaxed hash code into the final binary representation:

[0047]

[0048] in, In addition, in order to make full use of the label information of the sample, after obtaining the relaxed hash code, it is used to make a simple category probability prediction, which can be formally expressed as:

[0049]

[0050] Among them, FC represents the fully connected layer, σ is the sigmoid activation function, Each element of represents the probability that the sample belongs to the corresponding category. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the same-mode completion module in the present invention The overall structure diagram;

[0052] Figure 2 The cross-modal generation module in the present invention The overall structure diagram;

[0053] Figure 3 It is the overall structure diagram of the deep semantic information module in the present invention;

[0054] Figure 4 It is the overall structure diagram of the fusion and mapping module in the present invention;

[0055] Figure 5 It is the overall structure diagram of the model in the present invention; DETAILED DESCRIPTION

[0056] The present invention will be further explained below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0057] The present invention provides a partial multimodal hashing method based on fine-grained feature fusion, which specifically includes the following five steps:

[0058] Step 1: Prepare a multimodal retrieval dataset. The multimodal retrieval dataset contains data in two completely different modalities: images and texts, and contains the category information to which each multimodal sample belongs. Use the VGGNet network and bag-of-words vector to extract features from the original data to obtain a high-dimensional feature vector.

[0059] Step 2: Construct the same-modality completion module in the missing modality completion module Its input is the existing modal features of the sample and the complete modal features of the pre-sampled anchor points. By measuring the semantic similarity between the sample and the anchor point, the modal features of the anchor point with similar semantics are used to fill in the missing modal features and serve as the "true value" of the subsequent supervision cross-modal generation module.

[0060] Step 3: Construct the cross-modal generation module in the missing modality completion module Its input is the existing modal features of the sample. The result generated by the same modality completion module in step 2 is used as the "true value" for training this module. Multiple fully connected layers are used to directly map the existing modal features of the sample to the missing modal features through simple nonlinear transformations, so as to accurately model the similar semantic relationships between different modalities of the same sample.

[0061] Step 4: Build a deep semantic information module, whose input is the complete modal features of the sample after completion. Decompose each modal feature of the sample into K vectors, representing K shallow semantic information. Then, introduce a modality-specific Transformer encoder for each modality, model the relationship between shallow semantic information through self-attention, and adaptively capture deep semantic information.

[0062] Step 5: Construct a fusion and mapping module, whose input is the K deep semantic information obtained by the Transformer encoder. Add the K deep semantic information of each modality, use K different mapping functions to map them bit by bit to the corresponding hash bit, and obtain a K-bit relaxed hash code. Finally, use the sign function sign to convert the relaxed hash code into the final binary representation.

[0063] In step 1, we used two public datasets commonly used in the field of multimodal retrieval, namely the MIR Flickr dataset and the MS COCO dataset. Among them, the MIR Flickr dataset contains 25,000 image-text pairs collected from the Flickr website, with a total of 24 different categories. 17,772 samples to be retrieved, 5,000 training samples and 2,243 query samples were selected. The image modality features of the samples in the MIR Flickr dataset are 4,096 dimensions, and the text modality features are 1,386 dimensions. The MS COCO dataset contains samples of 80 different categories, and 82,783 samples to be retrieved, 18,000 training samples and 5,981 query samples were selected. The image modality features of the samples in the MS COCO dataset are 4,096 dimensions, and the text modality features are 2,000 dimensions.

[0064] In step 2, the same modality completion module The overall structure is as follows:

[0065]

[0066] in, is the text modality feature of the sample. In order to reduce the search range, it is necessary to first select N samples from the sample set with complete modality. a The anchor point set is composed of samples. The sampling of anchor points is random, and the number of anchor points should be greater than the number of types of samples in the training set. is the image modality feature composed of all anchor points in the anchor point set, is the text modality feature composed of all anchor points in the anchor point set, Represents the image modality features completed by this module.

[0067] There are two different methods to implement this module: one based on K-nearest neighbor (KNN) and the other based on self-attention.

[0068] The KNN-based method calculates the distance between the text modality features of the sample and the text modality features of all anchors in the anchor set, and sorts them based on the distance. The smaller the distance, the closer the sample is to the corresponding anchor in the anchor set in terms of text modality, which indicates that the sample has a similar semantic relationship with the anchor. Then the image modality features of the first K most semantically similar anchors are averaged, or the inverse of the distance is normalized and used as a weighted sum. The KNN-based method is expressed as follows:

[0069]

[0070] The method based on the self-attention mechanism measures the similarity between the query and the key as the attention score and takes the weighted sum of the values ​​as the encoding result to model the correlation between the input sequences. Specifically, the text modality features of the sample are As query, the text modality feature Y of the anchor set is a As the key, the image modality feature X of the anchor set a As value, the calculation formula is as follows:

[0071]

[0072] in, V=X a W V , ⊙ is the Hadamard product, is the result of encoding using the cross-modal self-attention mechanism, a i The vector representing the similarity between the sample and all anchor points in the anchor point set is used Perform calculations.

[0073] After obtaining the result of the cross-modal self-attention mechanism encoding, a simple fully connected layer is used as a decoder to produce the final output:

[0074]

[0075] in It is the output of the decoder and serves as the image modality feature completed by this module.

[0076] In step 3, the cross-modal generation module The overall structure is as follows:

[0077]

[0078] in, Indicates the missing image modality features of the sample generated by this module. Since the selection of anchor points in step 2 above is random, there will be a situation where the new multimodal sample in the query phase does not belong to the same category as all anchor points. In this case, step 2 will introduce a lot of noise. At the same time, there are similar semantic relationships between different modalities of the same sample, so the missing modality features can be directly generated across modalities using the existing modality features of the sample.

[0079] When implementing this module, multiple fully connected layers are directly stacked, that is, the text modality features of the sample are directly mapped to the image modality features through nonlinear transformation. Under the supervision of the "true value" generated in step 2, such a relatively simple structure can also accurately model the similar semantic relationships between different modalities of the same sample. In addition, such a module setting can reduce the time spent on filling in the missing modalities of the query sample in the online query stage, thereby effectively shortening the retrieval time of the database.

[0080] In step 4, the overall structure of the deep semantic information module is as follows:

[0081]

[0082] in, is the missing modal feature generated in step 3, is the existing modal feature of the sample. First, use the Multi-Layer Perceptron (MLP) to map different modal features into the same dimension:

[0083]

[0084] in, Represents the feature vector after mapping image and text features into the same dimension.

[0085] Afterwards Decompose into shallow semantic information corresponding to K hash bits: First, Mapping is (K×d common )-dimensional vector, and then transform it back into a K×d common dimensional matrix and is denoted by

[0086] Next, a modality-specific Transformer encoder is introduced for each modality to capture deep semantic information:

[0087]

[0088] in, Represents deep semantic information. Specifically,

[0089] In step 5, the overall structure of the fusion and mapping module is as follows:

[0090]

[0091] The input is the deep semantic information obtained in step 5. After capturing the deep semantic information, the two modal features can be fused in a fine-grained manner. The fusion method can be defined as follows:

[0092]

[0093] in, It represents the result of multimodal feature fusion of the i-th sample. Then, K different mapping functions are used to map it bit by bit to the corresponding hash bit:

[0094]

[0095] in, Used to combine the k-th fused features Mapped to the relaxation value on the corresponding hash bit

[0096] Finally, the sign function sign is used to convert the relaxed hash code into the final binary representation:

[0097]

[0098] in, In addition, in order to make full use of the label information of the sample, after obtaining the relaxed hash code, it is used to make a simple category probability prediction, which can be formally expressed as:

[0099]

[0100] Among them, FC represents the fully connected layer, σ is the sigmoid activation function, Each element of represents the probability that the sample belongs to the corresponding category.

[0101] In terms of data sets, the present invention uses two popular multimodal retrieval data sets for training, which enhances the real usability of the model and achieves the best results overall; in terms of network structure, the present invention uses the self-attention mechanism and the Transformer encoder module to complete the partial modal features missing from the sample, and mines the deep semantic information between the modalities to achieve multimodal feature fusion at a fine-grained level; in terms of applicable scenarios, the present invention focuses on scenarios where the sample is missing some modalities, and can better cope with complex and changeable real retrieval scenarios. Through experiments, it is found that the model provided by this method can improve the retrieval efficiency of new databases for multimodal data as a whole, and the model has strong robustness.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A partial multimodal hashing method based on fine-grained feature fusion, comprising the following steps: Step 1: Prepare a multimodal retrieval dataset. The multimodal retrieval dataset contains data in two completely different modalities: images and texts, and contains the category information to which each multimodal sample belongs. Use the VGGNet network and bag-of-words vector to extract features from the original data to obtain a high-dimensional feature vector. Step 2: Construct the same-modality completion module in the missing modality completion module Its input is the existing modal features of the sample and the complete modal features of the pre-sampled anchor points. By measuring the semantic similarity between the sample and the anchor point, the modal features of the anchor point with similar semantics are used to fill in the missing modal features and serve as the "true value" of the subsequent supervision cross-modal generation module. Step 3: Construct the cross-modal generation module in the missing modality completion module Its input is the existing modal features of the sample. The result generated by the same modality completion module in step 2 is used as the "true value" for training this module. Multiple fully connected layers are used to directly map the existing modal features of the sample to the missing modal features through simple nonlinear transformations, so as to accurately model the similar semantic relationships between different modalities of the same sample. Step 4: Build a deep semantic information module, whose input is the complete modal features of the sample after completion. Decompose each modal feature of the sample into K vectors, representing K shallow semantic information. Then, introduce a modality-specific Transformer encoder for each modality, model the relationship between shallow semantic information through self-attention, and adaptively capture deep semantic information. Step 5: Construct a fusion and mapping module, whose input is the K deep semantic information obtained by the Transformer encoder. Add the K deep semantic information of each modality, use K different mapping functions to map them bit by bit to the corresponding hash bit, and obtain a K-bit relaxed hash code. Finally, use the sign function sign to convert the relaxed hash code into the final binary representation.

2. The partial multimodal hashing method based on fine-grained feature fusion according to claim 1, characterized in that: In step 1, we used two public datasets commonly used in the field of multimodal retrieval, namely the MIR Flickr dataset and the MS COCO dataset. The MIR Flickr dataset contains 25,000 image-text pairs collected from the Flickr website, with a total of 24 different categories. 17,772 samples to be retrieved, 5,000 training samples, and 2,243 query samples were selected. The image modality features of the samples in the MIR Flickr dataset are 4,096 dimensions, and the text modality features are 1,386 dimensions. The MS COCO dataset contains samples of 80 different categories, and 82,783 samples to be retrieved, 18,000 training samples, and 5,981 query samples were selected. The image modality features of the samples in the MS COCO dataset are 4,096 dimensions, and the text modality features are 2,000 dimensions.

3. The partial multimodal hashing method based on fine-grained feature fusion according to claim 1, characterized in that: In step 2, the same modality completion module The overall structure is as follows: Y a ,X a →Same modality completion→ in, is the text modality feature of the sample. In order to reduce the search range, it is necessary to first select N samples from the sample set with complete modality. a The anchor point set is composed of samples. The sampling of anchor points is random, and the number of anchor points should be greater than the number of types of samples in the training set. is the image modality feature composed of all anchor points in the anchor point set, is the text modality feature composed of all anchor points in the anchor point set, Represents the image modality features completed by this module. There are two different methods to implement this module: one based on K-nearest neighbor (KNN) and the other based on self-attention. The KNN-based method calculates the distance between the text modality features of the sample and the text modality features of all anchors in the anchor set, and sorts them based on the distance. The smaller the distance, the closer the sample is to the corresponding anchor in the anchor set in terms of text modality, which indicates that the sample has a similar semantic relationship with the anchor. Then the image modality features of the first K most semantically similar anchors are averaged, or the inverse of the distance is normalized and used as a weighted sum. The KNN-based method is expressed as follows: The method based on the self-attention mechanism measures the similarity between the query and the key as the attention score and takes the weighted sum of the values ​​as the encoding result to model the correlation between the input sequences. Specifically, the text modality features of the sample are As query, the text modality feature Y of the anchor set is a As the key, the image modality feature X of the anchor set a As value, the calculation formula is as follows: in, K=Y a W K , V = X a W V , ⊙ is the Hadamard product, is the result of encoding using the cross-modal self-attention mechanism, a i The vector representing the similarity between the sample and all anchor points in the anchor point set is used Perform calculations. After obtaining the result of the cross-modal self-attention mechanism encoding, a simple fully connected layer is used as a decoder to produce the final output: in It is the output of the decoder and serves as the image modality feature completed by this module.

4. The traffic flow prediction method based on multi-head attention diffusion graph convolutional neural network according to claim 1 is characterized by: In step 3, the cross-modal generation module The overall structure is as follows: →Cross-modal generation→ in, Indicates the missing image modality features of the sample generated by this module. Since the selection of anchor points in step 2 above is random, there will be a situation where the new multimodal sample in the query phase does not belong to the same category as all anchor points. In this case, step 2 will introduce a lot of noise. At the same time, there are similar semantic relationships between different modalities of the same sample, so the missing modality features can be directly generated across modalities using the existing modality features of the sample. When implementing this module, multiple fully connected layers are directly stacked, that is, the text modality features of the sample are directly mapped to the image modality features through nonlinear transformation. Under the supervision of the "true value" generated in step 2, such a relatively simple structure can also accurately model the similar semantic relationships between different modalities of the same sample. In addition, such a module setting can reduce the time spent on filling in the missing modalities of the query sample in the online query stage, thereby effectively shortening the database retrieval time.

5. The partial multimodal hashing method based on fine-grained feature fusion according to claim 1, characterized in that: In step 4, the overall structure of the deep semantic information module is as follows: →Eigendecomposition→Transformer Encoder→ in, is the missing modal feature generated in step 3, is the existing modal feature of the sample. First, use the Multi-Layer Perceptron (MLP) to map different modal features into the same dimension: in, Represents the feature vector after mapping image and text features into the same dimension. Afterwards Decompose into shallow semantic information corresponding to K hash bits: First, Mapping is (K×d common )-dimensional vector, and then transform it back into a K×d common dimensional matrix and is denoted by Next, a modality-specific Transformer encoder is introduced for each modality to capture deep semantic information: in, Represents deep semantic information. Specifically, 6. The partial multimodal hashing method based on fine-grained feature fusion according to claim 1, characterized in that: In step 5, the overall structure of the fusion and mapping module is as follows: → Feature fusion → Relaxed hash code →Binary Hash Code The input is the deep semantic information obtained in step 5. After capturing the deep semantic information, the two modal features can be fused in a fine-grained manner. The fusion method can be defined as follows: in, It represents the result of multimodal feature fusion of the i-th sample. Then, K different mapping functions are used to map it bit by bit to the corresponding hash bit: in, Used to combine the k-th fused features Mapped to the relaxation value on the corresponding hash bit Finally, the sign function sign is used to convert the relaxed hash code into the final binary representation: in, In addition, in order to make full use of the label information of the sample, after obtaining the relaxed hash code, it is used to make a simple category probability prediction, which can be formally expressed as: Among them, FC represents the fully connected layer, σ is the sigmoid activation function, Each element of represents the probability that the sample belongs to the corresponding category.

Citation Information

Patent Citations

  • Classification feature learning method for incomplete multi-source heterogeneous data

    CN117609823A

Cited By

  • Distribution outward vector retrieval method and device

    CN120030192A

  • Cross-working-condition bearing fault diagnosis method based on prototype domain alignment feature coding

    CN120541740A

  • Multi-modal target re-identification method and device, electronic equipment and medium

    CN122416206A