Multi-modal hash retrieval method, system, device and medium based on deep semantic agent

Abstract

The application discloses a multi-modal hash retrieval method and system based on a deep semantic agent, a device and a medium, and belongs to the technical field of artificial intelligence and multi-modal hash retrieval. The technical problem to be solved by the application is the insufficient multi-modal fusion in the multi-modal hash retrieval process and the weak correlation between the ignored data itself and the supervised semantics. The technical solution is as follows: constructing a multi-modal data set: obtaining an image modal and a text modal data set, and preprocessing; using a CLIP pre-training model to perform deep feature extraction on the image modal and the text modal data set, and constructing a training set, a test set and a retrieval set according to a proportion, and then combining to generate a final data set; constructing a multi-modal hash retrieval model based on a deep semantic agent: using a neural network and a deep learning method to construct a multi-modal hash retrieval model based on a deep semantic agent; training the model: using the final data set to train the multi-modal hash retrieval model based on the deep semantic agent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal hash retrieval technology, in particular to a multi-modal hash retrieval method, system, device and medium based on deep semantic agent. BACKGROUND

[0002] With the vigorous development of the information age and the rapid progress of the network, multi-modal data presents in various forms such as images, texts and audios, showing an explosive growth trend. However, the traditional single-modal data retrieval method, such as image retrieval and text retrieval, has been unable to meet the needs of the increasingly diversified multi-modal data. Multi-modal data has the characteristics of low-level representation heterogeneity and high-level semantic homogeneity, that is, the same thing can be presented through different expression ways and forms. This diverse form can represent the same data instance from different aspects, compensate for the information loss of each modality, and thus enrich the understanding of the semantic information of the modality. Therefore, multi-modal retrieval needs to be able to cross the semantic gap between different modal data to achieve accurate and efficient retrieval.

[0003] Multi-modal hash retrieval is different from cross-modal hash retrieval. Cross-modal hash retrieval aims to query the modality instance related to the semantic of another modality from one modality through a series of hash functions. In contrast, multi-modal hash utilizes the complementary and consistent correlation between multi-modal data to learn hash codes for retrieval. Multi-modal hash compresses multi-modal heterogeneous data into compact binary hash codes, in which the semantic relationship of the original multi-modal instance is well preserved in the reconstructed Hamming space. Through the obtained multi-modal hash representation, the similarity calculation speed between modalities can be accelerated and the storage cost can be greatly reduced in the process of large-scale multimedia retrieval.

[0004] In recent years, many deep learning network models have been applied to multi-modal hash retrieval systems. Among them, methods based on label matrix supervision and pair-wise semantic supervision have shown good performance. The multi-modal hash retrieval method based on label matrix supervision constructs a label matrix supervision through semantic labels, and learns binary hash codes through multi-modal feature integration and label matrix supervision. The multi-modal hash retrieval method based on pair-wise semantic supervision learns multi-modal hash codes through collaborative multi-modal feature mapping by preserving multi-modal semantic correlation in the hash codes. Although they have achieved good results, so far, the multi-modal fusion in the above methods mainly relies on cascaded fusion, which lacks flexibility and cannot fully capture the correlation between different modalities, thereby limiting the effectiveness of the fusion. In addition, most supervised multi-modal hash methods only rely on labels for supervision, ignoring the inherent implicit information of the data, thereby leading to a weak connection between the data itself and the supervised semantics, resulting in insufficient supervision ability. SUMMARY

[0005] The technical task of the present application is to provide a multi-modal hash retrieval method, system, device and medium based on deep semantic agent, to solve the problem of insufficient multi-modal fusion and the problem of weak association between the ignored data itself and the supervised semantics in the multi-modal hash retrieval process.

[0006] The technical task of the present application is achieved in the following way, a multi-modal hash retrieval method based on deep semantic agent, which is as follows:

[0007] Constructing a multi-modal data set: obtaining image modal and text modal data sets, and preprocessing, then using a CLIP pre-training model to extract deep features of the image modal and text modal data sets, and constructing a training set, a test set and a retrieval set according to the proportion, and then combining to generate the final data set;

[0008] Constructing a multi-modal hash retrieval model based on deep semantic agent: using neural network and deep learning method to construct a multi-modal hash retrieval model based on deep semantic agent;

[0009] Training the model: using the final data set to train the multi-modal hash retrieval model based on deep semantic agent.

[0010] As a preferred, the multi-modal data set is constructed as follows:

[0011] Obtaining a public data set: obtaining and downloading a public data set containing image modal and text modal from public network resources as an original data set; wherein the public data set includes Mir-Flickr data set and NUS-WIDE data set;

[0012] Dividing the downloaded original data set; specifically as follows:

[0013] Randomly selecting any part of image-text pairs on the Mir-Flickr data set and the NUS-WIDE data set as a test set for evaluating the test performance;

[0014] Excluding the test set image-text pair data in the Mir-Flickr data set and the NUS-WIDE data set, and randomly extracting a part of the remaining image-text pairs as a retrieval data set for retrieval operation;

[0015] In the retrieval data set, the remaining part of the image-text pair data set is used as a training set for target function training to achieve better system retrieval performance;

[0016] Extracting image modal deep features, using a pre-training model CLIP as an image encoder to realize deep feature extraction of the image modal; specifically as follows:

[0017] Preprocessing image modal data: uniformly crop and resize the input image to HxH pixels; convert the resized image into a flattened two-dimensional plane image block, and normalize the image to eliminate differences between images;

[0018] Extracting deep image features: load the ViT-B / 32 model onto the GPU by downloading the CLIP pre-trained model, and use the ViT-B / 32 model to extract image deep features to capture high-level semantic information of the image;

[0019] Extracting deep text features: use the CLIP pre-trained model as a text encoder to extract deep features from the text modal; specifically as follows:

[0020] Preprocessing text modal data: perform text cleaning, text segmentation, and text vectorization preprocessing operations on the input text, and map the text information to a text vector;

[0021] Extracting deep text features: load the ViT-L / 14 model onto the GPU by downloading the CLIP pre-trained model, and use the ViT-L / 14 model to extract deep text features to capture high-level semantic information of the text.

[0022] More preferably, the multi-modal hash retrieval model based on deep semantic agent is constructed as follows:

[0023] Multi-modal fusion: after extracting deep image features and deep text features through a fine-grained feature extractor, obtain deep image feature weights and deep text feature weights, process the deep image feature weights and deep text feature weights through a feature refinement part to obtain refined image features and refined text features, generate a feature fusion matrix through multi-modal feature fusion, and output a hash code through a hash mapping part;

[0024] Semantic agent hashing: map pairs of deep image features, deep text features, and deep label features through a multi-layer perceptron into image encoding, text encoding, and label encoding, and then generate two proxy hash codes through loose coupling fusion and proxy hash mapping;

[0025] Hash retrieval query: the hash code is used for similarity retrieval by calculating the Hamming distance, and the average precision is calculated multiple times for iteration to evaluate the retrieval performance.

[0026] More preferably, the multi-modal fusion is as follows:

[0027] Feature refinement: process the image feature weights and text feature weights through a fine-grained feature extractor, multiply the deep image features and image feature weights, and the deep text features and text feature weights to obtain image refined features and text refined features, respectively, as follows:

[0028]

[0029]

[0030] where X and Y represent the depth image feature and the depth text feature, respectively; T x and T y represent the image fine-grained feature extractor and the text fine-grained feature extractor, respectively; C x and C y represent the image feature weight and the text feature weight, respectively; θ dx and θ dy represent the training parameters; and represent the image refined feature and the text refined feature, respectively;

[0031] Multimodal feature fusion: the image refined feature and the text refined feature are fused through multimodal feature fusion to generate a feature fusion matrix, as follows:

[0032]

[0033] where represents the feature fusion matrix; k1 and k2 represent the modal fusion parameters; and represent the image refined feature and the text refined feature, respectively;

[0034] Hash mapping: the feature fusion matrix is mapped through hash mapping to generate a hash code; the formula is as follows:

[0035]

[0036] where B i represents the hash code of the i-th pair of sample features; represents the feature fusion vector of the i-th pair of sample features.

[0037] More preferably, the semantic agent hash is as follows:

[0038] Feature encoding: the pairs of depth image features, depth text features and depth label features are mapped through a multilayer perceptron to generate image encoding, text encoding and label encoding, as follows:

[0039]

[0040]

[0041]

[0042] where X, Y and L represent the depth image feature, the depth text feature and the depth label feature, respectively; and represent image encoding, text encoding and label encoding, respectively; MLP X , MLP Y and MLP L represent a multi-layer perception;

[0043] Loosely coupled fusion: image encoding and label encoding and text encoding and label encoding are respectively subjected to L2 normalization and then subjected to loosely coupled fusion addition to obtain image fusion label matrix and text fusion label matrix, and the formula is as follows:

[0044]

[0045]

[0046] wherein, and represent the image fusion label matrix and the text fusion label matrix of the i-th pair of samples; and represent image encoding, text encoding and label encoding corresponding to the i-th pair of samples; ||·||2 represents L2 norm;

[0047] Proxy hash mapping: the image fusion label matrix and the text fusion label matrix are subjected to proxy hash mapping to output loose image modal proxy hash code and loose text modal proxy hash code, and then subjected to Sign function to generate image modal proxy hash code and text modal proxy hash code, and the formula is as follows:

[0048]

[0049]

[0050]

[0051]

[0052] wherein, and represent the image fusion label matrix and the text fusion label matrix of the i-th pair of samples; B I i and B T i represent image modal proxy hash code and text modal proxy hash code, respectively; h I i and h T i represent loose image modal proxy hash code and loose text modal proxy hash code, respectively;

[0053] The hash retrieval query is specifically as follows:

[0054] The average precision (AP) is calculated, and the formula is as follows:

[0055]

[0056] Wherein, N represents the total number associated with the retrieval sample; P(m) represents the precision of the relevant first M instances, and when the mth sample is relevant to the retrieval sample, δ(m)=1, otherwise δ(m)=0;

[0057] The mean average precision (MAP) is calculated, and the formula is as follows:

[0058]

[0059] Wherein, Q represents the number of retrieval sets; AP(i) represents the ith AP value;

[0060] For each modality, the average precision (AP) under different recall rates is calculated, and the average precision of each modality is averaged to obtain the MAP value of each modality.

[0061] As preferred, the training model is specifically as follows:

[0062] The loss function is constructed, and the goal of the multi-modal hash retrieval model based on deep semantic agent is to optimize the hash code by using the image modal agent hash code and the text modal agent hash code, so as to obtain high-quality hash code and realize more efficient multi-modal hash retrieval. The image modal agent hash code B I i , the text modal agent hash code B T i , the hash code B i , the loose image modal agent hash code h I i and the loose text modal agent hash code h T i The multi-modal hash retrieval model based on deep semantic agent is trained by constructing the loss function, and the specific process is as follows:

[0063] The image modal agent hash code B I i is minimized and optimized, and the formula is as follows:

[0064]

[0065] Wherein, α1 and α2 represent optimization parameters; B I i represents the image modal agent hash code; h I idenotes a loose image modal proxy hash code; k denotes the number of sample pairs;

[0066] minimizing the optimization text modal proxy hash code B T i , the formula is as follows:

[0067]

[0068] wherein, α3 and α4 represent optimization parameters; B T i denotes a text modal proxy hash code; h T i denotes a loose text modal proxy hash code; k denotes the number of sample pairs;

[0069] optimizing the hash code B i , the formula is as follows:

[0070]

[0071] wherein, μ1 and μ2 represent optimization parameters; B i denotes a hash code; B I i denotes an image modal proxy hash code; B T i denotes a text modal proxy hash code;

[0072] optimization model; the Adam algorithm is used as the optimization function of the model; wherein, the learning rate parameter is set to 0.0001, and other hyperparameters can use the default values in PyTorch.

[0073] A multi-modal hash retrieval system based on deep semantic proxy, which is used to realize the multi-modal hash retrieval method based on deep semantic proxy as described above; the system comprises:

[0074] A data set construction unit is configured to obtain image modal and text modal data sets, and pre-process them, then use a CLIP pre-training model to extract deep features of the image modal and text modal data sets, and construct a training set, a test set and a retrieval set according to a proportion, and then combine to generate a final data set.

[0075] A model construction unit is configured to construct a multi-modal hash retrieval model based on deep semantic proxy by using a neural network and a deep learning method.

[0076] A model training unit is configured to train the multi-modal hash retrieval model based on deep semantic proxy by using the final data set.

[0077] More preferably, the model construction unit comprises:

[0078] The multi-modal fusion module is used to extract key semantic information from different modal depth features by the pre-trained model to extract depth image features and depth text features, and to retain cross-modal relationships in the fused features to generate high-quality hash codes; specifically, the depth image features and the depth text features are extracted through a fine-grained feature extractor to extract image feature weights and text feature weights, the image feature weights and the text feature weights are crucial to understanding the association between different modalities, key information is allowed to be extracted from different modal samples for feature refinement according to the importance of the samples, the depth image features and the image feature weights are multiplied, and the depth text features and the text feature weights are multiplied to obtain refined image features and refined text features, respectively, and then the refined image features and the refined text features are combined by parameter addition to obtain a feature fusion matrix, and finally a hash code is output through hash mapping;

[0079] The semantic agent hashing module is used to obtain image modal agent hash codes and text modal agent hash codes by feature encoding, loosely coupled fusion and agent hash mapping of the depth image features, the depth text features and the depth label features, so as to realize better supervision information to guide the quality of the hash codes generated in the flexible multi-modal fusion module; specifically, the depth image features, the depth text features and the depth label features are respectively mapped into image encoding, text encoding and label encoding through a multi-layer perceptron, the image encoding and the label encoding are respectively fused with the text encoding and the label encoding to generate an image fusion label matrix and a text fusion label matrix, and then the image fusion label matrix and the text fusion label matrix are mapped through agent hash mapping to generate a loose image modal agent hash code and a loose text modal agent hash code, and finally the image modal agent hash code and the text modal agent hash code are output through a sign function.

[0080] The hash retrieval query module is used to retrieve and query the hash codes to generate a query result MAP as an output; specifically, the hash codes are retrieved and queried by calculating the Hamming distance, the average precision (AP) is calculated through multiple iterations, and then the average precision mean (MAP) is calculated, and finally the MAP value is obtained to evaluate the retrieval performance.

[0081] An electronic device, comprising a memory and at least one processor;

[0082] The memory has a computer program stored thereon;

[0083] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multi-modal hash retrieval method based on the deep semantic agent as described above.

[0084] A computer readable storage medium, the computer readable storage medium has a computer program stored therein, the computer program can be executed by a processor to realize the multi-modal hash retrieval method based on the deep semantic agent as described above.

[0085] The multi-modal hash retrieval method, system, device and medium based on the deep semantic agent of the application have the following advantages:

[0086] (1) The application solves the problem of insufficient multi-modal fusion and weak association between data itself and supervised semantics in the multi-modal hash retrieval process, realizes a neural network structure for multi-modal hash retrieval based on deep semantic agents, ensures adaptive dynamic fusion according to the importance of different modalities and strengthens the connection between data itself and supervised semantic data during multi-modal fusion, thereby improving the accuracy of multi-modal hash retrieval;

[0087] (2) The application proposes a neural network structure for deep semantic agent multi-modal hash retrieval: the neural network structure can realize efficient multi-modal hash retrieval while retaining semantic information of different modalities; the core idea is to use two semantic agent hash networks to generate image modal agent hash codes and text modal agent hash codes that fuse label information, thereby establishing a connection between data instances and supervised information;

[0088] (3) The application realizes retrieval effect superior to existing technologies through flexible multi-modal fusion networks and semantic agent hash networks, can more fully utilize semantic information of data instances themselves, and realizes more efficient multi-modal hash retrieval;

[0089] (4) The multi-modal hash retrieval based on deep semantic agents proposed by the application effectively improves the accuracy and efficiency of multi-modal hash retrieval, and provides a more superior solution for practical applications in the field of multi-modal retrieval;

[0090] (5) The application uses CLIP as a modal deep feature extraction encoder, which not only enriches the semantic information of multi-modal data, but also effectively obtains global information of input images and models long-distance dependencies of text, thereby improving the semantic expression ability of features;

[0091] (6) The application realizes task-based key information matching in the flexible multi-modal fusion process, realizes feature association between different samples, effectively learns cross-modal relationships, and improves the fusion effect;

[0092] (7) The application introduces two agent hash networks in the semantic agent hash process to generate image modal and text modal agent hash codes that fuse label information, establish a connection between data instances and corresponding supervised information, and improve the effectiveness of supervision;

[0093] (viii) This invention optimizes the hash function by using proxy hash codes, thereby obtaining binary hash codes with good semantic consistency and cross-modal alignment, which improves the accuracy of retrieval. Attached Figure Description

[0094] The invention will be further described below with reference to the accompanying drawings.

[0095] Appendix Figure 1 The flowchart shows the multimodal hash retrieval method based on deep semantic agents.

[0096] Figure 2 This is a schematic diagram of a multimodal hash retrieval method based on deep semantic agents;

[0097] Figure 3 A flowchart for training the model;

[0098] Figure 4 A flowchart for constructing a multimodal dataset;

[0099] Figure 5 A flowchart for multimodal fusion;

[0100] Figure 6 A flowchart for semantic proxy hashing;

[0101] Figure 7 This is a flowchart of a hash retrieval query. Detailed Implementation

[0102] The following detailed description of the multimodal hash retrieval method, system, device, and medium based on deep semantic proxy of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0103] Example 1:

[0104] As attached Figure 1 As shown in the figure, this embodiment provides a multimodal hash retrieval method based on deep semantic proxy, which is as follows:

[0105] S1. Construct a multimodal dataset: Obtain image modality and text modality datasets, preprocess them, then use the CLIP pre-trained model to extract deep features from the image modality and text modality datasets, and construct training set, test set and retrieval set according to proportions, and then combine them to generate the final dataset;

[0106] S2. Construct a multimodal hash retrieval model based on deep semantic agents: Utilize neural networks and deep learning methods to construct a multimodal hash retrieval model based on deep semantic agents;

[0107] S3. Training the model: Use the final dataset to train the multimodal hash retrieval model based on deep semantic agents.

[0108] As shown in the accompanying Figure 4 The construction of the multi-modal data set in step S1 of the embodiment is specifically as follows:

[0109] S101, acquiring a public data set: acquiring and downloading a public data set containing an image modality and a text modality from a public network resource as an original data set; wherein the public data set includes a Mir-Flickr data set and a NUS-WIDE data set;

[0110] S102, data division on the downloaded original data set; specifically as follows:

[0111] S10201, randomly selecting any part of image-text pairs on the Mir-Flickr data set and the NUS-WIDE data set as a test set for evaluating test performance;

[0112] S10202, excluding the test set image-text pair data in the Mir-Flickr data set and the NUS-WIDE data set, and randomly extracting a part of the remaining image-text pairs as a retrieval data set for retrieval operation;

[0113] S10203, in the retrieval data set, the remaining part of the image-text pairs in the data set are used as a training set for target function training to achieve better system retrieval performance;

[0114] S103, extracting image modality deep features, using a pre-trained model CLIP as an image encoder to realize deep feature extraction of the image modality; specifically as follows:

[0115] S10301, preprocessing the image modality data: uniformly crop and reshape the input image to HxH pixels; convert the reshaped image into a flattened two-dimensional plane image block, and normalize the image to eliminate differences between images;

[0116] S10302, extracting deep image features: load the ViT-B / 32 model on the GPU by downloading the CLIP pre-trained model, and use the ViT-B / 32 model to extract image deep features to capture high-level semantic information of the image;

[0117] For example: in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0118]

[0119] Wherein, the preprocess_image function represents the preprocessing of the image modality data; extract_image_features represents the extraction of deep image features;

[0120] S104, extract the text modal depth feature, use the CLIP pre-training model as the text encoder to realize the deep feature extraction of the text modal; the specific implementation is as follows:

[0121] S10401, preprocessing the text modal data: performing preprocessing operations such as text cleaning, text segmentation and text vectorization on the input text, and mapping the text information to a text vector;

[0122] S10402, extract deep text features: load the ViT-L / 14 model on the GPU by downloading the CLIP pre-training model, and use the ViT-L / 14 model to extract deep text features to capture high-level semantic information of the text.

[0123] For example: in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0124]

[0125]

[0126] Among them, the preprocess_text function represents the preprocessing of the text modal data; extract_text_features represents the extraction of deep text features.

[0127] As shown in the accompanying Figure 2 , the construction of the multi-modal hash retrieval model based on the deep semantic agent in step S2 of the embodiment is as follows:

[0128] S201, multi-modal fusion: after the deep image features and the deep text features are extracted by the fine-grained feature extractor, the deep image feature weight and the deep text feature weight are obtained, the deep image feature weight and the deep text feature weight are processed by the feature refinement part to obtain the refined image features and the refined text features, the feature fusion matrix is generated through multi-modal feature fusion, and the hash code is output through the hash mapping part;

[0129] S202, semantic agent hash: the pair of deep image features, deep text features and deep label features are mapped to image encoding, text encoding and label encoding through a multi-layer perceptron, and then two agent hash codes are generated through loose coupling fusion and agent hash mapping;

[0130] S203, hash retrieval query: the hash code is used for similarity retrieval by calculating the Hamming distance, and the average precision is calculated multiple times for iteration to evaluate the retrieval performance.

[0131] As shown in the accompanying Figure 5As shown, the multi-modal fusion in step S201 in the embodiment is specifically as follows:

[0132] S20101, feature refinement: obtain image feature weight and text feature weight by processing through fine-grained feature extractor, multiply deep image feature and image feature weight and deep text feature and text feature weight to obtain image refined feature and text refined feature, the formula is as follows:

[0133] C x =T x (X,θ dx );C y =T y (Y,θ dy );

[0134]

[0135] Wherein, X and Y represent deep image feature and deep text feature respectively; T x and T y respectively represent image fine-grained feature extractor and text fine-grained feature extractor; C x and C y respectively represent image feature weight and text feature weight; θ dx and θ dy represent training parameters; and respectively represent image refined feature and text refined feature;

[0136] S20102, multi-modal feature fusion: generate feature fusion matrix by multi-modal feature fusion of image refined feature and text refined feature, the formula is as follows:

[0137]

[0138] Wherein, represents feature fusion matrix; k1 and k2 represent modal fusion parameters; and respectively represent image refined feature and text refined feature;

[0139] S20103, hash mapping: generate hash code by hash mapping of feature fusion matrix; the formula is as follows:

[0140]

[0141] Wherein, B i represents the hash code of the i-th pair of sample features; represents the feature fusion vector of the i-th pair of sample features.

[0142] For example, in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0143]

[0144] Wherein, nn.Linear represents a linear layer; nn.ReLU represents a ReLU activation function; nn.Sigmoid represents a Sigmoid activation function; torch.sign represents a sign activation function; self.fusionnn represents multi-modal feature fusion; self.params[0] and self.params[1] represent modal fusion parameters.

[0145] As shown in the accompanying Figure 6 The semantic agent hash in step S202 of the embodiment is as follows:

[0146] S20201, feature encoding: the paired depth image features, depth text features and depth label features are mapped through a multi-layer perceptron to generate image encoding, text encoding and label encoding, and the formula is as follows:

[0147]

[0148]

[0149]

[0150] Wherein, X, Y and L represent depth image features, depth text features and depth label features, respectively; and represent image encoding, text encoding and label encoding, respectively; MLP X , MLP Y and MLP L represent a multi-layer perceptron.

[0151] S20202, loosely coupled fusion: the image encoding and label encoding and the text encoding and label encoding are respectively subjected to L2 normalization and then added after loosely coupled fusion to obtain an image fusion label matrix and a text fusion label matrix, and the formula is as follows:

[0152]

[0153]

[0154] Wherein, and represent the image fusion label matrix and the text fusion label matrix of the ith sample pair, respectively; and denote the image encoding, text encoding and label encoding corresponding to the i-th pair of samples; ||·||2 denotes the L2 norm;

[0155] S20203, proxy hash mapping: the image fused label matrix and the text fused label matrix are subjected to proxy hash mapping to output loose image modal proxy hash codes and loose text modal proxy hash codes, and then subjected to Sign function to generate image modal proxy hash codes and text modal proxy hash codes, and the formula is as follows:

[0156]

[0157]

[0158]

[0159]

[0160] wherein, and denote the image fused label matrix and the text fused label matrix of the i-th pair of samples; B I i and B T i denote the image modal proxy hash codes and the text modal proxy hash codes; h I i and h T i denote loose image modal proxy hash codes and loose text modal proxy hash codes.

[0161] For example: in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0162]

[0163] wherein, FeatureEncoder denotes the feature encoding operation; loose_coupling_fusion denotes the loose coupling fusion operation; proxy_hash_mapping denotes the proxy hash mapping operation; image_codes, text_codes and label_codes denote the image encoding, text encoding and label encoding; image_fused_labels and text_fused_labels denote loose image modal proxy hash codes and loose text modal proxy hash codes; image_hash_codes and text_hash_codes denote image modal proxy hash codes and text modal proxy hash codes.

[0164] As shown in the accompanying drawings Figure 7As shown, the hash retrieval query in step S203 of the embodiment is specifically as follows:

[0165] S20301, average precision (AP) calculation, the formula is as follows:

[0166]

[0167] Wherein, N represents the total number associated with the retrieval sample; P(m) represents the precision of the relevant first M instances, when the mth sample is related to the retrieval sample, δ(m) = 1, otherwise δ(m) = 0;

[0168] S20302, calculate the average precision mean (MAP), the formula is as follows:

[0169]

[0170] Wherein, Q represents the number of retrieval sets; AP(i) represents the ith AP value;

[0171] For each modality, calculate the average precision (AP) under different recall rates, and average the average precision of each modality to obtain the MAP value of each modality.

[0172] For example: in the PyTorch framework, the code implementation corresponding to the hash retrieval query described above is as shown below:

[0173]

[0174]

[0175] Wherein, compute_ap is S20301 average precision (AP) calculation; compute_map is S20302 average precision mean (MAP) calculation; torch.cdist is to calculate the Hamming distance.

[0176] As shown in the accompanying Figure 3 The training model in step S3 of the embodiment is specifically as follows:

[0177] S301, construct the loss function, the goal of the multi-modal hash retrieval model based on deep semantic agent is to optimize the hash code by using the image modal agent hash code and the text modal agent hash code, so as to obtain high-quality hash code and realize more efficient multi-modal hash retrieval, and the image modal agent hash code B I i , the text modal agent hash code B T i , the hash code B i , the loose image modal agent hash code h I iand loose text modal agent hash code h T i The multi-modal hash retrieval model based on deep semantic agent is trained by constructing loss function, specifically as follows:

[0178] S30101, minimizing the optimization image modal agent hash code B I i , the formula is as follows:

[0179]

[0180] Where, alpha1 and alpha2 represent optimization parameters; B I i Indicates the image modal agent hash code; h I i Indicates the loose image modal agent hash code; k represents the number of sample pairs;

[0181] S30102, minimizing the optimization text modal agent hash code B T i , the formula is as follows:

[0182]

[0183] Where, alpha3 and alpha4 represent optimization parameters; B T i Indicates the text modal agent hash code; h T i Indicates the loose text modal agent hash code; k represents the number of sample pairs;

[0184] S30103, optimizing hash code B i , the formula is as follows:

[0185]

[0186] Where, mu1 and mu2 represent optimization parameters; B i Indicates the hash code; B I i Indicates the image modal agent hash code; B T i Indicates the text modal agent hash code;

[0187] Illustration: the loss function described above is expressed as code in PyTorch as follows:

[0188] Loss1 = alpha_1 * torch.norm(image_hash_codes - proxy_image_hash_codes, dim = 1).pow(2) + alpha_2 * torch.mean(torch.abs(image_hash_codes))

[0189] Loss2 = alpha_3 * torch.norm(text_hash_codes - proxy_text_hash_codes, dim = 1).pow(2) + alpha_4 * torch.mean(torch.abs(text_hash_codes))

[0190] Loss3 = mu_1 * torch.norm(hash_codes - image_proxy_hash_codes, dim = 1).pow(2) + mu_2 * torch.norm(hash_codes - text_proxy_hash_codes, dim = 1).pow(2)

[0191] + mu_2 * torch.norm(hash_codes - text_proxy_hash_codes, dim = 1).pow(2)

[0192] Wherein, torch.norm represents L2 norm; torch.mean represents mean; torch.abs represents absolute value;

[0193] S302, optimize the model; use Adam algorithm as the optimization function of the model; wherein, the learning rate parameter is set to 0.0001, and other hyperparameters can use the default value in PyTorch.

[0194] This embodiment selects MAP value as evaluation index.

[0195] For example: the optimization function described above is expressed as code in PyTorch as follows:

[0196] optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

[0197] Wherein, torch.optim.Adam is the Adam optimization function embedded in pytorch; model.parameters() is the parameter set of model training; learning_rate is the learning rate.

[0198] The model in this embodiment was tested on two public datasets, the Mir-Flickr dataset and the NUS-WIDE dataset. The experiments show that the model in this embodiment outperforms existing models. The comparison of the MAP results is shown in the table below:

[0199]

[0200]

[0201] The model in this embodiment was compared with existing models, and it can be seen that the performance of this embodiment is the best compared with other methods.

[0202] Example 2:

[0203] This embodiment provides a multimodal hash retrieval system based on deep semantic proxy, which is used to implement the multimodal hash retrieval method based on deep semantic proxy as described in Embodiment 1; the system includes:

[0204] The dataset construction unit is used to acquire image modality and text modality datasets, preprocess them, and then use the CLIP pre-trained model to extract deep features from the image modality and text modality datasets. Training sets, test sets and retrieval sets are constructed according to the proportions, and then combined to generate the final dataset.

[0205] The model building unit is used to construct a multimodal hash retrieval model based on deep semantic agents using neural networks and deep learning methods;

[0206] The model training unit is used to train the multimodal hash retrieval model based on deep semantic agents using the final dataset.

[0207] As attached Figure 2 As shown, the model building unit in this embodiment includes:

[0208] The multimodal fusion module is used to extract key semantic information from the deep image and deep text features extracted by the pre-trained model, preserve cross-modal relationships in the fused features, and generate high-quality hash codes; as shown in the appendix. Figure 5 As shown, specifically: the deep image features and deep text features are processed by a fine-grained feature extractor to extract image feature weights and text feature weights. Image feature weights and text feature weights are crucial for understanding the correlation between different modalities. Based on the importance of the samples, key information can be extracted from samples of different modalities for feature refinement. The deep image features and image feature weights are multiplied by the key, and the deep text features and text feature weights are multiplied by the key to obtain refined image features and refined text features, respectively. Then, the refined image features and refined text features are combined by adding parameters to obtain a feature fusion matrix. Finally, a hash code is output through hash mapping.

[0209] The semantic agent hash module is used to obtain image modal agent hash codes and text modal agent hash codes through feature coding, loose coupling fusion and agent hash mapping of the depth image features, the depth text features and the depth label features, so as to realize better supervision information to guide the quality of the hash codes generated in the flexible multi-modal fusion module; as shown in the accompanying Figure 6 The depth image features, the depth text features and the depth label features are respectively mapped into image coding, text coding and label coding through a multi-layer perception, the image coding and the label coding are respectively subjected to loose coupling fusion with the text coding and the label coding to generate image fusion label matrices and text fusion label matrices, the image fusion label matrices and the text fusion label matrices are then subjected to agent hash mapping to generate loose image modal agent hash codes and loose text modal agent hash codes, and finally the image modal agent hash codes and the text modal agent hash codes are output through a sign function;

[0210] The hash retrieval query module is used to perform retrieval query on the hash codes to generate a query result MAP as an output; as shown in the accompanying Figure 7 The hash codes are subjected to similarity retrieval through calculation of Hamming distances, the average precision (AP) is calculated through multiple iterations, and then the average precision mean (MAP) is calculated, and finally the MAP value obtained is used to evaluate the retrieval performance.

[0211] Embodiment 3

[0212] The embodiment also provides an electronic device, including a memory and a processor.

[0213] The memory stores computer execution instructions.

[0214] The processor executes the computer execution instructions stored in the memory, so that the processor executes the multi-modal hash retrieval method based on the depth semantic agent in any embodiment of the application.

[0215] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), ready-to-program gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor, or the processor can be any conventional processor.

[0216] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the terminal, and the like. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory device.

[0217] Embodiment 4:

[0218] The embodiment also provides a computer readable storage medium, in which a plurality of instructions are stored, the instructions are loaded by a processor to make the processor execute the deep semantic agent-based multi-modal hash retrieval method in any embodiment of the application. Specifically, a system or device equipped with a storage medium can be provided, and software program code for realizing the functions of any one of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0219] In this case, the program code read from the storage medium can itself realize the functions of any one of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the application.

[0220] Embodiments of the storage medium for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0221] In addition, it should be clear that not only the program code read by the computer can be executed, but also part or all of the actual operations can be completed by the operating system and the like operating on the computer based on the instructions of the program code, so as to realize the functions of any one of the above embodiments.

[0222] Further, it is understood that the program code read by the storage medium can be written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU or the like mounted on the expansion board or the expansion unit is caused to perform part or all of the actual operation based on the instruction of the program code, thereby realizing the function of any of the above-described embodiments.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal hash retrieval method based on deep semantic proxy, characterized in that, The method is as follows: Constructing a multimodal dataset: Obtain image modality and text modality datasets, preprocess them, then use the CLIP pre-trained model to extract deep features from the image modality and text modality datasets, and construct training set, test set and retrieval set according to proportions, and then combine them to generate the final dataset; Constructing a multimodal hash retrieval model based on deep semantic agents: Utilizing neural networks and deep learning methods to construct a multimodal hash retrieval model based on deep semantic agents; Training the model: The multimodal hash retrieval model based on deep semantic agents is trained using the final dataset; The specific steps for constructing a multimodal hash retrieval model based on deep semantic agents are as follows: Multimodal fusion: After extracting depth image features and depth text features through a fine-grained feature extractor, feature weights for depth image features and feature weights for depth text features are obtained. The feature weights for depth image features and feature weights for depth text features are then processed by a feature refinement part to obtain refined image features and refined text features. A feature fusion matrix is ​​generated through multimodal feature fusion, and a hash code is output through a hash mapping part. Semantic surrogate hashing: Pairs of deep image features, deep text features, and deep label features are mapped to image codes, text codes, and label codes through a multilayer perceptron, and then two surrogate hash codes are generated through loose coupling fusion and surrogate hash mapping. Hash retrieval query: Hash codes are used to perform similarity retrieval by calculating Hamming distance, and the average precision is calculated through multiple iterations to evaluate retrieval performance; Multimodal fusion is detailed below: Feature Refinement: Image feature weights and text feature weights are obtained through a fine-grained feature extractor. The image refinement features and text refinement features are then multiplied by the depth image features and their respective weights, as well as by the depth text features and their respective weights, as shown in the following formulas: C x =T x (X,θ dx );C y =T y (Y,θ dy ); Where X and Y represent depth image features and depth text features, respectively; T x and T y These represent the image fine-grained feature extractor and the text fine-grained feature extractor, respectively; C x and C y θ represents the image feature weights and text feature weights, respectively; dx and θ dy Indicates training parameters; and These represent image thinning features and text thinning features, respectively. Multimodal feature fusion: Image thinning features and text thinning features are fused using multimodal feature fusion to generate a feature fusion matrix, as shown in the following formula: in, k1 and k2 represent the feature fusion matrix and modality fusion parameters, respectively. and These represent image thinning features and text thinning features, respectively. Hash mapping: The feature fusion matrix is ​​processed through a hash mapping to generate hash codes; the formula is as follows: Among them, B i Represents the hash code of the i-th pair of sample features; This represents the feature fusion vector of the i-th pair of sample features.

2. The multimodal hash retrieval method based on deep semantic proxy according to claim 1, characterized in that, The construction of the multimodal dataset is as follows: Obtain public datasets: Obtain and download public datasets containing image and text modalities from public online resources as the raw datasets; among them, public datasets include the Mir-Flickr dataset and the NUS-WIDE dataset; The downloaded raw dataset is then partitioned; the specific steps are as follows: Randomly select a subset of image-text pairs from the Mir-Flickr and NUS-WIDE datasets as the test set to evaluate test performance; Excluding the test set image-text pairs from the Mir-Flickr and NUS-WIDE datasets, a portion of the remaining image-text pairs is randomly selected as the retrieval dataset for retrieval operations. In the retrieval dataset, the remaining image-text pairs in the dataset are used as the training set for training the objective function; To extract image modal depth features, a pre-trained model CLIP is used as the image encoder to achieve deep feature extraction of image modalities; the details are as follows: Preprocessing image modal data: uniformly crop and reshape the input image to H×H pixel size; convert the reshaped image into a flat two-dimensional planar image block, and perform normalization processing to eliminate differences between images; Extracting deep image features: By downloading the CLIP pre-trained model, the ViT-B / 32 model is loaded onto the GPU, and the ViT-B / 32 model is used to extract deep image features, thereby capturing the high-level semantic information of the image; Deep features of text modalities are extracted using the CLIP pre-trained model as a text encoder; the specifics are as follows: Preprocessing text modal data: The input text undergoes preprocessing operations such as text cleaning, text segmentation, and text vectorization, and the text information is mapped to text vectors; Extracting deep text features: By downloading the CLIP pre-trained model and loading the ViT-L / 14 model onto the GPU, the ViT-L / 14 model is used to extract deep text features, thereby capturing the high-level semantic information of the text.

3. The multimodal hash retrieval method based on deep semantic proxy according to claim 1, characterized in that, The semantic proxy hash is as follows: Feature encoding: Pairs of deep image features, deep text features, and deep label features are mapped using a multilayer perceptron to generate image codes, text codes, and label codes, as shown in the following formula: Where X, Y, and L represent depth image features, depth text features, and depth label features, respectively; and These represent image encoding, text encoding, and tag encoding, respectively; MLP X MLP Y and MLP L This represents a multilayer perceptron; Loosely Coupled Fusion: Image encoding and label encoding are L2 normalized separately, then loosely coupled and added together to obtain the image fusion label matrix and the text fusion label matrix, as shown in the following formula: in, and Let represent the image fusion label matrix and the text fusion label matrix of the i-th pair of samples, respectively; and This represents the image encoding, text encoding, and label encoding corresponding to the i-th pair of samples; ||·||2 represents the L2 norm. Proxy hashing: The image fusion label matrix and text fusion label matrix are processed through a proxy hashing method to output loose image modality proxy hash codes and loose text modality proxy hash codes. These are then processed by the Sign function to generate the image modality proxy hash codes and text modality proxy hash codes, as shown in the following formula: in, and B represents the image fusion label matrix and the text fusion label matrix of the i-th sample, respectively; I i With B T i These represent the image modal surrogate hash code and the text modal surrogate hash code, respectively; h I i with h T i These represent loose image modal surrogate hash codes and loose text modal surrogate hash codes, respectively. The hash search query is as follows: The average accuracy is calculated using the following formula: Where N represents the total number of instances associated with the retrieved sample; P(m) represents the precision of the first M instances associated with the retrieved sample, and δ(n) = 1 when the m-th sample is associated with the retrieved sample, otherwise δ(n) = 0. The formula for calculating the mean precision is as follows: Where Q represents the number of retrieval sets; AP(i) represents the i-th AP value; For each modality, calculate the average precision under different recall rates, and average the average precision of each modality to obtain the MAP value of each modality.

4. The multimodal hash retrieval method based on deep semantic proxy according to claim 1, characterized in that, The specific training model is as follows: The goal of constructing a loss function for the deep semantic proxy-based multimodal hash retrieval model is to optimize the hash code using image modal proxy hash codes and text modal proxy hash codes, thereby obtaining high-quality hash codes and achieving more efficient multimodal hash retrieval. This is achieved by utilizing image modal proxy hash codes B... I i Text modal proxy hash code B T i Hash code B i Loose image modality proxy hash code h I i and loose text modal proxy hash code h T i The loss function is constructed and a multimodal hash retrieval model based on deep semantic agents is trained as follows: Minimize and optimize image modal proxy hash code B I i The formula is as follows: Where α1 and α2 represent the optimization parameters; B I i Represents the image modality surrogate hash code; h I i This represents a loosely defined image modality surrogate hash code; k represents the number of sample pairs. Minimize and optimize text modal proxy hash code B T i The formula is as follows: Where α3 and α4 represent the optimization parameters; B T i Represents the text modal surrogate hash code; h T i Represents a loosely defined text modality proxy hash code; k represents the number of sample pairs; Optimized hash code B i The formula is as follows: Where μ1 and μ2 represent the optimization parameters; B i Represents hash code; B I i Represents the image modality surrogate hash code; B T i Represents the text modal proxy hash code; Optimize the model; use the Adam algorithm as the model's optimization function; the learning rate parameter is set to 0.0001, and other hyperparameters can use the default values ​​in PyTorch.

5. A multimodal hash retrieval system based on deep semantic proxy, characterized in that, This system is used to implement the multimodal hash retrieval method based on deep semantic agents as described in any one of claims 1 to 4; the system includes: The dataset construction unit is used to acquire image modality and text modality datasets, preprocess them, and then use the CLIP pre-trained model to extract deep features from the image modality and text modality datasets. Training sets, test sets and retrieval sets are constructed according to the proportions, and then combined to generate the final dataset. The model building unit is used to construct a multimodal hash retrieval model based on deep semantic agents using neural networks and deep learning methods; The model training unit is used to train the multimodal hash retrieval model based on deep semantic agents using the final dataset.

6. The multimodal hash retrieval system based on deep semantic proxy according to claim 5, characterized in that, The model building unit includes: The multimodal fusion module is used to extract key semantic information from the deep image features and deep text features extracted by the pre-trained model from the deep features of different modalities, and to preserve cross-modal relationships in the fused features to generate high-quality hash codes. Specifically, the deep image features and deep text features are processed by a fine-grained feature extractor to extract image feature weights and text feature weights. Image feature weights and text feature weights are crucial for understanding the correlation between different modalities. Based on the importance of the samples, key information can be extracted from samples of different modalities for feature refinement. Deep image features and image feature weights are multiplied by key points, and deep text features and text feature weights are multiplied by key points to obtain refined image features and refined text features, respectively. The refined image features and refined text features are then combined by adding parameters to obtain a feature fusion matrix. Finally, a hash code is output through hash mapping. The semantic proxy hashing module is used to obtain image modality proxy hash codes and text modality proxy hash codes by performing feature encoding, loose coupling fusion, and proxy hash mapping on deep image features, deep text features, and deep label features. Specifically, the deep image features, deep text features, and deep label features are mapped to image codes, text codes, and label codes respectively through a multilayer perceptron. The image codes and label codes are then loosely coupled with the text codes and label codes to generate image fusion label matrices and text fusion label matrices, respectively. These are then used for proxy hash mapping to generate loose image modality proxy hash codes and loose text modality proxy hash codes. Finally, the image modality proxy hash codes and text modality proxy hash codes are output through the sign function. The hash retrieval module is used to retrieve hash codes and generate a MAP (Map of Query Results) as output. Specifically, the hash codes are used to perform similarity retrieval by calculating Hamming distance, and the average precision is calculated through multiple iterations. Then, the mean of the average precision is calculated, and the final MAP value is used to evaluate the retrieval performance.

7. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal hash retrieval method based on deep semantic agents as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the multimodal hash retrieval method based on deep semantic agents as described in any one of claims 1 to 4.

Images