Cross-modal retrieval method and device based on attention network adversarial hashing

Abstract

The application provides a cross-modal retrieval method and device based on an attention network and an anti-hash, and the method comprises the following steps: obtaining image-text pair data, extracting initial global features of image data and text data, processing the initial global features of the image data and the initial global features of the text data through a shared attention module respectively to obtain final feature representations of the image data and the text data; a modal discriminator interacts with a feature extractor in an anti-hash manner to promote the modal feature extractor to learn better image-text features; the final feature representations of the image data and the text data are converted into binary codes, the features are mapped into a common Hamming space, cross-modal retrieval is performed, and the first K cross-modal retrieval results are obtained. The application combines an anti-hash network with a hash learning network, utilizes the anti-hash network learning to promote the feature extractor to learn modal invariant representation, simultaneously utilizes the characteristics of hash fast retrieval, and thus cross-modal retrieval is realized.

Description

Technical Field

[0001] This invention relates to the field of cross-modal image and text retrieval in computer technology, specifically to a cross-modal retrieval method and device based on attention network adversarial hashing. Background Technology

[0002] In cross-modal retrieval, extracting high-quality modal features and achieving fast and accurate retrieval are current research goals and directions. Traditional models based on handcrafted features cannot effectively extract features from different modalities, while modal features extracted using deep learning have stronger expressive power. Simultaneously, shared attention modules can effectively learn high-quality features, focusing on relevant information and ignoring irrelevant information, generating more discriminative features and laying the foundation for hash learning. Cross-modal hashing is popular due to its fast retrieval speed. Cross-modal hashing utilizes feature extractors to extract features from different modalities, while also extracting easily overlooked information from modal features. By learning their hash transformations, this data information is projected into a Hamming space, and then similarity is calculated in this space to ultimately achieve cross-modal retrieval. However, hash learning methods have not fully explored the semantic relevance of modalities during the hash learning process, nor have they adequately considered the distribution variations of the data. Therefore, how to efficiently extract attention features with masked weights and combine adversarial and hashing approaches for cross-modal retrieval has become a problem that needs to be considered. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a cross-modal retrieval method and device based on adversarial hashing using attention networks. It utilizes deep learning and a shared attention module to extract features from image and text data. Simultaneously, a modality discriminator works adversarially with the feature extractor to adapt to changes in data distribution. Through a hash learning network, binary encoding is obtained, ultimately achieving the goal of fast and accurate retrieval.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a cross-modal retrieval method based on attention network adversarial hashing, comprising the following steps:

[0005] Extract initial features from image and text data in paired image-text data;

[0006] Mask weights are learned for the initial features to obtain high-quality image and text features for the retrieval task. Specifically, the initial global features of the image data and the initial global features of the text data are used as inputs to a shared attention module. After processing by the attention module, the activation outputs of the image data and the text data are obtained. The activation outputs of the image data and the text data are then fed into a fully connected layer to obtain the final feature representations of the image data and the text data, respectively.

[0007] Based on the high-quality image and text features, a modality discriminator is used to determine whether the image and text features are image features or text features. The modality feature extractor and the modality discriminator interact in an adversarial manner, and the modality feature extractor learns better image and text features.

[0008] The image and text features are used as input to a hash learner, and a hash learning network is used to non-linearly convert the better image and text features into binary codes, i.e., hash codes.

[0009] The image or text data to be queried is mapped to a public Hamming space. Similarity is measured between the binary encoding of the query data and the binary encoding of the initial data in the public Hamming space. The similarity measurement results are sorted to obtain the top K cross-modal retrieval results.

[0010] Extracting initial features from the image and text data in paired image-text data includes:

[0011] For image data, a ResNet-101 network model pre-trained on ImageNet is used to extract initial features. The 2048-dimensional feature vector extracted from the pool 5 layer is mapped to a 1024-dimensional feature vector, and this is used as the initial global feature vector for the image data.

[0012] P∈R r×n ,P i =f(v i ;θ v );

[0013] For text data, word vectors are first obtained through preprocessing. Then, the word vectors are used as input to a bidirectional long short-term memory neural network for feature extraction, and their output is used as the initial global features Q∈R of the text data. r×n Q j =f(t) j ;θ t ).

[0014] The initial global features P of the image data i and the initial global features Q of the text data j As input to the shared attention module, f is constructed using a fully connected layer with a sigmoid activation function. mask The function learns the mask weights at each location in the feature map, and the mask weights M of the image data. i (v) and the mask weights M of the text data j (t) is:

[0015] M i (v)=f mask (P i )

[0016] M j (t)=f mask (Q j )

[0017] Fully connected layers are used to learn non-linear transformations of features and extract semantic information from the data;

[0018] The sigmoid activation function is used to apply mask weights M to image data. i (v) and the mask weights M of the text data j The value of (t) is mapped to the range [0,1].

[0019] The mask weight M of the image data i (v) and the initial global features P of the image data i Perform an inner product operation, and then combine the value obtained from the inner product with the initial global features P of the image data. i The operation and manipulation ultimately yield the activation output H of the image. i (v);

[0020] The mask weight M of the text data j (t) and the initial global feature Q of the text data j Perform an inner product operation; then combine the value obtained from the inner product with the initial global feature Q of the text data. j The operation and manipulation ultimately yield the activation output H of the text. j (t);

[0021] The parameters of the inner layer are shared for both modality samples, and the activation function output H of the image is... i (v) and the activation output H of the text j (t) is:

[0022] H i (v)=(1+M i (v))(P i )

[0023] H j (t)=(1+M j (t))(Q j )

[0024] Output H of the activation function of the image i (v) and the activation output H of the text j (t) are input into a fully connected layer respectively to obtain the final feature representation of the image data. The final feature representation of text data

[0025] Determining whether the image-text features are image features or text features using a modality discriminator includes:

[0026] A three-layer feedforward neural network is used to perform modality discrimination on the final representation of the above image data and text data. The output of the modality discriminator represents the modality of the input features, where 1 represents image data features and 0 represents text data features.

[0027] The modality discriminator is used to correctly distinguish the modality of a sample, that is, whether the input sample features belong to image sample features or text sample features. The difference between image sample features and text sample features is the corresponding loss. The smaller the loss, the greater the difference between image sample features and text sample features.

[0028] The modality discriminator consists of a three-layer feedforward neural network, and its output is:

[0029] D∈R n D i =D(y) i ;θ D )∈{0,1}

[0030] Where y i θ represents the image or text features of the projection. D As a parameter, if D i A value of 1 indicates that the projected features are classified as image data features; conversely, if D is 1, the projected features are classified as image data features. i A value of 0 indicates that the projected features are classified as text data features;

[0031] Using the likelihood function to represent y i and l i The relationship between them

[0032]

[0033] in The probability that an image or text is a feature of an image is represented by l. i Here, we define the true modality label for each image-text pair; the difference between image features and text features represents the corresponding loss. A smaller loss indicates a greater difference between image and text features. The discriminative loss function is:

[0034]

[0035] Where n is the number of image-text pairs in the cross-modal dataset, θ D These are the parameters of the modality discriminator.

[0036] Using the image and text features as input to a hash learner, the hash learning network non-linearly converts the better image and text features into binary codes, including:

[0037] Together, better image and text features are used to form a set of all feature representations with two modalities;

[0038] The set of all feature representations is used as the input to the hash learning network, which consists of three fully connected layers. The binary code is calculated based on the first two fully connected layers.

[0039] Hash learning networks utilize a loss function with semantic supervision to ensure the semantic relationship between the generated binary code and the initial data.

[0040] The hash learning network consists of three fully connected layers. The binary encoding is calculated based on the first two fully connected layers, and the binary encoding is learned by minimizing the following formula:

[0041]

[0042] stB∈{-1,1} r×2n

[0043] Where, B = {b1, b2, ..., b} 2n} represents the binary encoding of the data sample, n is the number of image-text pairs in the cross-modal dataset, and S h denoted as pairwise similarity in binary encoding learning, where a is the length of the binary encoding;

[0044] The above formula ensures that the learned binary codes should be close to the pairwise similarity S. h Equivalent to:

[0045]

[0046] stB∈{-1,1} r×2n

[0047] Where M = {m1, m2, ..., m} 2n} represents the value of the second fully connected layer in hash learning;

[0048] Based on an asymmetric relaxation strategy, the quantization error is further reduced by forcing the binary code B to be similar to the feature, and minimizing the feature and the pairwise similarity matrix S. h The differences between them;

[0049] The objective function of the hash learning network is:

[0050]

[0051] stB∈{-1,1} r×2n

[0052] in, It is an independent term. These are the balancing terms. α1, α2, and α3 are hyperparameters. α1 controls the discrete constraints of the binary code, α2 controls the independence of the binary code, and α3 controls the balance of the binary code. θ h These are the parameters of the hash learning network;

[0053] Ultimately, the overall objective function can be expressed as:

[0054]

[0055] Where β, γ, δ are hyperparameters, and θ v ,θ t These are the parameters of the image feature extraction network and the text feature extraction network, θ. D θ is the parameter of the modal discriminator. h These are the parameters of the hash learning network.

[0056] Image and text data are mapped to a common Hamming space. Similarity is measured between the binary encoding of the query data and the binary encoding of the initial data within the common Hamming space. The similarity measurement results are then ranked to obtain the top K cross-modal retrieval results, including:

[0057] The query text is used to retrieve images. The query text is converted into binary code through a cross-modal retrieval model. The distance between the obtained binary code and the binary code in the common Hamming space is calculated and sorted to obtain the top K image retrieval results.

[0058] The process of retrieving text from an image involves converting the image into binary code using a cross-modal retrieval model, calculating the distance between the resulting binary code and the binary codes in the common Hamming space, and then ranking them to obtain the top K text retrieval results.

[0059] Another computer device is provided, including a processor and a memory. The memory is used to store a computer executable program, and the processor reads the computer executable program from the memory and executes it. When the processor executes the program, it can implement the cross-modal retrieval method described in this invention.

[0060] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the cross-modal retrieval method described in the present invention.

[0061] Compared with existing cross-modal methods, the present invention has at least the following advantages:

[0062] Modal features extracted using deep learning have stronger expressive power, are more efficient and accurate in representing datasets, and the extracted image features are more robust and have better generalization ability. By embedding features into the same space using a shared attention module, heterogeneity gaps can be reduced, semantic features can be extracted, relevant information can be focused on and irrelevant information can be ignored, and more discriminative features can be generated. Modal discriminators can distinguish data types, and when they cannot distinguish data types, it means that the heterogeneity gap between different modal data has been minimized. By using cross-modal hashing learning methods to convert different modal features into binary codes, fast and accurate retrieval can be achieved, which improves the quality and efficiency of cross-modal retrieval methods to a certain extent.

[0063] Furthermore, when using different neural networks to extract initial features from image data and text data respectively, the long short-term memory function of bidirectional long short-term memory networks can solve the gradient explosion problem and preserve semantic consistency within modalities. Bidirectional long short-term memory networks can also pay attention to contextual information and rich semantic information, thereby fully extracting the semantic features of text data. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the structure of a novel cross-modal retrieval method.

[0065] Figure 2 This is a schematic diagram of a novel cross-modal retrieval method.

[0066] Figure 3 This is a schematic diagram of the shared attention module.

[0067] Figure 4 This is a flowchart of a novel cross-modal retrieval process. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making inventive technical solutions are within the scope of protection of the present invention.

[0069] This invention provides a cross-modal retrieval method based on attention network adversarial hashing, comprising the following steps:

[0070] Extract initial features from image and text data in paired image-text data;

[0071] Mask weights are learned for the initial features to obtain high-quality image and text features for the retrieval task. Specifically, the initial global features of the image data and the initial global features of the text data are used as inputs to a shared attention module. After processing by the attention module, the activation outputs of the image data and the text data are obtained. The activation outputs of the image data and the text data are then fed into a fully connected layer to obtain the final feature representations of the image data and the text data, respectively.

[0072] Based on the high-quality image and text features, a modality discriminator is used to determine whether the image and text features are image features or text features. The modality feature extractor and the modality discriminator interact in an adversarial manner, and the modality feature extractor learns better image and text features.

[0073] The image and text features are used as input to a hash learner, and a hash learning network is used to non-linearly convert the better image and text features into binary codes, i.e., hash codes.

[0074] The image or text data to be queried is mapped to a public Hamming space. Similarity is measured between the binary encoding of the query data and the binary encoding of the initial data in the public Hamming space. The similarity measurement results are sorted to obtain the top K cross-modal retrieval results.

[0075] Figure 1 The cross-modal retrieval described in this invention is based on a cross-modal retrieval model. A schematic diagram of the cross-modal retrieval model structure is shown below:

[0076] The cross-modal retrieval model consists of three parts:

[0077] The first part, the feature extractor, first uses different neural networks to perform initial feature extraction on the image and text. Then, the extracted features are input into a shared attention module to obtain image features and text features with attention, as detailed below:

[0078] For image data, a ResNet-101 network model pre-trained on ImageNet is used to extract initial features. The 2048-dimensional feature vector extracted from the pool 5 layer is mapped to a 1024-dimensional feature vector, and this is used as the initial global feature P∈R of the image data. r×n ,P i =f(v i ;θ v );

[0079] For text data, word vectors are first obtained through preprocessing. Then, the word vectors are used as input to a bidirectional long short-term memory neural network for feature extraction, and their output is used as the initial global features Q∈R of the text data. r×n Q j =f(t) j;θ t );

[0080] The initial global features of the image data and the initial global features of the text data are used as input to the shared attention module. After passing through the attention module, the activation outputs of the image data and the text data are obtained.

[0081] The activation outputs of the image data and text data are fed into a fully connected layer to obtain the final feature representations of the image data and text data, respectively.

[0082] The second part, the modality discriminator, is used to perform modality discrimination on the attention-bearing image features and text features obtained by the feature extractor, and to encourage the feature extractor to learn better image and text features in an adversarial manner, as detailed below:

[0083] A three-layer feedforward neural network is used to perform modality discrimination on the final representation of the above image data and text data. The output of the modality discriminator represents the modality of the input features, where 1 represents image data features and 0 represents text data features.

[0084] The purpose of a modality discriminator is to correctly distinguish the modality of a sample, that is, whether the input sample features belong to image sample features or text sample features. The difference between image sample features and text sample features is the corresponding loss. The smaller the loss, the greater the difference between the two.

[0085] Modal feature extractors aim to prevent modal discriminators from distinguishing the modalities of input sample features. The two interact in an adversarial manner to adapt to distribution changes, thereby enabling the modal feature extractor to learn better image and text features.

[0086] The third part, the hash learner, is used to non-linearly convert the attention-laden final image feature representation and the final text feature representation obtained by the feature extractor into binary codes, and uses a semantically supervised loss function to ensure the semantic relationship between the generated binary codes and the initial data, as follows:

[0087] The final feature representations of the image data and the final feature representations of the text data are combined to form a set of all feature representations with two modalities;

[0088] The set of all feature representations is used as the input to the hash learning network, which consists of three fully connected layers. The binary encoding is calculated based on the first two fully connected layers, and the binary encoding is finally obtained.

[0089] Hash learning networks utilize a loss function with semantic supervision to ensure the semantic relationship between the generated binary code and the initial data.

[0090] Figure 2 This is a schematic diagram of the cross-modal retrieval method of the present invention. The implementation of the present invention will now be described in detail.

[0091] Step 1: Use paired image and text data as input to the cross-modal retrieval model.

[0092] Step 2 involves using different neural networks to extract initial features from the image and text data, as detailed below:

[0093] Step 2.1: For the image data, the ResNet-101 network model pre-trained on ImageNet is used to extract initial features. The 2048-dimensional feature vector extracted from the pool5 layer is mapped to a 1024-dimensional feature vector, and this is used as the initial global feature P∈R of the image data. r×n ,P i =f(v i ;θ v );

[0094] The ResNet-101 network model has a strong ability to express local pixels, and can make full use of the two-dimensional structure of the image and the correlation between adjacent pixels.

[0095] The ResNet-101 residual network, pre-trained on the ImageNet dataset, exhibits excellent performance in image feature extraction.

[0096] Step 2.2: For text data, word vectors are first obtained through preprocessing. Then, the word vectors are used as input to a bidirectional long short-term memory neural network for feature extraction, and their output is used as the initial global features Q∈R of the text data. r×n Q j =f(t) j ;θ t ).

[0097] First, the word2vec method is used to extract word vectors, which are then mapped to a d-dimensional vector X.

[0098] Then, each word is embedded into a word vector space and input into a bidirectional long short-term memory network in sequence to extract the features and semantics of the text vectors. The process is represented as follows:

[0099]

[0100]

[0101] Where e t This represents the word vectors embedded in the word vector space. and Let represent the hidden states of the forward and backward LSTM at step t, respectively. These two hidden states are added at each time step t. A set of text feature vectors were constructed.

[0102] Finally, the initial global features Q of the text are obtained through average pooling.

[0103] The long short-term memory function of bidirectional long short-term memory networks can be used to solve the gradient explosion problem and preserve semantic consistency within modalities.

[0104] Bidirectional long short-term memory networks can also focus on contextual information and rich semantic information, thereby fully extracting the semantic features of text data.

[0105] Step 3: The initial features extracted in Step 2 are used as input to the shared attention module. The shared attention module learns mask weights for the initial features to obtain high-quality image and text features for the retrieval task, as detailed below:

[0106] The initial global features P of the image data i and the initial global features Q of the text data j As input to the shared attention module, f is constructed using a fully connected layer with a sigmoid activation function. mask The function learns the mask weights at each location in the feature map, and the mask weights M of the image data. i (v) and the mask weights M of the text data j (t) is defined as:

[0107] M i (v)=f mask (P i )

[0108] M j (t)=f mask (Q j )

[0109] Fully connected layers are used to learn non-linear transformations of features and extract semantic information from the data.

[0110] The sigmoid activation function is used to apply mask weights M to image data. i (v) and the mask weights M of the text data j The value of (t) is mapped to the range [0,1].

[0111] The mask weight M of the image data i (v) and the initial global features P of the image data i Perform an inner product operation, and then combine the value obtained from the inner product with the initial global features P of the image data. iThe operation and manipulation ultimately yield the activation output H of the image. i (v);

[0112] The mask weight M of the text data j (t) and the initial global feature Q of the text data j Perform an inner product operation; then combine the value obtained from the inner product with the initial global feature Q of the text data. j The operation and manipulation ultimately yield the activation output H of the text. j (t);

[0113] The parameters of the inner layer are shared for both modality samples, and the activation function output H of the image is... i (v) and the activation output H of the text j (t) is defined as:

[0114] H i (v)=(1+M i (v))(P i )

[0115] H j (t)=(1+M j (t))(Q j )

[0116] Output H of the activation function of the image i (v) and the activation output H of the text j (t) are input into a fully connected layer respectively to obtain the final feature representation of the image data. The final feature representation of text data

[0117] Step 4: Use the image and text features obtained in Step 3 as input to the modality discriminator, and use the modality discriminator to determine whether the input features are image features or text features.

[0118] A three-layer feedforward neural network is used to perform modality discrimination on the final representation of the above image data and text data. The output of the modality discriminator represents the modality of the input features, where 1 represents image data features and 0 represents text data features.

[0119] The modality discriminator is used to correctly distinguish whether the input sample features belong to image sample features or text sample features. The difference between image sample features and text sample features is the corresponding loss. The smaller the loss, the greater the difference between image sample features and text sample features.

[0120] Step 5: The modality feature extractor aims to prevent the modality discriminator from distinguishing the modalities of the input sample features. The two interact adversarially, prompting the modality feature extractor to learn better image and text features, as detailed below:

[0121] The modality discriminator consists of a three-layer feedforward neural network, and its output is:

[0122] D∈R n D i =D(y) i ;θ D )∈{0,1}

[0123] Where y i θ represents the image or text features of the projection. D Let D be the parameters of the modal discriminator. i A value of 1 indicates that the projected features are classified as image data features; conversely, if D is 1, the projected features are classified as image data features. i A value of 0 indicates that the projected features are classified as text data features;

[0124] Using the likelihood function to represent the feature y i and tag l i The relationship between them

[0125]

[0126] in The probability that an image or text is a feature of an image is represented by l. i Real modal labels for each image-text pair.

[0127] The difference between image features and text features represents the corresponding loss. The smaller the loss, the greater the difference between image features and text features. Therefore, the discriminant loss function is:

[0128]

[0129] Where n is the number of image-text pairs in the cross-modal dataset, θ D These are the parameters of the modality discriminator.

[0130] Step 6: The final image and text features obtained in Step 3 are also used as input to the hash learner. The hash learning network is used to non-linearly convert pairs of samples into binary codes, i.e., hash codes, as follows:

[0131] The hash learning network consists of three fully connected layers. The binary code is calculated based on the first two fully connected layers, and is learned by minimizing the following formula:

[0132]

[0133] stB∈{-1,1} r×2n

[0134] Where, B = {b1, b2, ..., b} 2n} represents the binary encoding of the data sample, n is the number of image-text pairs in the cross-modal dataset, and S h Let represent pairwise similarity in binary encoding learning, and 'a' be the length of the binary encoding.

[0135] The above formula ensures that the learned binary code should be close to the pairwise similarity S. h This allows for a better understanding of binary encoding.

[0136] The above formula is also equivalent to:

[0137]

[0138] stB∈{-1,1} r×2n

[0139] Where M = {m1, m2, ..., m} 2n} represents the value of the second fully connected layer in hash learning.

[0140] Based on an asymmetric relaxation strategy, the quantization error is further reduced by forcing the binary code B to be similar to the feature, and minimizing the feature and the pairwise similarity matrix S. h The differences between them.

[0141] Therefore, the objective function of the hash learning network is:

[0142]

[0143] stB∈{-1,1} r×2n

[0144] in, It is an independent term. These are the balancing terms. α1, α2, and α3 are hyperparameters. α1 controls the discrete constraints of the binary code, α2 controls the independence of the binary code, and α3 controls the balance of the binary code. θ h These are the parameters of the hash learning network.

[0145] Ultimately, the overall objective function can be expressed as:

[0146]

[0147] Where β, γ, δ are hyperparameters, and θ v ,θ t These are the parameters of the image feature extraction network and the text feature extraction network, θ. D θ is the parameter of the modal discriminator. h These are the parameters of the hash learning network.

[0148] Step 7: The cross-modal retrieval model maps the query image data and text data to a common Hamming space. Similarity is measured between the binary encoding of the query data and the binary encoding of the initial data in the common Hamming space to obtain the top K cross-modal retrieval results. Specifically:

[0149] Step 7.1: Retrieve images from the text to be queried. The text to be queried is converted into binary code through a cross-modal retrieval model. The distance between the obtained binary code and the binary code in the common Hamming space is calculated and sorted to obtain the top K image retrieval results.

[0150] Step 7.2: Retrieve text from the image to be queried. Convert the image to be queried into binary code using a cross-modal retrieval model. Calculate the distance between the obtained binary code and the binary code in the common Hamming space and sort them to obtain the top K text retrieval results.

[0151] In summary, this invention proposes a cross-modal retrieval method and device based on adversarial hashing using attention networks. The method includes: a feature extraction step: obtaining image-text pair data, extracting initial global features of the image data using a ResNet-101 network, extracting initial global features of the text data using a bidirectional long short-term memory neural network, and processing the initial global features of the image data and text data respectively through a shared attention module to obtain the final feature representations of the image data and text data; a modality discrimination step: interacting with the feature extractor in an adversarial manner to adapt to distribution changes and prompt the modality feature extractor to learn better image and text features; and a hash learning step: converting the final feature representations of the image data and text data into binary codes, successfully mapping their features to a common Hamming space, performing cross-modal retrieval, and obtaining the top K cross-modal retrieval results. This invention combines adversarial networks with hash learning networks, utilizing both the adversarial learning to prompt the feature extractor to learn better image and text feature representations and the fast retrieval characteristics of hashing to achieve cross-modal retrieval.

[0152] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the cross-modal retrieval method based on attention network adversarial hashing described in the present invention when executing the computer executable program.

[0153] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the cross-modal retrieval method based on attention network adversarial hashing as described in the present invention.

[0154] The computer device may be a laptop, a desktop computer, or a workstation.

[0155] The processor described in this invention may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0156] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0157] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0158] Finally, it should be noted that the above description is only for illustrating specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that any modifications or variations made based on the technical solutions and inventive concepts of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A cross-modal retrieval method based on attention network adversarial hashing, characterized in that, The method comprises the following steps: extracting initial features from image data and text data in pairs of image-text data; learning mask weights from the initial features to obtain high-quality image-text features for retrieval tasks; specifically, taking the initial global features of the image data and the initial global features of the text data as inputs of a shared attention module, processing the inputs through the attention module to obtain activation outputs of the image data and the text data; and sending the activation outputs of the image data and the text data to a fully connected layer to obtain final feature representations of the image data and the text data respectively; using a modal discriminator to determine whether the image-text features are image features or text features based on the high-quality image-text features; the modal feature extractor and the modal discriminator interact in an adversarial manner, and the modal feature extractor learns better image-text features; taking the image-text features as inputs of a hash learning network, and using the hash learning network to nonlinearly convert the better image-text features into binary codes, i.e. hash codes; mapping the query image data or text data to a common Hamming space, measuring the similarity between the binary codes of the query data and the binary codes of the initial data in the common Hamming space, sorting the similarity measurement results, and obtaining the top K cross-modal retrieval results; the hash learning network comprises three fully connected layers, and the calculation of the binary codes is based on the first two fully connected layers, and the binary codes are learned by minimizing the following formula: wherein, is a binary encoding of data samples, n is the number of image-text pairs in a cross-modal dataset, is a pairwise similarity in binary encoding learning, a is the length of binary encoding; The above equation ensures that the learned binary codes should be close to pairwise similarities , which is equivalent to: wherein, is the value of the second layer of fully connected in hash learning; Based on an asymmetric relaxation strategy, further reducing quantization error, forcing binary codes B to be similar to the features, and minimizing the difference between the features and the pairwise similarity matrix ; The objective function of the hash learning network is: wherein, is an independent term, is a balancing term, is a hyperparameter, is a hyperparameter controlling the binary encoding of discrete constraints, is a hyperparameter controlling the binary encoding independence, is a hyperparameter controlling the binary encoding balance, is a parameter of the hash learning network; Finally, the total objective function can be expressed as: wherein, is a hyper-parameter, are parameters of the image feature extraction network and the text feature extraction network, respectively, are parameters of the modality discriminator, are parameters of the hash learning network, is a discriminative loss function.

2. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, extracting initial features from image data and text data in pairs of image-text data respectively comprises: For image data, the ResNet-101 network model pre-trained on ImageNet is used to extract initial features, and a 2048-dimensional feature vector extracted from the pool5 layer is mapped to a 1024-dimensional feature, which is taken as the initial global feature of the image data ; For text data, first, word vectors are obtained through preprocessing, and then the word vectors are taken as inputs of a bidirectional long short-term memory neural network for feature extraction, and the output thereof is taken as initial global features of the text data .

3. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, initial global features of the image data and initial global features of the text data As the input of the shared attention module, a fully connected layer with sigmoid activation function is used to construct function, learning the mask weight of each position in the feature map, the mask weight of the image data and the mask weight of the text data is: The fully connected layer is used to learn the nonlinear transformation of the features and extract the semantic information of the data; sigmoid activation function is used to map the values of the mask weights for image data and text data in the range [0, 1]; and text data in the range [0, 1]; masking the image data with an initial global feature of the image data performing an inner product operation, and then performing a sum operation on the value resulting from the inner product and an initial global feature of the image data to ultimately obtain an activation output of the image ; masking weights for the text data with the initial global feature for the text data ; then performing an inner product operation on the inner product value and the initial global feature for the text data ; and finally performing a sum operation to obtain the activation output of the text ; The parameters of the inner product layer are shared for both modalities samples, the activation function output of the image and the activation output of the text are: Output the activation function of the image Activation output of text The data is input into a fully connected layer to obtain the final feature representation of the image data. The final feature representation of text data .

4. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, using a modal discriminator to determine whether the image-text features are image features or text features comprises: using a three-layer feedforward neural network to perform modal discrimination on the final representations of the image data and the text data, and the output of the modal discriminator represents the modality of the input features, wherein 1 represents the image data features and 0 represents the text data features; The modal discriminator is used to correctly distinguish the modality of the samples, i.e. whether the input sample features belong to image sample features or text sample features, and the difference between the image sample features and the text sample features is the corresponding loss, and the smaller the loss is, the greater the difference between the image sample features and the text sample features is.

5. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, The modal discriminator comprises a three-layer feedforward neural network, and the output of the modal discriminator is: wherein represents a projected image feature or text feature, is a parameter, if is 1, it means that the projected feature is identified as an image data feature, otherwise, if is 0, it means that the projected feature is identified as a text data feature; Using the likelihood function representation and the relationship between wherein denotes the probability that an image or text becomes an image feature, is the true modality label for each image-text pair; the difference between the image feature and the text feature is the corresponding loss, the smaller the loss, the greater the difference between the image feature and the text feature, and the discriminant loss function is: where n is the number of image-text pairs in the cross-modal dataset, are parameters of the modality discriminator.

6. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, taking the image-text features as inputs of a hash learning network, and using the hash learning network to nonlinearly convert the better image-text features into binary codes comprises: forming a set of all feature representations with two modalities together from the better image-text features; taking the set of all feature representations as inputs of the hash learning network, and the hash learning network comprises three fully connected layers, and the binary codes are calculated based on the first two fully connected layers; The hash learning network uses a loss function with semantic supervision to ensure the semantic relationship between the generated binary codes and the initial data.

7. The cross-modal retrieval method based on attention network adversarial hashing according to claim 1, characterized in that, Mapping the image data and the text data to a common Hamming space, performing a similarity measurement between a binary encoding of the query data and a binary encoding of the initial data in the common Hamming space, ranking the similarity measurement results, and obtaining a top K cross-modal retrieval result include: Retrieving an image from a text, converting the text to be queried into a binary code through a cross-modal retrieval model, performing distance calculation and ranking on the binary code obtained and the binary code in the common Hamming space, and finally obtaining a top K image retrieval result; Retrieving a text from an image, converting the image to be queried into a binary code through a cross-modal retrieval model, performing distance calculation and ranking on the binary code obtained and the binary code in the common Hamming space, and finally obtaining a top K text retrieval result.

8. A computer device, comprising: The method comprises a processor and a memory, the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes, and the processor can implement the cross-modal retrieval method of any one of claims 1-7 when executing the program.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the cross-modal retrieval method of any one of claims 1-7.

Images