A hierarchical supervised cross-modal image-text retrieval method

By constructing a feature extraction network, a modal adversarial network, and a hash code generation network, the problems of large memory consumption, insufficient mining of semantic information, and semantic category interference in existing cross-modal retrieval methods are solved, and efficient and accurate cross-modal data retrieval is achieved.

CN114168784BActive Publication Date: 2024-11-01GUILIN UNIV OF ELECTRONIC TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111511008.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-11-01
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

Existing cross-modal retrieval methods suffer from problems such as large memory consumption of fixed-length hash encoding, inability to fully mine the rich semantic information of tags, lack of constraints during the feature learning stage, and failure to consider the interference of different semantic category data on retrieval.

Method used

A hierarchical supervised cross-modal image and text retrieval method is adopted. By constructing a feature extraction network, a modal adversarial network, and a hash code generation network, and utilizing adversarial learning and label layer similarity loss function, the optimal hash code is generated to achieve efficient retrieval of cross-modal data.

Benefits of technology

It improves the accuracy and efficiency of cross-modal retrieval, reduces data storage costs, avoids interference from data of different semantic categories, and improves retrieval speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114168784B_ABST
    Figure CN114168784B_ABST
Patent Text Reader

Abstract

The present invention discloses a hierarchical supervised cross-modal image-text retrieval method, wherein the method comprises the following steps: S1: constructing a feature extraction network for extracting image features and text features; S2: extracting image and text features using the feature extraction network, and obtaining preliminary high-dimensional feature values ​​of the image and text respectively; S3: constructing a modal adversarial network, inputting the preliminary high-dimensional feature values ​​of the image and text into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest in the public space; S4: constructing a hash code generation network, and using the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the preliminary high-dimensional feature values ​​of the image and text through the last fully connected layer are used to generate an optimal hash code, thereby realizing cross-modal data retrieval. The present invention can realize the retrieval of cross-modal data with hierarchical supervision and improve the efficiency of cross-modal retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal image-text retrieval, and more specifically, to a hierarchical supervised cross-modal image-text retrieval method. Background Art

[0002] With the rapid development of the Internet and the Internet of Things, a huge amount of valuable multimodal data has been generated. How to quickly and efficiently find relevant multimodal information in massive data is extremely important, which makes cross-modal retrieval have application scenarios and research significance.

[0003] Most existing cross-modal retrieval methods target non-hierarchical supervision information and cannot fully mine semantic information with rich labels. However, in many real-world application scenarios, the label supervision information of cross-modal data often has a certain hierarchical structure and contains rich semantic information. Therefore, constructing a cross-modal retrieval method that fully mines hierarchical supervision information is extremely important in the field of information retrieval.

[0004] Prior art An image-text cross-modal retrieval method based on a multi-layer semantic deep hashing algorithm defines the similarity between data through the co-occurrence relationship between multi-label data, and uses this as the supervisory information for network training. A loss function that comprehensively considers multi-layer semantic similarity and binary similarity is designed to train the network, so that the feature extraction and hash code learning processes are unified in one framework to achieve end-to-end learning. This algorithm makes full use of the semantic correlation information between data and improves the retrieval accuracy, such as Figure 1 As shown in the figure, the overall framework consists of three modules: deep feature extraction module, similarity matrix generation module, and hash code learning module. Two deep neural networks are used to extract image and text features respectively, and the feature learning and hash code learning processes are unified in one framework. The multi-level semantic supervision information based on label co-occurrence is introduced to guide the entire training process, so that the binary code obtained not only retains the basic similarity / dissimilarity relationship of the original sample space, but also can distinguish the similarity between samples, retain the high-level semantics between samples to a greater extent, and improve the retrieval accuracy. In terms of structure, the network is trained by imposing the constraint that "similar images and texts in the semantic space have similar hash codes in the Hamming space", and the hash code is directly used as the output of the network to achieve end-to-end learning, thereby ensuring that the learned features are suitable for specific retrieval tasks.

[0005] However, the above prior art still has the following shortcomings:

[0006] 1. Fixed-length hash codes are used to represent cross-modal data, which takes up a lot of memory.

[0007] 2. Cross-modal retrieval designed for non-hierarchical supervised information cannot fully mine semantic information with rich labels.

[0008] 3. Constraints are only imposed in the hash code learning stage, but not in the feature learning stage, which fails to fully minimize the distance between multiple modal data containing the same semantic information in the common space.

[0009] 4. The interference of data of different semantic categories on cross-modal retrieval is not considered. Summary of the invention

[0010] In order to solve the above problems existing in the prior art, the present invention provides a hierarchical supervised cross-modal image-text retrieval method, which can realize the retrieval of cross-modal data with hierarchical supervision and improve the efficiency of cross-modal retrieval.

[0011] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0012] A hierarchical supervised cross-modal image-text retrieval method, the method comprising the following steps:

[0013] S1: Construct a feature extraction network for extracting image features and text features;

[0014] S2: Use the feature extraction network to extract image and text features, and obtain preliminary high-dimensional feature values ​​of the image and text respectively;

[0015] S3: In the feature extraction stage, a modal adversarial network is constructed, and the preliminary high-dimensional feature values ​​of the image and text are input into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest to each other in the common space;

[0016] S4: Construct a hash code generation network, and use the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the preliminary high-dimensional feature values ​​of the image and text of the last fully connected layer can generate the optimal hash code to achieve cross-modal data retrieval.

[0017] Preferably, the feature extraction network includes a first deep neural network for extracting image features and a text extraction network for extracting text features;

[0018] Modify the last layer of the first deep neural network into a hash code output layer, whose number of neurons is the hash code length, and use an activation function to map the output to between -1 and 1;

[0019] The text extraction network includes:

[0020] Bag-of-words model for vectorizing all text in the dataset,

[0021] A multi-scale feature stacking model constructed by five parallel mean pooling layers is used to extract high-order and low-order features under different receptive fields, and its window sizes are 1×1, 2×2, 3×3, 5×5, and 10×10, respectively.

[0022] The second deep neural network consists of three fully connected layers.

[0023] Furthermore, the second deep neural network is as follows: the number of network nodes in the first fully connected layer is the output dimension of the multi-scale feature stacking model; the second layer is a fully connected layer with a network node number of 4096 dimensions, and the second fully connected layer outputs preliminary high-dimensional feature values; the third layer is a fully connected layer with a node number of hash code length h t , the activation function is the fully connected layer of Sigmoid, that is, the third fully connected layer is used as the hash code output layer.

[0024] Furthermore, the modal adversarial network uses E networks with three fully connected layers, the number of nodes in the first hidden layer is the same as the feature dimension of the input; the number of nodes in the second hidden layer is the same as the total number of labels in the e-th label layer Φ e The third layer has 2 nodes, and its activation function is the Sigmoid function. The output is a binary code, where 0 represents image mode and 1 represents text mode.

[0025] Among them, E represents the number of label layers of image-text pairs, and the index of the label layer is {1, 2, ..., E} from top to bottom.

[0026] Furthermore, the adversarial loss of the modal adversarial network uses a cross entropy loss function, which is defined as follows:

[0027]

[0028]

[0029] Among them, L Adv Represents the overall objective function of the modal adversarial network, L Adv_e represents the adversarial loss corresponding to the e-th layer label, v i represents the true label supervision information of each data, G(*;ε) is the modal probability distribution generated by the data set β(i) in the modal adversarial network, and ε is the parameter of the modal adversarial network.

[0030] Furthermore, the hash code generation network includes

[0031] The submodules for associating different label layers fully exploit the intrinsic similarity of labels in each layer and the correlation between label layers by introducing similarity loss within the label layer and correlation loss between label layers.

[0032] Different semantic category distinction submodule, by designing a total objective function for different semantic category distinction, the modal data with different semantics can keep a certain distance from each other in the public space, thus avoiding the interference of different semantic data on retrieval;

[0033] The cross-modal hash loss submodule converts the preliminary high-dimensional feature values ​​into hash codes with optimal coding length to compress the data dimension and reduce the data storage space, thus achieving accurate and fast retrieval of cross-modal data.

[0034] Furthermore, the different label layer association submodule includes two branches, namely, similarity loss within the label layer and correlation loss between label layers;

[0035] In the label hierarchy, in order to maintain the similarity of each layer of labels, a label hash code C is generated for each label in each layer. e ∈{-1,1} h×Φe , e∈{1,2,...,E}, define the similarity loss function within the label layer:

[0036]

[0037] Among them, C e is the category hash code of the e-th layer; Represents the label vector of each set of image-text pair dataset β(i), where Indicates that the i-th image-text pair data is labeled by the j-th label of the e-th layer, otherwise, f(x i ,W f ) represents the output of the original image after the second fully connected layer in the feature extraction network, where W f represents the network parameters of the network for extracting image features; g(t i ,W g ) represents the output of the text after the second fully connected layer in the feature extraction network, where W g Represents the network parameters of the network for extracting text features; e is the confidence of the e-th layer label, and the sum of the confidences of all label layers is 1;

[0038] In order to fully obtain the cross-layer correlation, the cross-layer label similarity matrix is ​​defined, namely:

[0039]

[0040] Among them, e∈{1,2,...,E-1}, e i represents the i-th label of the e-th layer, and E represents the last layer of labels; Indicates that the jth label of the Eth layer is the descendant node of the ith label of the eth layer, otherwise it is 0;

[0041] Then the label layer correlation loss function is expressed as:

[0042]

[0043] Among them, C E is the hash matrix of the E-th layer label, η e represents a hyperparameter, and

[0044] Furthermore, in different semantic category distinction submodules, an overall objective function for different semantic category distinction is designed as follows:

[0045] In the common space, the distances between different semantic categories and different modal data should be as far as possible, so the first objective function is defined as:

[0046]

[0047] Where dis(W,V) represents two points W = (w 1 ,w 2 ,...,w n ) and V=(v 1 ,v 2 ,...,v n ), F(x i ) and F(x j ) represents the image features, G(t i ) and G(t j ) represents text features;

[0048] In the common subspace, data of the same modality should be separated according to different semantic categories, so the second objective function is defined as:

[0049]

[0050] The total loss function of different semantic category distinction modules is expressed as:

[0051]

[0052] Where N represents the number of image-text pairs in the dataset.

[0053] Furthermore, the output result of the second fully connected layer in the feature extraction network is introduced into the cross-modal hashing module, and the cross-modal hashing loss function is defined as follows:

[0054]

[0055] in, These are the hash codes learned from image and text modality data respectively; Represents the Fibonacci norm.

[0056] Furthermore, the feature extraction network, modal adversarial network, and hash code generation network are used to form a cross-modal image-text retrieval model, and the final loss function of the cross-modal image-text retrieval model is expressed as:

[0057] Loss = αL Adv +βL intra_layer +xL inter_layer +δL dis +φL hash (10)

[0058] Among them, α, β, χ, δ, and φ are hyperparameters; L Adv , L intra_layer , L inter_layer , L dis , L hash They represent the cross entropy loss function, the similarity loss function within the label layer, the label cross-layer correlation loss function, the total loss function of different semantic category distinction modules, and the cross-modal hash loss function respectively;

[0059] The cross-modal image-text retrieval model is trained and optimized according to the final loss function, so that the cross-modal image-text retrieval model reaches the optimal value.

[0060] The beneficial effects of the present invention are as follows:

[0061] In view of the problem that the current cross-modal retrieval algorithm does not fully minimize the distance between multiple modal data containing the same semantic information in the common space, this invention borrows the adversarial idea and solves it by imposing constraints in the deep feature extraction stage. The feature extraction network is used as a generator and the modal adversarial network is used as an adversary. After the two are adversarially learned, until the adversary is difficult to distinguish the modal type of the modal features extracted by the feature extraction network, this makes the distance between different modalities with the same semantics in the common space the shortest.

[0062] To address the problem that hierarchical supervision information has not been fully mined, the present invention simultaneously introduces similarity loss within the label layer and correlation loss between label layers to fully mine the intrinsic similarity of each layer of labels and the correlation between label layers, thereby improving the accuracy of cross-modal retrieval.

[0063] To address the problem of dissimilar data interfering with cross-modal retrieval, the present invention redesigns a total loss function for the distance between data of different semantic categories to solve the problem. This approach allows modal data of different semantic categories to maintain a certain distance from each other in the common space.

[0064] In order to solve the problems of high storage cost and slow retrieval speed of current cross-modal retrieval algorithms, the present invention utilizes a cross-modal hash loss submodule to represent multimodal data features with low-dimensional binary hash codes with optimal coding length, so that the Hamming distance of hash codes of similar data in the original space is the shortest; conversely, the hash codes of dissimilar data in the original space are also dissimilar, and the Hamming distance is as far as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a framework diagram of the existing image-text cross-modal retrieval method based on a multi-layer semantic deep hashing algorithm.

[0066] Figure 2 It is a flowchart of the hierarchical supervised cross-modal image-text retrieval method.

[0067] Figure 3 This is the overall structural diagram of the hierarchical supervised cross-modal image and text retrieval method. DETAILED DESCRIPTION

[0068] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Example 1

[0070] like Figure 1 , Figure 2 As shown, a hierarchical supervised cross-modal image-text retrieval method is provided, wherein the method comprises the following steps:

[0071] S1: Construct a feature extraction network for extracting image features and text features;

[0072] S2: Use the feature extraction network to extract image and text features, and obtain preliminary high-dimensional feature values ​​of the image and text respectively;

[0073] S3: In the feature extraction stage, the feature extraction network is used as a generator and the modal adversarial network is used as an adversary. The preliminary high-dimensional feature values ​​of the image and text generated by the feature extraction network are input into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest to each other in the common space.

[0074] S4: Construct a hash code generation network, and use the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the preliminary high-dimensional feature values ​​of the image and text of the last fully connected layer can generate the optimal hash code to achieve accurate and fast retrieval of cross-modal data.

[0075] In a specific embodiment, the feature extraction network includes a first deep neural network for extracting image features and a text extraction network for extracting text features;

[0076] Modify the last layer of the first deep neural network into a hash code output layer, whose number of neurons is the hash code length,

[0077] Use an activation function to map the output to between -1 and 1;

[0078] Each image in the data set is resized to 224×224 and input into the pre-trained first deep neural network to extract preliminary high-dimensional feature values ​​of the image; the first deep neural network described in this embodiment is a deep neural network VGG-16, and the last three layers of the deep neural network VGG-16 are fully connected layers. The last layer of the first deep neural network was originally used for classification tasks, and this embodiment uses the last layer to generate hash codes. Specifically, the number of network nodes in the first and second fully connected layers of the deep neural network VGG-16 is 4096 dimensions, and the number of neurons in the third fully connected layer is the hash code length value h i , using the Tanh activation function, the output is mapped to the range between -1 and 1.

[0079] The text extraction network includes:

[0080] Bag-of-words model for vectorizing all text in the dataset,

[0081] A multi-scale feature stacking model constructed by five parallel mean pooling layers.

[0082] The second deep neural network consists of three fully connected layers.

[0083] In text feature extraction, first, each text is represented by a bag-of-words model, and the bag-of-words model of the text modality is used as the input of the text extraction network.

[0084] Then, a multi-scale feature stacking model constructed by five parallel mean pooling layers is used to extract high-order and low-order features under different receptive fields, and the window sizes are 1×1, 2×2, 3×3, 5×5, and 10×10 respectively;

[0085] Finally, the second deep neural network consisting of three fully connected layers extracts features and outputs the hash code of the text features. The configuration of the second deep neural network is shown in Table 1.

[0086] Table 1 Detailed configuration of the fully connected layer feature extraction network for text modality

[0087]

[0088] The details are as follows: the number of network nodes in the first fully connected layer fc1 is the output dimension of the multi-scale feature stacking model; the second fully connected layer is a fully connected layer with a network node number of 4096 dimensions, and the second fully connected layer fc2 outputs preliminary high-dimensional feature values; the number of nodes in the third fully connected layer fc3 is the hash code length h t , the activation function is the fully connected layer of Sigmoid, that is, the third fully connected layer is used as the hash code output layer.

[0089] In this embodiment, the dataset composed of image-text pairs β(i)={(x i , t i )|i∈1,2,...,N} is input into the feature extraction network, where x i Represents the original feature vector of the i-th image data, t i Represents the i-th text data.

[0090] In this embodiment, the preliminary high-dimensional feature values ​​of the image output by the second fully connected layer of the first deep neural network and the preliminary high-dimensional feature values ​​of the text output by the second connected layer in the text extraction network are input into the modal adversarial network for adversarial learning, and the features of the image and text are updated so that the features of the image and text pairs are brought closest in the common space; at the same time, the second connected layers of the image and text are also input into the third fully connected layer of the first deep neural network and the third connected layer in the text extraction network, respectively, to generate hash code features under the constraints of the hash code generation module.

[0091] In a specific embodiment, the modal adversarial network is based on the adversarial idea and learns the common space of the image modality and the text modality. The task of the feature extraction network is to learn the representation of the image and text modalities and map the image and text modalities into a common subspace. Its purpose is to confuse the discrimination of the modal adversarial module as an opponent of the modal adversarial module, thereby improving the discrimination ability of the modal adversarial module. The task of the modal adversarial module is to discriminate the modal type of data in the feature extraction network to improve the representation ability of the feature extraction network and further minimize the distance between different modal data with the same semantic information in the common space.

[0092] The modal adversarial network uses E networks with three fully connected layers, the number of nodes in the first hidden layer is the same as the feature dimension of the input; the number of nodes in the second hidden layer is the same as the total number of labels in the e-th label layer Φ e The number of nodes in the third layer is 2, its activation function is the Sigmoid function, and the output is a binary code, 0 represents image mode, and 1 represents text mode;

[0093] Among them, E represents the number of label layers of image-text pairs, and the index of the label layer is {1, 2, ..., E} from top to bottom, Φ eIndicates the total number of labels in the e-th label layer.

[0094] The network configuration of the modal adversarial network is shown in Table 2.

[0095] Table 2 Detailed network configuration of the modality adversarial module

[0096]

[0097] In a specific embodiment, the adversarial loss of the modal adversarial network uses a cross entropy loss function, which is defined as follows:

[0098]

[0099]

[0100] Among them, L Adv Represents the overall objective function of the modal adversarial network, L Adv_e represents the adversarial loss corresponding to the e-th layer label, v i represents the true label supervision information of each data, G(*;ε) is the modal probability distribution generated by the data set β(i) in the modal adversarial network, and ε is the parameter of the modal adversarial network.

[0101] In a specific embodiment, the hash code generation network includes

[0102] The submodules for associating different label layers fully exploit the intrinsic similarity of labels in each layer and the correlation between label layers by introducing similarity loss within the label layer and correlation loss between label layers.

[0103] The sub-module for distinguishing different semantic categories designs an overall objective function for distinguishing different semantic categories, so that the modal data with different semantics keep a certain distance from each other in the public space, thereby avoiding the interference of different semantic data on retrieval.

[0104] The cross-modal hash loss submodule converts the preliminary high-dimensional feature values ​​into hash codes with optimal coding length to compress the data dimension and reduce the data storage space, thus achieving accurate and fast retrieval of cross-modal data.

[0105] In a specific embodiment, the different label layer association submodule includes two branches, namely, similarity loss within the label layer and correlation loss between label layers.

[0106] In the label hierarchy, in order to maintain the similarity of each layer of labels, a label hash code is generated for each label in each layer. e∈{1,2,...,E}, define the similarity loss function within the label layer:

[0107]

[0108] Among them, C e is the category hash code of the e-th layer; Represents the label vector of each set of image-text pair dataset β(i), where Indicates that the i-th image-text pair data is labeled by the j-th label of the e-th layer, otherwise, f(x i ,W f ) represents the output of the original image after the second fully connected layer in the feature extraction network, where W f represents the network parameters of the network for extracting image features; g(t i ,W g ) represents the output of the text after the second fully connected layer in the feature extraction network, where W g Represents the network parameters of the network for extracting text features; e is the confidence of the e-th layer label, and the sum of the confidences of all label layers is 1.

[0109] In order to fully obtain the cross-layer correlation, the cross-layer label similarity matrix is ​​defined, namely:

[0110]

[0111] Among them, e∈{1,2,...,E-1}, e i represents the i-th label of the e-th layer, and E represents the last layer of labels; Indicates that the j-th label of the E-th layer is the descendant node of the ith label of the e-th layer, otherwise it is 0.

[0112] Then the label layer correlation loss function is expressed as:

[0113]

[0114] Among them, C E is the hash matrix of the E-th layer label, η e represents a hyperparameter, and

[0115] In a specific embodiment, in different semantic category distinguishing submodules, a total objective function for different semantic category distinguishing is designed as follows:

[0116] In the common space, the distances between different semantic categories and different modal data should be as far as possible, so the first objective function is defined as:

[0117]

[0118] Where dis(W,V) represents two points W = (w 1 ,w 2 ,...,w n ) and V=(v 1 ,v 2 ,...,v n ), F(x i ) and F(x j ) represents the image features, G(t i ) and G(t j ) represents text features;

[0119] In the common subspace, data of the same modality according to different semantic categories should be separated in the common subspace, so the second objective function is defined as:

[0120]

[0121] The total loss function of different semantic category distinction modules is expressed as:

[0122]

[0123] Where N represents the number of image-text pairs in the dataset.

[0124] In a specific embodiment, the output result of the second fully connected layer in the feature extraction network is introduced into the cross-modal hash module, and the cross-modal hash loss function is defined as follows:

[0125]

[0126] in, These are the hash codes learned from image and text modality data respectively; Represents the Fibonacci norm.

[0127] This embodiment uses the cross-modal hash loss submodule to propose an optimal coding length hash learning algorithm, which represents the multimodal data features with a low-dimensional binary hash code of the optimal coding length, so that the Hamming distance of the hash codes of similar data in the original space is the shortest; conversely, the hash codes of dissimilar data in the original space are also dissimilar, and the Hamming distance is as far as possible.

[0128] In a specific embodiment, a cross-modal image-text retrieval model is constructed using a feature extraction network, a modal adversarial network, and a hash code generation network. The final loss function of the cross-modal image-text retrieval model is expressed as:

[0129] Loss = αL Adv +βL intra_layer +xL inter_layer +δL dis +φLhash (10)

[0130] Among them, α, β, χ, δ, and φ are hyperparameters; L Adv , L intra_layer , L inter_layer , L dis , L hash They respectively represent the cross entropy loss function, the similarity loss function within the label layer, the label cross-layer correlation loss function, the total loss function of different semantic category distinction modules, and the cross-modal hash loss function.

[0131] The cross-modal image-text retrieval model is trained and optimized according to the final loss function, so that the cross-modal image-text retrieval model reaches the optimal value.

[0132] In this embodiment, there are four parameters that need to be trained and optimized, namely, the network parameter W for extracting image features f , network parameters W for extracting text features g and hash code D x , D t In the optimization of the cross-modal image-text retrieval model, the forward propagation network is used to calculate f(x i ,W f ) and g(t i ,W g ), the Adam gradient descent algorithm is used to update the parameter W in the back propagation f and W g After passing through the hash code generation module, the hash code D is updated according to formula (9): x , D t . The hyperparameters α, β, χ, δ, φ, h are given by the grid search method. i and h t Assignment makes the cross-modal image and text retrieval model optimal.

[0133] The optimization process of this embodiment is shown in Algorithm 1, where sgn(·) is a binary judgment function expressed as:

[0134]

[0135]

[0136] Example 2

[0137] A computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following method steps are implemented:

[0138] S1: Construct a feature extraction network for extracting image features and text features;

[0139] S2: Use the feature extraction network to extract image and text features, and obtain preliminary high-dimensional feature values ​​of the image and text respectively;

[0140] S3: In the feature extraction stage, a modal adversarial network is constructed, and the preliminary high-dimensional feature values ​​of the image and text are input into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest to each other in the common space;

[0141] S4: Construct a hash code generation network and use the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the preliminary high-dimensional feature values ​​of the image and text through the last fully connected layer can generate the optimal hash code to achieve cross-modal data retrieval.

[0142] Example 3

[0143] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method steps implemented are as follows:

[0144] S1: Construct a feature extraction network for extracting image features and text features;

[0145] S2: Use the feature extraction network to extract image and text features, and obtain preliminary high-dimensional feature values ​​of the image and text respectively;

[0146] S3: In the feature extraction stage, a modal adversarial network is constructed, and the preliminary high-dimensional feature values ​​of the image and text are input into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest to each other in the common space;

[0147] S4: Construct a hash code generation network and use the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the preliminary high-dimensional feature values ​​of the image and text through the last fully connected layer can generate the optimal hash code to achieve cross-modal data retrieval.

[0148] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk), etc.

[0150] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical supervised cross-modal image-text retrieval method, Features: The method comprises the following steps: S1: Construct a feature extraction network for extracting image features and text features; S2: Use the feature extraction network to extract image and text features, and obtain preliminary high-dimensional feature values ​​of the image and text respectively; S3: Construct a modal adversarial network, input the preliminary high-dimensional feature values ​​of the image and text into the modal adversarial network for adversarial learning, so that different modalities with the same semantics are closest to each other in the common space; the modal adversarial network uses E networks with three fully connected layers, the number of nodes in the first hidden layer is the same as the feature dimension of the input; the number of nodes in the second hidden layer is the same as the total number of labels in the e-th label layer Φ e The third layer has 2 nodes, and its activation function is the Sigmoid function. The output is a binary code, 0 for image mode and 1 for text mode. E represents the number of label layers of the image-text pair, and the index of the label layer is {1, 2, ..., E} from top to bottom. S4: constructing a hash code generation network, and using the hash code generation network to constrain the last fully connected layer of the feature extraction network, so that the initial high-dimensional feature values ​​of the image and text of the last fully connected layer are used to generate an optimal hash code to achieve cross-modal data retrieval; the hash code generation network includes: The submodule for associating different label layers fully exploits the intrinsic similarity of each label layer and the correlation between label layers by introducing similarity loss within the label layer and correlation loss between label layers. The submodule for associating different label layers includes two branches, namely similarity loss within the label layer and correlation loss between label layers. In the label hierarchy, in order to maintain the similarity of each layer of labels, a label hash code is generated for each label in each layer. Define the similarity loss function within the label layer: Among them, C e is the category hash code of the e-th layer; Represents the label vector of each set of image-text pair dataset β(i), where Indicates that the i-th image-text pair data is labeled by the j-th label of the e-th layer, otherwise, f(x i ,W f ) represents the output of the original image after the second fully connected layer in the feature extraction network, where W f represents the network parameters of the network for extracting image features; g(t i ,W g ) represents the output of the text after the second fully connected layer in the feature extraction network, where Wg represents the network parameters of the network for extracting text features; e is the confidence of the e-th layer label, and the sum of the confidences of all label layers is 1; In order to fully obtain the cross-layer correlation, the cross-layer label similarity matrix is ​​defined, namely: Among them, e∈{1,2,...,E-1}, e i represents the i-th label of the e-th layer, and E represents the last layer of labels; Indicates that the jth label of the Eth layer is the descendant node of the ith label of the eth layer, otherwise it is 0; Then the label layer correlation loss function is expressed as: Among them, C E is the hash matrix of the E-th layer label, η e represents a hyperparameter, and Different semantic category distinguishing submodules, by designing a total objective function for different semantic category distinguishing, make the modal data with different semantics keep a certain distance from each other in the public space, so as to avoid the interference of different semantic data on retrieval; in different semantic category distinguishing submodules, designing a total objective function for different semantic category distinguishing is as follows: In the common space, the distances between different semantic categories and different modal data should be as far as possible, so the first objective function is defined as: Where dis(W,V) represents two points W = (w 1 ,w 2 ,...,w n ) and V=(v 1 ,v 2 ,...,v n ), F(x i ) and F(x j ) represents the image features, G(t i ) and G(t j ) represents text features; In the common subspace, data of the same modality should be separated according to different semantic categories, so the second objective function is defined as: The total loss function of different semantic category distinction modules is expressed as: Where N represents the number of image-text pairs in the dataset; The cross-modal hash loss submodule converts the preliminary high-dimensional feature values ​​into the optimal coding length hash code to compress the data dimension and reduce the data storage space, so as to achieve accurate and fast retrieval of cross-modal data. The output result of the second fully connected layer in the feature extraction network is introduced into the cross-modal hash module, and the cross-modal hash loss function is defined as follows: in, These are the hash codes learned from image and text modality data respectively; Represents the Fibonacci norm.

2. The hierarchical supervised cross-modal image-text retrieval method according to claim 1, Features: The feature extraction network includes a first deep neural network for extracting image features and a text extraction network for extracting text features; Modify the last layer of the first deep neural network into a hash code output layer, whose number of neurons is the hash code length, and use an activation function to map the output to between -1 and 1; The text extraction network includes Bag-of-words model for vectorizing all text in the dataset, A multi-scale feature stacking model constructed by five parallel mean pooling layers is used to extract high-order and low-order features under different receptive fields, and its window sizes are 1×1, 2×2, 3×3, 5×5, and 10×10, respectively. The second deep neural network consists of three fully connected layers.

3. The hierarchical supervised cross-modal image-text retrieval method according to claim 2, Features: The second deep neural network is as follows: the number of network nodes in the first fully connected layer is the output dimension of the multi-scale feature stacking model; the second layer is a fully connected layer with a network node number of 4096 dimensions, and the second fully connected layer outputs preliminary high-dimensional feature values; the third layer is a fully connected layer with a node number of hash code length h t , the activation function is the fully connected layer of Sigmoid, that is, the third fully connected layer is used as the hash code output layer.

4. The hierarchical supervised cross-modal image-text retrieval method according to claim 1, Features: The adversarial loss of the modal adversarial network uses the cross entropy loss function, which is defined as follows: Among them, L Adv Represents the overall objective function of the modal adversarial network, L Adv_e represents the adversarial loss corresponding to the e-th layer label, v i represents the true label supervision information of each data, G(*;ε) is the modal probability distribution generated by the data set β(i) in the modal adversarial network, and ε is the parameter of the modal adversarial network.

5. The hierarchical supervised cross-modal image-text retrieval method according to claim 4, Features: The cross-modal image-text retrieval model is constructed by using feature extraction network, modal adversarial network and hash code generation network. The final loss function of the cross-modal image-text retrieval model is expressed as: Loss=αL Adv +βL intra_layer +xL inter_layer +δL dis +φL hash (10) Among them, α, β, χ, δ, and φ are hyperparameters; L Adv , L intra_layer , L inter_layer , L dis , L hash They represent the cross entropy loss function, the similarity loss function within the label layer, the label cross-layer correlation loss function, the total loss function of different semantic category distinction modules, and the cross-modal hash loss function respectively; The cross-modal image-text retrieval model is trained and optimized according to the final loss function, so that the cross-modal image-text retrieval model reaches the optimal value.