Cross-modal retrieval method based on efficient hierarchical information aggregation hash and related equipment

Abstract

The invention provides a cross-modal retrieval method based on efficient hierarchical information aggregation hash and related equipment, and the method comprises the steps: obtaining a multi-modal data set which comprises image modal data and text modal data; inputting the image modal data and the text modal data into a hierarchical information aggregation network, and obtaining an associated hash code between the image modal data and the text modal data through the hierarchical information aggregation network; and inputting the image modal data, the text modal data and the associated hash code into a full-connection network, obtaining a target hash code by using the full-connection network, and performing cross-modal retrieval by using the target hash code. According to the method, rich semantic information in the multi-modal data is considered, complementarity between different samples and different modals can be considered at the same time, and the problem that fine-grained semantic information and semantic association of the multi-modal data are missing is effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a cross-modal retrieval method and related equipment based on efficient hierarchical information aggregation hashing. Background Art

[0002] With the rapid development of Internet technology, massive multimedia data requires more accurate and efficient multimodal retrieval systems to support large-scale retrieval scenarios. In recent years, thanks to the extremely low storage cost and high retrieval speed of hash technology, hash-based retrieval methods have received more and more attention. Compared with single-modal retrieval strategies, multimodal retrieval technology can capture semantic associations in multimodal data and fuse multimodal information to learn more semantically rich binary hash codes.

[0003] Although existing research has achieved many remarkable achievements, there are still some problems to be solved: how to construct a multimodal alignment space based on a deep neural network structure to effectively model multimodal deep semantic associations and reduce heterogeneous multimodal differences; how to save rich multimodal semantic associations into binary hash codes under the deep learning framework; based on the existing feature-level multimodal fusion strategy, how to further explore a more fine-grained fusion process; it is difficult to model fine-grained semantic associations between samples using a coarse-grained label similarity learning strategy. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to propose a cross-modal retrieval method and related equipment based on efficient hierarchical information aggregation hashing to solve or partially solve the above problems.

[0005] Based on the above purpose, the first aspect of the present disclosure provides a cross-modal retrieval method based on efficient hierarchical information aggregation hashing, the method comprising:

[0006] Acquire a multimodal data set, wherein the multimodal data set includes image modality data and text modality data;

[0007] Inputting the image modality data and the text modality data into a hierarchical information aggregation network, and obtaining an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network;

[0008] The image modality data, the text modality data and the associated hash code are input into a fully connected network, a target hash code is obtained by using the fully connected network, and a cross-modality retrieval is performed by using the target hash code.

[0009] Specifically, the hierarchical information aggregation network includes a feature encoding network, an intra-modal information aggregation network, and a cross-modal information aggregation network.

[0010] The step of inputting the image modality data and the text modality data into a hierarchical information aggregation network and obtaining an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network comprises:

[0011] Inputting the image modality data and the text modality data into a feature encoding network to obtain image features and text features;

[0012] Inputting the image features and the text features into an intra-modal information aggregation network to obtain intra-modal aggregation features;

[0013] Inputting the image features and the text features into a cross-modal information aggregation network to obtain inter-modal aggregation features;

[0014] The intra-modality aggregate features and the inter-modality aggregate features are concatenated to obtain complementary features;

[0015] The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data.

[0016] Specifically, the inputting the image features and the text features into an intra-modal information aggregation network to obtain intra-modal aggregation features includes:

[0017] Obtaining a first activation function corresponding to the intra-modal information aggregation network;

[0018] The image feature and the text feature are input into an intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula:

[0019]

[0020] Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g+1th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is the text feature, x is the image, and y is the text.

[0021] Specifically, the inputting the image features and the text features into a cross-modal information aggregation network to obtain inter-modal aggregation features includes:

[0022] The similarity between the image modality data and the text modality data is calculated, and a hierarchical similarity matrix is ​​determined based on the similarity. The similarity matrix is ​​expressed by the formula:

[0023]

[0024] Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, t represents the modality of sample i, T is the period, l i is the annotation information of the i-th image modality data, is the annotation information of the j-th text modal data in the T-th period;

[0025] Concatenate the image feature and the text feature row by row to obtain a first feature;

[0026] Obtain a second activation function corresponding to the inter-modal information aggregation network, input the first feature into the cross-modal information aggregation network, and determine the inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula:

[0027]

[0028] in, represents the input of the g-th layer image modality in the cross-modal information aggregation network, represents the output of the g-th layer image modality in the cross-modal information aggregation network, represents the input of the g-th layer text modality in the cross-modal information aggregation network, represents the output of the g-th layer text modality in the cross-modal information aggregation network, As the first feature, is the second activation function.

[0029] Specifically, the intra-modality aggregate features and the inter-modality aggregate features are concatenated to obtain complementary features, including:

[0030] The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula:

[0031]

[0032]

[0033] T S=[C x ,C y ]

[0034] Among them, H # , # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projected into a low-dimensional latent space, C x As part of the complementary feature, C y The other part of the complementary feature, T S are complementary features.

[0035] Specifically, the quantizing process of the complementary features to obtain the associated hash code between the image modality data and the text modality data includes:

[0036] The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula:

[0037] B tea =sign(T S )

[0038] Among them, B tea is the associated hash code and sign() is the sign function.

[0039] Specifically, the fully connected network includes a hash module and a bit-by-bit fusion module.

[0040] The step of inputting the image modality data, the text modality data, and the associated hash code into a fully connected network and obtaining a target hash code using the fully connected network includes:

[0041] Inputting the image modality data and the text modality data into a hash module to obtain image hash features and text hash features;

[0042] Input the image hash feature and the text hash feature into the bit-by-bit fusion module to obtain the joint feature;

[0043] According to the joint feature and the associated hash code, a fully connected network is inputted, and cosine similarity is reconstructed using the fully connected network. When the preset conditions are met, a target hash code is obtained, wherein the target hash code is expressed by the formula:

[0044]

[0045] in, is the target hash code, is a joint feature.

[0046] Specifically, the image hash feature and the text hash feature are input into the bit-by-bit fusion module to obtain the joint feature, including:

[0047] The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula:

[0048]

[0049] Among them, W f is the preset fusion weight vector, is the image hash feature, is the text hash feature.

[0050] Based on the same inventive concept, the second aspect of the present disclosure proposes a cross-modal retrieval device based on efficient hierarchical information aggregation hashing, comprising:

[0051] A data acquisition module is configured to acquire a multimodal data set, wherein the multimodal data set includes image modality data and text modality data;

[0052] an associated hash code determination module, configured to input the image modality data and the text modality data into a hierarchical information aggregation network, and obtain an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network;

[0053] The retrieval module is configured to input the image modality data, the text modality data and the associated hash code into a fully connected network, obtain a target hash code using the fully connected network, and perform cross-modal retrieval using the target hash code.

[0054] Specifically, the hierarchical information aggregation network includes a feature encoding network, an intra-modal information aggregation network and a cross-modal information aggregation network.

[0055] The associated hash code determination module specifically includes:

[0056] A feature determination unit is configured to input the image modality data and the text modality data into a feature encoding network to obtain image features and text features;

[0057] An intra-modality aggregate feature determination unit is configured to input the image feature and the text feature into an intra-modality information aggregation network to obtain an intra-modality aggregate feature;

[0058] An inter-modal aggregate feature determination unit is configured to input the image feature and the text feature into a cross-modal information aggregation network to obtain an inter-modal aggregate feature;

[0059] A complementary feature determination unit is configured to perform splicing processing on the intra-modality aggregate feature and the inter-modality aggregate feature to obtain a complementary feature;

[0060] The quantization processing unit is configured to perform quantization processing on the complementary features to obtain an associated hash code between the image modality data and the text modality data.

[0061] Specifically, the intra-modality aggregate feature determination unit is specifically configured as follows:

[0062] Obtaining a first activation function corresponding to the intra-modal information aggregation network;

[0063] The image feature and the text feature are input into an intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula:

[0064]

[0065] Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g+1th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, W * (g) is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is the text feature, x is the image, and y is the text.

[0066] Specifically, the inter-modal aggregate feature determination unit is specifically configured as follows:

[0067] The similarity between the image modality data and the text modality data is calculated, and a hierarchical similarity matrix is ​​determined based on the similarity. The similarity matrix is ​​expressed by the formula:

[0068]

[0069] Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, t represents the modality of sample i, T is the period, l i is the annotation information of the i-th image modality data, is the annotation information of the j-th text modal data in the T-th period;

[0070] Concatenate the image feature and the text feature row by row to obtain a first feature;

[0071] Obtain a second activation function corresponding to the inter-modal information aggregation network, input the first feature into the cross-modal information aggregation network, and determine the inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula:

[0072]

[0073] in, represents the input of the g-th layer image modality in the cross-modal information aggregation network, represents the output of the g-th layer image modality in the cross-modal information aggregation network, represents the input of the g-th layer text modality in the cross-modal information aggregation network, represents the output of the g-th layer text modality in the cross-modal information aggregation network, As the first feature, is the second activation function.

[0074] Specifically, the complementary feature determination unit is specifically configured as follows:

[0075] The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula:

[0076]

[0077] T S =[C x ,C y ]

[0078] Among them, H # , # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projected into a low-dimensional latent space, C x As part of the complementary feature, C y The other part of the complementary feature, T S are complementary features.

[0079] Specifically, the quantization processing unit is specifically configured as follows:

[0080] The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula:

[0081] B tea=sign(T S )

[0082] Among them, B tea is the associated hash code and sign() is the sign function.

[0083] Specifically, the fully connected network includes a hash module and a bit-by-bit fusion module, and the retrieval module specifically includes:

[0084] A feature extraction unit is configured to input the image modality data and the text modality data into a hash module to obtain image hash features and text hash features;

[0085] A joint feature determination unit is configured to input the image hash feature and the text hash feature into a bit-by-bit fusion module to obtain a joint feature;

[0086] The target hash code determination unit is configured to input the joint feature and the associated hash code into a fully connected network, perform cosine similarity reconstruction using the fully connected network, and obtain a target hash code when a preset condition is met, wherein the target hash code is expressed by a formula:

[0087]

[0088] in, is the target hash code, is a joint feature.

[0089] Specifically, the joint feature determination unit is specifically configured as follows:

[0090] The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula:

[0091]

[0092] Among them, W f is the preset fusion weight vector, is the image hash feature, is the text hash feature.

[0093] Based on the same inventive concept, the third aspect of the present disclosure proposes an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the cross-modal retrieval method based on efficient hierarchical information aggregation hashing as described above when executing the computer program.

[0094] Based on the same inventive concept, the fourth aspect of the present disclosure proposes a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the cross-modal retrieval method based on efficient hierarchical information aggregation hashing as described above.

[0095] As can be seen from the above, the present disclosure proposes a cross-modal retrieval method and related equipment based on efficient hierarchical information aggregation hashing, obtains a multimodal data set, wherein the multimodal data set contains image modal data and text modal data; the image modal data and text modal data are input into a hierarchical information aggregation network, and the associated hash code between the image modal data and the text modal data is obtained through the hierarchical information aggregation network. By performing information aggregation across different modalities to construct a multimodal complementary space, heterogeneous modalities can be better aligned. The image modal data, text modal data and the associated hash code are input into a fully connected network, and the target hash code is obtained using the fully connected network, and the target hash code is used for cross-modal retrieval. Fine-grained semantic supervision knowledge is provided by associating hash codes, thereby enhancing the semantic expression ability of hash functions. Thereby, a cross-modal retrieval method based on efficient hierarchical information aggregation hashing is realized, supporting more efficient cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 It is a flowchart of a cross-modal retrieval method based on efficient hierarchical information aggregation hashing according to an embodiment of the present disclosure;

[0098] Figure 2 It is a structural block diagram of a cross-modal retrieval device based on efficient hierarchical information aggregation hashing according to an embodiment of the present disclosure;

[0099] Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0100] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0101] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0102] The terms used in this disclosure are explained as follows:

[0103] ReLU: Rectified Linear Unit, is an activation function commonly used in artificial neural networks.

[0104] With the rapid development of the Internet, different types of multimedia data such as text, images, videos and audio are growing rapidly on the Internet. In social networks, different forms of multimedia data are usually used to describe the same or similar subject content. For example, in existing search engines such as Baidu and 360, when a specific topic is queried, the system can retrieve data in multiple modes; in most web pages on the Internet, in addition to a single text description, it usually contains a variety of pictures or video content to more comprehensively display the subject of the current web page; on real-time social media platforms, multimedia data uploaded by users in large quantities, such as Weibo, Xiaohongshu and other platforms, the data posted by these users is usually a combination of text, video and pictures. These different types of multimedia data are collectively referred to as multimodal data, that is, multimodal data is information data that describes the same subject or entity from different perspectives. Therefore, with the rapid growth of multimodal data in social networks and the heterogeneous characteristics between different modalities, some conventional search algorithms in the past cannot provide users with more accurate and efficient retrieval guarantees. Therefore, these massive amounts of multimodal data and the increasing demand for retrieval provide opportunities and directions for many new research tasks. For example, by developing retrieval algorithms specific to cross-modal scenario retrieval tasks to enable searches between different modalities, users can easily use text descriptions to search for desired images on the Internet; and by exploring the complementarity of information between different modalities or views, multimodal information can be integrated to improve the ability to describe specific topics.

[0105] Information retrieval technology in multimodal data scenarios is a method of finding the desired target from a database through a specific query sample. Generally speaking, the retrieval algorithm first needs to preprocess the data sample, convert its internal key information into a feature form, and select a specific measurement strategy to evaluate the similarity between different samples, so that a set of samples with similar content can be returned for a specific query. Approximate Nearest Neighbor (ANN) is one of the most basic strategies in multimodal retrieval tasks. It can improve the computational efficiency of the search process without sacrificing too much retrieval accuracy. Among them, hash quantization technology is one of the typical methods in the approximate nearest neighbor search algorithm.

[0106] Existing cross-modal hash retrieval methods are mainly divided into shallow models and deep models. Shallow models tend to train an efficient and simple nonlinear model, but the feature expression ability is often limited. Compared with shallow models, deep cross-modal hash retrieval methods can better model the information of the original data nonlinearly, and therefore have attracted widespread attention from researchers. Although existing deep cross-modal hash retrieval methods have achieved great success, most methods generally assume that samples are independent and unrelated during the model training process, and only consider the process of generating the final hash code, imposing pairwise similarity constraints, and retaining the semantic relevance of the original multimodal data into the hash code. This strategy is unable to deeply model the semantic associations between multimodal data samples to a certain extent.

[0107] Based on the above description, this embodiment proposes a cross-modal retrieval method based on efficient hierarchical information aggregation hashing, such as Figure 1 As shown, the method includes:

[0108] Step 101: Acquire a multimodal dataset, wherein the multimodal dataset includes image modality data and text modality data.

[0109] Step 102: input the image modality data and the text modality data into a hierarchical information aggregation network, and obtain an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network.

[0110] Step 103: input the image modality data, the text modality data and the associated hash code into a fully connected network, use the fully connected network to obtain a target hash code, and use the target hash code to perform cross-modal retrieval.

[0111] In a specific implementation, a multimodal data set is obtained, wherein the multimodal data set includes image modal data and text modal data. In this embodiment, the hierarchical information network can be regarded as a teacher module, and the fully connected network can be regarded as a student module.

[0112] In order to avoid the problem of using mandatory modality alignment constraints to learn a shared cross-modal space, which inevitably leads to different degrees of information loss in each modality, a hierarchical information aggregation network is designed to construct a multimodal complementary space to align heterogeneous modalities and model potential multimodal association relationships. The image modality data and text modality data are input into the hierarchical information aggregation network, and the association hash code between the image modality data and the text modality data is obtained through the hierarchical information aggregation network.

[0113] The cross-modal association knowledge distillation strategy is to transfer the deep association knowledge within and between modalities from the teacher module to the student module. The student module consists of multiple layers of fully connected layers, which are used to learn the final hash code and can preserve the semantic association relationship between and within modalities in the original space. The image modality data, text modality data and the associated hash code are input into the fully connected network, and the target hash code is obtained using the fully connected network, and the target hash code is used for cross-modal retrieval.

[0114] Through the above scheme, a multimodal data set is obtained, wherein the multimodal data set includes image modal data and text modal data; the image modal data and text modal data are input into a hierarchical information aggregation network, and the associated hash code between the image modal data and the text modal data is obtained through the hierarchical information aggregation network. By performing information aggregation across different modalities to construct a multimodal complementary space, heterogeneous modalities can be better aligned. The image modal data, text modal data and the associated hash code are input into a fully connected network, the target hash code is obtained using the fully connected network, and cross-modal retrieval is performed using the target hash code. Fine-grained semantic supervision knowledge is provided by associating hash codes, thereby enhancing the semantic expression ability of hash functions. Thereby, a cross-modal retrieval method based on efficient hierarchical information aggregation hashing is realized, supporting more efficient cross-modal retrieval. The present application proposes a cross-modal retrieval method based on efficient hierarchical information aggregation hashing, which can make multimodal retrieval capabilities more accurate and intelligent. Compared with existing solutions, it has the following advantages: considering the rich semantic information in multimodal data, it can simultaneously consider the complementarity between different samples and different modalities, and effectively solve the problem of missing fine-grained semantic information and semantic association in multimodal data. It explores the semantic information in different modal data of samples, establishes semantic associations between different samples, and makes the retrieval system more accurate and intelligent.

[0115] In some embodiments, the hierarchical information aggregation network includes a feature encoding network, an intra-modal information aggregation network, and a cross-modal information aggregation network, and step 102 specifically includes:

[0116] Step 1021, inputting the image modality data and the text modality data into a feature encoding network to obtain image features and text features;

[0117] Step 1022: Input the image features and the text features into an intra-modal information aggregation network to obtain intra-modal aggregation features.

[0118] Step 1023: input the image features and the text features into a cross-modal information aggregation network to obtain inter-modal aggregation features.

[0119] Step 1024 , concatenating the intra-modality aggregate features and the inter-modality aggregate features to obtain complementary features.

[0120] Step 1025, quantizing the complementary features to obtain an associated hash code between the image modality data and the text modality data.

[0121] In specific implementation, the feature encoding network includes an image encoding network and a text encoding network. The image modality data is input into the image encoding network to obtain image features. The image features are expressed by the formula:

[0122] F x = ImageNet(X;θ x )

[0123] Among them, F x is the image feature, θ x are the training parameters of the image encoding network.

[0124] The text modality data is input into the text encoding network to obtain text features, which are expressed by the formula:

[0125] F y = TextNet(X;θ y )

[0126] Among them, F y is the text feature, θ y Training parameters for the text encoding network.

[0127] The image features and the text features are input into a cross-modal information aggregation network to obtain inter-modal aggregation features. The image features and the text features are input into a cross-modal information aggregation network to obtain inter-modal aggregation features. The intra-modal aggregation features and the inter-modal aggregation features are concatenated to obtain complementary features. The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data.

[0128] In some embodiments, step 1022 specifically includes:

[0129] Step 10221, obtaining a first activation function corresponding to the intra-modal information aggregation network;

[0130] Step 10222: Input the image features and the text features into an intra-modal information aggregation network, and determine an intra-modal aggregation feature based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula:

[0131]

[0132]

[0133] Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g+1th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is a text feature.

[0134] In a specific implementation, a first activation function corresponding to the intra-modal information aggregation network is obtained, the image feature and the text feature are input into the intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula:

[0135]

[0136] Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g-th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is a text feature.

[0137] In this embodiment, is a normalized similarity matrix. Specifically, the normalized similarity matrix is ​​expressed as follows:

[0138]

[0139] D *ii =∑jS *ij

[0140] In this embodiment, the first activation function is preferably ReLU(). At the same time, in a supervised learning scenario, S(ij) is a multi-label similarity matrix. At the same time, This computational strategy is able to explicitly model fine-grained correlations between samples.

[0141] In some embodiments, step 1023 specifically includes:

[0142] Step 10231, calculate the similarity between the image modality data and the text modality data, and determine a hierarchical similarity matrix based on the similarity. The similarity matrix is ​​expressed by the formula:

[0143]

[0144] Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, t represents the modality of sample i, l i is the annotation information of the i-th image modality data, It is the annotation information of the j-th text modal data in the T-th period.

[0145] Step 10232, concatenating the image feature and the text feature row by row to obtain a first feature;

[0146] Step 10233, obtaining a second activation function corresponding to the inter-modal information aggregation network, inputting the first feature into the cross-modal information aggregation network, and determining an inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula:

[0147]

[0148] in, and represents the input and output of the g-th layer image modality in CM-MAN, As the first feature, is the second activation function.

[0149] In specific implementation, for the cross-modal information aggregation network, this embodiment introduces the similarity matrix S C , specifically, the similarity matrix is ​​expressed as:

[0150]

[0151] Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, t represents the modality of sample i, and T is the period.

[0152] In this embodiment, in a supervised learning scenario, I is the identity matrix, and

[0153] The image feature and the text feature are concatenated row by row to obtain a first feature. A second activation function corresponding to the inter-modal information aggregation network is obtained, the first feature is input into the cross-modal information aggregation network, and an inter-modal aggregation feature is determined based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula:

[0154]

[0155]

[0156] in, and represents the input and output of the g-th layer image modality in CM-MAN, As the first feature, is the second activation function, For S C The calculated normalized similarity matrix.

[0157] Through the above scheme, the output combines the features of the modality itself and the neighborhood features of other modalities. Therefore, the hierarchical information aggregation network can aggregate semantic information in a hierarchical form across multiple modalities and model multimodal association relationships.

[0158] In some embodiments, step 1024 specifically includes:

[0159] The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula:

[0160]

[0161] T S =[C x ,C y ]

[0162] Among them, H #, # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projection to low-dimensional latent space middle, represents the set of real numbers, T S is a complementary feature, C x As part of the complementary feature, C y is another part of the complementary feature, specifically,

[0163] In some embodiments, step 1025 specifically includes:

[0164] The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula:

[0165] B tea =sign(T S )

[0166] Among them, B tea is the associated hash code and sign() is the sign function.

[0167] In the specific implementation, in this embodiment, in order to construct a modality-invariant hash code, complementary features of different modalities are spliced ​​and then quantized to obtain an associated hash code between image modality data and text modality data, wherein the associated hash code is expressed by the formula:

[0168] B tea =sign(T S )

[0169] Among them, B tea is the associated hash code, and sign() is the sign function. In this embodiment, T S The space is the multimodal complementary space proposed by this method, which can effectively model the fine-grained association relationship within and between modalities. Therefore, the information in this space can be further used to guide the training process of the student module.

[0170] In some embodiments, in order to ensure that the hash code can retain the semantic correlation between paired samples in the original space, the similarity matrix calculated by the labels is reconstructed by calculating the cosine similarity between the hash codes, which is expressed as follows:

[0171]

[0172] in,

[0173] B tea is calculated by the symbolic function, so B tea is not differentiable. tea Replaced by the relaxed feature T S , which can be expressed by the formula:

[0174]

[0175] Among them, θ tea are all trainable parameters in the teacher module, S is the similarity matrix, if S ij = 0, which means that the i-th sample is not similar to the j-th sample, then S ij It constrains the cosine similarity between the joint complementary features of the i-th sample and the j-th sample to be equal to 0, indicating that the paired samples are orthogonal in space.

[0176] In this embodiment, in order to ensure no loss of generality, a quantization error is set, which is specifically expressed by the formula:

[0177]

[0178] In some embodiments, a discriminator is established Constraining joint complementary representations via adversarial learning strategies It obeys Gaussian distribution, which can be expressed by the formula:

[0179]

[0180] Among them, R c is randomly sampled from a Gaussian distribution, and its dimension is the same as T S Stay consistent. is the trainable parameter of the discriminator. The entire teacher module is regarded as a generator and the discriminator Considered as a classifier, it is used to distinguish the output features T of the teacher module S and R sampled from a Gaussian distribution c .

[0181] In this embodiment, the overall objective function of the teacher module is:

[0182]

[0183] Among them, α and β are the balance coefficients of the teacher module.

[0184] In some embodiments, the fully connected network includes a hash module and a bit-by-bit fusion module, and step 103 specifically includes:

[0185] Step 1031, inputting the image modality data and the text modality data into a hash module to obtain image hash features and text hash features;

[0186] Step 1032, inputting the image hash feature and the text hash feature into a bit-by-bit fusion module to obtain a joint feature;

[0187] Step 1033, input the joint feature and the associated hash code into a fully connected network, use the fully connected network to perform cosine similarity reconstruction, and obtain a target hash code when a preset condition is met, wherein the target hash code is expressed by the formula:

[0188]

[0189] in, is the target hash code, is a joint feature.

[0190] In specific implementation, the student module consists of multiple layers of fully connected layers, which is used to learn the final hash code and can preserve the semantic associations between and within modalities in the original space.

[0191] For any pair of data in the multimodal dataset, such as the i-th pair of data (x i ,y i ), the student module learns two hash functions for each of them to project the multimodal data into a common Hamming space. Specifically:

[0192] The image modality data is transformed using the first hash function to obtain an image hash feature, wherein the image hash feature is expressed using the formula:

[0193]

[0194] in, is the first hash function;

[0195] The text modal data is transformed using the second hash function to obtain a text hash feature, wherein the text hash feature is expressed using the formula:

[0196]

[0197] in, is the second hash function.

[0198] The image hash feature and the text hash feature are input into the bit-by-bit fusion module to obtain the joint feature. The joint feature and the associated hash code are input into the fully connected network, and the cosine similarity is reconstructed using the fully connected network. When the preset conditions are met, the target hash code is obtained, wherein the target hash code is expressed by the formula:

[0199]

[0200]

[0201] in, is the target hash code, is the joint feature, B tea is the associated hash code, is the image hash feature, is the text hash feature, is the preset condition, T S For complementary features, is the image part of the complementary feature, is the text part of the complementary feature, μ is the balance coefficient, θ stu are all trainable parameters in the student module.

[0202] In this embodiment, The target hash code obtained in this embodiment The relevance of and The cross-modal correlations between them all retain the cross-modal association relationship, and each mode retains its own internal association relationship. By calculating the continuous real value T S The cosine similarity between them is used to construct a fine-grained similarity matrix to guide the hash function learning process of the student module. S The fine-grained multimodal association knowledge in can be transferred to the student module from both inter-modal and intra-modal perspectives.

[0203] In some embodiments, step 1032 specifically includes:

[0204] The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula:

[0205]

[0206] Among them, W f is the preset fusion weight vector, W f ∈{0,1} 1×k , ⊙ is the Hadamard product. Different samples share the same fusion weight vector, while different hash bits have different fusion weights.

[0207] In this embodiment, the feature encoding network consists of three fully connected layers (dimension: d x →2048→1024→512,d y→300→512→512). Each fully connected layer is followed by a batch normalization layer and an activation function Tanh.

[0208] In this embodiment, both the intra-modal information aggregation network and the cross-modal information aggregation network are set to a single aggregation layer, and the input and output dimensions are set to 512. The discriminator consists of three fully connected layers (dimensions: k→64→32→1), k is the highest dimension, and each layer contains an activation function Relu.

[0209] In this embodiment, the hash function consists of three fully connected layers (dimension: d x →2048→512→k,d y →300→512→k), k is the highest dimension, and each layer contains a batch normalization layer and an activation function Tanh. α and β are the balance coefficients of the teacher module, and μ is the balance coefficient of the student module. The balance coefficient values ​​are set to {α=100, β=0.001, μ=1}, and the learning rates of the teacher and student modules are both set to 0.0001.

[0210] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0211] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0212] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a cross-modal retrieval device based on efficient hierarchical information aggregation hashing.

[0213] refer to Figure 2 , Figure 2 The cross-modal retrieval device based on efficient hierarchical information aggregation hashing of the embodiment includes:

[0214] The data acquisition module 201 is configured to acquire a multimodal data set, wherein the multimodal data set includes image modality data and text modality data;

[0215] The associated hash code determination module 202 is configured to input the image modality data and the text modality data into a hierarchical information aggregation network, and obtain an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network;

[0216] The retrieval module 203 is configured to input the image modality data, the text modality data and the associated hash code into a fully connected network, obtain a target hash code using the fully connected network, and perform cross-modality retrieval using the target hash code.

[0217] In some embodiments, the hierarchical information aggregation network includes a feature encoding network, an intra-modal information aggregation network, and a cross-modal information aggregation network, and the associated hash code determination module 202 specifically includes:

[0218] A feature determination unit is configured to input the image modality data and the text modality data into a feature encoding network to obtain image features and text features;

[0219] An intra-modality aggregate feature determination unit is configured to input the image feature and the text feature into an intra-modality information aggregation network to obtain an intra-modality aggregate feature;

[0220] An inter-modal aggregate feature determination unit is configured to input the image feature and the text feature into a cross-modal information aggregation network to obtain an inter-modal aggregate feature;

[0221] A complementary feature determination unit is configured to perform splicing processing on the intra-modality aggregate feature and the inter-modality aggregate feature to obtain a complementary feature;

[0222] The quantization processing unit is configured to perform quantization processing on the complementary features to obtain an associated hash code between the image modality data and the text modality data.

[0223] In some embodiments, the intra-modality aggregate feature determination unit is specifically configured to:

[0224] Obtaining a first activation function corresponding to the intra-modal information aggregation network;

[0225] The image feature and the text feature are input into an intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula:

[0226]

[0227] Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g-th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is a text feature.

[0228] In some embodiments, the inter-modality aggregate feature determination unit is specifically configured as follows:

[0229] The similarity between the image modality data and the text modality data is calculated, and a hierarchical similarity matrix is ​​determined based on the similarity. The similarity matrix is ​​expressed by the formula:

[0230]

[0231]

[0232] Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, t represents the modality of sample i, T is the period, l i is the annotation information of the i-th image modality data, is the annotation information of the j-th text modal data in the T-th period;

[0233] Concatenate the image feature and the text feature row by row to obtain a first feature;

[0234] Obtain a second activation function corresponding to the inter-modal information aggregation network, input the first feature into the cross-modal information aggregation network, and determine the inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula:

[0235]

[0236] in, and represents the input and output of the g-th layer image modality in CM-MAN, The first feature is is the second activation function.

[0237] In some embodiments, the complementary feature determination unit is specifically configured to:

[0238] The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula:

[0239]

[0240] T S =[C x ,C y ]

[0241] Among them, H # , # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projected into a low-dimensional latent space, T S is a complementary feature, C x As part of the complementary feature, C y Another part of the complementary feature.

[0242] In some embodiments, the quantization processing unit is specifically configured to:

[0243] The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula:

[0244] B tea =sign(T S )

[0245] Among them, B tea is the associated hash code and sign() is the sign function.

[0246] In some embodiments, the fully connected network includes a hash module and a bit-by-bit fusion module, and the retrieval module 203 specifically includes:

[0247] A feature extraction unit is configured to input the image modality data and the text modality data into a hash module to obtain image hash features and text hash features;

[0248] A joint feature determination unit is configured to input the image hash feature and the text hash feature into a bit-by-bit fusion module to obtain a joint feature;

[0249] The target hash code determination unit is configured to input the joint feature and the associated hash code into a fully connected network, perform cosine similarity reconstruction using the fully connected network, and obtain a target hash code when a preset condition is met, wherein the target hash code is expressed by a formula:

[0250]

[0251] in, is the target hash code, is the joint feature, B tea is the associated hash code, is the image hash feature, is the text hash feature, is the preset condition, T S For complementary features, is the image part of the complementary feature, is the text part of the complementary feature, and μ is the balance coefficient.

[0252] In some embodiments, the joint feature determination unit is specifically configured to:

[0253] The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula:

[0254]

[0255] Among them, W f is the preset fusion weight vector.

[0256] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0257] The device of the above embodiment is used to implement the corresponding cross-modal retrieval method based on efficient hierarchical information aggregation hash in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0258] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cross-modal retrieval method based on efficient hierarchical information aggregation hashing as described in any of the above embodiments is implemented.

[0259] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0260] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0261] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0262] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0263] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0264] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0265] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0266] The electronic device of the above embodiment is used to implement the corresponding cross-modal retrieval method based on efficient hierarchical information aggregation hash in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0267] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the cross-modal retrieval method based on efficient hierarchical information aggregation hashing as described in any of the above embodiments.

[0268] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0269] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the cross-modal retrieval method based on efficient hierarchical information aggregation hash as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0270] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0271] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0272] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0273] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0274] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0275] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0276] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0277] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A cross-modal retrieval method based on efficient hierarchical information aggregation hashing, characterized in that: include: Acquire a multimodal data set, wherein the multimodal data set includes image modality data and text modality data; Inputting the image modality data and the text modality data into a hierarchical information aggregation network, and obtaining an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network; The image modality data, the text modality data and the associated hash code are input into a fully connected network, a target hash code is obtained by using the fully connected network, and a cross-modality retrieval is performed by using the target hash code.

2. The method according to claim 1, characterized in that The hierarchical information aggregation network includes feature encoding network, intra-modal information aggregation network and cross-modal information aggregation network. The step of inputting the image modality data and the text modality data into a hierarchical information aggregation network and obtaining an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network comprises: Inputting the image modality data and the text modality data into a feature encoding network to obtain image features and text features; Inputting the image features and the text features into an intra-modal information aggregation network to obtain intra-modal aggregation features; Inputting the image features and the text features into a cross-modal information aggregation network to obtain inter-modal aggregation features; The intra-modality aggregate features and the inter-modality aggregate features are concatenated to obtain complementary features; The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data.

3. The method according to claim 2, characterized in that The step of inputting the image features and the text features into an intra-modal information aggregation network to obtain intra-modal aggregation features includes: Obtaining a first activation function corresponding to the intra-modal information aggregation network; The image feature and the text feature are input into an intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula: Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g+1th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, W * (g) is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is the text feature, x is the image, and y is the text.

4. The method according to claim 2, characterized in that: The step of inputting the image features and the text features into a cross-modal information aggregation network to obtain inter-modal aggregation features includes: The similarity between the image modality data and the text modality data is calculated, and a hierarchical similarity matrix is ​​determined based on the similarity. The similarity matrix is ​​expressed by the formula: Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, t represents the modality of sample i, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, T is the period, l i is the annotation information of the i-th image modality data, is the annotation information of the j-th text modal data in the T-th period; Concatenate the image feature and the text feature row by row to obtain a first feature; Obtain a second activation function corresponding to the inter-modal information aggregation network, input the first feature into the cross-modal information aggregation network, and determine the inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula: in, represents the input of the g-th layer image modality in the cross-modal information aggregation network, represents the output of the g-th layer image modality in the cross-modal information aggregation network, represents the input of the g-th layer text modality in the cross-modal information aggregation network, represents the output of the g-th layer text modality in the cross-modal information aggregation network, The first feature is is the second activation function.

5. The method according to claim 4, characterized in that The step of performing splicing processing on the intra-modality aggregate features and the inter-modality aggregate features to obtain complementary features includes: The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula: T S =[C x ,C y ] Among them, H # , # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projected into a low-dimensional latent space, T S is a complementary feature, C x As part of the complementary feature, C y Another part of the complementary feature.

6. The method according to claim 2, characterized in that The quantizing process of the complementary features to obtain the associated hash code between the image modality data and the text modality data includes: The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula: B tea =sign(T S ) Among them, B tea is the associated hash code and sign() is the sign function.

7. The method according to claim 1, characterized in that The fully connected network includes a hash module and a bit-by-bit fusion module. The step of inputting the image modality data, the text modality data, and the associated hash code into a fully connected network and obtaining a target hash code using the fully connected network includes: Inputting the image modality data and the text modality data into a hash module to obtain image hash features and text hash features; Input the image hash feature and the text hash feature into the bit-by-bit fusion module to obtain the joint feature; According to the joint feature and the associated hash code, a fully connected network is inputted, and cosine similarity is reconstructed using the fully connected network. When the preset conditions are met, a target hash code is obtained, wherein the target hash code is expressed by the formula: in, is the target hash code, is a joint feature.

8. The method according to claim 7, characterized in that The image hash feature and the text hash feature are input into the bit-by-bit fusion module to obtain the joint feature, including: The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula: Among them, W f is the preset fusion weight vector, is the image hash feature, is the text hash feature.

9. A cross-modal retrieval device based on efficient hierarchical information aggregation hashing, characterized in that: include: A data acquisition module is configured to acquire a multimodal data set, wherein the multimodal data set includes image modality data and text modality data; an associated hash code determination module, configured to input the image modality data and the text modality data into a hierarchical information aggregation network, and obtain an associated hash code between the image modality data and the text modality data via the hierarchical information aggregation network; The retrieval module is configured to input the image modality data, the text modality data and the associated hash code into a fully connected network, obtain a target hash code using the fully connected network, and perform cross-modality retrieval using the target hash code.

10. The device according to claim 9, characterized in that The hierarchical information aggregation network includes a feature encoding network, an intra-modal information aggregation network and a cross-modal information aggregation network. The associated hash code determination module specifically includes: A feature determination unit is configured to input the image modality data and the text modality data into a feature encoding network to obtain image features and text features; An intra-modality aggregate feature determination unit is configured to input the image feature and the text feature into an intra-modality information aggregation network to obtain an intra-modality aggregate feature; An inter-modal aggregate feature determination unit is configured to input the image feature and the text feature into a cross-modal information aggregation network to obtain an inter-modal aggregate feature; A complementary feature determination unit is configured to perform splicing processing on the intra-modality aggregate feature and the inter-modality aggregate feature to obtain a complementary feature; The quantization processing unit is configured to perform quantization processing on the complementary features to obtain an associated hash code between the image modality data and the text modality data.

11. The device according to claim 10, characterized in that The intra-modality aggregate feature determination unit is specifically configured as follows: Obtaining a first activation function corresponding to the intra-modal information aggregation network; The image feature and the text feature are input into an intra-modal information aggregation network, and an intra-modal aggregation feature is determined based on the first activation function, wherein the intra-modal aggregation feature is expressed by the formula: Where * = {x, y}, is the input feature of the g-th layer of the intra-modal information aggregation network, is the output feature of the g+1th layer of the intra-modal information aggregation network, is the normalized similarity matrix, is the first activation function, W * (g) is the parameter of the g-th convolution filter of the intra-modal information aggregation network, F x is the image feature, F y is the text feature, x is the image, and y is the text.

12. The device according to claim 10, characterized in that The inter-modality aggregate feature determination unit is specifically configured as follows: The similarity between the image modality data and the text modality data is calculated, and a hierarchical similarity matrix is ​​determined based on the similarity. The similarity matrix is ​​expressed by the formula: Among them, S C is the similarity matrix, I is the identity matrix, S t2i and S i2t is the cross-modal similarity matrix, t represents the modality of sample i, S i2t (ij) is the similarity between the i-th image modality data and the j-th text modality data, T is the period, l i is the annotation information of the i-th image modality data, is the annotation information of the j-th text modal data in the T-th period; Concatenate the image feature and the text feature row by row to obtain a first feature; Obtain a second activation function corresponding to the inter-modal information aggregation network, input the first feature into the cross-modal information aggregation network, and determine the inter-modal aggregation feature based on the hierarchical similarity matrix and the second activation function, wherein the inter-modal aggregation feature is expressed by the formula: in, represents the input of the g-th layer image modality in the cross-modal information aggregation network, represents the output of the g-th layer image modality in the cross-modal information aggregation network, represents the input of the g-th layer text modality in the cross-modal information aggregation network, represents the output of the g-th layer text modality in the cross-modal information aggregation network, The first feature is is the second activation function.

13. The device according to claim 12, characterized in that The complementary feature determination unit is specifically configured as follows: The intra-modality aggregation features and the inter-modality aggregation features are concatenated by columns to obtain complementary features, wherein the complementary features are expressed by the formula: T S =[C x ,C y ] Among them, H # , # = {x, xc, y, yc}, represents the output of the above hierarchical information aggregation network, is the relaxed hashing layer, used to convert H # Projected into a low-dimensional latent space, T S is a complementary feature, C x As part of the complementary feature, C y Another part of the complementary feature.

14. The device according to claim 10, characterized in that The quantization processing unit is specifically configured as follows: The complementary features are quantized to obtain an associated hash code between the image modality data and the text modality data, wherein the associated hash code is expressed by the formula: B tea =sign(T S ) Among them, B tea is the associated hash code and sign() is the sign function.

15. The device according to claim 9, characterized in that The fully connected network includes a hash module and a bit-by-bit fusion module, and the retrieval module specifically includes: A feature extraction unit is configured to input the image modality data and the text modality data into a hash module to obtain image hash features and text hash features; A joint feature determination unit is configured to input the image hash feature and the text hash feature into a bit-by-bit fusion module to obtain a joint feature; The target hash code determination unit is configured to input the joint feature and the associated hash code into a fully connected network, perform cosine similarity reconstruction using the fully connected network, and obtain a target hash code when a preset condition is met, wherein the target hash code is expressed by a formula: in, is the target hash code, is a joint feature.

16. The device according to claim 15, characterized in that The joint feature determination unit is specifically configured as follows: The image hash feature and the text hash feature are input into the bit-by-bit fusion module, and the bit-by-bit fusion module is used to construct the joint feature bit by bit. The joint feature is expressed by the formula: Among them, W f is the preset fusion weight vector, is the image hash feature, is the text hash feature.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

18. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 8.

Images