A cross-modal data retrieval method, device and equipment

Abstract

The application relates to a cross-modal data retrieval method, device and equipment, which comprises the following steps: acquiring a data set containing first modal data and second modal data; processing the data set to generate a training set; constructing a specific feature extraction network and a modal conversion network; inputting the training set into the specific feature extraction network to generate first features corresponding to the first modal data and second features corresponding to the second modal; training the modal conversion network by using the first features and the second features; and performing cross-modal retrieval in the respective feature spaces of each modal by using the specific feature extraction network and the trained modal conversion network. It can be understood that the specific feature extraction network is used to extract the features of the double-modal data respectively, the modal conversion network is used for conversion, the features of each modal can be measured in the respective feature spaces, and information loss caused by projection in a public space is avoided.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a cross-modal data retrieval method, apparatus, and device. Background Technology

[0002] Currently, the amount of media data on the internet, including text, images, and videos, is growing rapidly, making cross-modal retrieval increasingly important in real-world applications. Given a query task and databases from different modalities, cross-modal retrieval is the task of retrieving the most relevant data from the database. Most research in cross-modal retrieval focuses on image and text modalities.

[0003] Typically, existing cross-modal data retrieval methods map visual and text inputs to embedding functions in a common embedding space, simplifying the cross-modal retrieval task into a familiar nearest neighbor retrieval task in Euclidean space.

[0004] However, existing methods that project data from different modalities into a common feature space tend to focus too much on the shared information of the data from different modalities, thus losing specific characteristics and resulting in low similarity in cross-modal retrieval. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a cross-modal data retrieval method, apparatus and device to solve the problem that the prior art focuses too much on the shared information of different modal data, thereby losing the specific characteristics and resulting in low similarity in cross-modal retrieval.

[0006] According to a first aspect of the present invention, a cross-modal data retrieval method is provided, comprising:

[0007] Obtain the dataset containing the first modality data and the second modality data;

[0008] The dataset is processed to generate a training set;

[0009] Construct a specific feature extraction network and a mode conversion network;

[0010] The training set is input into the special feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality.

[0011] The modality conversion network is trained using the first feature and the second feature;

[0012] Using the specific feature extraction network and the trained modality transformation network, cross-modal retrieval is performed in the feature space of each modality.

[0013] Preferably, the step of processing the dataset to generate a training set includes:

[0014] Analyze the first modal data and the second modal data to obtain second modal data of the same type as the first modal data and second modal data of the same or different types as the first modal data, respectively;

[0015] The first modal data, the second modal data of the same type as the first modal data, and the second modal data of different types from the first modal data are integrated into a first data group in the form of triplets;

[0016] Analyze the first modal data and the second modal data to obtain first modal data of the same type as the second modal data and first modal data of the same or different types as the second modal data;

[0017] The second modal data, the first modal data of the same type as the second modal data, and the first modal data of different types from the second modal data are integrated into a triplet to form the second data group;

[0018] Use the first data set and the second data set as the training set.

[0019] Preferably, the construction of the specific feature extraction network includes:

[0020] Construct a specific feature extraction network that includes a first feature extractor and a second feature extractor;

[0021] The first and second feature extractors have the same structure, each containing a max pooling layer and seven convolutional layers.

[0022] Preferably, the construction of the specific feature extraction network, which includes a first feature extractor and a second feature extractor, comprises:

[0023] The first and second feature extractors are constructed according to a specific structure;

[0024] The specific structure is as follows: a first convolutional layer, a max pooling layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer;

[0025] The input of the second convolutional layer is staggered with the output of the fourth convolutional layer;

[0026] The input of the fifth convolutional layer is staggered with the output of the seventh convolutional layer.

[0027] Preferably, the step of inputting the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality includes:

[0028] The first feature is generated by extracting features from the first data set using the first feature extractor.

[0029] The second feature extractor extracts features from the second data set to generate a second feature.

[0030] Preferably, training the mode conversion network using the first feature and the second feature includes:

[0031] Apply intramodal ranking loss constraints and cross-entropy constraints to the first feature and the second feature;

[0032] The constrained data is processed through the mode transformation matrix to obtain the first transformation feature and the second transformation feature;

[0033] The first feature and the second transformation feature are subjected to triplet constraints, and the second feature and the first transformation feature are subjected to triplet constraints to obtain the trained mode transformation matrix, and then the trained mode transformation network is obtained.

[0034] Preferably, the method further includes:

[0035] Based on the dataset, generate a test set with randomized data;

[0036] Using the test set as input, cross-modal retrieval is performed to generate test results.

[0037] According to a second aspect of the present invention, a cross-modal data retrieval apparatus is provided, comprising:

[0038] The training data acquisition module is used to acquire a dataset containing first modality data and second modality data;

[0039] The data processing module is used to process the dataset and generate a training set;

[0040] The building blocks are used to construct specific feature extraction networks and mode transformation networks;

[0041] The training module is used to input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; it is also used to train the modality conversion network using the first feature and the second feature.

[0042] The cross-modal retrieval module is used to perform cross-modal retrieval in the feature space of each modality by utilizing the specific feature extraction network and the trained modality transformation network.

[0043] Preferably, the device further includes:

[0044] The testing module is used to generate a randomized test set based on the dataset; and to perform cross-modal retrieval using the test set as input to generate test results.

[0045] According to a third aspect of the present invention, a cross-modal data retrieval device is provided, comprising:

[0046] The main controller and the memory connected to the main controller;

[0047] Memory, which stores program instructions;

[0048] The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0049] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0050] It is understood that this invention can acquire a dataset containing first modality data and second modality data; process the dataset to generate a training set; construct a specific feature extraction network and a modality transformation network; input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; train the modality transformation network using the first and second features; and perform cross-modality retrieval in the feature space of each modality using the specific feature extraction network and the trained modality transformation network. It is understood that this invention utilizes the constructed specific feature extraction network to extract features from both modalities, and uses the modality transformation network to transform them, enabling the similarity measurement of data features of each modality in their respective feature spaces, avoiding information loss due to common space projection.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 This is a schematic diagram illustrating the steps of a cross-modal data retrieval method according to an exemplary embodiment;

[0054] Figure 2 This is a flowchart illustrating a cross-modal data retrieval process according to an exemplary embodiment;

[0055] Figure 3 This is a schematic block diagram of a cross-modal data retrieval device according to an exemplary embodiment. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0057] Example 1

[0058] Figure 1 This is a schematic diagram illustrating the steps of a cross-modal data retrieval method according to an exemplary embodiment. See also: Figure 1 A cross-modal data retrieval method is provided, including:

[0059] Step S11: Obtain a dataset containing first modality data and second modality data;

[0060] Step S12: Process the dataset to generate a training set;

[0061] Step S13: Construct a specific feature extraction network and a mode conversion network;

[0062] Step S14: Input the training set into the special feature extraction network to generate the first feature corresponding to the first modality data and the second feature corresponding to the second modality;

[0063] Step S15: Train the mode conversion network using the first feature and the second feature;

[0064] Step S16: Using the specific feature extraction network and the trained modality transformation network, perform cross-modal retrieval in the feature space of each modality.

[0065] In practical application, the above dataset can be represented as X = {(a i ,b i ,y i )|i=1,2,...,N}where, a i For the first modal data, b i For the second modality data, y i These are the labels of the dataset. For the first modality data, the second modality data includes second modality data of the same type as the first modality data, and second modality data of different types. For the second modality data, the first modality data includes first modality data of the same type as the second modality data, and first modality data of different types.

[0066] In specific application scenarios, this embodiment first needs to acquire a dataset, then construct a specific feature extraction network and a mode conversion network, and then train the network using the dataset. Figure 2 This is a flowchart illustrating a cross-modal data retrieval process according to an exemplary embodiment. In a specific application scenario, the network trained after the above steps is as follows: Figure 2 As shown, suppose a user wants to perform cross-modal retrieval on data from modality 1 and modality 2. The data from modality 1 and modality 2 are input into a specific feature extraction network. Then, the specific feature extraction network generates a first feature x based on the data from modality 1. 1 The second feature z is generated based on the data from mode 2. 2 By inputting the first feature and the second feature into the modality transfer network, the first feature x can be transformed into a modality transfer network. 1 The feature x converted to mode 2 1(2) , the second feature z 2 The feature z converted to mode 1 2(1) Then, the first feature x can be performed in the feature space of mode 1. 1 and z 2(1) The similarity measurement is performed, and simultaneously, in the feature space of modality 2, the second feature z is calculated. 2 and x 1(2) Similarity metric.

[0067] It is understood that this invention can acquire a dataset containing first modality data and second modality data; process the dataset to generate a training set; construct a specific feature extraction network and a modality transformation network; input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; train the modality transformation network using the first and second features; and perform cross-modality retrieval in the feature space of each modality using the specific feature extraction network and the trained modality transformation network. It is understood that this invention utilizes the constructed specific feature extraction network to extract features from both modalities, and uses the modality transformation network to transform them, enabling the similarity measurement of data features of each modality in their respective feature spaces, avoiding information loss due to common space projection.

[0068] It should be noted that the process of processing the dataset to generate the training set includes:

[0069] Analyze the first modal data and the second modal data to obtain second modal data of the same type as the first modal data and second modal data of the same or different types as the first modal data, respectively;

[0070] The first modal data, the second modal data of the same type as the first modal data, and the second modal data of different types from the first modal data are integrated into a first data group in the form of triplets;

[0071] Analyze the first modal data and the second modal data to obtain first modal data of the same type as the second modal data and first modal data of the same or different types as the second modal data;

[0072] The second modal data, the first modal data of the same type as the second modal data, and the first modal data of different types from the second modal data are integrated into a triplet to form the second data group;

[0073] Use the first data set and the second data set as the training set.

[0074] In practice, for the training set, the data is input into the network in the form of triples. That is, the training set is divided into two data groups. The first data group contains the data of the first modality, which can be represented by {a i ,b p ,b n} indicates that, where a i For the first modal data, b p b represents second modal data of the same type as the first modal data. n This represents the second modal data, which is of the same or different class as the first modal data. Similarly, for the second data set, {b} can be used. i ,a p ,a n} indicates that, where b i The second modal data, a p a represents the first modal data of the same type as the second modal data. n The first modal data represents the same or different class as the second modal data.

[0075] Understandably, by processing the dataset into a training set in the form of triples, the specific feature extraction network, and the modality transfer network, can process the data more accurately and quickly.

[0076] It should be noted that the construction of the specific feature extraction network includes:

[0077] Construct a specific feature extraction network that includes a first feature extractor and a second feature extractor;

[0078] The first and second feature extractors have the same structure, each containing a max pooling layer and seven convolutional layers.

[0079] In practice, when constructing a specific feature extraction network, the specific feature extraction network includes two feature extractors with identical structures. In specific application scenarios, the two feature extractors can extract features of a set of modal data respectively.

[0080] Specifically, the network is constructed in the following manner, comprising a specific feature extraction network of a first feature extractor and a second feature extractor, including:

[0081] The first and second feature extractors are constructed according to a specific structure;

[0082] The specific structure is as follows: a first convolutional layer, a max pooling layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer;

[0083] The input of the second convolutional layer is staggered with the output of the fourth convolutional layer;

[0084] The input of the fifth convolutional layer is staggered with the output of the seventh convolutional layer.

[0085] In practice, the first feature extractor and the second feature extractor have the same structure, both containing a specific structure: first convolutional layer → max pooling layer → second convolutional layer → third convolutional layer → fourth convolutional layer → fifth convolutional layer → sixth convolutional layer → seventh convolutional layer. The input of the second convolutional layer is connected to the output residual of the fourth convolutional layer, and the input of the fifth convolutional layer is connected to the output residual of the seventh convolutional layer.

[0086] Preferably, the kernel size of the first convolutional layer is 7×7, the number of kernels is 64, and the stride is 2; the pooling window size of the max pooling layer is 3×3, and the stride is 2; the kernel size of the second, fourth, fifth, and seventh convolutional layers is 1×1, the number of kernels is 64, 256, 128, and 512, respectively, and the stride is 1; the kernel size of the third and sixth convolutional layers is 3×3, the number of kernels is 128 and 256, respectively, and the stride is 2.

[0087] Understandably, constructing a feature extractor in the manner described above can lead to more accurate final results.

[0088] It should be noted that the step of inputting the training set into the specific feature extraction network to generate the first feature corresponding to the first modality data and the second feature corresponding to the second modality includes:

[0089] The first feature is generated by extracting features from the first data set using the first feature extractor.

[0090] The second feature extractor extracts features from the second data set to generate a second feature.

[0091] In practice, the specific feature extraction network includes two structurally identical feature extractors. During network training, the first feature extractor extracts features from the first data set to generate a first feature; the second feature extractor extracts features from the second data set to generate a second feature. Similarly, in cross-modal data retrieval tasks, the first feature extractor extracts features from modality 1, and the second feature extractor extracts features from modality 2. This setup not only increases operational efficiency but also makes the results more accurate.

[0092] It should be noted that training the mode conversion network using the first feature and the second feature includes:

[0093] Apply intramodal ranking loss constraints and cross-entropy constraints to the first feature and the second feature;

[0094] The constrained data is processed through the mode transformation matrix to obtain the first transformation feature and the second transformation feature;

[0095] The first feature and the second transformation feature are subjected to triplet constraints, and the second feature and the first transformation feature are subjected to triplet constraints to obtain the trained mode transformation matrix, and then the trained mode transformation network is obtained.

[0096] In practice, applying cross-entropy constraints to the first and second features ensures sufficient discriminability of the bimodal sample features. The cross-entropy formula is as follows:

[0097]

[0098] Where n represents the number of samples, x i z represents the features of a sample in modality 1. i p(y) represents the feature of the modality 2 sample. i |x i p(y) represents the predicted feature value for a sample in modality 1. i ∣z i ) represents the predicted feature value for the modality 2 sample.

[0099] Applying intra-modal ranking loss constraints to the first and second features ensures feature discriminability within the same modality. For each sample labeled i, the distance between positive and negative samples within the same modality should be greater than a predefined value ρ2, as shown in the formula below:

[0100]

[0101] Where ρ2 is a predefined boundary value, and j and k represent labels of different categories.

[0102] Inspired by the concept of transition matrices in matrix theory—that basis vectors in different linear spaces can be mutually represented by transition matrices, and the eigenvalues ​​corresponding to basis vectors can also be mutually represented by transition matrices—image data is Euclidean data. Bimodal image data corresponds to different basis vectors, which can be represented by the mode transformation matrix W. t Transform the two image features so that the data from the two modalities can be similarly measured in their respective feature spaces:

[0103] x 1(2) =W t ·x 1 ,

[0104]

[0105] st||W t ||2=1

[0106] The transformed features are constrained using triplet loss, as shown in the following formula:

[0107]

[0108] For mode 1, the label is y i Anchor point sample features It and similar sample features when transitioning from mode 2 to mode 1 The distance should be less than the distance between it and the features of the modality 2 outlier samples. The distance. Similarly, for samples of modality 2, such constraints are also needed. x and z represent sample features of different modalities, i and j represent samples of the same class, and i and k represent samples of the same class. ρ i represents the predefined boundary value, and D(·) represents the Euclidean distance. To ensure stable training, all features x and z are standardized.

[0109] The loss function of this method is:

[0110] L = L c +L intra +L trip

[0111] st||W t ||2=1

[0112] During training, the pre-defined bimodal data is input into the designed specific feature extraction network in the form of triples to obtain the first feature x. 1 Second feature z 2 For x 1 and z 2 Apply intramodal ranking loss constraints and cross-entropy constraints respectively, x1 Through the mode transition matrix W t Obtain the second transformation feature x 1(2) , z 2 Through the mode transformation matrix The second transformation feature z is obtained 2(1) For the first feature x 1 Second transformation feature z 2(1) Apply triplet constraints to the second feature z 2 and the first transformation feature x 1(2) By applying triplet constraints, the trained mode transition matrix is ​​obtained.

[0113] It should be noted that the method also includes:

[0114] Based on the dataset, generate a test set with randomized data;

[0115] Using the test set as input, cross-modal retrieval is performed to generate test results.

[0116] In practice, a randomized test set can be generated based on the dataset. The test set can be randomly input without the need for triples. The test set can then be used as input to perform cross-modal retrieval and generate test results.

[0117] After testing this embodiment, it was compared with existing cross-modal retrieval models, and the performance evaluation table is shown below:

[0118]

[0119] Table 1

[0120] The simulation experiments above show that the AP of multi-channel retrieval of single channel in this embodiment is 99.01%, and the AP of single-channel retrieval of multi-channel is 98.9%; the MAP of multi-channel retrieval of single channel is 98.79%, and the MAP of single-channel retrieval of multi-channel is 98.74%. Both of these indicators are higher than those of TONE, SIDHCNN and Distillation_50 in the prior art.

[0121] Therefore, this embodiment utilizes a constructed specific feature extraction network to extract bimodal data features separately and designs a modality transformation matrix, enabling the similarity measurement of data features of each modality in its respective feature space, thus avoiding information loss caused by common space projection. Experiments demonstrate that this invention is a very practical cross-modal retrieval method.

[0122] Example 2

[0123] Figure 3This is a schematic block diagram of a cross-modal data retrieval device according to an exemplary embodiment. See also: Figure 3 A cross-modal data retrieval device is provided, comprising:

[0124] The training data acquisition module 101 is used to acquire a dataset containing first modality data and second modality data;

[0125] Data processing module 102 is used to process the dataset to generate a training set;

[0126] Module 103 is used to construct a specific feature extraction network and a mode conversion network;

[0127] Training module 104 is used to input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; it is also used to train the modality conversion network using the first feature and the second feature.

[0128] The cross-modal retrieval module 105 is used to perform cross-modal retrieval in the feature space of each modality by utilizing the specific feature extraction network and the trained modality transformation network.

[0129] It is understood that in this embodiment, the training data acquisition module 101 acquires a dataset containing first modality data and second modality data; the data processing module 102 processes the dataset to generate a training set; the construction module 103 constructs a specific feature extraction network and a modality transformation network; the training module 104 inputs the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; the modality transformation network is trained using the first and second features; and the cross-modality retrieval module 105 uses the specific feature extraction network and the trained modality transformation network to perform cross-modality retrieval in the feature space of each modality. It is understood that this invention utilizes the constructed specific feature extraction network to extract features from both modalities and uses the modality transformation network for transformation, enabling similarity measurement of data features of each modality in their respective feature spaces, avoiding information loss due to common space projection.

[0130] It should be noted that the device further includes:

[0131] The testing module is used to generate a randomized test set based on the dataset; and to perform cross-modal retrieval using the test set as input to generate test results.

[0132] Understandably, by setting up a testing module, the trained special feature extraction network and modality transformation network can be tested and test results can be generated, so that the training results can be roughly understood before using the trained special feature extraction network and modality transformation network for cross-modal retrieval.

[0133] Example 3

[0134] According to a third aspect of the present invention, a cross-modal data retrieval device is provided, comprising:

[0135] The main controller and the memory connected to the main controller;

[0136] Memory, which stores program instructions;

[0137] The main controller is used to execute program instructions stored in the memory and perform any of the methods described above.

[0138] It is understood that this embodiment can acquire a dataset containing first modality data and second modality data; process the dataset to generate a training set; construct a specific feature extraction network and a modality transformation network; input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality; train the modality transformation network using the first and second features; and perform cross-modality retrieval in the feature space of each modality using the specific feature extraction network and the trained modality transformation network. It is understood that this invention utilizes the constructed specific feature extraction network to extract features from both modalities, and uses the modality transformation network to transform them, enabling the data features of each modality to be similar in their respective feature spaces, avoiding information loss due to common space projection.

[0139] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0140] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0141] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0142] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0143] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0145] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0146] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0147] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A cross-modal data retrieval method, characterized in that, include: Obtain the dataset containing the first modality data and the second modality data; The first modal data is an image, and the second modal data is text; The dataset is processed to generate a training set, including: integrating the first modality data, second modality data of the same class as the first modality data, and second modality data of the same or different classes as the first modality data into a first data set in the form of triplets; integrating the second modality data, first modality data of the same class as the second modality data, and first modality data of the same or different classes as the second modality data into a second data set in the form of triplets; Construct a specific feature extraction network and a mode conversion network; The training set is input into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality, including: extracting features from the first data group through the first feature extractor of the specific feature extraction network to generate the first feature; and extracting features from the second data group through the second feature extractor of the specific feature extraction network to generate the second feature. The modality transformation network is trained using the first feature and the second feature, including: applying intra-modal ranking loss constraints and cross-entropy constraints to the first feature and the second feature; passing the constrained data through the modality transformation matrix to obtain the first transformation feature and the second transformation feature; applying triplet constraints to the first feature and the second transformation feature, and applying triplet constraints to the second feature and the first transformation feature to obtain the trained modality transformation matrix, thereby obtaining the trained modality transformation network; Using the specific feature extraction network and the trained modality transformation network, cross-modal retrieval is performed in the feature space of each modality.

2. The method according to claim 1, characterized in that, The step of processing the dataset to generate a training set further includes: Analyze the first modal data and the second modal data to obtain second modal data of the same type as the first modal data and second modal data of the same or different types as the first modal data, respectively; Analyze the first modal data and the second modal data to obtain first modal data of the same type as the second modal data and first modal data of the same or different types as the second modal data; Use the first data set and the second data set as the training set.

3. The method according to claim 2, characterized in that, The construction of the specific feature extraction network includes: Construct a specific feature extraction network that includes a first feature extractor and a second feature extractor; The first and second feature extractors have the same structure, each containing a max pooling layer and seven convolutional layers.

4. The method according to claim 3, characterized in that, The construction of the specific feature extraction network, which includes a first feature extractor and a second feature extractor, includes: The first and second feature extractors are constructed according to a specific structure; The specific structure is as follows: a first convolutional layer, a max pooling layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer; The input of the second convolutional layer is staggered with the output of the fourth convolutional layer; The input of the fifth convolutional layer is staggered with the output of the seventh convolutional layer.

5. The method according to claim 1, characterized in that, Also includes: Based on the dataset, generate a test set with randomized data; Using the test set as input, cross-modal retrieval is performed to generate test results.

6. A cross-modal data retrieval device, characterized in that, include: The training data acquisition module is used to acquire a dataset containing first modality data and second modality data; The first modal data is an image, and the second modal data is text; The data processing module is used to process the dataset to generate a training set, including: integrating the first modality data, second modality data of the same type as the first modality data, and second modality data of the same or different types as the first modality data into a first data set in the form of triplets; and integrating the second modality data, first modality data of the same type as the second modality data, and first modality data of the same or different types as the second modality data into a second data set in the form of triplets. The building blocks are used to construct specific feature extraction networks and mode transformation networks; The training module is used to input the training set into the specific feature extraction network to generate a first feature corresponding to the first modality data and a second feature corresponding to the second modality, including: extracting features from the first data group through a first feature extractor of the specific feature extraction network to generate a first feature; and extracting features from the second data group through a second feature extractor of the specific feature extraction network to generate a second feature. The training module is also used to train the modality transformation network using the first feature and the second feature, including: applying intra-modal ranking loss constraints and cross-entropy constraints to the first feature and the second feature; passing the constrained data through the modality transformation matrix to obtain the first transformation feature and the second transformation feature; applying triplet constraints to the first feature and the second transformation feature, and applying triplet constraints to the second feature and the first transformation feature to obtain the trained modality transformation matrix, thereby obtaining the trained modality transformation network; The cross-modal retrieval module is used to perform cross-modal retrieval in the feature space of each modality by utilizing the specific feature extraction network and the trained modality transformation network.

7. The apparatus according to claim 6, characterized in that, Also includes: The testing module is used to generate a randomized test set based on the dataset; and to perform cross-modal retrieval using the test set as input to generate test results.

8. A cross-modal data retrieval device, characterized in that, include: The main controller and the memory connected to the main controller; Memory, which stores program instructions; The main controller is used to execute program instructions stored in the memory and to perform the method as described in any one of claims 1 to 5.

Images