Cross-modal retrieval method, network training method, device, equipment and medium

Abstract

The present disclosure relates to a cross-modal retrieval method, a network training method, an apparatus, a device and a medium. The cross-modal retrieval method includes: obtaining data to be retrieved and candidate data; the data to be retrieved and the candidate data correspond to different modalities; extracting a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network; and retrieving data matching the data to be retrieved from the candidate data according to the matching degree of the first feature and the second feature. The cross-modal retrieval network in the present disclosure can accurately capture the local information of the input data to be retrieved and the candidate data, and output more effective features. The more effective features are more distinguishable between the same modality and more identifiable between different modalities, thereby improving the fine-grained cross-modal retrieval performance.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a cross-modal retrieval method, a network training method, a device, a equipment and a medium. Background Art

[0002] Cross-modal retrieval refers to a retrieval method in which the modality of the retrieval results is different from the modality of the query data. For example, using images to retrieve text, video, audio, etc.

[0003] In related technologies, cross-modal retrieval usually calculates the similarity of features of different modalities output by the retrieval network, obtains a correlation score, and performs retrieval based on the correlation score. When training the retrieval network, the two modal data are usually mapped to a high-dimensional representation space of the same dimension, and after obtaining the features of the two modalities, the contrast loss function is directly used for training. The features output by the retrieval network obtained by this training method are not fine enough, and can only make a coarse-grained judgment on whether the features are related. The retrieval effect for fine-grained cross-modal retrieval is poor. Summary of the invention

[0004] The present disclosure provides a cross-modal retrieval method, a network training method, an apparatus, a device and a medium to at least solve the problem that the features output by the retrieval network in the related art are not fine enough and can only judge whether the features are related in a coarse-grained manner, resulting in poor retrieval effect for fine-grained cross-modal retrieval. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a cross-modal retrieval method is provided, comprising:

[0006] Acquire data to be retrieved and candidate data; the data to be retrieved and the candidate data correspond to different modalities;

[0007] Extracting a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network;

[0008] Retrieving data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature;

[0009] Among them, the cross-modal retrieval network is obtained by adversarially training a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data in the adversarial network based on sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data that does not match the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality.

[0010] In an exemplary embodiment, the cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network, and extracting the first feature of the data to be retrieved and the second feature of the candidate data based on the cross-modal retrieval network includes:

[0011] Inputting the data to be retrieved and the candidate data into the cross-modal retrieval network;

[0012] extracting the first feature based on the first generative network;

[0013] The second feature is extracted according to the second generation network and the third generation network.

[0014] In an exemplary embodiment, the retrieving data matching the data to be retrieved from the candidate data according to the matching degree of the first feature and the second feature includes:

[0015] determining the degree of match between the first feature and the second feature;

[0016] The candidate data corresponding to the target second feature is used as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature meets a preset condition.

[0017] According to a second aspect of an embodiment of the present disclosure, a method for training a cross-modal retrieval network is provided, comprising:

[0018] Acquire sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality;

[0019] Input the sample data to be retrieved, the associated sample data, and the non-associated sample data into a first sample generation network, a second sample generation network, and a third sample generation network in the adversarial network to obtain sample features of the sample data to be retrieved, associated sample features of the associated sample data, and non-associated sample features of the non-associated sample data;

[0020] The adversarial network is adversarially trained based on the sample features, the associated sample features and the unassociated sample features to obtain a cross-modal retrieval network.

[0021] In an exemplary embodiment, the step of inputting the sample data to be retrieved, the associated sample data, and the non-associated sample data into a first sample generation network, a second sample generation network, and a third sample generation network in an adversarial network to obtain sample features of the sample data to be retrieved, associated sample features of the associated sample data, and non-associated sample features of the non-associated sample data includes:

[0022] Inputting the sample data to be retrieved into the first sample generation network, and extracting the sample features based on the first sample generation network;

[0023] Inputting the associated sample data into the second sample generation network, and extracting the associated sample features according to the second sample generation network;

[0024] The non-associated sample data is input into the third sample generation network, and the non-associated sample features are extracted based on the third sample generation network.

[0025] In an exemplary embodiment, the performing adversarial training on the adversarial network based on the sample features, the associated sample features, and the unassociated sample features to obtain a cross-modal retrieval network includes:

[0026] Obtaining first loss information based on the sample feature, the associated sample feature, and the unassociated sample feature;

[0027] Inputting the sample features, the associated sample features, and the unassociated sample features into a discriminant network in the adversarial network to obtain second loss information;

[0028] The adversarial network is trained based on the first loss information and the second loss information to obtain a first generation network, a second generation network and a third generation network; the first generation network is used to extract features of the data to be retrieved, and the second generation network and the third generation network are used to extract features of the candidate data;

[0029] A cross-modal retrieval network is generated according to the first generation network, the second generation network, and the third generation network.

[0030] In an exemplary embodiment, inputting the sample feature, the associated sample feature, and the unassociated sample feature into a discriminant network in the adversarial network to obtain second loss information includes:

[0031] Inputting the sample features, the associated sample features and the non-associated sample features into the discriminant network;

[0032] Based on the discriminant network, the matching degree between the sample feature and the associated sample feature is discriminated to obtain a first discriminant result, and the matching degree between the sample feature and the non-associated sample feature is discriminated to obtain a second discriminant result;

[0033] The second loss information is obtained according to the first discrimination result and the second discrimination result.

[0034] In an exemplary embodiment, obtaining the second loss information according to the first discrimination result and the second discrimination result includes:

[0035] Calculating a first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result;

[0036] The second loss information is obtained according to the first logarithm and the second logarithm.

[0037] In an exemplary embodiment, the number of the sample data to be retrieved is multiple, and the method further includes:

[0038] Determine target sample data to be retrieved from multiple sample data to be retrieved;

[0039] Determine target associated sample data matching the target sample data to be retrieved from the associated sample data, and determine target non-associated sample data not matching the target sample data to be retrieved from the non-associated sample data;

[0040] The step of inputting the sample features, the associated sample features, and the unassociated sample features into a discriminant network in the adversarial network to obtain second loss information includes:

[0041] The sample features of the target sample to-be-retrieved data, the associated sample features of the target associated sample data, and the non-associated sample features of the target non-associated sample data are input into the discriminant network to obtain the second loss information.

[0042] According to a third aspect of an embodiment of the present disclosure, a cross-modal retrieval device is provided, including:

[0043] A data acquisition module is configured to acquire data to be retrieved and candidate data; the data to be retrieved and the candidate data correspond to different modalities;

[0044] A feature extraction module, configured to extract a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network;

[0045] A data retrieval module is configured to retrieve data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature;

[0046] Among them, the cross-modal retrieval network is obtained by adversarially training a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data in the adversarial network based on sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data that does not match the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality.

[0047] In an exemplary embodiment, the cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network, and the feature extraction module includes:

[0048] an input unit, configured to input the to-be-retrieved data and the candidate data into the cross-modal retrieval network;

[0049] A first feature extraction unit is configured to extract the first feature based on the first generation network;

[0050] The second feature extraction unit is configured to extract the second feature according to the second generation network and the third generation network.

[0051] In an exemplary embodiment, the data retrieval module includes:

[0052] a matching degree determining unit, configured to determine the matching degree between the first feature and the second feature;

[0053] The matching data determination unit is configured to execute candidate data corresponding to a target second feature as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature meets a preset condition.

[0054] According to a third aspect of an embodiment of the present disclosure, a cross-modal retrieval device is provided, including:

[0055] A data acquisition module is configured to acquire data to be retrieved and candidate data; the data to be retrieved and the candidate data correspond to different modalities;

[0056] A feature extraction module, configured to extract a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network;

[0057] A data retrieval module is configured to retrieve data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature;

[0058] Among them, the cross-modal retrieval network is obtained by adversarially training a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data in the adversarial network based on sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data that does not match the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality.

[0059] In an exemplary embodiment, the cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network, and the feature extraction module includes:

[0060] an input unit, configured to input the to-be-retrieved data and the candidate data into the cross-modal retrieval network;

[0061] A first feature extraction unit is configured to extract the first feature based on the first generation network;

[0062] The second feature extraction unit is configured to extract the second feature according to the second generation network and the third generation network.

[0063] In an exemplary embodiment, the data retrieval module includes:

[0064] a matching degree determining unit, configured to determine the matching degree between the first feature and the second feature;

[0065] The matching data determination unit is configured to execute candidate data corresponding to a target second feature as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature meets a preset condition.

[0066] According to a fourth aspect of an embodiment of the present disclosure, a training device for a cross-modal retrieval network is provided, comprising:

[0067] A sample data acquisition module is configured to acquire sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality;

[0068] The sample feature determination module is configured to input the sample data to be retrieved, the associated sample data, and the non-associated sample data into the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network to obtain the sample features of the sample data to be retrieved, the associated sample features of the associated sample data, and the non-associated sample features of the non-associated sample data;

[0069] The cross-modal retrieval network determination module is configured to perform adversarial training on the adversarial network based on the sample features, the associated sample features and the unassociated sample features to obtain a cross-modal retrieval network.

[0070] In an exemplary embodiment, the sample feature determination module includes:

[0071] a sample feature extraction unit, configured to input the sample to-be-retrieved data into the first sample generation network, and extract the sample feature based on the first sample generation network;

[0072] an associated sample feature extraction unit, configured to input the associated sample data into the second sample generation network, and extract the associated sample feature according to the second sample generation network;

[0073] The non-correlated sample feature extraction unit is configured to input the non-correlated sample data into the third sample generation network, and extract the non-correlated sample features based on the third sample generation network.

[0074] In an exemplary embodiment, the cross-modal retrieval network determination module includes:

[0075] A first loss information determining unit is configured to obtain first loss information based on the sample feature, the associated sample feature and the unassociated sample feature;

[0076] A second loss information determining unit is configured to execute a discriminant network that inputs the sample feature, the associated sample feature, and the non-associated sample feature into the adversarial network to obtain second loss information;

[0077] A training unit is configured to train the adversarial network based on the first loss information and the second loss information to obtain a first generation network, a second generation network, and a third generation network; the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data;

[0078] The cross-modal retrieval network generating unit is configured to generate a cross-modal retrieval network according to the first generating network, the second generating network and the third generating network.

[0079] In an exemplary embodiment, the second loss information determining unit includes:

[0080] A sample feature input subunit, configured to input the sample feature, the associated sample feature and the non-associated sample feature into the discriminant network;

[0081] A discrimination result determination subunit is configured to perform discrimination based on the discrimination network on the matching degree between the sample feature and the associated sample feature to obtain a first discrimination result, and discriminate the matching degree between the sample feature and the non-associated sample feature to obtain a second discrimination result;

[0082] The second loss information determination subunit is configured to obtain the second loss information according to the first discrimination result and the second discrimination result.

[0083] In an exemplary embodiment, the second loss information determining subunit includes:

[0084] a logarithm determination submodule, configured to calculate a first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result;

[0085] The second loss information determination submodule is configured to obtain the second loss information according to the first logarithm and the second logarithm.

[0086] In an exemplary embodiment, the number of the sample data to be retrieved is multiple, and the device further includes:

[0087] A first determination module is configured to determine target sample data to be retrieved from a plurality of sample data to be retrieved;

[0088] The second determination module is configured to determine target associated sample data matching the target sample data to be retrieved from the associated sample data, and determine target non-associated sample data not matching the target sample data to be retrieved from the non-associated sample data;

[0089] The second loss information determining unit is configured to input sample features of the target sample to be retrieved data, associated sample features of the target associated sample data, and non-associated sample features of the target non-associated sample data into the discriminant network to obtain the second loss information.

[0090] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0091] processor;

[0092] A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the cross-modal retrieval method or the training method of a cross-modal retrieval network as described in any of the above embodiments.

[0093] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the cross-modal retrieval method or the training method of a cross-modal retrieval network as described in any of the above embodiments.

[0094] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the cross-modal retrieval method or the training method of a cross-modal retrieval network described in any of the above embodiments.

[0095] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0096] The disclosed embodiment is based on sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data that does not match the sample data to be retrieved, and conducts adversarial training on a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data to obtain a cross-modal retrieval network. That is, the disclosed embodiment proposes a multi-modal representation training framework with a three-tower structure based on adversarial training. The cross-modal retrieval network trained using the framework can accurately capture the local information of the input data to be retrieved and the candidate data, and output more effective features. The more effective features are more distinguishable between the same modality and more identifiable between different modalities, thereby improving the fine-grained cross-modal retrieval performance.

[0097] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0099] Figure 1 It is a schematic diagram of an implementation environment of a cross-modal retrieval method or a training method of a cross-modal retrieval network according to an exemplary embodiment.

[0100] Figure 2 The figure is a flowchart of a cross-modal retrieval method according to an exemplary embodiment.

[0101] Figure 3 The figure is a flow chart showing a method of extracting a first feature and a second feature according to an exemplary embodiment.

[0102] Figure 4 The present invention is a flowchart showing a method of retrieving data matching the data to be retrieved from candidate data according to the matching degree of the first feature and the second feature according to an exemplary embodiment.

[0103] Figure 5 The present invention is a flowchart of a method for training a cross-modal retrieval network according to an exemplary embodiment.

[0104] Figure 6 The present invention is a flowchart showing a method of obtaining sample features of sample data to be retrieved, associated sample features of associated sample data, and non-associated sample features of non-associated sample data according to an exemplary embodiment.

[0105] Figure 7 The present invention is a flowchart of a method for training a cross-modal retrieval network according to an exemplary embodiment.

[0106] Figure 8 The present invention is a flowchart of obtaining a cross-modal retrieval network according to an exemplary embodiment.

[0107] Fig. 9 The present invention is a flowchart showing a method of inputting sample features, associated sample features and non-associated sample features into a discriminant network in an adversarial network to obtain second loss information according to an exemplary embodiment.

[0108] Fig.10 The present invention is a flowchart showing a method of determining target sample data to be retrieved, target associated sample data and target non-associated sample data according to an exemplary embodiment.

[0109] Fig.11 It is a block diagram of a cross-modal retrieval device according to an exemplary embodiment.

[0110] Fig.12It is a block diagram of a training device for a cross-modal retrieval network according to an exemplary embodiment.

[0111] Fig.13 It is a block diagram of an electronic device for cross-modal retrieval or training of a cross-modal retrieval network according to an exemplary embodiment. DETAILED DESCRIPTION

[0112] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0113] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0114] Figure 1 is a schematic diagram of an implementation environment of a cross-modal retrieval method or a cross-modal retrieval network training method according to an exemplary embodiment. Figure 1 As shown, the implementation environment may include at least a terminal 01 and a server 02. The terminal 01 and the server 2 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present disclosure.

[0115] Specifically, the terminal can be used to collect data to be retrieved. Optionally, the terminal 01 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart watch, etc., but is not limited thereto.

[0116] Specifically, the server 02 can be used to train a cross-modal retrieval network, and to retrieve data matching the data to be retrieved from the candidate data based on the cross-modal retrieval network. Optionally, the server 02 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0117] It should be noted that Figure 1This is just an example. In other scenarios, other implementation environments may also be included. For example, the implementation environment may include a terminal, a cross-modal retrieval network is obtained through terminal training, and the cross-modal retrieval network is used to retrieve data matching the data to be retrieved from the candidate data based on the cross-modal retrieval network.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0119] In order to better understand the cross-modal detection method provided by the embodiment of the present disclosure, the technical scenarios in which the cross-modal detection method can be applied are first introduced. Specifically, the cross-modal detection method of the embodiment of the present disclosure can be applied in the following scenarios A, B, C, and D, but the cross-modal detection method of the embodiment of the present disclosure is not limited to these three application scenarios. The following is a brief introduction to scenario A, scenario B, scenario C, and scenario D.

[0120] Scenario A:

[0121] Retrieval and matching are performed on text data and video data. Through the cross-modal retrieval method of the embodiment of the present disclosure, the text to be retrieved can be used as a query input to automatically generate videos matching the text to be retrieved and recommend them to the terminal account.

[0122] Scenario B:

[0123] Retrieval and matching are performed on image data and text data. Through the cross-modal retrieval method of the embodiment of the present disclosure, the image to be retrieved can be used as a query input to automatically generate a text description matching the image to be retrieved and recommend it to the terminal account.

[0124] Scenario C:

[0125] Retrieval and matching are performed for image data and voice data. For example, intelligent voice search for images on mobile phones. Through the cross-modal retrieval method of the disclosed embodiment, the voice of the terminal account can be used as a query input, and the image content in the mobile phone album is retrieved and matched to automatically generate an image that matches the voice of the terminal account.

[0126] Scenario D:

[0127] Retrieval and matching of video data and music data. For example, composing music for entertainment short videos / advertising videos. Through the cross-modal retrieval method of the disclosed embodiment, the cross-modal retrieval technology can be used to automatically find the music that best matches the video to be retrieved as a query, and make personalized recommendations, for example, mainly considering the matching of the video's actions and shot transitions with the music beat, etc.

[0128] Figure 2 FIG. 1 is a flow chart of a cross-modal retrieval method according to an exemplary embodiment. Figure 2 As shown, this method is used to Figure 1 In a system including a terminal and a server, the following steps are included.

[0129] In step S11, the data to be retrieved and the candidate data are obtained; the data to be retrieved and the candidate data correspond to different modalities.

[0130] Optionally, the data to be retrieved can be used as a search term to retrieve data matching the data to be retrieved from the candidate data. Exemplarily, the modality of the data to be retrieved includes, but is not limited to, text, video, image, voice, etc. The modality of the candidate data includes, but is not limited to, text, video, image, voice, etc., but it is different from the modality of the data to be retrieved. For example, if the modality of the data to be retrieved is text, then the modality of the candidate data can be video, image, voice, etc. If the modality of the data to be retrieved is image, then the modality of the candidate data can be text, video, voice, etc.

[0131] In one embodiment, in the above step S11, the terminal account may send a data retrieval request to the server, the data retrieval request selects a target modality that can carry the data to be retrieved and the data to be retrieved, and the server may respond to the data retrieval request and obtain candidate data corresponding to the target modality from the database according to the target modality. The server obtains candidate data in response to the data retrieval request sent by the terminal account, which can realize personalized recommendation of the terminal account.

[0132] In another embodiment, in the above step S11, the server can also automatically obtain the historically browsed data of the terminal account, use the historically browsed data as the data to be retrieved, and automatically obtain candidate data with a modality different from the modality of the data to be retrieved from the database. Since the historically browsed data may be data of interest to the terminal account, the server automatically obtains candidate data based on the historically browsed data, which can automatically recommend the data of interest to the terminal account and improve the experience of the terminal account.

[0133] In step S13, the first feature of the data to be retrieved and the second feature of the candidate data are extracted based on the cross-modal retrieval network. The cross-modal retrieval network is obtained by performing adversarial training on the first sample generation network corresponding to the sample data to be retrieved, the second sample generation network corresponding to the associated sample data, and the third sample generation network corresponding to the non-associated sample data in the adversarial network based on the sample data to be retrieved, the associated sample data matching the sample data to be retrieved, and the non-associated sample data not matching the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality.

[0134] In the disclosed embodiment, sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved can be pre-acquired, wherein the modality of the associated sample data and the modality of the non-associated sample data are both different from the modality of the sample data to be retrieved, while the modality of the associated sample data and the modality of the non-associated sample data are the same. The cross-modal retrieval network is obtained by performing adversarial training on the first sample generation network, the second sample generation network, and the third sample generation network based on the sample data to be retrieved, the associated sample data, and the non-associated sample data, wherein the first sample generation network is a generation network that takes the sample data to be retrieved as input, the second sample generation network is a generation network that takes the associated sample data as input, and the third sample generation network is a generation network that takes the non-associated sample data as input.

[0135] Optionally, the cross-modal retrieval network obtained by the above training method includes a first generation network corresponding to the above first sample generation network, a second generation network corresponding to the above second sample generation network, and a third generation network corresponding to the above third sample generation network. The first generation network can be used to extract features of the data to be retrieved, and the second generation network and the third generation network can be used to extract features of candidate data.

[0136] Figure 3 FIG. 1 is a flowchart of extracting a first feature and a second feature according to an exemplary embodiment. Figure 3 As shown, in an optional embodiment, in the above step S13, the above extracting the first feature of the above data to be retrieved and the second feature of the above candidate data based on the cross-modal retrieval network includes:

[0137] In step S131, the data to be retrieved and the candidate data are input into the cross-modal retrieval network.

[0138] In step S133, the first feature is extracted based on the first generation network.

[0139] In step S135, the second feature is extracted according to the second generation network and the third generation network.

[0140] Optionally, after the data to be retrieved is input into the above-mentioned cross-modal retrieval network, the first feature of the data to be retrieved can be extracted through the first generation network, and the second feature of the candidate data can be extracted through the second generation network and the third generation network.

[0141] It should be noted that the second generation network corresponds to the second sample generation network. Since the second sample generation network is a generation network that takes associated sample data as input, the second generation network can be used to extract features of data that matches the data to be retrieved and has a modality different from that of the data to be retrieved. Since the third sample generation network is a generation network that takes non-associated sample data as input, the third generation network can be used to extract features of data that does not match the data to be retrieved and has a modality different from that of the data to be retrieved.

[0142] If a certain candidate data matches the data to be retrieved, the second feature of the candidate data can be extracted through the second generation network, or a more identifiable second feature of the candidate data can be extracted, while the third generation network cannot extract the feature of the candidate data or can only extract a less identifiable second feature of the candidate data.

[0143] Similarly, if a certain candidate data does not match the data to be retrieved, the third generation network can extract the second feature of the candidate data or a more identifiable second feature of the candidate data, while the second generation network cannot extract the second feature of the candidate data or can only extract a less identifiable second feature of the candidate data.

[0144] Assuming that the candidate data includes candidate data 1 and candidate data 2, candidate data 1 matches the data to be retrieved, and candidate data 2 does not match the data to be retrieved, then candidate data 1 and candidate data 2 are input into the second generation network and the third generation network respectively, and the second generation network can extract the second feature with higher recognizability of candidate data 1, but cannot extract the second feature of candidate data 2, or the second generation network extracts the second feature with lower recognizability of candidate data 2, the third generation network can extract the second feature with higher recognizability of candidate data 2, but cannot extract the second feature of candidate data 1, or the second generation network extracts the second feature with lower recognizability of candidate data 1.

[0145] Exemplarily, the first feature or the second feature may be a one-dimensional feature vector, which may also be referred to as a vector representation.

[0146] It can be seen that the cross-modal retrieval network in the embodiment of the present disclosure is a three-tower structure based on the first generation network, the second generation network and the third generation network. For the data to be retrieved, a finer-grained first feature can be extracted. For the candidate data, regardless of whether a certain candidate data matches the data to be retrieved, a more effective second feature can always be extracted through the second generation network and the third generation network. The more effective feature is more distinguishable between the same modality and more identifiable between different modalities, thereby improving the fine-grained cross-modal retrieval performance and accuracy.

[0147] In step S15, data matching the data to be retrieved is retrieved from the candidate data according to the matching degree between the first feature and the second feature.

[0148] In the disclosed embodiment, after obtaining the first feature and the second feature, the matching degree between the first feature and the second feature may be calculated, and data matching the data to be retrieved may be retrieved from the candidate data according to the matching degree calculation result.

[0149] Figure 4 FIG. 1 is a flowchart showing a method of retrieving data matching the data to be retrieved from candidate data according to the matching degree of the first feature and the second feature according to an exemplary embodiment. Figure 4 As shown, in an optional embodiment, in the above step S15, retrieving data matching the data to be retrieved from the candidate data according to the matching degree of the first feature and the second feature may include:

[0150] In step S151, the matching degree between the first feature and the second feature is determined.

[0151] In step S153, the candidate data corresponding to the target second feature is used as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature satisfies a preset condition.

[0152] Optionally, in the above step S151, the present disclosure may adopt a variety of methods to calculate the matching degree between the first feature and the second feature, which are not specifically limited here.

[0153] In one implementation, the cosine similarity between the first feature and the second feature may be calculated to obtain the matching degree.

[0154] In another implementation, the Euclidean distance between the first feature and the second feature may also be calculated to obtain the matching degree.

[0155] In the third embodiment, the Pearson correlation coefficient between the first feature and the second feature may also be calculated, and the matching degree may be characterized by the Pearson correlation coefficient. The larger the Pearson correlation coefficient, the higher the matching degree, and vice versa.

[0156] Optionally, in the above step S153, the present disclosure may use a variety of methods to retrieve data matching the above data to be retrieved from the above candidate data, which are not specifically limited here.

[0157] In one implementation, the second feature with the highest matching degree with the first feature may be used as the target second feature, and the candidate data corresponding to the target second feature may be used as the data matching the data to be retrieved.

[0158] In another embodiment, the second features can also be sorted according to the degree of match with the first feature to obtain a second feature sequence, and the first preset number of second features in the second feature sequence are used as the target second features, and the candidate data corresponding to the target second features are used as data matching the data to be retrieved.

[0159] In the third embodiment, a matching degree threshold may be set, and a second feature whose matching degree with the first feature is greater than the matching degree threshold is used as the target second feature, and candidate data corresponding to the target second feature is used as data matching the data to be retrieved.

[0160] Among them, the data that matches the above-mentioned data to be retrieved means that the retrieved data can match the data to be retrieved. For example, the data to be retrieved is "badminton racket usage skills". If the retrieved result is a video of "xxx brand badminton racket usage skills", then the two are considered to match. If the retrieved result is a video of "xxx brand badminton racket, but not usage skills, but a buying guide", then the two are considered to not match.

[0161] In the disclosed embodiment, the first feature is extracted by the first generating network in the three-tower structure, and the second feature is extracted by the second generating network and the third generating network in the three-tower structure. If the candidate data is data that matches the data to be retrieved, the second generating network can extract a second feature that is more relevant to the first feature, so that the matching degree between the second feature and the first feature is higher. If the candidate data is data that does not match the data to be retrieved, the third generating network can extract a second feature that is less relevant to the first feature, so that the matching degree between the second feature and the first feature is lower. When calculating the matching degree, since the candidate data either has a higher matching degree with the data to be retrieved or a lower matching degree with the data to be retrieved, the data that matches the above-mentioned data to be retrieved can be accurately determined, thereby improving the performance and accuracy of fine-grained cross-modal retrieval.

[0162] Taking the scenario of text retrieval video as an example, the beneficial effects achieved by the embodiment of the present disclosure are described as follows:

[0163] Assume that the data to be retrieved is "tips on using xxx brand badminton racket".

[0164] The candidate data are:

[0165] Video 1: It introduces badminton rackets and usage tips, but it is not xxx brand.

[0166] Video 2: It introduces xxx brand badminton racket, but it does not provide usage tips, but a buying guide.

[0167] Video 3: Introduces the usage techniques of xxx brand badminton racket.

[0168] Since the cross-modal retrieval network of the present invention can output feature vectors with better quality, the same modality is more distinguishable and the different modalities are more identifiable. The result retrieved using the cross-modal retrieval method in the embodiment of the present invention is the above-mentioned video 3. The retrieval result has a high correlation with the data to be retrieved and the retrieval effect is better.

[0169] Since the representation obtained by the related technology lacks fine-grained information, it is unable to distinguish the relevance of the above three videos well, resulting in the retrieved video being video 1 or video 2. The retrieval result has a low correlation with the data to be retrieved, and the retrieval effect is poor.

[0170] Figure 5 FIG. 1 is a flow chart of a method for training a cross-modal retrieval network according to an exemplary embodiment. Figure 5 As shown, it may include:

[0171] In step S21, sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved are obtained.

[0172] Optionally, the associated sample data that matches the above sample data to be retrieved refers to data whose modality is different from that of the sample data to be retrieved and can be matched with the sample data to be retrieved. The non-associated sample data that does not match the above sample data to be retrieved refers to data whose modality is different from that of the sample data to be retrieved and cannot be matched with the sample data to be retrieved. For example, if the sample data to be retrieved is the "Peppa Pig" text, the associated sample data may be the "Peppa Pig Season 1" video, the "Peppa Pig Season 2" video, the "Peppa Pig Season 3" video, etc., and the non-associated sample data may be the "Snow White" video, the "Ultraman" video, etc.

[0173] Exemplarily, a data group consisting of sample data to be retrieved and associated sample data (i.e., related pairs) and a data group consisting of sample data to be retrieved and non-associated sample data (i.e., unrelated pairs) can be prepared respectively. The related pairs and unrelated pairs are called data set A.

[0174] Taking the sample data to be retrieved as text, and the associated sample data and the unassociated sample data as video as an example, the above dataset A is explained as follows:

[0175] Assuming that the sample data to be retrieved includes "Peppa Pig" text and "Ultraman" text, the relevant pairs can be:

[0176] "Peppa Pig" text—"Peppa Pig Season 1" video, "Peppa Pig" text—"Peppa Pig Season 2" video, "Peppa Pig" text—"Peppa Pig Season 3" video, "Ultraman" text—"Ultraman Orb" video, "Ultraman" text—"Ultraman Belia" video, etc.

[0177] Unrelated pairs:

[0178] "Peppa Pig" text—"Ultraman" video, "Peppa Pig" text—"Snow White" video, "Ultraman" text—"Peppa Pig" video, "Ultraman" text—"Bald Qiang" video, etc.

[0179] The above-mentioned related pairs and unrelated pairs are referred to as data set A. It can be seen that data set A may include multiple different texts, as well as associated videos matching each text and unassociated videos not matching each text.

[0180] In step S23, the sample data to be retrieved, the associated sample data, and the unassociated sample data are input into the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network to obtain sample features of the sample data to be retrieved, associated sample features of the associated sample data, and unassociated sample features of the unassociated sample data.

[0181] Optionally, the adversarial network includes a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data. The three sample data can be input into the adversarial network, and feature vectors are extracted from the corresponding sample data through the corresponding generation networks to obtain sample features corresponding to the three sample data.

[0182] Taking the sample data to be retrieved as text, and the associated sample data and the unassociated sample data as video as an example, the first sample generation network can be a Bidirectional Enoceder Representations from Transformers (BERT) model or other models that can extract text features and back-propagation training. The second sample generation network and the third sample generation network can be models that extract video representations and back-propagation such as video recognition networks (C3D), 3D convolutional neural networks (3D Convolutional Neural Networks, 3D CNN).

[0183] Optionally, the adversarial network can be a generative adversarial network (GAN), where GAN is a deep learning model. The model includes at least two modules: one module is a generative model (also referred to as a generative network in the disclosed embodiment), and the other module is a discriminative model (also referred to as a discriminative network in the disclosed embodiment). These two modules learn from each other through game, thereby generating better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, assume that there are two networks, G (generator) and D (discriminator), where G is a network that generates pictures, which receives a random noise z and generates pictures through this noise, recorded as G (z); D is a discriminative network, which is used to determine whether a picture is "real". Its input parameter is a picture, and the output is the probability that the picture is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0184] Exemplarily, the discriminant network can be a model that can be back-propagated, such as a multilayer perceptron neural network (MLP).

[0185] Among them, back propagation can correct the size of the parameters in the initial sample generation network during the training process, so that the reconstruction error loss of the initial sample generation network becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial sample generator are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the target sample generation network, such as the weight matrix.

[0186] Figure 6 The flowchart of obtaining sample features of sample data to be retrieved, associated sample features of associated sample data, and non-associated sample features of non-associated sample data according to an exemplary embodiment is shown. Figure 6 As shown, in an optional embodiment, in the above step S23, the above inputting the sample data to be retrieved, the associated sample data, and the non-associated sample data into the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network to obtain the sample features of the sample data to be retrieved, the associated sample features of the associated sample data, and the non-associated sample features of the non-associated sample data may include:

[0187] In step S231, the sample data to be retrieved is input into the first sample generation network, and the sample features are extracted based on the first sample generation network.

[0188] In step S233, the associated sample data is input into the second sample generation network, and the associated sample features are extracted according to the second sample generation network.

[0189] In step S235, the non-correlated sample data is input into the third sample generation network, and non-correlated sample features are extracted based on the third sample generation network.

[0190] Figure 7 FIG. 1 is a flow chart of a method for training a cross-modal retrieval network according to an exemplary embodiment. Figure 7 As shown, the sample data to be retrieved can be input into the first sample generation network corresponding to it, and the sample features of the sample data to be retrieved can be extracted based on the first sample generation network. The associated sample data can be input into the second sample generation network corresponding to it, and the associated sample features of the associated sample data can be extracted based on the second sample generation network. The non-associated sample data can be input into the third sample generation network corresponding to it, and the non-associated sample features of the non-associated sample data can be extracted based on the third sample generation network.

[0191] Exemplarily, the above sample features, associated sample features, and unassociated sample features may be one-dimensional feature vectors, which may also be referred to as vector representations.

[0192] The disclosed embodiment provides a three-tower structure including a first sample generation network, a second sample generation network and a third sample generation network. For sample data to be retrieved, more fine-grained sample features can be extracted. For associated sample data, more fine-grained associated sample features can be extracted through the associated second sample generation network. For non-associated sample data, more fine-grained non-associated sample features can be extracted through the non-associated third sample generation network. For sample features, associated sample features and non-associated sample features, more identifiable sample features can always be extracted, which can reduce the difficulty of training difficult negative samples, thereby making different modalities more identifiable and the same modalities more distinguishable, thereby improving the training accuracy of cross-modal retrieval.

[0193] In step S25, adversarial training is performed on the adversarial network based on sample features, associated sample features and unassociated sample features to obtain a cross-modal retrieval network.

[0194] In the disclosed embodiment, after obtaining the sample features, the above-mentioned associated sample features and the above-mentioned non-associated sample features, the adversarial network can be trained adversarially based on the three sample features until the adversarial network converges to obtain the cross-modal retrieval network.

[0195] Optionally, the embodiments of the present disclosure may adopt a variety of methods to train the adversarial network, which are not specifically limited here.

[0196] Figure 8 FIG. 1 is a flowchart of obtaining a cross-modal retrieval network according to an exemplary embodiment. Figure 8 As shown, in an optional embodiment, in the above step S25, the adversarial training of the adversarial network based on the sample features, the associated sample features and the unassociated sample features to obtain the cross-modal retrieval network may include:

[0197] In step S251, first loss information is obtained based on the sample features, the associated sample features and the unassociated sample features.

[0198] Optionally, the first loss information may be a contrast loss function. Contrastive loss is a dimensionality reduction learning method that can learn a mapping relationship, which can make points of the same category but far away in high-dimensional space become closer after being mapped to low-dimensional space by a function, and points of different categories but close distances become farther in low-dimensional space after mapping. As a result, in low-dimensional space, points of the same category will produce a clustering effect, and points of different categories will be separated.

[0199] In a feasible implementation manner, in the above step S251, the first loss information L1 can be calculated by the following formula:

[0200]

[0201] Among them, L1 refers to the first loss information, q refers to the sample characteristics, v + refers to the associated sample features, v - refers to the non-correlated sample features, and τ refers to the hyperparameter.

[0202] In another feasible implementation manner, q, v + 、v - Set the corresponding weights a, b, and c, and substitute the corresponding weights into the above formula, that is, use q·a (i.e., the product of q and a) instead of q, and use v + b(i.e. v + The product of v and b) + , use v - ·c(i.e. v - The product of v and c) - .

[0203] In a third feasible implementation, other functions that represent the similarity between vectors may be designed to obtain the first loss information. For example, the first loss information may be obtained by designing the cosine similarity.

[0204] In step S253, the sample features, the associated sample features and the unassociated sample features are input into the discriminant network in the adversarial network to obtain the second loss information.

[0205] Optionally, the embodiment of the present disclosure may obtain the second loss information in a variety of ways, which are not specifically limited herein.

[0206] Fig. 9 FIG. 1 is a flow chart showing a method of inputting sample features, associated sample features and non-associated sample features into a discriminant network in an adversarial network to obtain second loss information according to an exemplary embodiment. Fig. 9 As shown, in an optional embodiment, in the above step S253, the above-mentioned inputting the sample features, the associated sample features and the non-associated sample features into the discriminant network in the adversarial network to obtain the second loss information may include:

[0207] In step S2531, the sample features, associated sample features and unassociated sample features are input into the discriminant network.

[0208] In one approach, continue as Figure 7 As shown, the sample features and associated sample features can be combined in pairs, and the sample features and unassociated sample features can be combined in pairs and input into the discriminant network.

[0209] In another way, the sample features, associated sample features and unassociated sample features can be directly input into the discriminant network.

[0210] In step S2533, based on the discriminant network, the matching degree between the sample feature and the associated sample feature is discriminated to obtain a first discriminant result, and the matching degree between the sample feature and the non-associated sample feature is discriminated to obtain a second discriminant result.

[0211] In step S2535, the second loss information is obtained according to the first discrimination result and the second discrimination result.

[0212] In an exemplary embodiment, in the above step S2535, obtaining the second loss information according to the first discrimination result and the second discrimination result may include:

[0213] A first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result are calculated.

[0214] The second loss information is obtained according to the first logarithm and the second logarithm.

[0215] In one embodiment, in the above steps S2531 to S2535, the second loss information L2 may be calculated using the following formula:

[0216] L2=L(q,v + , v - )=-logD(q,v + )-log(1-D(q,v - ));

[0217] Among them, L2 refers to the second loss information, q refers to the sample characteristics, v + refers to the associated sample features, v - Refers to the non-correlated sample features, D(q,v + ) refers to the first judgment result, D(q,v - ) refers to the second discrimination result, logD(q,v + ) refers to the first logarithm, log(1-D(q,v + )) refers to the second logarithm.

[0218] In another way, it can also be D(q,v + ) and D(q,v - ) respectively set weights c, d, and use D(q,v + )·c(i.e. D(q,v + ) and c) to replace D(q,v + ), using D(q,v- )·d(i.e. D(q,v - ) and d) to replace D(q,v - ).

[0219] In the disclosed embodiment, the second loss information is calculated by the first discrimination result between the sample feature and the above-mentioned associated sample feature and the second discrimination result between the sample feature and the above-mentioned non-associated sample feature, which can fully consider the matching degree between the sample feature and the above-mentioned associated sample feature, as well as the matching degree between the sample feature and the above-mentioned non-associated sample feature, so that the data between different modalities are more identifiable, and the determination accuracy of the second loss information is improved, thereby improving the training accuracy of cross-modal retrieval. In addition, since taking the logarithm not only does not change the nature and correlation of the data, but also compresses the scale of the variable, reduces the absolute value of the data, and facilitates calculation, therefore, the second loss function is calculated by the first logarithm corresponding to the first discrimination result and the second logarithm corresponding to the above-mentioned second discrimination result, which can not only ensure the accuracy of the second loss function, but also improve the calculation efficiency of the second loss function.

[0220] In step S255, the adversarial network is trained based on the first loss information and the second loss information to obtain a first generation network, a second generation network and a third generation network; the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data.

[0221] In step S257, a cross-modal retrieval network is generated according to the first generation network, the second generation network and the third generation network.

[0222] In one embodiment, in the above steps S255-S257, the sum of the first loss information and the second loss information can be used as the total loss information, and the adversarial network can be trained based on the total loss information, and the model can be optimized using the gradient descent algorithm until the adversarial network converges to obtain the first generative network, the above second generative network and the above third generative network, and the first generative network, the above second generative network and the above third generative network are used as the cross-modal retrieval network.

[0223] In another embodiment, in the above steps S255-S257, the first loss information and the second loss information may be weighted to obtain the total loss information, and the weighting formula may be as follows: L=A*L1+B*L2; wherein L is the total loss information, A is the weight of L1, and B is the weight of L2. The adversarial network is trained according to the total loss information, and the model is optimized using a gradient descent algorithm until the adversarial network converges to obtain the first generative network, the above second generative network, and the above third generative network, and the first generative network, the above second generative network, and the above third generative network are used as the cross-modal retrieval network.

[0224] In the disclosed embodiment, adversarial training is performed on the adversarial network by using the first loss information obtained from the sample features, the associated sample features and the unassociated sample features, and the second loss information obtained by inputting the sample features, the associated sample features and the unassociated sample features into the discriminant network in the adversarial network. Since the first loss information can be a contrast loss function, and the contrast loss is a dimensionality reduction learning method, it can make the points of the same category but far away in the high-dimensional space become closer after being mapped to the low-dimensional space through the function, and the points of different categories but close distances become farther in the low-dimensional space after mapping. The second loss information can fully consider the matching degree between the sample features and the associated sample features, as well as the matching degree between the sample features and the unassociated sample features, so that different modalities are more identifiable and the same modalities are more distinguishable. Therefore, by training the adversarial network with the above-mentioned first loss information and the second loss information, the training accuracy of the cross-modal retrieval network can be improved, so that the trained cross-modal retrieval network can output a more effective vector representation.

[0225] Fig.10 FIG. 1 is a flowchart showing a method for determining target sample data to be retrieved, target associated sample data, and target non-associated sample data according to an exemplary embodiment. Fig.10 As shown, in an optional embodiment, the number of the sample data to be retrieved is multiple, and the method may further include:

[0226] In step S31, target sample data to be retrieved is determined from a plurality of sample data to be retrieved.

[0227] In step S33, target associated sample data matching the target sample data to be retrieved is determined from the associated sample data, and target non-associated sample data not matching the target sample data to be retrieved is determined from the non-associated sample data.

[0228] Accordingly, in the step S253, the step of inputting the sample features, the associated sample features, and the unassociated sample features into the discriminant network in the adversarial network to obtain the second loss information may include:

[0229] The sample features of the target sample to be retrieved data, the associated sample features of the target associated sample data and the non-associated sample features of the target non-associated sample data are input into the discriminant network to obtain the second loss information.

[0230] Optionally, the above-mentioned data set A may include multiple different sample data to be retrieved, and each sample data to be retrieved may correspond to multiple associated sample data and multiple non-associated sample data. In steps S31-S33, a target sample data to be retrieved can be determined from the multiple different sample data to be retrieved, and the target associated sample data under the target sample data to be retrieved can be determined from the multiple associated sample data, and the target non-associated sample data under the target sample data to be retrieved can be determined from the multiple non-associated sample data, thereby obtaining a data set B, which is a subset of data set A. By constructing data set B in this way, the reuse rate of data can be improved, costs can be saved, and the burden on the server can be reduced, thereby improving the training speed of the cross-modal retrieval network.

[0231] It should be noted that any sample data to be retrieved in the data set A can be used as the target sample data to be retrieved. For each sample data to be retrieved, a data set B including the sample data to be retrieved, corresponding associated sample data and corresponding non-associated sample data can be generated.

[0232] The following is an introduction to the process of generating dataset B from the above dataset A:

[0233] The related pairs of the same text (i.e. the target sample to be retrieved data) and the associated video (i.e. the target associated sample data):

[0234] "Peppa Pig" text—"Peppa Pig Season 1" video, "Peppa Pig" text—"Peppa Pig Season 2" video, "Peppa Pig" text—"Peppa Pig Season 3" video, etc.

[0235] Unrelated pairs consisting of the same text (i.e., target sample data to be retrieved) and unrelated videos (i.e., target unrelated sample data):

[0236] "Peppa Pig" text—"Ultraman" video, "Peppa Pig" text—"Snow White" video, "Peppa Pig" text—"Bald Qiang" video, etc.

[0237] The above-mentioned related pairs and unrelated pairs are called data set B. It can be seen that data set B contains the same text, related videos matching the same text, and unrelated videos not matching the same text.

[0238] Accordingly, in the above step S253, the sample features of the target sample data to be retrieved in the data set B and the associated features of the target associated sample data can be combined in pairs, and the sample features of the target sample data to be retrieved and the non-associated features of the non-target associated sample data can be combined in pairs, and input into the discriminant network to obtain the second loss information. Among them, the extraction process of the sample features of the target sample data to be retrieved, the associated features of the target associated sample data, and the non-associated features of the non-target associated sample data in the data set B is similar to that of the sample data to be retrieved, and will not be repeated here.

[0239] The disclosed embodiments determine target sample data to be retrieved from multiple sample data to be retrieved, and obtain target-associated sample data and target-unassociated sample data under the target sample data to be retrieved, which can improve the data reuse rate, save costs, and reduce the server burden, thereby improving the training speed of the cross-modal retrieval network; in addition, the target-associated sample data and target-unassociated sample data under the same target sample data to be retrieved are input into the discriminant network to obtain the second loss information, which can reduce the training difficulty of the discriminant network and improve the determination accuracy of the second loss information, thereby improving the training accuracy of the cross-modal retrieval network.

[0240] Of course, in another way, in step S31-step S33, data set B can be collected from a database different from data set A, thereby improving the richness of the data and further improving the training accuracy of the cross-modal retrieval network.

[0241] The disclosed embodiment is based on sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data that does not match the sample data to be retrieved, and conducts adversarial training on a first sample generation network corresponding to the sample data to be retrieved, a second sample generation network corresponding to the associated sample data, and a third sample generation network corresponding to the non-associated sample data to obtain a cross-modal retrieval network. That is, the disclosed embodiment proposes a multi-modal representation training framework with a three-tower structure based on adversarial training, which can extract local characteristics during the training process, making data between the same modality more distinguishable and data between different modalities more recognizable, thereby improving the training accuracy of the cross-modal retrieval network, and then making the trained cross-modal retrieval network able to accurately capture the local information of the input data to be retrieved and candidate data, and output more effective features, making data between the same modality more distinguishable and data between different modalities more recognizable, thereby improving fine-grained retrieval performance. In addition, the training process uses an adversarial network. By inputting sample data into the adversarial network, the cross-modal retrieval network can be trained. That is, the training process is an end-to-end training process, which reduces the complexity of training and reduces the consumption of system resources by the training process.

[0242] Fig.11 is a block diagram of a cross-modal retrieval device according to an exemplary embodiment. Fig.11 The device includes a data acquisition module 41, a feature extraction module 43 and a data retrieval module 45.

[0243] The data acquisition module 41 is configured to acquire the data to be retrieved and the candidate data; the data to be retrieved and the candidate data correspond to different modalities;

[0244] The feature extraction module 43 is configured to extract the first feature of the data to be retrieved and the second feature of the candidate data based on the cross-modal retrieval network.

[0245] The data retrieval module 45 is configured to retrieve data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature.

[0246] Among them, the above-mentioned cross-modal retrieval network is obtained by adversarially training the first sample generation network corresponding to the above-mentioned sample data to be retrieved, the second sample generation network corresponding to the above-mentioned associated sample data, and the third sample generation network corresponding to the above-mentioned non-associated sample data in the adversarial network based on the sample data to be retrieved, the associated sample data matching the above-mentioned sample data to be retrieved, and the non-associated sample data that does not match the above-mentioned sample data to be retrieved; the above-mentioned associated sample data and the above-mentioned sample data to be retrieved correspond to different modalities, and the above-mentioned associated sample data and the above-mentioned non-associated sample data correspond to the same modality.

[0247] In an optional embodiment, the cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network, and the feature extraction module includes:

[0248] The input unit is configured to input the data to be retrieved and the candidate data into the cross-modal retrieval network.

[0249] The first feature extraction unit is configured to extract the first feature based on the first generation network.

[0250] The second feature extraction unit is configured to extract the second feature according to the second generation network and the third generation network.

[0251] In an optional embodiment, the data retrieval module includes:

[0252] The matching degree determining unit is configured to determine the matching degree between the first feature and the second feature.

[0253] The matching data determination unit is configured to execute candidate data corresponding to a target second feature as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature satisfies a preset condition.

[0254] Fig.12 is a block diagram of a training device for a cross-modal retrieval network according to an exemplary embodiment. Fig.12 , including: a sample data acquisition module 51, a sample feature determination module 53, and a cross-modal retrieval network determination module 55.

[0255] The sample data acquisition module 51 is configured to execute acquisition of sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality.

[0256] The sample feature determination module 53 is configured to execute the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network by inputting the above-mentioned sample data to be retrieved, the above-mentioned associated sample data, and the above-mentioned non-associated sample data to obtain the sample features of the above-mentioned sample data to be retrieved, the associated sample features of the above-mentioned associated sample data, and the non-associated sample features of the above-mentioned non-associated sample data.

[0257] The cross-modal retrieval network determination module 55 is configured to perform adversarial training on the adversarial network based on the sample features, the associated sample features and the unassociated sample features to obtain a cross-modal retrieval network.

[0258] In an optional embodiment, the sample feature determination module includes:

[0259] The sample feature extraction unit is configured to input the sample data to be retrieved into the first sample generation network, and extract the sample feature based on the first sample generation network.

[0260] The associated sample feature extraction unit is configured to input the associated sample data into the second sample generation network and extract the associated sample feature according to the second sample generation network.

[0261] The non-correlated sample feature extraction unit is configured to input the non-correlated sample data into the third sample generation network, and extract the non-correlated sample features based on the third sample generation network.

[0262] In an optional embodiment, the cross-modal retrieval network determination module includes:

[0263] The first loss information determining unit is configured to obtain the first loss information based on the sample feature, the associated sample feature and the unassociated sample feature.

[0264] The second loss information determining unit is configured to execute the discriminant network in which the sample features, the associated sample features and the unassociated sample features are input into the adversarial network to obtain the second loss information.

[0265] The training unit is configured to train the adversarial network based on the first loss information and the second loss information to obtain a first generation network, a second generation network and a third generation network; the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data.

[0266] The cross-modal retrieval network generating unit is configured to generate a cross-modal retrieval network according to the first generating network, the second generating network and the third generating network.

[0267] In an optional embodiment, the second loss information determining unit includes:

[0268] The sample feature input subunit is configured to input the sample feature, the associated sample feature and the non-associated sample feature into the discriminant network.

[0269] The discrimination result determination subunit is configured to perform the discrimination network based on the matching degree between the sample feature and the associated sample feature to obtain a first discrimination result, and to discriminate the matching degree between the sample feature and the non-associated sample feature to obtain a second discrimination result.

[0270] The second loss information determination subunit is configured to obtain the second loss information according to the first discrimination result and the second discrimination result.

[0271] In an optional embodiment, the second loss information determining subunit includes:

[0272] The logarithm determination submodule is configured to calculate a first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result.

[0273] The second loss information determination submodule is configured to obtain the second loss information according to the first logarithm and the second logarithm.

[0274] In an optional embodiment, the number of the sample data to be retrieved is multiple, and the device further includes:

[0275] The first determining module is configured to determine target sample data to be retrieved from a plurality of sample data to be retrieved.

[0276] The second determination module is configured to determine target associated sample data matching the target sample data to be retrieved from the associated sample data, and determine target non-associated sample data not matching the target sample data to be retrieved from the non-associated sample data.

[0277] The second loss information determination unit is configured to input the sample features of the target sample to be retrieved data, the associated sample features of the target associated sample data and the non-associated sample features of the target non-associated sample data into the discriminant network to obtain the second loss information.

[0278] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0279] In an exemplary embodiment, an electronic device is also provided, comprising a processor; a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions stored in the memory, the steps of any cross-modal retrieval method or cross-modal retrieval network training method in the above-mentioned embodiments are implemented.

[0280] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.13 It is a block diagram of an electronic device for cross-modal retrieval or training of a cross-modal retrieval network according to an exemplary embodiment. The electronic device 60 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 61 (the central processing unit 61 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 63 for storing data, and one or more storage media 62 (such as one or more mass storage devices) for storing application programs 623 or data 622. Among them, the memory 63 and the storage medium 62 can be short-term storage or persistent storage. The program stored in the storage medium 62 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 61 may be configured to communicate with the storage medium 62 to execute a series of instruction operations in the storage medium 62 on the electronic device 60. The electronic device 60 may also include one or more power supplies 66, one or more wired or wireless network interfaces 65, one or more input and output interfaces 64, and / or one or more operating systems 621, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0281] The input / output interface 64 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 60. In one example, the input / output interface 64 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In an exemplary embodiment, the input / output interface 64 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0282] It can be understood by those skilled in the art that Fig.13 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown.

[0283] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of any cross-modal retrieval method or cross-modal retrieval network training method in the above-mentioned embodiments.

[0284] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the cross-modal retrieval method or the training method of a cross-modal retrieval network provided in any of the above-mentioned embodiments.

[0285] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0286] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0287] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A cross-modal retrieval method, characterized in that: include: Obtaining data to be retrieved and candidate data; The data to be retrieved and the candidate data correspond to different modalities; Extracting a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network; Retrieving data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature; The cross-modal retrieval network is generated by a first generation network, a second generation network and a third generation network obtained by training an adversarial network based on first loss information and second loss information, the first loss information is obtained based on sample features of sample data to be retrieved, associated sample features of associated sample data and non-associated sample features of non-associated sample data, the second loss information is obtained by inputting the sample features, the associated sample features and the non-associated sample features into a discriminant network in the adversarial network, and the sample features, the associated sample features and the non-associated sample features are obtained by inputting the sample data to be retrieved, the associated sample data and the non-associated sample data into the first sample generation network, the second sample generation network and the third sample generation network in the adversarial network; The associated sample data and the sample data to be retrieved correspond to different modalities, the associated sample data and the non-associated sample data correspond to the same modality, the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data.

2. The cross-modal retrieval method according to claim 1, characterized in that: The cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network. The extracting the first feature of the data to be retrieved and the second feature of the candidate data based on the cross-modal retrieval network includes: Inputting the data to be retrieved and the candidate data into the cross-modal retrieval network; extracting the first feature based on the first generative network; The second feature is extracted according to the second generation network and the third generation network.

3. The cross-modal retrieval method according to claim 1 or 2, characterized in that: The retrieving data matching the to-be-searched data from the candidate data according to the matching degree between the first feature and the second feature includes: determining the degree of match between the first feature and the second feature; The candidate data corresponding to the target second feature is used as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature meets a preset condition.

4. A method for training a cross-modal retrieval network, characterized in that: include: Acquire sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved; The associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality; Input the sample data to be retrieved, the associated sample data, and the non-associated sample data into a first sample generation network, a second sample generation network, and a third sample generation network in the adversarial network to obtain sample features of the sample data to be retrieved, associated sample features of the associated sample data, and non-associated sample features of the non-associated sample data; The adversarial network is trained adversarially based on the sample features, the associated sample features and the unassociated sample features to obtain a cross-modal retrieval network, including: obtaining first loss information based on the sample features, the associated sample features and the unassociated sample features; inputting the sample features, the associated sample features and the unassociated sample features into a discriminant network in the adversarial network to obtain second loss information; training the adversarial network based on the first loss information and the second loss information to obtain a first generation network, a second generation network and a third generation network; the first generation network is used to extract features from data to be retrieved, and the second generation network and the third generation network are used to extract features from candidate data; generating a cross-modal retrieval network based on the first generation network, the second generation network and the third generation network.

5. The method for training a cross-modal retrieval network according to claim 4, characterized in that: The step of inputting the sample data to be retrieved, the associated sample data, and the non-associated sample data into the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network to obtain sample features of the sample data to be retrieved, associated sample features of the associated sample data, and non-associated sample features of the non-associated sample data includes: Inputting the sample data to be retrieved into the first sample generation network, and extracting the sample features based on the first sample generation network; Inputting the associated sample data into the second sample generation network, and extracting the associated sample features according to the second sample generation network; The non-associated sample data is input into the third sample generation network, and the non-associated sample features are extracted based on the third sample generation network.

6. The method for training a cross-modal retrieval network according to claim 4, characterized in that: The step of inputting the sample features, the associated sample features, and the unassociated sample features into a discriminant network in the adversarial network to obtain second loss information includes: Inputting the sample features, the associated sample features and the non-associated sample features into the discriminant network; Based on the discriminant network, the matching degree between the sample feature and the associated sample feature is discriminated to obtain a first discriminant result, and the matching degree between the sample feature and the non-associated sample feature is discriminated to obtain a second discriminant result; The second loss information is obtained according to the first discrimination result and the second discrimination result.

7. The method for training a cross-modal retrieval network according to claim 6, characterized in that: The obtaining the second loss information according to the first discrimination result and the second discrimination result includes: Calculating a first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result; The second loss information is obtained according to the first logarithm and the second logarithm.

8. The method for training a cross-modal retrieval network according to claim 4, characterized in that: The number of sample data to be retrieved is multiple, and the method further includes: Determine target sample data to be retrieved from multiple sample data to be retrieved; Determine target associated sample data matching the target sample data to be retrieved from the associated sample data, and determine target non-associated sample data not matching the target sample data to be retrieved from the non-associated sample data; The step of inputting the sample features, the associated sample features, and the unassociated sample features into a discriminant network in the adversarial network to obtain second loss information includes: The sample features of the target sample to-be-retrieved data, the associated sample features of the target associated sample data, and the non-associated sample features of the target non-associated sample data are input into the discriminant network to obtain the second loss information.

9. A cross-modal retrieval device, characterized in that: include: A data acquisition module is configured to acquire the data to be retrieved and the candidate data; The data to be retrieved and the candidate data correspond to different modalities; A feature extraction module, configured to extract a first feature of the data to be retrieved and a second feature of the candidate data based on a cross-modal retrieval network; A data retrieval module is configured to retrieve data matching the data to be retrieved from the candidate data according to the matching degree between the first feature and the second feature; The cross-modal retrieval network is generated by a first generation network, a second generation network and a third generation network obtained by training an adversarial network based on first loss information and second loss information, the first loss information is obtained based on sample features of sample data to be retrieved, associated sample features of associated sample data and non-associated sample features of non-associated sample data, the second loss information is obtained by inputting the sample features, the associated sample features and the non-associated sample features into a discriminant network in the adversarial network, and the sample features, the associated sample features and the non-associated sample features are obtained by inputting the sample data to be retrieved, the associated sample data and the non-associated sample data into the first sample generation network, the second sample generation network and the third sample generation network in the adversarial network; The associated sample data and the sample data to be retrieved correspond to different modalities, the associated sample data and the non-associated sample data correspond to the same modality, the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data.

10. The cross-modal search device according to claim 9, characterized in that: The cross-modal retrieval network includes a first generation network corresponding to the first sample generation network, a second generation network corresponding to the second sample generation network, and a third generation network corresponding to the third sample generation network. The feature extraction module includes: an input unit, configured to input the to-be-retrieved data and the candidate data into the cross-modal retrieval network; A first feature extraction unit is configured to extract the first feature based on the first generation network; The second feature extraction unit is configured to extract the second feature according to the second generation network and the third generation network.

11. The cross-modal search device according to claim 9 or 10, characterized in that: The data retrieval module comprises: a matching degree determining unit, configured to determine the matching degree between the first feature and the second feature; The matching data determination unit is configured to execute candidate data corresponding to a target second feature as the data matching the data to be retrieved; the target second feature represents a second feature whose matching degree with the first feature meets a preset condition.

12. A training device for a cross-modal retrieval network, characterized in that: include: A sample data acquisition module is configured to acquire sample data to be retrieved, associated sample data matching the sample data to be retrieved, and non-associated sample data not matching the sample data to be retrieved; the associated sample data and the sample data to be retrieved correspond to different modalities, and the associated sample data and the non-associated sample data correspond to the same modality; The sample feature determination module is configured to input the sample data to be retrieved, the associated sample data, and the non-associated sample data into the first sample generation network, the second sample generation network, and the third sample generation network in the adversarial network to obtain the sample features of the sample data to be retrieved, the associated sample features of the associated sample data, and the non-associated sample features of the non-associated sample data; A cross-modal retrieval network determination module is configured to perform adversarial training on the adversarial network based on the sample features, the associated sample features and the unassociated sample features to obtain a cross-modal retrieval network; Wherein, the cross-modal retrieval network determination module includes: A first loss information determining unit is configured to obtain first loss information based on the sample feature, the associated sample feature and the unassociated sample feature; A second loss information determining unit is configured to execute a discriminant network that inputs the sample feature, the associated sample feature, and the non-associated sample feature into the adversarial network to obtain second loss information; A training unit is configured to train the adversarial network based on the first loss information and the second loss information to obtain a first generation network, a second generation network, and a third generation network; the first generation network is used to extract features from the data to be retrieved, and the second generation network and the third generation network are used to extract features from the candidate data; The cross-modal retrieval network generating unit is configured to generate a cross-modal retrieval network according to the first generating network, the second generating network and the third generating network.

13. The cross-modal retrieval network training device according to claim 12, characterized in that: The sample feature determination module comprises: a sample feature extraction unit, configured to input the sample to-be-retrieved data into the first sample generation network, and extract the sample feature based on the first sample generation network; an associated sample feature extraction unit, configured to input the associated sample data into the second sample generation network, and extract the associated sample feature according to the second sample generation network; The non-correlated sample feature extraction unit is configured to input the non-correlated sample data into the third sample generation network, and extract the non-correlated sample features based on the third sample generation network.

14. The cross-modal retrieval network training device according to claim 12, characterized in that: The second loss information determining unit includes: A sample feature input subunit, configured to input the sample feature, the associated sample feature and the non-associated sample feature into the discriminant network; A discrimination result determination subunit is configured to perform discrimination based on the discrimination network on the matching degree between the sample feature and the associated sample feature to obtain a first discrimination result, and discriminate the matching degree between the sample feature and the non-associated sample feature to obtain a second discrimination result; The second loss information determination subunit is configured to obtain the second loss information according to the first discrimination result and the second discrimination result.

15. The cross-modal retrieval network training device according to claim 14, characterized in that: The second loss information determining subunit includes: a logarithm determination submodule, configured to calculate a first logarithm corresponding to the first discrimination result and a second logarithm corresponding to the second discrimination result; The second loss information determination submodule is configured to obtain the second loss information according to the first logarithm and the second logarithm.

16. The cross-modal retrieval network training device according to claim 12, characterized in that: The number of the sample data to be retrieved is multiple, and the device further includes: A first determination module is configured to determine target sample data to be retrieved from a plurality of sample data to be retrieved; The second determination module is configured to determine target associated sample data matching the target sample data to be retrieved from the associated sample data, and determine target non-associated sample data not matching the target sample data to be retrieved from the non-associated sample data; The second loss information determining unit is configured to input sample features of the target sample to be retrieved data, associated sample features of the target associated sample data, and non-associated sample features of the target non-associated sample data into the discriminant network to obtain the second loss information.

17. An electronic device, characterized in that: include: processor; A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the cross-modal retrieval method as described in any one of claims 1 to 3 or the training method of a cross-modal retrieval network as described in any one of claims 4 to 8.

18. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the cross-modal retrieval method as described in any one of claims 1 to 3 or the training method of a cross-modal retrieval network as described in any one of claims 4 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the cross-modal retrieval method described in any one of claims 1 to 3 or the training method of a cross-modal retrieval network described in any one of claims 4 to 8.

Images