A training method and device of a cross-modal retrieval model, equipment and a storage medium

Abstract

The application provides a training method and device of a cross-modal retrieval model, equipment and a storage medium. The method comprises the following steps: inputting first modal data and second modal data of each sample object into an initial model respectively to obtain first modal features of each sample object under the first modal data and second modal features of each sample object under the second modal data; and adjusting model parameters of the initial model according to a first data feature difference of the same sample object under different modal data, a second data feature difference of the same sample object in a plurality of sample objects under different modal data, a third data feature difference of different sample objects in the plurality of sample objects under different modal data, and a group feature difference of the plurality of sample objects under different modal data until convergence is achieved. In this way, the cross-modal retrieval flexibility of the cross-modal retrieval model can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of model training technology, and more specifically, to a training method, apparatus, device, and storage medium for a cross-modal retrieval model. Background Technology

[0002] Cross-modal retrieval is a retrieval scheme that uses the association between different modal samples of the same retrieval object to retrieve samples of another different modality of the retrieval object based on a certain modal sample of the retrieval object.

[0003] Currently, existing cross-modal retrieval models typically use data from one fixed modality as query data and data from another fixed modality as index data to train the model to perform cross-modal retrieval of the target object between the two fixed modalities. For example, existing cross-modal retrieval models generally use the text information of the target object as query data and retrieve target images from the image information stored in the retrieval database that have a similarity higher than a threshold with the text information of the target object as the cross-modal retrieval result between text and image. Therefore, existing cross-modal retrieval models can only achieve the effect of using one modality of data to retrieve data from another modality, both during the model training and application phases. That is, the data types of query data and index data used in actual retrieval cannot be interchanged, resulting in low flexibility in cross-modal retrieval. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a training method, apparatus, device and storage medium for a cross-modal retrieval model, so as to solve the problem that the data types of query data and index data that can be used in actual retrieval in the prior art cannot be interchanged, thereby effectively improving the cross-modal retrieval flexibility of the cross-modal retrieval model.

[0005] In a first aspect, embodiments of this application provide a training method for a cross-modal retrieval model, the training method comprising:

[0006] For multiple sample objects included in the same batch of training samples, the first modal data of each sample object is input into the first feature extraction sub-network in the initial model to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network.

[0007] The second modal data of each sample object is input into the second feature extraction sub-network within the initial model to obtain the second modal features of each sample object under the second modal data, as output by the second feature extraction sub-network.

[0008] Based on the first modal features and the second modal features of each sample object, the differences in the first data features exhibited by the same sample object under different modal data, the differences in the second data features exhibited by similar sample objects under different modal data, the differences in the third data features exhibited by dissimilar sample objects under different modal data, and the differences in the group features exhibited by the multiple sample objects under different modal data are determined.

[0009] Based on the differences in the first data features, the second data features, the third data features, and the group features, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0010] In one optional implementation, adjusting the model parameters of the initial model based on the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference includes:

[0011] For each sample object, based on the differences in the first data features exhibited by the sample object under different modal data, the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs is determined;

[0012] For each sample object, based on the differences in second data features exhibited by the sample object and the first sample object under different modal data, and the differences in third data features exhibited by the sample object and the second sample object under different modal data, the cross-modal triplet loss between the sample object and similar sample objects under different modal data, as well as dissimilar sample objects under different modal data, is determined; wherein, the first sample object is used to characterize the sample object belonging to the same entity type as the sample object among the plurality of sample objects; the second sample object is used to characterize the sample object belonging to a different entity type than the sample object among the plurality of sample objects;

[0013] For the plurality of sample objects, based on the first group feature differences exhibited by the plurality of sample objects under the first modal data and the second group feature differences exhibited by the plurality of sample objects under the second modal data, the mutual learning loss between the first group feature differences and the second group feature differences is determined;

[0014] The model parameters of the initial model are adjusted based on the classification prediction loss for each sample object, the cross-modal triplet loss for each sample object, and the mutual learning loss for the multiple sample objects, until the initial model converges.

[0015] In one optional implementation, determining the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs, based on the differences in the first data features exhibited by the sample object under different modal data, includes:

[0016] Based on the first modal features of the sample object output by the first feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the first classification prediction result of the sample object.

[0017] Based on the second modal features of the sample object output by the second feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the second classification prediction result of the sample object;

[0018] Based on the first classification loss between the first classification prediction result and the true classification result, and the second classification loss between the second classification prediction result and the true classification result, the sum of the first classification loss and the second classification loss is used as the classification prediction loss corresponding to the sample object; wherein, the true classification result is used to characterize the entity type to which the sample object belongs.

[0019] In one optional implementation, the difference in second data features between the sample object and the first sample object under different modal data is determined by the following method:

[0020] For the sample object, based on the entity type to which the sample object belongs, one sample object of the same entity type as the sample object is randomly selected from the plurality of sample objects as the first sample object;

[0021] Based on the first modal features of the sample object output by the first feature extraction subnetwork and the second modal features of the first sample object output by the second feature extraction subnetwork, the feature distance between the first modal features of the sample object and the second modal features of the first sample object is calculated to obtain the first similar feature distance between the sample object and the first sample object.

[0022] Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the first sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the first sample object is calculated to obtain the second similar feature distance between the sample object and the first sample object.

[0023] The first similar feature distance and the second similar feature distance are used as the second data feature differences between the sample object and the first sample object under different modal data.

[0024] In one optional implementation, the differences in third data features exhibited by the sample object and the second sample object under different modal data are determined by the following method:

[0025] For the sample object, based on the entity type to which the sample object belongs, each sample object belonging to a different entity type from the plurality of sample objects is determined as the target sample object;

[0026] For each target sample object, based on the first modal features of the sample object output by the first feature extraction sub-network and the second modal features of the target sample object output by the second feature extraction sub-network, the feature distance between the first modal features of the sample object and the second modal features of the target sample object is calculated to obtain the third feature distance between the sample object and the target sample object.

[0027] Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the target sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the target sample object is calculated to obtain the fourth feature distance between the sample object and the target sample object.

[0028] The minimum value among the third feature distances between the sample object and each of the target sample objects is taken as the first outlier feature distance, and the minimum value among the fourth feature distances between the sample object and each of the target sample objects is taken as the second outlier feature distance. The first outlier feature distance and the second outlier feature distance are taken as the third data feature difference between the sample object and the second sample object under different modal data.

[0029] In one optional implementation, determining the cross-modal triplet loss between the sample object and similar sample objects under different modal data, and between the sample object and the first sample object and a third sample object under different modal data, based on the differences in second data features exhibited by the sample object and the first sample object under different modal data, and the differences in third data features exhibited by the sample object and the second sample object under different modal data, includes:

[0030] Calculate the first sum between the first preset boundary threshold and the first distance of the same type of feature, and take the difference between the first sum and the first distance of the different type of feature as the first target difference;

[0031] Calculate the second sum between the second preset boundary threshold and the distance of the second similar feature, and take the difference between the second sum and the distance of the second dissimilar feature as the second target difference;

[0032] The maximum value among the first target difference and the specified parameters is used as the first cross-modal loss of the sample object under the first modal data, and the maximum value among the second target difference and the specified parameters is used as the second cross-modal loss of the sample object under the second modal data. The sum of the first cross-modal loss and the second cross-modal loss is used as the cross-modal triplet loss corresponding to the sample object.

[0033] In one optional implementation, the first group characteristic differences exhibited by the plurality of sample objects under the first modality data and the second group characteristic differences exhibited by the plurality of sample objects under the second modality data are determined by the following method:

[0034] For the plurality of sample objects, the first classification prediction result of each sample object is used as a vertex, and the first modal feature of each sample object is used as the vertex attribute of each vertex to construct the first adjacency matrix of the plurality of sample objects under the first modal data.

[0035] Using the second classification prediction result of each sample object as a vertex and the second modal feature of each sample object as the vertex attribute of each vertex, a second adjacency matrix of the multiple sample objects under the second modal data is constructed.

[0036] The graph structure features exhibited by the first adjacency matrix under the graph structure are used as the differences in the first group features exhibited by the multiple sample objects under the first modal data.

[0037] The graph structure features exhibited by the second adjacency matrix under the graph structure are used as the differences in the second group features exhibited by the multiple sample objects under the second modal data.

[0038] In one optional implementation, when the first modal data and the second modal data belong to different types of data, it is determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a non-parameter-sharing manner during the model training process of the cross-modal retrieval model; when the first modal data and the second modal data belong to the same type of data with different data parameters, it is determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a parameter-sharing manner during the model training process of the cross-modal retrieval model.

[0039] Secondly, embodiments of this application provide a cross-modal retrieval method, the cross-modal retrieval method comprising:

[0040] First sample data of multiple search objects are input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model extracts features from the data characteristics exhibited by each search object under the first modal data to obtain the first modal features of each search object. The first sample data includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object. The first search object and the search object belong to the same entity type, while the second search object and the search object belong to different entity types.

[0041] The second modal data of the target object is input into the cross-modal retrieval model. The cross-modal retrieval model is used to extract the data features of the target object under the second modal data to obtain the second modal features of the target object.

[0042] Using the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature, the cross-modal retrieval model calculates the similarity between the first retrieval data feature and each first index data feature, and outputs the retrieval object whose similarity calculation result meets the retrieval display conditions as the cross-modal retrieval result for the target object.

[0043] Thirdly, embodiments of this application provide a training apparatus for a cross-modal retrieval model, the training apparatus comprising:

[0044] The first extraction module is used to input the first modal data of each sample object into the first feature extraction sub-network in the initial model for multiple sample objects included in the same batch of training samples, so as to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network.

[0045] The second extraction module is used to input the second modal data of each sample object into the second feature extraction sub-network in the initial model, so as to obtain the second modal features of each sample object under the second modal data output by the second feature extraction sub-network;

[0046] The first determining module is used to determine, based on the first modal features of each sample object and the second modal features of each sample object, the differences in first data features exhibited by the same sample object under different modal data, the differences in second data features exhibited by similar sample objects under different modal data, the differences in third data features exhibited by dissimilar sample objects under different modal data, and the differences in group features exhibited by the multiple sample objects under different modal data.

[0047] The parameter adjustment module is used to adjust the model parameters of the initial model based on the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference, so as to obtain a cross-modal retrieval model including the adjusted model parameters.

[0048] Fourthly, embodiments of this application provide a cross-modal retrieval device, the cross-modal retrieval device comprising:

[0049] The first index extraction module is used to input first sample data of multiple search objects into a pre-trained cross-modal retrieval model, and to extract features from the data features exhibited by each search object under the first modal data through the cross-modal retrieval model to obtain the first modal features of each search object; wherein, the first sample data includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object; the first search object and the search object belong to the same entity type, and the second search object and the search object belong to different entity types;

[0050] The first query extraction module is used to input the second modal data of the target object into the cross-modal retrieval model, and extract the data features of the target object under the second modal data through the cross-modal retrieval model to obtain the second modal features of the target object;

[0051] The first retrieval module is used to use the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature. The module calculates the similarity between the first retrieval data feature and each first index data feature through the cross-modal retrieval model, and outputs the retrieval objects whose similarity calculation results meet the retrieval display conditions as cross-modal retrieval results for the target object.

[0052] Fifthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the training method for any of the cross-modal retrieval models described above, or executes the steps of any of the cross-modal retrieval methods described above.

[0053] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs steps of the training method for any of the cross-modal retrieval models described above, or performs steps of any of the cross-modal retrieval methods described above.

[0054] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0055] This application provides a training method, apparatus, device, and storage medium for a cross-modal retrieval model. For multiple sample objects included in the same batch of training samples, the first modality data of each sample object is input into a first feature extraction sub-network within an initial model to obtain the first modality features of each sample object under the first modality data, output by the first feature extraction sub-network. The second modality data of each sample object is input into a second feature extraction sub-network within the initial model to obtain the second modality features of each sample object under the second modality data, output by the second feature extraction sub-network. Based on the first modality features and second modality features of each sample object, the differences in first data features exhibited by the same sample object under different modality data, the differences in second data features exhibited by similar sample objects under different modality data, the differences in third data features exhibited by dissimilar sample objects under different modality data, and the differences in group features exhibited by multiple sample objects under different modality data are determined. Based on the differences in first data features, second data features, third data features, and group features, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0056] In this way, this application solves the problem that the data types of query data and index data that can be used in actual retrieval cannot be interchanged by training the model to learn from the data features of similar sample objects and dissimilar sample objects under different modal data. This effectively improves the cross-modal retrieval flexibility of the cross-modal retrieval model. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 The diagram illustrates a flowchart of a training method for a cross-modal retrieval model provided in an embodiment of this application.

[0059] Figure 2 The illustration shows a flowchart of a method for calculating the classification prediction loss of the same sample object under different modal data, provided in an embodiment of this application.

[0060] Figure 3 The illustration shows a flowchart of a method for determining the difference in second data features between a sample object and a first sample object under different modal data, according to an embodiment of this application.

[0061] Figure 4 The illustration shows a flowchart of a method for determining the differences in third data features between a sample object and a second sample object under different modal data, according to an embodiment of this application.

[0062] Figure 5 A flowchart illustrating a method for calculating cross-modal triplet loss provided in an embodiment of this application is shown.

[0063] Figure 6 A flowchart illustrating a method for determining the characteristic differences of a first group and the characteristic differences of a second group, provided in an embodiment of this application, is shown.

[0064] Figure 7 A flowchart illustrating the first cross-modal retrieval method provided in an embodiment of this application is shown;

[0065] Figure 8 A flowchart illustrating the second cross-modal retrieval method provided in an embodiment of this application is shown;

[0066] Figure 9 This paper shows a schematic diagram of the structure of a training device for a cross-modal retrieval model provided in an embodiment of this application;

[0067] Figure 10 This illustration shows a schematic diagram of the structure of a cross-modal retrieval device provided in an embodiment of this application;

[0068] Figure 11 This is a schematic diagram of the structure of a computer device 1100 provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0070] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0071] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0072] Currently, existing cross-modal retrieval models typically use data from one fixed modality as query data and data from another fixed modality as index data to train the model to perform cross-modal retrieval of the target object between the two fixed modalities. For example, existing cross-modal retrieval models generally use the text information of the target object as query data and retrieve target images from the image information stored in the retrieval database that have a similarity higher than a threshold with the text information of the target object as the cross-modal retrieval result between text and image. Therefore, existing cross-modal retrieval models can only achieve the effect of using one modality of data to retrieve data from another modality, both during the model training and application phases. That is, the data types of query data and index data used in actual retrieval cannot be interchanged, resulting in low flexibility in cross-modal retrieval.

[0073] Based on this, embodiments of this application provide a training method, apparatus, device, and storage medium for a cross-modal retrieval model. For multiple sample objects included in the same batch of training samples, the first modal data of each sample object is input into a first feature extraction sub-network within an initial model to obtain the first modal features of each sample object under the first modal data, output by the first feature extraction sub-network. The second modal data of each sample object is input into a second feature extraction sub-network within the initial model to obtain the second modal features of each sample object under the second modal data, output by the second feature extraction sub-network. Based on the first modal features and second modal features of each sample object, the differences in first data features exhibited by the same sample object under different modal data, the differences in second data features exhibited by similar sample objects under different modal data, the differences in third data features exhibited by dissimilar sample objects under different modal data, and the differences in group features exhibited by multiple sample objects under different modal data are determined. Based on the differences in first data features, second data features, third data features, and group features, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0074] In this way, this application solves the problem that the data types of query data and index data that can be used in actual retrieval cannot be interchanged by training the model to learn from the data features of similar sample objects and dissimilar sample objects under different modal data. This effectively improves the cross-modal retrieval flexibility of the cross-modal retrieval model.

[0075] The following provides a detailed description of a training method, apparatus, device, and storage medium for a cross-modal retrieval model provided in the embodiments of this application.

[0076] Reference Figure 1 As shown, Figure 1 The diagram illustrates a flowchart of a training method for a cross-modal retrieval model provided in an embodiment of this application. The training method includes steps S101-S104; specifically:

[0077] S101, for multiple sample objects included in the same batch of training samples, the first modal data of each sample object is input into the first feature extraction sub-network in the initial model to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network.

[0078] It should be noted that this application embodiment does not impose any limitation on the number of sample objects included in the same batch of training samples (that is, the number of samples in the same batch of training samples).

[0079] Here, the specific entity type to which the above sample objects belong can be determined based on the actual application scenario of the cross-modal retrieval model; this application embodiment does not impose any limitation on the specific entity type to which the above sample objects belong.

[0080] Specifically, when a trained cross-modal retrieval model is used in a search engine, since the search engine does not have a specific requirement for limiting the scope of entity types for the search objects, various types of entity objects can be obtained as the sample objects mentioned above. For example, different types of entity objects such as animals, plants, household items (e.g., tables, chairs, computers, mobile phones, etc.), and vehicles (e.g., cars, ships, trains, etc.) can be obtained as the sample objects mentioned above.

[0081] Specifically, when a trained cross-modal retrieval model is used in retrieval scenarios such as vehicle retrieval / pedestrian re-identification within a certain type of entity object, the aforementioned sample objects can be selected from the range of certain types of entity objects that may appear in the retrieval scenario. For example, taking vehicle retrieval as an example, various types of vehicles (such as cars, motorcycles, bicycles, etc.) can be obtained as the aforementioned sample objects.

[0082] Here, the first modal data mentioned above can be text data of the sample object, image data of the sample object, or audio data of the sample object. This application embodiment does not limit the specific data type of the first modal data mentioned above.

[0083] It should be noted that the first feature extraction sub-network in the initial model is used to extract features from the first modality data of each sample object in the same batch of training samples. That is, the specific network structure of the first feature extraction sub-network is determined according to the data type of the first modality data. This application embodiment does not limit the specific network structure of the first feature extraction sub-network.

[0084] For example, taking the image data of the sample object as the first modality data, as an optional embodiment, and taking the first feature extraction sub-network as containing 7 convolutional layers, the specific structure of the first feature extraction sub-network can be: 1st convolutional layer → max pooling layer → 2nd convolutional layer → 3rd convolutional layer → 4th convolutional layer → 5th convolutional layer → 6th convolutional layer → 7th convolutional layer; wherein, the input of the 2nd convolutional layer is residually connected to the output of the 4th convolutional layer; The input of the 5th convolutional layer is concatenated with the output residual of the 7th convolutional layer; the kernel size of the 1st convolutional layer is 7×7, the number of kernels is 64, and the stride is 2; the pooling window size of the max pooling layer is 3×3, and the stride is 2; the kernel size of the 2nd, 4th, 5th, and 7th convolutional layers is 1×1, the number of kernels is 64, 256, 128, and 512 respectively, and the stride is 1; the kernel size of the 3rd and 6th convolutional layers is 3×3, the number of kernels is 128 and 256 respectively, and the stride is 2.

[0085] S102, the second modal data of each sample object is input into the second feature extraction sub-network in the initial model to obtain the second modal features of each sample object under the second modal data output by the second feature extraction sub-network.

[0086] Here, the second modal data is used to represent data different from the first modal data. As an optional embodiment, the second modal data can be data of a different type than the first modal data; for example, the first modal data is text data, and the second modal data is image data. As another optional embodiment, the second modal data can also be data of the same type but with different data parameters than the first modal data; for example, the first modal data is an RGB image containing parameter information for three color channels, while the second modal data is an Infrared image containing only single-channel parameters. Therefore, this application embodiment does not limit the specific data type of the second modal data.

[0087] It should be noted that, similar to the first feature extraction sub-network described above, the second feature extraction sub-network in the initial model is used to extract features from the second modality data of each sample object in the same batch of training samples. That is, the specific network structure of the second feature extraction sub-network is determined according to the data type of the second modality data. This application embodiment does not impose any limitations on the specific network structure of the second feature extraction sub-network.

[0088] Specifically, taking the first modality data as belonging to an RGB image and the second modality data as belonging to an Infrared image as an example, the specific structure of the second feature extraction sub-network can be the same as the specific structure of the first feature extraction sub-network given in the example content of step S101 above. The repetition will not be repeated here.

[0089] S103, based on the first modal features and the second modal features of each sample object, determine the first data feature differences exhibited by the same sample object under different modal data, the second data feature differences exhibited by similar sample objects under different modal data, the third data feature differences exhibited by dissimilar sample objects under different modal data, and the group feature differences exhibited by the multiple sample objects under different modal data.

[0090] Here, based on the implementation of the above steps S101-S102, the first modal feature and the second modal feature of each sample object in the same batch of multiple sample objects can be obtained. Based on this, for each sample object, the first data feature difference can be determined according to the feature difference between the first modal feature and the second modal feature of the sample object.

[0091] Here, for each sample object, the second data feature difference can be determined based on the feature difference between the first modal feature of the sample object and the second modal feature of similar sample objects (i.e., sample objects of the same entity type as the sample object among multiple sample objects), and the feature difference between the second modal feature of the sample object and the first modal feature of the aforementioned similar sample objects.

[0092] Here, for each sample object, the third data feature difference can be determined based on the feature difference between the first modal feature of the sample object and the second modal feature of the heterogeneous sample object (i.e., a sample object of a different entity type than the sample object among multiple sample objects), and the feature difference between the second modal feature of the sample object and the first modal feature of the heterogeneous sample object.

[0093] Here, for the same batch of training samples (i.e., the multiple sample objects mentioned above), the difference in the first group characteristics of the multiple sample objects under the first modal data, which is composed of the first modal features of the multiple sample objects, and the difference in the second group characteristics of the multiple sample objects under the second modal data, which is composed of the second modal features of the multiple sample objects, can be used to determine the above-mentioned group characteristic differences.

[0094] It should be noted that, similar to the specific entity types to which the sample objects belong, the specific classification methods for the same type of sample objects and different type of sample objects can also be determined according to the actual application scenario of the cross-modal retrieval model; the embodiments of this application do not impose any limitations on the specific classification methods for the same type of sample objects and different type of sample objects.

[0095] Specifically, when a trained cross-modal retrieval model is used in a search engine, since the search engine does not have a specific requirement for limiting the scope of entity types for the search objects, the sample objects can be classified based on a coarse-grained hierarchical division. For example, sample objects such as cars, ships, and trains can be classified as similar sample objects under the entity type of "transportation", while sample objects of the animal class and other sample objects with low correlation, such as plants, can be classified as dissimilar sample objects.

[0096] Specifically, when a well-trained cross-modal retrieval model is used in retrieval scenarios within a certain type of entity object, such as vehicle retrieval / pedestrian re-identification, the model can classify whether sample objects belong to the same category based on fine-grained hierarchical classification. For example, taking vehicle retrieval as an example, private cars of different brands / models can be classified as the same type of sample object under the entity type "sedan", while sample objects of the sedan category can be classified as different types of sample objects from other types of vehicles such as motorcycles and bicycles.

[0097] S104, based on the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0098] Here, in this embodiment of the application, as an optional embodiment, step S104 can be performed according to the steps shown in steps a1-a4, specifically:

[0099] Step a1: For each sample object, based on the differences in the first data features exhibited by the sample object under different modal data, determine the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs.

[0100] Specifically, for an explanation of the first data feature difference in step a1 above, please refer to the relevant explanation in S103 above. Repeated points will not be repeated here.

[0101] Step a2: For each sample object, based on the difference in second data features between the sample object and the first sample object under different modal data, and the difference in third data features between the sample object and the second sample object under different modal data, determine the cross-modal triplet loss between the sample object and similar sample objects under different modal data, as well as dissimilar sample objects under different modal data.

[0102] Here, the first sample object is used to characterize a sample object that belongs to the same entity type as the sample object among the plurality of sample objects; the second sample object is used to characterize a sample object that belongs to a different entity type than the sample object among the plurality of sample objects.

[0103] Specifically, the explanation of the differences in the second data features in step a2 above, as well as the specific classification methods for similar and dissimilar sample objects, can be found in the relevant explanations in S103 above. Repeated points will not be repeated here.

[0104] Step a3: For the multiple sample objects, based on the first group feature differences exhibited by the multiple sample objects under the first modal data and the second group feature differences exhibited by the multiple sample objects under the second modal data, determine the mutual learning loss between the first group feature differences and the second group feature differences.

[0105] Specifically, for the explanation of the differences in characteristics of the first group and the second group in step a3 above, please refer to the relevant explanation in S103 above. The repeated parts will not be repeated here.

[0106] Step a4: Adjust the model parameters of the initial model according to the classification prediction loss corresponding to each sample object, the cross-modal triplet loss corresponding to each sample object, and the mutual learning loss corresponding to the multiple sample objects, until the initial model converges.

[0107] Specifically, for training samples in the same batch, the sum of the classification prediction loss for each sample object, the cross-modal triplet loss for each sample object, and the mutual learning loss for multiple sample objects can be used as the model loss of the initial model. During the training process of the initial model, the model parameters of the initial model (such as the weight coefficients of each feature extraction sub-network) are adjusted based on the above-mentioned loss sum until the initial model converges (such as when the above-mentioned loss sum reaches its minimum). The converged initial model is then used as the trained cross-modal retrieval model.

[0108] Based on this, during the initial model training process, based on the first data feature differences exhibited by the same sample object under different modal data, the initial model can be trained to learn how to distinguish the features of the same sample object under different modalities; based on the aforementioned second and third data feature differences, the initial model can be trained to learn how to narrow the feature distance between different modal features of similar sample objects and how to widen the feature distance between different modal features of dissimilar sample objects; based on the aforementioned group feature differences, the initial model can be trained to learn the overall feature differences exhibited by multiple sample objects under different modal data; thus, this embodiment of the application solves the problem in the prior art that the data types of query data and index data that can be used in actual retrieval cannot be interchanged by training the model to learn from each other the data features exhibited by similar and dissimilar sample objects under different modal data, thereby effectively improving the cross-modal retrieval flexibility of the cross-modal retrieval model.

[0109] Here, it should be noted regarding steps S101-S104 above: In this embodiment of the application, the initial model may include multiple different feature extraction sub-networks, wherein each feature extraction sub-network is used to extract one modality of data of the sample object; that is, the number of feature extraction sub-networks included in the initial model can be determined according to the different types of modal data that the user needs to use in cross-modal retrieval; the above steps S101-S104 are only specific descriptions using two different modal data as examples, and this embodiment of the application does not limit the number of feature extraction sub-networks included in the initial model.

[0110] The specific implementation process of each of the above steps in the embodiments of this application will be described in detail below:

[0111] Regarding the specific implementation process of steps S101-S102 above, and considering the two possible data partitioning methods between the second modal data and the first modal data given in step S102, the specific model structure of the initial model can be set in the following two ways:

[0112] 1. When the first modal data and the second modal data belong to different types of data, in this case, as an optional embodiment, it can be determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a non-shared parameter manner during the model training process of the cross-modal retrieval model.

[0113] Specifically, when the first modality data and the second modality data belong to different types of data (e.g., the first modality data is text data, and the second modality data is image data), the focus of the first feature extraction sub-network when extracting features of the first modality (e.g., text features) is different from the focus of the second feature extraction sub-network when extracting features of the second modality (e.g., image features) (equivalent to the difference between the focus points exceeding a certain threshold). Based on this, when setting the specific model structure of the above initial model, it can be set as an independent structure for the first and second feature extraction sub-networks, so that the first and second feature extraction sub-networks are trained in a non-shared parameter manner during the model training process of the cross-modal retrieval model.

[0114] 2. When the first modal data and the second modal data belong to the same type of data with different data parameters, in this case, as another optional embodiment, it can be determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in the manner of parameter sharing during the model training process of the cross-modal retrieval model.

[0115] Specifically, when the first modality data and the second modality data belong to the same type of data with different data parameters (e.g., the first modality data is an RGB image, and the second modality data is an Infrared image), the focus points of the first feature extraction subnetwork when extracting features of the first modality (e.g., image features of an RGB image) are similar to the focus points of the second feature extraction subnetwork when extracting features of the second modality (e.g., image features of an Infrared image) (equivalent to the difference between the focus points being less than a certain threshold). Based on this, when setting the specific model structure of the above initial model, the first feature extraction subnetwork and the second feature extraction subnetwork can be concatenated with a shared parameter subnetwork, so that the first feature extraction subnetwork and the second feature extraction subnetwork can be trained in a parameter-sharing manner during the model training process of the cross-modal retrieval model.

[0116] For example, taking a first modality data belonging to an RGB image and a second modality data belonging to an Infrared image, if the specific structures of the first feature extraction subnetwork and the second feature extraction subnetwork are the same as the specific structure of the first feature extraction subnetwork given in the example of step S101 above, then as an optional embodiment, the shared parameter subnetwork connected in series with the first feature extraction subnetwork and the second feature extraction subnetwork can include: multiple convolutional layers arranged in series and an attention module, wherein the attention module includes a feature sampling layer, a normalization layer and multiple convolutional layers arranged in series. Taking the specific number of the multiple convolutional layers arranged in series as 6 as an example, then the shared parameter subnetwork... The specific structure of the parameter subnetwork can be: Convolutional Layer 1 → Convolutional Layer 2 → Convolutional Layer 3 → Convolutional Layer 4 → Convolutional Layer 5 → Convolutional Layer 6. The kernel size of Convolutional Layers 1, 3, 4, and 6 is 1×1, with 256, 1024, 512, and 2048 kernels respectively, and a stride of 1. The kernel size of Convolutional Layers 2 and 3 is 3×3, with 256 and 512 kernels respectively, and a stride of 2. In this case, the attention module in the shared parameter subnetwork can contain two convolutional layers. The kernel size of the first convolutional layer in the attention module is 3×3, with 1 kernel and a stride of 1. The kernel size of the second convolutional layer in the attention module is 1×1, with 1 kernel and a stride of 1.

[0117] It should be noted that the specific structure of the above-mentioned shared parameter subnetwork is only used as an example, and the embodiments of this application do not limit the specific structure of the above-mentioned shared parameter subnetwork.

[0118] Regarding the specific implementation process of steps S103-S104 above, and in conjunction with steps a1-a4 given in step S104, the specific methods for determining the various data feature differences and various types of losses involved are explained in detail as follows:

[0119] 1. Regarding the differences in the first data characteristic mentioned above, refer to Figure 2 As shown, Figure 2 This illustration shows a flowchart of a method for calculating the classification prediction loss of the same sample object under different modal data, as provided in an embodiment of this application. When performing step S104, the method includes steps S201-S203; specifically:

[0120] S201, based on the first modal features of the sample object output by the first feature extraction sub-network, classify and predict the entity type to which the sample object belongs, and obtain the first classification prediction result of the sample object.

[0121] Here, the entity type of the sample object can be determined based on the entity type label for each sample object in this batch of training samples. For example, if the entity type label corresponding to sample object i is y... i If the sample object i is a vehicle, then the entity type can be determined to be "vehicle".

[0122] Specifically, for the specific annotation methods of the above entity types, please refer to the coarse-grained classification method and fine-grained classification method for similar or dissimilar sample objects in step S103 above. The repeated parts will not be repeated here.

[0123] S202, based on the second modal features of the sample object output by the second feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the second classification prediction result of the sample object.

[0124] Here, the implementation method of step S202 is the same as that of step S201 above, and the repeated parts will not be described again.

[0125] S203, based on the first classification loss between the first classification prediction result and the true classification result and the second classification loss between the second classification prediction result and the true classification result, the sum of the first classification loss and the second classification loss is used as the classification prediction loss corresponding to the sample object.

[0126] Here, the actual classification result is used to characterize the entity type to which the sample object belongs.

[0127] It should be noted that when calculating the first classification loss and the second classification loss, any commonly used loss function (such as the softmax function, the multi-class cross-entropy loss function, etc.) used by the model in multi-class prediction can be used. This application does not limit the specific type of loss function used in calculating the first classification loss and the second classification loss.

[0128] Here, taking the cross-entropy loss function as an example, the above classification prediction loss can be calculated according to the following formula:

[0129]

[0130] Where n represents the number of sample objects in this batch of training samples, and i takes the value from 1 to n;

[0131] x i The first modal feature used to characterize the i-th sample object;

[0132] z iThe second modal feature used to characterize the i-th sample object;

[0133] y i Used to characterize the entity type to which the i-th sample object belongs;

[0134] p(y i |x i ) is used to characterize the first classification prediction result of the i-th sample object;

[0135] p(y i |z i The second classification prediction result is used to characterize the i-th sample object;

[0136] L c The classification prediction loss is used to characterize the i-th sample object.

[0137] 2. Regarding the differences in the second data characteristic mentioned above, refer to... Figure 3 As shown, Figure 3 This illustration shows a flowchart of a method for determining the difference in second data features between a sample object and a first sample object under different modal data, according to an embodiment of this application. When performing step S103, the method includes steps S301-S304; specifically:

[0138] S301, for the sample object, according to the entity type to which the sample object belongs, randomly select one sample object from the plurality of sample objects that belongs to the same entity type as the sample object as the first sample object.

[0139] For example, if the current i-th sample object belongs to the entity type "vehicle", then a sample object j belonging to the entity type "vehicle" can be randomly selected from the n sample objects in the same batch as the first sample object mentioned above.

[0140] S302, based on the first modal features of the sample object output by the first feature extraction sub-network and the second modal features of the first sample object output by the second feature extraction sub-network, calculate the feature distance between the first modal features of the sample object and the second modal features of the first sample object, and obtain the first similar feature distance between the sample object and the first sample object.

[0141] It should be noted that the feature distance between the first modal feature of the sample object and the second modal feature of the first sample object is used to characterize the feature differences (i.e., the similarity) exhibited by similar sample objects under different modal data. Based on this, when calculating the first similar feature distance, any feature distance calculation method that can characterize the similarity between two features can be used (e.g., Euclidean distance, Mahalanobis distance, etc.). This application embodiment does not limit the specific calculation method of the first similar feature distance.

[0142] S303, based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the first sample object output by the first feature extraction sub-network, calculate the feature distance between the second modal features of the sample object and the first modal features of the first sample object, and obtain the second similar feature distance between the sample object and the first sample object.

[0143] It should be noted that, similar to the calculation method of the first type of feature distance described above, the specific calculation method of the second type of feature distance described above is not limited in this application embodiment.

[0144] S304, the first similar feature distance and the second similar feature distance are used as the second data feature difference between the sample object and the first sample object under different modal data.

[0145] Here, taking the calculation of feature distances using Euclidean distance (i.e., Euclidean distance) as an example, the first type of feature distance and the second type of feature distance can be calculated according to the following formula:

[0146] First similar feature distance = D(x) i ,z j );

[0147] Where D(x) is used to characterize the Euclidean distance calculation formula, x i z is the first modal feature used to characterize the i-th sample object. j The second modal feature used to characterize the j-th sample object, which belongs to the same entity type as the i-th sample object;

[0148] Second similar feature distance = D(z) i ,x j );

[0149] Among them, z i The second modal feature used to characterize the i-th sample object, x j The first modal feature used to characterize the j-th sample object.

[0150] 3. Regarding the aforementioned third data feature difference, in the first optional embodiment, a sample object belonging to a different entity type from the sample object can still be randomly selected as the second sample object. Then, following a method similar to steps S301-S304 above, the feature distance between the second modal feature of the sample object and the first modal feature of the second sample object is calculated to obtain the first heterogeneous feature distance between the sample object and the second sample object. The feature distance between the second modal feature of the sample object and the first modal feature of the second sample object is calculated to obtain the second heterogeneous feature distance between the sample object and the second sample object. Thus, the calculated first heterogeneous feature distance and the second heterogeneous feature distance are used as the third data feature difference between the sample object and the second sample object under different modal data.

[0151] In the second alternative implementation, refer to Figure 4 As shown, Figure 4 This illustration shows a flowchart of a method for determining the differences in third data features between a sample object and a second sample object under different modal data, according to an embodiment of this application. When performing step S103, the method includes steps S401-S404; specifically:

[0152] S401, for the sample object, based on the entity type to which the sample object belongs, determine each sample object belonging to a different entity type from the plurality of sample objects as the target sample object.

[0153] For example, if the current i-th sample object belongs to the entity type "vehicle", and among the n sample objects in the same batch, there are 5 sample objects that do not belong to the entity type "vehicle", then it is determined that these 5 sample objects all belong to the target sample object corresponding to the i-th sample object.

[0154] S402, for each target sample object, based on the first modal features of the sample object output by the first feature extraction sub-network and the second modal features of the target sample object output by the second feature extraction sub-network, calculate the feature distance between the first modal features of the sample object and the second modal features of the target sample object, and obtain the third feature distance between the sample object and the target sample object.

[0155] It should be noted that, similar to the calculation method of the first type of feature distance mentioned above, the specific calculation method of the third feature distance mentioned above is not limited in this application embodiment.

[0156] Specifically, taking Euclidean distance as an example for feature distance calculation, the third feature distance mentioned above can be calculated according to the following formula:

[0157] Third feature distance = D(x) i ,z k );

[0158] Where D(x) is used to characterize the Euclidean distance calculation formula, x i z is the first modal feature used to characterize the i-th sample object. k The second modal feature used to characterize the k-th sample object, which belongs to a different entity type than the i-th sample object.

[0159] S403, based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the target sample object output by the first feature extraction sub-network, calculate the feature distance between the second modal features of the sample object and the first modal features of the target sample object to obtain the fourth feature distance between the sample object and the target sample object.

[0160] It should be noted that, similar to the calculation method of the first type of feature distance mentioned above, the specific calculation method of the fourth feature distance mentioned above is not limited in this application embodiment.

[0161] Here, taking Euclidean distance as an example for feature distance calculation, the fourth feature distance mentioned above can be calculated according to the following formula:

[0162] Fourth feature distance = D(z) i ,x k );

[0163] Where D(x) is used to characterize the Euclidean distance calculation formula, z i The second modal feature used to characterize the i-th sample object, x k The first modal feature used to characterize the k-th sample object.

[0164] S404, the minimum value among the third feature distances between the sample object and each of the target sample objects is taken as the first outlier feature distance, and the minimum value among the fourth feature distances between the sample object and each of the target sample objects is taken as the second outlier feature distance. The first outlier feature distance and the second outlier feature distance are taken as the third data feature difference between the sample object and the second sample object under different modal data.

[0165] Specifically, taking Euclidean distance for feature distance calculation as an example, the first out-of-class feature distance mentioned above can be expressed as:

[0166] The aforementioned second out-of-class feature distance can be expressed as:

[0167] 4. Regarding the aforementioned cross-modal triplet loss, combining the methods for calculating the second data feature difference shown in steps S301-S304 and the methods for calculating the third data feature difference shown in steps S401-S404, refer to... Figure 5 As shown, Figure 5 This paper presents a flowchart illustrating a method for calculating cross-modal triplet loss according to an embodiment of this application. When performing step S104, the method includes steps S501-S503; specifically:

[0168] S501, calculate the first sum between the first preset boundary threshold and the first similar feature distance, and take the difference between the first sum and the first dissimilar feature distance as the first target difference.

[0169] Here, the first preset boundary threshold is used to characterize the preset boundary threshold corresponding to the data retrieval of the second modality data of the sample object when the first modality data is used as the query data for retrieval; wherein, the initial value of the first preset boundary threshold can be set according to the actual retrieval needs of the model, and the first preset boundary threshold is one of the model parameters that need to be adjusted during the initial model training process.

[0170] S502, calculate the second sum between the second preset boundary threshold and the distance of the second similar feature, and take the difference between the second sum and the distance of the second dissimilar feature as the second target difference.

[0171] Here, the second preset boundary threshold is used to characterize the preset boundary threshold corresponding to the data retrieval of the first modality data of the sample object when the second modality data is used as the query data for retrieval; wherein, the initial value of the second preset boundary threshold can also be set according to the actual retrieval needs of the model, and the second preset boundary threshold is also one of the model parameters that need to be adjusted during the initial model training process.

[0172] S503, the maximum value among the first target difference and the specified parameters is used as the first cross-modal loss of the sample object under the first modal data, the maximum value among the second target difference and the specified parameters is used as the second cross-modal loss of the sample object under the second modal data, and the sum of the first cross-modal loss and the second cross-modal loss is used as the cross-modal triplet loss corresponding to the sample object.

[0173] It should be noted that the specific values ​​of the specified parameters can be set according to the actual retrieval needs of the model (usually they can be set to 0). This application embodiment does not impose any limitations on the specific values ​​of the specified parameters.

[0174] Here, taking Euclidean distance for feature distance calculation as an example, when the specified parameter is 0, the cross-modal triplet loss can be calculated according to the following formula:

[0175]

[0176] Among them, L c_trip Used to characterize the cross-modal triplet loss corresponding to the i-th sample object;

[0177] ρ1 is used to characterize the aforementioned first preset boundary threshold;

[0178] ρ2 is used to characterize the aforementioned second preset boundary threshold;

[0179] D(x i ,z i ) is used to characterize the distance of the first similar feature corresponding to the i-th sample object;

[0180] D(z i ,x j ) is used to characterize the distance of the second similar feature corresponding to the i-th sample object;

[0181] Used to characterize the first heterogeneous feature distance corresponding to the i-th sample object;

[0182] The distance used to characterize the second out-of-class feature corresponding to the i-th sample object.

[0183] 5. Regarding the differences in characteristics of the first group and the differences in characteristics of the second group mentioned above, refer to... Figure 6 As shown, Figure 6 This illustration shows a flowchart of a method for determining the characteristic differences of a first group and the characteristic differences of a second group, according to an embodiment of this application. When performing step S103, the method includes steps S601-S604; specifically:

[0184] S601, for the plurality of sample objects, using the first classification prediction result of each sample object as a vertex and the first modal feature of each sample object as the vertex attribute of each vertex, construct the first adjacency matrix of the plurality of sample objects under the first modal data.

[0185] Specifically, taking the current batch of training samples containing n sample objects as an example, the first modal data of the n sample objects are input into the first feature extraction sub-network in the initial model to obtain the first modal features of each of the n sample objects. Then, based on the first modal features of each sample object, the entity type to which each sample object belongs can be classified and predicted to obtain the first classification prediction result p1 of each of the n sample objects.

[0186] Based on this, using the first classification prediction result p1 of each of the n sample objects as vertices, and the first modal features of each of the n sample objects as the corresponding vertex attributes of each of the n vertices, the first adjacency matrix A1 of these n sample objects under the first modal data can be constructed; wherein, in the first adjacency matrix A1, the connection between any two vertices can be used to characterize whether the sample objects corresponding to these two vertices belong to the same entity type.

[0187] S602, using the second classification prediction result of each sample object as a vertex and the second modal feature of each sample object as the vertex attribute of each vertex, construct the second adjacency matrix of the multiple sample objects under the second modal data.

[0188] Specifically, taking the current batch of training samples containing n sample objects as an example, the second modal data of the n sample objects are input into the second feature extraction sub-network in the initial model to obtain the second modal features of each of the n sample objects. Then, based on the second modal features of each sample object, the entity type to which each sample object belongs can be classified and predicted to obtain the second classification prediction result p2 of each of the n sample objects.

[0189] Based on this, using the second classification prediction result p2 of each of the n sample objects as vertices, and the second modal features of each of the n sample objects as the corresponding vertex attributes of each of the n vertices, a second adjacency matrix A2 of these n sample objects under the second modal data can be constructed; wherein, in the second adjacency matrix A2, the connection between any two vertices can still be used to characterize whether the sample objects corresponding to these two vertices belong to the same entity type.

[0190] S603, the graph structure features exhibited by the first adjacency matrix under the graph structure are used as the differences in the first group features exhibited by the multiple sample objects under the first modal data.

[0191] S604, the graph structure features exhibited by the second adjacency matrix under the graph structure are used as the differences in the second group features exhibited by the multiple sample objects under the second modal data.

[0192] Here, based on the first group feature difference and the second group feature difference shown in steps S601-S604 above, in this embodiment of the application, as an optional embodiment, the mutual learning loss between the first group feature difference and the second group feature difference can be calculated according to the calculation method shown in the following formula, specifically:

[0193] 1. Two graph-based feature representations are denoted as H. 1 (i.e., the graph structure characteristics exhibited by the first adjacency matrix under a graph structure, which is also the difference in the characteristics of the first group mentioned above) and H 2 (That is, the graph structure characteristics exhibited by the second adjacency matrix under the graph structure, i.e., the differences in the characteristics of the second group mentioned above), which can be calculated as follows:

[0194]

[0195]

[0196] Where D1 is used to characterize the first adjacency matrix A 1 The angle matrix;

[0197] D2 is used to characterize the second adjacency matrix A. 2 The angle matrix;

[0198] F 1 Used to characterize the calculation of H 1 Features of the upper layer of the time-map convolution;

[0199] F 2 Used to characterize the calculation of H 2 Features of the upper layer of the time-map convolution;

[0200] The weight coefficients used to characterize the first feature extraction subnetwork as it is learned and adjusted during model training (equivalent to model parameters adjusted based on the loss during model training);

[0201] The weight coefficients used to characterize the second feature extraction subnetwork as it is learned and adjusted during model training (equivalent to model parameters adjusted based on the loss during model training);

[0202] L can be set to 1.

[0203] 2. To reduce the modal differences between two types of data, a loss function L that maximizes mutual information can be designed. HOL (that is, used to calculate the characteristic differences H of the first group) 1 Differences in characteristics with the second group H 2 The specific loss function for the mutual learning loss between them is as follows:

[0204] L HOL =-I(H 1 H 2 )

[0205] in,

[0206] f(x) is used to characterize the cosine similarity function;

[0207] Graph-based feature representation for characterizing the first modality data of sample object i;

[0208] Graph-based feature representation for characterizing the second modality data of sample object i;

[0209] I(x) is used to characterize the mutual information of the distributions of two modalities (i.e., the first modal data and the second modal data). The mutual information of the two modalities is large when they are in the same category, and small when they are in different categories.

[0210] E(x) is used to characterize the cross-entropy function.

[0211] The training method for the cross-modal retrieval model provided in this application embodiment, for multiple sample objects included in the same batch of training samples, inputs the first modal data of each sample object into the first feature extraction sub-network in the initial model to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network; inputs the second modal data of each sample object into the second feature extraction sub-network in the initial model to obtain the second modal features of each sample object under the second modal data output by the second feature extraction sub-network; based on the first modal features and the second modal features of each sample object, determines the first data feature differences of the same sample object under different modal data, the second data feature differences of similar sample objects under different modal data, the third data feature differences of dissimilar sample objects under different modal data, and the group feature differences of multiple sample objects under different modal data; based on the first data feature differences, second data feature differences, third data feature differences, and group feature differences, adjusts the model parameters of the initial model to obtain a cross-modal retrieval model including the adjusted model parameters.

[0212] In this way, this application solves the problem that the data types of query data and index data that can be used in actual retrieval cannot be interchanged by training the model to learn from the data features of similar sample objects and dissimilar sample objects under different modal data. This effectively improves the cross-modal retrieval flexibility of the cross-modal retrieval model.

[0213] In this embodiment of the application, after training the cross-modal retrieval model described above, during the application phase of the cross-modal retrieval model, the following cross-modal retrieval method is also provided:

[0214] Reference Figure 7 As shown, Figure 7 This paper illustrates a flowchart of a first cross-modal retrieval method provided in an embodiment of this application. The method includes steps S701-S703; specifically:

[0215] S701, input the first sample data of multiple search objects into a pre-trained cross-modal retrieval model, and extract the data features of each search object under the first modal data through the cross-modal retrieval model to obtain the first modal features of each search object.

[0216] Here, the first sample data mentioned above includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object; the first search object and the search object belong to the same entity type, and the second search object and the search object belong to different entity types.

[0217] Specifically, the cross-modal retrieval model can extract features from the first modal data of each retrieval object in the input first sample data based on the trained first feature extraction sub-network to obtain the first modal features of each retrieval object; and extract features from the second modal data of each first retrieval object and each second retrieval object in the input first sample data based on the trained second feature extraction sub-network to obtain the second modal features of each first retrieval object and each second retrieval object.

[0218] It should be noted that the data processing method of the cross-modal retrieval model for the first sample data can refer to the feature extraction method shown in steps S101-S102 above, and the repeated parts will not be repeated here.

[0219] It should be noted that the cross-modal retrieval model is obtained after training according to the above steps S101-S104; the specific training process of the cross-modal retrieval model can be referred to the specific implementation process of the above steps, and the repeated parts will not be repeated here.

[0220] S702, the second modal data of the target object is input into the cross-modal retrieval model, and the data features exhibited by the target object under the second modal data are extracted by the cross-modal retrieval model to obtain the second modal features of the target object.

[0221] Here, the aforementioned target object is used to characterize the entity object that actually needs to be retrieved during the model application stage. This application embodiment does not impose any limitation on the entity type to which the aforementioned target object belongs.

[0222] It should be noted that, unlike the first sample data that needs to be input for multiple retrieval objects mentioned above, here, only the second modality data of the target object needs to be input, and there is no need to input data in the form of cross-modal triples as shown in the first sample data above; for example, if the second modality data is image data, then only the single frame image data of the target object needs to be input as the second modality data of the target object into the cross-modal retrieval model.

[0223] S703, using the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature, the cross-modal retrieval model is used to calculate the similarity between the first retrieval data feature and each first index data feature, and the retrieval object whose similarity calculation result meets the retrieval display conditions is output as the cross-modal retrieval result for the target object.

[0224] Here, when calculating the similarity between the first retrieval data feature and each of the first index data features, the similarity can be obtained by calculating the Euclidean distance between the first retrieval data feature and each of the first index data features; alternatively, it can be obtained by calculating other feature distances such as the Mahalanobis distance between the first retrieval data feature and each of the first index data features. This application embodiment does not limit the specific similarity calculation method.

[0225] Specifically, when performing step S703 above, in a first optional implementation, based on the similarity calculation result between the first retrieval data feature and each of the first index data features, retrieval objects whose similarity calculation result is greater than or equal to a preset similarity threshold can be determined from all the retrieval objects as retrieval objects that meet the retrieval display conditions.

[0226] In a second optional implementation, the first modal data of each search object can be output as a cross-modal search result for the target object based on the similarity calculation results between the first search data feature and each of the first index data features, in descending order of the similarity calculation results.

[0227] Regarding the specific implementation process of steps S701-S703 above, it should be noted that steps S701-S703 above show: when using the second modality data as the query data used in actual retrieval and using the first modality data as the index data of the retrieval database, how to use the pre-trained cross-modal retrieval model above to perform cross-modal retrieval between the second modality data and the first modality data for the target object.

[0228] Because the cross-modal retrieval model trained in this application can solve the problem in existing technologies where the data types of query data and index data used in actual retrieval are not interchangeable, based on this, and referring to Figure 8 As shown, Figure 8 This paper illustrates a flowchart of a second cross-modal retrieval method provided in an embodiment of this application. The method includes steps S801-S803; specifically:

[0229] S801, input the second sample data of multiple search objects into a pre-trained cross-modal retrieval model, and extract the data features of each search object under the second modal data through the cross-modal retrieval model to obtain the second modal features of each search object.

[0230] Here, the second sample data includes: the second modal data of each of the search objects, the first modal data of the first search object, and the first modal data of the second search object.

[0231] Specifically, the implementation process of step S801 is similar to that of step S701, and the repetitions will not be repeated here.

[0232] S802, the first modal data of the target object is input into the cross-modal retrieval model, and the data features exhibited by the target object under the first modal data are extracted by the cross-modal retrieval model to obtain the first modal features of the target object.

[0233] Specifically, the implementation process of step S802 is similar to that of step S702, and the repetitions will not be repeated here.

[0234] S803, using the first modal feature of the target object as the second retrieval data feature, and the second modal feature of each retrieval object as the second index data feature, the similarity between the second retrieval data feature and each second index data feature is calculated through the cross-modal retrieval model, and the retrieval object whose similarity calculation result meets the retrieval display conditions is output as the cross-modal retrieval result for the target object.

[0235] Specifically, the implementation process of step S803 is similar to that of step S703, and the repetitions will not be repeated here.

[0236] Based on this, according to the two cross-modal retrieval methods shown in steps S701-S703 and S801-S803 provided in the embodiments of this application, when performing actual retrieval of a target object, the pre-trained cross-modal retrieval model can be used to perform cross-modal retrieval between the first modality data and the second modality data of the target object, using the first modality data as the query data for retrieval and the second modality data as the index data of the retrieval database; or the second modality data of the target object can be used as the query data for retrieval and the first modality data as the index data of the retrieval database, performing cross-modal retrieval between the second modality data and the first modality data of the target object. This solves the problem that the data types of query data and index data that can be used in actual retrieval in the prior art cannot be interchanged, effectively improving the cross-modal retrieval flexibility of the cross-modal retrieval model.

[0237] Based on the same inventive concept, this application also provides a training device for a cross-modal retrieval model corresponding to the training method of the cross-modal retrieval model in the above embodiments. Since the principle of the training device in this application is similar to that of the training method in the above embodiments of this application, the implementation of the training device can refer to the implementation of the aforementioned training method, and the repeated parts will not be described again.

[0238] Reference Figure 9 As shown, Figure 9 This illustration shows a schematic diagram of a training device for a cross-modal retrieval model provided in an embodiment of this application; the training device includes:

[0239] The first extraction module 901 is used to input the first modal data of each sample object into the first feature extraction sub-network in the initial model for multiple sample objects included in the same batch of training samples, so as to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network.

[0240] The second extraction module 902 is used to input the second modal data of each sample object into the second feature extraction sub-network in the initial model, so as to obtain the second modal features of each sample object under the second modal data output by the second feature extraction sub-network;

[0241] The first determining module 903 is used to determine, based on the first modal features of each sample object and the second modal features of each sample object, the differences in first data features exhibited by the same sample object under different modal data, the differences in second data features exhibited by similar sample objects under different modal data, the differences in third data features exhibited by dissimilar sample objects under different modal data, and the differences in group features exhibited by the multiple sample objects under different modal data.

[0242] The parameter adjustment module 904 is used to adjust the model parameters of the initial model according to the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference, so as to obtain a cross-modal retrieval model including the adjusted model parameters.

[0243] In one optional implementation, the parameter adjustment module 904 is specifically used for:

[0244] For each sample object, based on the differences in the first data features exhibited by the sample object under different modal data, the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs is determined;

[0245] For each sample object, based on the differences in second data features exhibited by the sample object and the first sample object under different modal data, and the differences in third data features exhibited by the sample object and the second sample object under different modal data, the cross-modal triplet loss between the sample object and similar sample objects under different modal data, as well as dissimilar sample objects under different modal data, is determined; wherein, the first sample object is used to characterize the sample object belonging to the same entity type as the sample object among the plurality of sample objects; the second sample object is used to characterize the sample object belonging to a different entity type than the sample object among the plurality of sample objects;

[0246] For the plurality of sample objects, based on the first group feature differences exhibited by the plurality of sample objects under the first modal data and the second group feature differences exhibited by the plurality of sample objects under the second modal data, the mutual learning loss between the first group feature differences and the second group feature differences is determined;

[0247] The model parameters of the initial model are adjusted based on the classification prediction loss for each sample object, the cross-modal triplet loss for each sample object, and the mutual learning loss for the multiple sample objects, until the initial model converges.

[0248] In an optional implementation, when determining the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs, based on the differences in the first data features exhibited by the sample object under different modal data, the parameter adjustment module 904 is specifically used for:

[0249] Based on the first modal features of the sample object output by the first feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the first classification prediction result of the sample object.

[0250] Based on the second modal features of the sample object output by the second feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the second classification prediction result of the sample object;

[0251] Based on the first classification loss between the first classification prediction result and the true classification result, and the second classification loss between the second classification prediction result and the true classification result, the sum of the first classification loss and the second classification loss is used as the classification prediction loss corresponding to the sample object; wherein, the true classification result is used to characterize the entity type to which the sample object belongs.

[0252] In an optional implementation, the first determining module 903 is configured to determine the difference in second data features exhibited by the sample object and the first sample object under different modal data by the following method:

[0253] For the sample object, based on the entity type to which the sample object belongs, one sample object of the same entity type as the sample object is randomly selected from the plurality of sample objects as the first sample object;

[0254] Based on the first modal features of the sample object output by the first feature extraction subnetwork and the second modal features of the first sample object output by the second feature extraction subnetwork, the feature distance between the first modal features of the sample object and the second modal features of the first sample object is calculated to obtain the first similar feature distance between the sample object and the first sample object.

[0255] Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the first sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the first sample object is calculated to obtain the second similar feature distance between the sample object and the first sample object.

[0256] The first similar feature distance and the second similar feature distance are used as the second data feature differences between the sample object and the first sample object under different modal data.

[0257] In an optional implementation, the first determining module 903 is used to determine the difference in third data features exhibited by the sample object and the second sample object under different modal data by the following method:

[0258] For the sample object, based on the entity type to which the sample object belongs, each sample object belonging to a different entity type from the plurality of sample objects is determined as the target sample object;

[0259] For each target sample object, based on the first modal features of the sample object output by the first feature extraction sub-network and the second modal features of the target sample object output by the second feature extraction sub-network, the feature distance between the first modal features of the sample object and the second modal features of the target sample object is calculated to obtain the third feature distance between the sample object and the target sample object.

[0260] Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the target sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the target sample object is calculated to obtain the fourth feature distance between the sample object and the target sample object.

[0261] The minimum value among the third feature distances between the sample object and each of the target sample objects is taken as the first outlier feature distance, and the minimum value among the fourth feature distances between the sample object and each of the target sample objects is taken as the second outlier feature distance. The first outlier feature distance and the second outlier feature distance are taken as the third data feature difference between the sample object and the second sample object under different modal data.

[0262] In an optional implementation, when determining the cross-modal triplet loss between the sample object and similar sample objects under different modal data and dissimilar sample objects under different modal data based on the difference in second data features exhibited by the sample object and the first sample object under different modal data, and the difference in third data features exhibited by the sample object and the second sample object under different modal data, the parameter adjustment module 904 is specifically used for:

[0263] Calculate the first sum between the first preset boundary threshold and the first distance of the same type of feature, and take the difference between the first sum and the first distance of the different type of feature as the first target difference;

[0264] Calculate the second sum between the second preset boundary threshold and the distance of the second similar feature, and take the difference between the second sum and the distance of the second dissimilar feature as the second target difference;

[0265] The maximum value among the first target difference and the specified parameters is used as the first cross-modal loss of the sample object under the first modal data, and the maximum value among the second target difference and the specified parameters is used as the second cross-modal loss of the sample object under the second modal data. The sum of the first cross-modal loss and the second cross-modal loss is used as the cross-modal triplet loss corresponding to the sample object.

[0266] In an optional implementation, the parameter adjustment module 904 is used to determine the first group characteristic differences exhibited by the plurality of sample objects under the first modal data and the second group characteristic differences exhibited by the plurality of sample objects under the second modal data by the following method:

[0267] For the multiple sample objects, the first classification prediction result of each sample object is used as a vertex, and the first modal feature of each sample object is used as the vertex attribute of each vertex to construct the first adjacency matrix of the multiple sample objects under the first modal data.

[0268] Using the second classification prediction result of each sample object as a vertex and the second modal feature of each sample object as the vertex attribute of each vertex, a second adjacency matrix of the multiple sample objects under the second modal data is constructed.

[0269] The graph structure features exhibited by the first adjacency matrix under the graph structure are used as the differences in the first group features exhibited by the multiple sample objects under the first modal data.

[0270] The graph structure features exhibited by the second adjacency matrix under the graph structure are used as the differences in the second group features exhibited by the multiple sample objects under the second modal data.

[0271] In one optional implementation, when the first modal data and the second modal data belong to different types of data, it is determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a non-parameter-sharing manner during the model training process of the cross-modal retrieval model; when the first modal data and the second modal data belong to the same type of data with different data parameters, it is determined that the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a parameter-sharing manner during the model training process of the cross-modal retrieval model.

[0272] Based on the same inventive concept, this application also provides a cross-modal retrieval device corresponding to the cross-modal retrieval method in the above embodiments. Since the principle of the cross-modal retrieval device in this application is similar to that of the cross-modal retrieval method in the above embodiments of this application, the implementation of the cross-modal retrieval device can refer to the implementation of the aforementioned cross-modal retrieval method, and the repeated parts will not be described again.

[0273] Reference Figure 10 As shown, Figure 10 This illustration shows a structural schematic diagram of a cross-modal retrieval device provided in an embodiment of this application; the cross-modal retrieval device includes:

[0274] The first index extraction module 1001 is used to input first sample data of multiple search objects into a pre-trained cross-modal retrieval model, and to extract features of the data features exhibited by each search object under the first modal data through the cross-modal retrieval model to obtain the first modal features of each search object; wherein, the first sample data includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object; the first search object and the search object belong to the same entity type, and the second search object and the search object belong to different entity types;

[0275] The first query extraction module 1002 is used to input the second modal data of the target object into the cross-modal retrieval model, and extract the data features of the target object under the second modal data through the cross-modal retrieval model to obtain the second modal features of the target object;

[0276] The first retrieval module 1003 is used to use the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature. The similarity between the first retrieval data feature and each first index data feature is calculated through the cross-modal retrieval model, and the retrieval object whose similarity calculation result meets the retrieval display conditions is output as the cross-modal retrieval result for the target object.

[0277] In one optional implementation, the cross-modal retrieval device further includes:

[0278] The second index extraction module is used to input the second sample data of multiple search objects into a pre-trained cross-modal retrieval model, and to extract the data features of each search object under the second modal data through the cross-modal retrieval model to obtain the second modal features of each search object; wherein, the second sample data includes: the second modal data of each search object, the first modal data of the first search object, and the first modal data of the second search object;

[0279] The second query extraction module is used to input the first modal data of the target object into the cross-modal retrieval model, and extract the data features of the target object under the first modal data through the cross-modal retrieval model to obtain the first modal features of the target object;

[0280] The second retrieval module is used to use the first modal feature of the target object as the second retrieval data feature, and the second modal feature of each retrieval object as the second index data feature. The module calculates the similarity between the second retrieval data feature and each second index data feature through the cross-modal retrieval model, and outputs the retrieval objects whose similarity calculation results meet the retrieval display conditions as cross-modal retrieval results for the target object.

[0281] In one optional implementation, the first retrieval module 1003 is used to determine the retrieval objects whose similarity calculation results meet the retrieval display conditions by the following method:

[0282] Based on the similarity calculation results between the first retrieval data feature and each of the first index data features, from all the retrieval objects, the retrieval objects whose similarity calculation results are greater than or equal to a preset similarity threshold are determined as the retrieval objects that meet the retrieval display conditions.

[0283] In an optional implementation, when the search object whose output similarity calculation result meets the search display conditions is used as the cross-modal search result for the target object, the first search module 1003 is further configured to:

[0284] Based on the similarity calculation results between the first retrieval data feature and each of the first index data features, the first modal data of each retrieval object is output as a cross-modal retrieval result for the target object in descending order of the similarity calculation results.

[0285] In one optional implementation, the cross-modal retrieval model is obtained by training according to the training method in any of the optional implementations described in the above embodiments; repeated details will not be repeated here.

[0286] Based on the same inventive concept, such as Figure 11 As shown, this application provides a computer device 1100 for executing the steps of the training method of any of the cross-modal retrieval models described in this application, or executing the steps of any of the cross-modal retrieval methods described above. The device includes a memory 1101, a processor 1102, and a computer program stored in the memory 1101 and executable on the processor 1102. When the processor 1102 executes the computer program, it implements the steps of the training method of any of the cross-modal retrieval models described above, or executes the steps of any of the cross-modal retrieval methods described above.

[0287] Specifically, the memory 1101 and processor 1102 described above can be general-purpose memory and processor, without any specific limitation. When the processor 1102 runs the computer program stored in the memory 1101, it can execute the steps of the training method of any of the cross-modal retrieval models described above, or execute the steps of any of the cross-modal retrieval methods described above.

[0288] Corresponding to the steps of the training method of any of the cross-modal retrieval models described in this application, or the steps of executing any of the cross-modal retrieval methods described above, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the training method of any of the cross-modal retrieval models described above, or executes the steps of any of the cross-modal retrieval methods described above.

[0289] Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or a hard disk. When the computer program on the storage medium is run, it can execute the steps of the training method of any of the cross-modal retrieval models described above, or execute the steps of any of the cross-modal retrieval methods described above.

[0290] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0291] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0292] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0293] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0294] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0295] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A training method for a cross-modal retrieval model, characterized in that, The training method includes: For multiple sample objects included in the same batch of training samples, the first modal data of each sample object is input into the first feature extraction sub-network in the initial model to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network. The second modal data of each sample object is input into the second feature extraction sub-network within the initial model to obtain the second modal features of each sample object under the second modal data, as output by the second feature extraction sub-network. Based on the first modal features and the second modal features of each sample object, the differences in the first data features exhibited by the same sample object under different modal data, the differences in the second data features exhibited by similar sample objects under different modal data, the differences in the third data features exhibited by dissimilar sample objects under different modal data, and the differences in the group features exhibited by the multiple sample objects under different modal data are determined. Based on the differences in the first data features, the second data features, the third data features, and the group features, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. Specifically, when the first modal data and the second modal data belong to different types of data, the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a non-parameter-sharing manner during the model training process of the cross-modal retrieval model; when the first modal data and the second modal data belong to the same type of data with different data parameters, the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a parameter-sharing manner during the model training process of the cross-modal retrieval model.

2. The training method according to claim 1, characterized in that, The step of adjusting the model parameters of the initial model based on the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference includes: For each sample object, based on the differences in the first data features exhibited by the sample object under different modal data, the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs is determined; For each sample object, based on the differences in second data features exhibited by the sample object and the first sample object under different modal data, and the differences in third data features exhibited by the sample object and the second sample object under different modal data, the cross-modal triplet loss between the sample object and similar sample objects under different modal data, as well as dissimilar sample objects under different modal data, is determined; wherein, the first sample object is used to characterize the sample object belonging to the same entity type as the sample object among the plurality of sample objects; the second sample object is used to characterize the sample object belonging to a different entity type than the sample object among the plurality of sample objects; For the plurality of sample objects, based on the first group feature differences exhibited by the plurality of sample objects under the first modal data and the second group feature differences exhibited by the plurality of sample objects under the second modal data, the mutual learning loss between the first group feature differences and the second group feature differences is determined; The model parameters of the initial model are adjusted based on the classification prediction loss for each sample object, the cross-modal triplet loss for each sample object, and the mutual learning loss for the multiple sample objects, until the initial model converges.

3. The training method according to claim 2, characterized in that, The step of determining the classification prediction loss between the data features exhibited by the sample object under different modal data and the entity type to which the sample object belongs, based on the differences in the first data features exhibited by the sample object under different modal data, includes: Based on the first modal features of the sample object output by the first feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the first classification prediction result of the sample object. Based on the second modal features of the sample object output by the second feature extraction sub-network, the entity type to which the sample object belongs is classified and predicted to obtain the second classification prediction result of the sample object; Based on the first classification loss between the first classification prediction result and the true classification result, and the second classification loss between the second classification prediction result and the true classification result, the sum of the first classification loss and the second classification loss is used as the classification prediction loss corresponding to the sample object; wherein, the true classification result is used to characterize the entity type to which the sample object belongs.

4. The training method according to claim 2, characterized in that, The differences in the second data features exhibited by the sample object and the first sample object under different modal data were determined using the following method: For the sample object, based on the entity type to which the sample object belongs, one sample object of the same entity type as the sample object is randomly selected from the plurality of sample objects as the first sample object; Based on the first modal features of the sample object output by the first feature extraction subnetwork and the second modal features of the first sample object output by the second feature extraction subnetwork, the feature distance between the first modal features of the sample object and the second modal features of the first sample object is calculated to obtain the first similar feature distance between the sample object and the first sample object. Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the first sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the first sample object is calculated to obtain the second similar feature distance between the sample object and the first sample object. The first similar feature distance and the second similar feature distance are used as the second data feature differences between the sample object and the first sample object under different modal data.

5. The training method according to claim 4, characterized in that, The differences in third data features exhibited by the sample object and the second sample object under different modal data were determined using the following method: For the sample object, based on the entity type to which the sample object belongs, each sample object belonging to a different entity type from the plurality of sample objects is determined as the target sample object; For each target sample object, based on the first modal features of the sample object output by the first feature extraction sub-network and the second modal features of the target sample object output by the second feature extraction sub-network, the feature distance between the first modal features of the sample object and the second modal features of the target sample object is calculated to obtain the third feature distance between the sample object and the target sample object. Based on the second modal features of the sample object output by the second feature extraction sub-network and the first modal features of the target sample object output by the first feature extraction sub-network, the feature distance between the second modal features of the sample object and the first modal features of the target sample object is calculated to obtain the fourth feature distance between the sample object and the target sample object. The minimum value among the third feature distances between the sample object and each of the target sample objects is taken as the first outlier feature distance, and the minimum value among the fourth feature distances between the sample object and each of the target sample objects is taken as the second outlier feature distance. The first outlier feature distance and the second outlier feature distance are taken as the third data feature difference between the sample object and the second sample object under different modal data.

6. The training method according to claim 5, characterized in that, The determination of the cross-modal triplet loss between the sample object and similar sample objects under different modal data, and between the sample object and the second sample object and the third sample object under different modal data, based on the differences in second data features exhibited by the sample object and the second sample object under different modal data, includes: Calculate the first sum between the first preset boundary threshold and the first distance of the same type of feature, and take the difference between the first sum and the first distance of the different type of feature as the first target difference; Calculate the second sum between the second preset boundary threshold and the distance of the second similar feature, and take the difference between the second sum and the distance of the second dissimilar feature as the second target difference; The maximum value among the first target difference and the specified parameters is used as the first cross-modal loss of the sample object under the first modal data, and the maximum value among the second target difference and the specified parameters is used as the second cross-modal loss of the sample object under the second modal data. The sum of the first cross-modal loss and the second cross-modal loss is used as the cross-modal triplet loss corresponding to the sample object.

7. The training method according to claim 3, characterized in that, The following methods are used to determine the first group characteristic differences exhibited by the plurality of sample objects under the first modality data and the second group characteristic differences exhibited by the plurality of sample objects under the second modality data: For the multiple sample objects, the first classification prediction result of each sample object is used as a vertex, and the first modal feature of each sample object is used as the vertex attribute of each vertex to construct the first adjacency matrix of the multiple sample objects under the first modal data. Using the second classification prediction result of each sample object as a vertex and the second modal feature of each sample object as the vertex attribute of each vertex, a second adjacency matrix of the multiple sample objects under the second modal data is constructed. The graph structure features exhibited by the first adjacency matrix under the graph structure are used as the differences in the first group features exhibited by the multiple sample objects under the first modal data. The graph structure features exhibited by the second adjacency matrix under the graph structure are used as the differences in the second group features exhibited by the multiple sample objects under the second modal data.

8. A cross-modal retrieval method, characterized in that, The cross-modal retrieval method includes: First sample data of multiple search objects are input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model extracts features from the data characteristics exhibited by each search object under the first modal data to obtain the first modal features of each search object. The first sample data includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object. The first search object and the search object belong to the same entity type, while the second search object and the search object belong to different entity types. The second modal data of the target object is input into the cross-modal retrieval model. The cross-modal retrieval model is used to extract the data features of the target object under the second modal data to obtain the second modal features of the target object. Using the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature, the cross-modal retrieval model calculates the similarity between the first retrieval data feature and each first index data feature, and outputs the retrieval object whose similarity calculation result meets the retrieval display conditions as the cross-modal retrieval result for the target object.

9. The cross-modal retrieval method according to claim 8, characterized in that, The cross-modal retrieval method further includes: The second sample data of multiple search objects are input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model extracts the data features of each search object under the second modal data to obtain the second modal features of each search object. The second sample data includes: the second modal data of each search object, the first modal data of the first search object, and the first modal data of the second search object. The first modal data of the target object is input into the cross-modal retrieval model, and the cross-modal retrieval model is used to extract the data features of the target object under the first modal data to obtain the first modal features of the target object. Using the first modal feature of the target object as the second retrieval data feature, and the second modal feature of each retrieval object as the second index data feature, the cross-modal retrieval model is used to calculate the similarity between the second retrieval data feature and each second index data feature. The retrieval objects whose similarity calculation results meet the retrieval display conditions are output as the cross-modal retrieval results for the target object.

10. The cross-modal retrieval method according to claim 8, characterized in that, The following method is used to determine the search objects whose similarity calculation results meet the search display conditions: Based on the similarity calculation results between the first retrieval data feature and each of the first index data features, from all the retrieval objects, the retrieval objects whose similarity calculation results are greater than or equal to a preset similarity threshold are determined as the retrieval objects that meet the retrieval display conditions.

11. The cross-modal retrieval method according to claim 8, characterized in that, The search objects whose output similarity calculation results meet the search display conditions are used as cross-modal search results for the target object, and also include: Based on the similarity calculation results between the first retrieval data feature and each of the first index data features, the first modal data of each retrieval object is output as a cross-modal retrieval result for the target object in descending order of the similarity calculation results.

12. The cross-modal retrieval method according to claim 8, characterized in that, The cross-modal retrieval model is obtained by training according to any one of the training methods described in claims 1 to 7.

13. A training device for a cross-modal retrieval model, characterized in that, The training device includes: The first extraction module is used to input the first modal data of each sample object into the first feature extraction sub-network in the initial model for multiple sample objects included in the same batch of training samples, so as to obtain the first modal features of each sample object under the first modal data output by the first feature extraction sub-network. The second extraction module is used to input the second modal data of each sample object into the second feature extraction sub-network in the initial model, so as to obtain the second modal features of each sample object under the second modal data output by the second feature extraction sub-network; The first determining module is used to determine, based on the first modal features of each sample object and the second modal features of each sample object, the differences in first data features exhibited by the same sample object under different modal data, the differences in second data features exhibited by similar sample objects under different modal data, the differences in third data features exhibited by dissimilar sample objects under different modal data, and the differences in group features exhibited by the multiple sample objects under different modal data. The parameter adjustment module is used to adjust the model parameters of the initial model based on the first data feature difference, the second data feature difference, the third data feature difference, and the group feature difference, so as to obtain a cross-modal retrieval model including the adjusted model parameters. Specifically, when the first modal data and the second modal data belong to different types of data, the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a non-parameter-sharing manner during the model training process of the cross-modal retrieval model; when the first modal data and the second modal data belong to the same type of data with different data parameters, the first feature extraction subnetwork and the second feature extraction subnetwork are trained in a parameter-sharing manner during the model training process of the cross-modal retrieval model.

14. A cross-modal retrieval device, characterized in that, The cross-modal retrieval device includes: The first index extraction module is used to input first sample data of multiple search objects into a pre-trained cross-modal retrieval model, and to extract features from the data features exhibited by each search object under the first modal data through the cross-modal retrieval model to obtain the first modal features of each search object; wherein, the first sample data includes: first modal data of each search object, second modal data of the first search object, and second modal data of the second search object; the first search object and the search object belong to the same entity type, and the second search object and the search object belong to different entity types; The first query extraction module is used to input the second modal data of the target object into the cross-modal retrieval model, and extract the data features of the target object under the second modal data through the cross-modal retrieval model to obtain the second modal features of the target object; The first retrieval module is used to use the second modal feature of the target object as the first retrieval data feature, and the first modal feature of each retrieval object as the first index data feature. The module calculates the similarity between the first retrieval data feature and each first index data feature through the cross-modal retrieval model, and outputs the retrieval objects whose similarity calculation results meet the retrieval display conditions as cross-modal retrieval results for the target object.

15. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the training method for the cross-modal retrieval model as described in any one of claims 1 to 7, or the steps of the cross-modal retrieval method as described in any one of claims 8 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the training method for the cross-modal retrieval model as described in any one of claims 1 to 7, or the steps of the cross-modal retrieval method as described in any one of claims 8 to 12.

Images