Method, apparatus and computer program product for matching cross-modal data

Abstract

The present disclosure provides a cross-modal data matching method and device, and the specific implementation scheme is as follows: a training sample set is obtained, wherein the training sample includes first modal data, second modal data, and a label representing whether the multi-modal data matches; first level feature information and second level feature information of the first modal data and the second modal data in the training sample are extracted respectively; a matching loss function is used to constrain the matching result obtained based on the first level feature information between the first modal data and the second modal data, and the matching result based on the second level feature information; a classification loss function is used to constrain the classification result obtained based on the first level feature information and the second level feature information of the first modal data respectively, and the classification result obtained based on the first level feature information and the second level feature information of the second modal data respectively, so as to train a cross-modal matching model. The present disclosure improves the matching speed and accuracy of the cross-modal matching model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to computer vision, image recognition and deep learning technology, and more particularly to a cross-modal data matching method and device, electronic equipment, storage medium and computer program product, which can be used in a cross-modal retrieval scenario. BACKGROUND

[0002] Cross-modal retrieval is an important research field in multimedia content understanding. Cross-modal retrieval aims to establish a matching relationship for data of different modalities, so as to realize retrieval of other modal data with the same or similar semantics by taking certain modal data as query data. Among them, cross-modal retrieval between image and text, i.e. image-text matching, is a task with important research and application value, because image and text are the two most widely existing information in the Internet. In traditional cross-modal retrieval, the text keyword method is mainly used, and its essence is still single-modal retrieval. For example, for the cross-modal retrieval task of searching image by text, a query text is given, and before retrieval, manual text annotation is performed on the pictures in the candidate set, then the matching degree between the query text and the annotated text is calculated, and finally the pictures corresponding to the annotated text are returned. SUMMARY

[0003] The present disclosure provides a cross-modal matching model training method and device, and a cross-modal data matching method and device, electronic equipment, storage medium and computer program product.

[0004] According to a first aspect, a cross-modal matching model training method is provided, comprising: obtaining a training sample set, wherein each training sample in the training sample set comprises first modal data, second modal data and a label representing whether the first modal data and the second modal data match; extracting first level feature information and second level feature information of the first modal data and the second modal data in each training sample respectively; training a cross-modal matching model by constraining, with a preset matching loss function, a matching result between the first modal data and the second modal data based on the first level feature information and a matching result based on the second level feature information, and by constraining, with a preset classification loss function, a classification result based on the first level feature information and the second level feature information of the first modal data respectively, and a classification result based on the first level feature information and the second level feature information of the second modal data respectively.

[0005] According to a second aspect, a cross-modal data matching method is provided, comprising: obtaining to-be-matched modal data; determining target modal data matched with the to-be-matched modal data by a cross-modal matching model, wherein the cross-modal matching model is trained by the method described in any of the implementations of the first aspect.

[0006] According to a third aspect, a training apparatus of a cross-modal matching model is provided, comprising: a first obtaining unit configured to obtain a training sample set, wherein each training sample in the training sample set comprises first modal data, second modal data, and a label representing whether the first modal data and the second modal data match; an extracting unit configured to extract first-level feature information and second-level feature information of the first modal data and the second modal data in each training sample respectively; and a training unit configured to train a cross-modal matching model by constraining, with a preset matching loss function, a matching result between the first modal data and the second modal data based on the first-level feature information and a matching result based on the second-level feature information, and by constraining, with a preset classification loss function, a classification result based on the first-level feature information and the second-level feature information of the first modal data respectively, and a classification result based on the first-level feature information and the second-level feature information of the second modal data respectively.

[0007] According to a fourth aspect, a matching apparatus of cross-modal data is provided, comprising: a second obtaining unit configured to obtain modal data to be matched; and a determining unit configured to determine, by a cross-modal matching model, target modal data matching the modal data to be matched, wherein the cross-modal matching model is trained by the apparatus of any implementation manner of the third aspect.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any implementation manner of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform the method of any implementation manner of the first aspect or the second aspect.

[0010] According to a seventh aspect, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the method of any implementation manner of the first aspect or the second aspect.

[0011] According to the technology of the present disclosure, a method for training a cross-modal matching model by combining multi-level feature information is provided, in the training process, a matching loss function is used to constrain the matching task between the feature information of the corresponding level of the first modal data and the second modal data, and a classification loss function is used to constrain the classification task of the feature information of each level of the first modal data and the classification task of the feature information of each level of the second modal data, thereby improving the matching speed and accuracy of the cross-modal matching model.

[0012] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0014] Figure 1 is an exemplary system architecture diagram that can be applied according to an embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of an embodiment of a training method of a cross-modal matching model according to the present disclosure;

[0016] Figure 3 is a model structure diagram suitable for the training method of the cross-modal matching model of the present disclosure;

[0017] Figure 4 is a cognitive graph for an image-text pair;

[0018] Figure 5 is a schematic diagram of an application scenario of the training method of the cross-modal matching model according to the present embodiment;

[0019] Figure 6 is a flowchart of an embodiment of a matching method of cross-modal data according to the present disclosure;

[0020] Figure 7 is a structural diagram of an embodiment of a training device of a cross-modal matching model according to the present disclosure;

[0021] Figure 8 is a structural diagram of an embodiment of a matching device of cross-modal data according to the present disclosure;

[0022] Figure 9 is a structural diagram of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0024] The collection, storage, use, processing, transmission, provision and disclosure of user personal information in the technical solutions of the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0025] Figure 1 An exemplary architecture 100 of the training method and device of the cross-modal matching model, and the matching method and device of the cross-modal data to which the present disclosure can be applied is shown.

[0026] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connections between the terminal devices 101, 102, 103 constitute a topological network, and the network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] The terminal devices 101, 102, 103 can be hardware devices or software that support network connections for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connections, information acquisition, interaction, display, processing, etc., including but not limited to smartphones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made herein.

[0028] The server 105 can be a server that provides various services, for example, a background processing server that trains a cross-modal matching model in combination with multi-level feature information according to operation instructions of the terminal devices 101, 102, 103. In the training process, a matching loss function is used to constrain the matching task between the corresponding level feature information between the first modal data and the second modal data, and a classification loss function is used to constrain the classification task of the level feature information of the first modal data and the classification task of the level feature information of the second modal data. As an example, the server 105 can be a cloud server.

[0029] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0030] It should also be noted that the training method of the cross-modal matching model and the matching method of the cross-modal data provided by the embodiments of the present disclosure can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the training apparatus of the cross-modal matching model and the matching apparatus of the cross-modal data can include all parts (for example, units) in the server, or all parts (for example, units) in the terminal device, or parts (for example, units) in the server and parts (for example, units) in the terminal device respectively.

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system architecture is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers. When the electronic device on which the training method of the cross-modal matching model and the matching method of the cross-modal data run does not need to transmit data with other electronic devices, the system architecture can only include the electronic device (for example, a server or a terminal device) on which the training method of the cross-modal matching model and the matching method of the cross-modal data run.

[0032] Please refer to Figure 2 , Figure 2 A flowchart of a training method of a cross-modal matching model provided by an embodiment of the present disclosure is shown in FIG. 2. The flowchart 200 includes the following steps:

[0033] In step 201, a training sample set is obtained.

[0034] In this embodiment, the execution subject (for example, a terminal device or a server) of the training method of the cross-modal matching model can obtain the training sample set from a remote or from a local based on a wired network connection mode or a wireless network connection mode. The training sample in the training sample set includes first modal data, second modal data and a label representing whether the first modal data and the second modal data match. Figure 1

[0035] The first modal data and the second modal data can be any two data with different modalities. The modality of the data includes but is not limited to text, voice, image and video.

[0036] As an example, the first modal data is voice data, and the second modal data is image data. As another example, the first modal data is voice data, and the second modal data is text data.

[0037] ​In some optional implementations of the embodiment, the first modality data is image data, and the second modality data is text data. Images and texts are the two most widely existing types of information on the Internet, and cross-modality retrieval between images and texts, i.e., image-text matching, has wide application value. In the implementation, corresponding to the cross-modality data of images and texts, the cross-modality matching model is an image-text matching model, which improves the practicability and application range of the disclosure.

[0038] In step 202, first-level feature information and second-level feature information of the first modality data and the second modality data in each training sample are extracted, respectively.

[0039] In the embodiment, the execution subject can extract the first-level feature information and the second-level feature information of the first modality data and the second modality data in each training sample, respectively.

[0040] As an example, the execution subject can perform feature extraction on the first modality data and the second modality data in each training sample through a feature extraction network in the initial cross-modality matching model to obtain the first-level feature information and the second-level feature information of the first modality data, and the first-level feature information and the second-level feature information of the second modality data.

[0041] The levels of the first-level feature information and the second-level feature information are different. The level is used to represent the semantic expression capability of the feature information. When the level is lower, it indicates that the feature information mainly includes more concrete information. Taking the feature information of an image as an example, the low-level feature information mainly includes the appearance features of objects in the image, and the low-level feature information can be defined as appearance-level feature information. When the level is higher, it indicates that the feature information mainly includes more abstract information. Taking the feature information of an image as an example, the high-level feature information mainly includes the intrinsic features of objects in the image (for example, the category to which each object belongs), and the high-level feature information can be defined as concept-level feature information.

[0042] It can be understood that the feature extraction network generally includes multiple convolution layers, and a subsequent convolution layer further convolves the feature information output by the current convolution layer to obtain higher-level feature information. In the embodiment, the feature information of different convolution layers in the feature extraction network can be determined as the first-level feature information and the second-level feature information. As an example, the feature information obtained by the middle convolution layer can be determined as the first-level feature information, and the feature information obtained by the last convolution layer can be determined as the second-level feature information.

[0043] As Figure 3As shown, a model structure 300 suitable for the training method of the cross-modal matching model of the present disclosure is shown. Among them, the feature extraction network adopts a residual network (for example, ResNet50), including a first residual network 301 for extracting feature information of the first modal data and a second residual network 302 for extracting feature information of the second modal data. The residual network includes 5 feature extraction stages, and the feature information obtained in the third stage is taken as the first level feature information, and the feature information obtained in the fifth stage is taken as the second level feature information.

[0044] In the present embodiment, the above execution subject decomposes the feature information of the modal data into two levels that are semantically complementary, namely, an exterior representation, i.e., exterior level feature information, and a concept representation, i.e., concept level feature information. As shown in the following figure, Figure 4 As shown, a cognitive graph 400 for the image-text pairs is shown. When the 6 image-text pairs 401, 402, 403, 404, 405, 406 shown in the cognitive graph, humans usually first classify them into 3 higher level categories, among which the horse corresponding to the image-text pair 405 and the dog corresponding to the image-text pair 406 belong to the animal category, i.e., ((horse, dog)—>animal); the bus corresponding to the image-text pair 403 and the car corresponding to the image-text pair 402 belong to the vehicle category, i.e., ((bus, car)—>vehicle); the soccer player corresponding to the image-text pair 401 and the tennis player corresponding to the image-text pair 404 belong to the athlete category, i.e., ((tennis player, soccer player)—>athlete). This is because, when people understand or learn a thing, they usually build a category mechanism similar to the "family-class-order-family-genus-species" in biology in the brain, so as to better understand and remember things. Following this mechanism, we use the feature extraction network to extract the exterior level feature information of the image-text pairs 405 and 406 under the animal category, the image-text pairs 402 and 403 under the vehicle category, and the image-text pairs 401 and 404 under the athlete category based on the above classification, and the concept level feature information of each image-text pair, and according to the characteristics of different semantic levels to which they belong, set different learning goals and semantic supervision information through subsequent steps, so as to realize the hierarchical representation of multi-modal data.

[0045] It should be noted that the number of levels of feature information used in the present disclosure is not limited to two, and more levels of feature information can be used according to actual needs.

[0046] In some optional implementations of the present embodiment, the above execution subject can extract the first level feature information and the second level feature information of the first modal data in each training sample in the following way:

[0047] First, the first modal data is subjected to feature extraction by the first modal feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the first modal data, wherein the initial second-level feature information is extracted on the basis of the initial first-level feature information; second, the first-level feature information corresponding to the initial first-level feature information is obtained by the first convolutional layer and the first mapping unit; and third, the second-level feature information corresponding to the initial second-level feature information is obtained by the second mapping unit.

[0048] With reference to the foregoing Figure 3 In the feature extraction process of the second-level feature information of the first modal data, a first residual network (for example, ResNet-50) 301 pre-trained on an ImageNet dataset is used as an image encoder. In order to map the feature information of the first modal data to a common semantic space, we first remove the end full connection layer for classification on the ResNet-50, and the feature information extracted by the ResNet-50 is taken as the initial second-level feature information. Then two stacked second mapping units 303 are added after it for further enhancing its semantic representation ability, which is composed of a full connection layer, a batch normalization (BN) layer and a ReLU activation function in sequence. After sequentially passing through the first residual network 301 and the second mapping unit 303, the second-level feature information of the first modal data can be obtained.

[0049] In the feature extraction process of the first-level feature information of the first modal data, in order to obtain fine-grained first-level feature information that is semantically complementary to the second-level feature information, the middle-level feature of the first residual network 301, that is, the feature information output by the third residual module of the ResNet50 network, is used as the initial first-level feature information. After the initial first-level feature information is input to the first convolutional layer 304, followed by a global average pooling (GAP) layer and the first mapping unit 305, the first-level feature information of the first modal data can be obtained. The first mapping unit 305 is the same as the second mapping unit 303 in structure.

[0050] In the present implementation, a specific way of extracting the first-level feature information and the second-level feature information of the first modal data is provided, which improves the accuracy and expressiveness of the obtained feature information.

[0051] In some optional implementations of the present embodiment, the execution subject can extract the first-level feature information and the second-level feature information of the second modal data in each training sample in the following manner:

[0052] First, the second modal data is subjected to feature extraction by a second modal feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the second modal data, wherein the initial second-level feature information is extracted on the basis of the initial first-level feature information; second, the first-level feature information corresponding to the initial first-level feature information is obtained through a second convolutional layer and a third mapping unit; and third, the second-level feature information corresponding to the initial second-level feature information is obtained through a fourth mapping unit.

[0053] With reference to the foregoing Figure 3 In the feature extraction process of the second-level feature information of the second modal data, a second residual network 302 similar to the ResNet-50 network is designed to extract the feature information of the second modal data. Similar to the first residual network 301, the second residual network 302 also contains 5 residual modules, and the difference lies in that the second residual network 302 replaces the 3x 3 convolution kernel in the basic convolutional layer of the ResNet-50 with a 1x 2 size convolution kernel, which helps to mine the relationship between adjacent elements in the text sequence (when the second modal data is text data). The initial second-level feature information is obtained by removing the classification layer of the second residual network. Then, after passing through a fourth mapping unit 306, the second-level feature information of the second modal data can be obtained. The fourth mapping unit 306 has the same structure as the second mapping unit 303.

[0054] In the feature extraction process of the first-level feature information of the second modal data, in order to obtain first-level feature information (which can be understood as word group level semantic information when the second modal data is text data) that is semantically complementary to the second-level feature information, similar to the first residual network 301, the corresponding middle-level feature of the second residual network 302, i.e., the feature information output by the third residual module of the ResNet50 network, is used as the initial first-level feature information. The initial first-level feature information is input into a second convolutional layer 307, followed by a global average pooling (GAP) layer and a third mapping unit 308, and the first-level feature information of the second modal data can be obtained after processing. The third mapping unit 308 has the same structure as the second mapping unit 303.

[0055] In the present implementation, a specific way of extracting the first-level feature information and the second-level feature information of the second modal data is provided, which improves the accuracy and expressiveness of the obtained feature information.

[0056] Step 203, training to obtain the cross-modal matching model by constraining the matching results between the first modal data and the second modal data based on the first level feature information and the second level feature information with a preset matching loss function, and constraining the classification results based on the first level feature information and the second level feature information of the first modal data and the first level feature information and the second level feature information of the second modal data respectively with a preset classification loss function.

[0057] In the embodiment, the execution subject can constrain the matching results between the first modal data and the second modal data based on the first level feature information and the second level feature information with a preset matching loss function, and constrain the classification results based on the first level feature information and the second level feature information of the first modal data and the first level feature information and the second level feature information of the second modal data respectively with a preset classification loss function, to train to obtain the cross-modal matching model.

[0058] For the matching task between the first modal data and the second modal data, first, the execution subject can determine the first matching result between the first level feature information of the first modal data and the first level feature information of the second modal data, and the second matching result between the second level feature information of the first modal data and the second level feature information of the second modal data; then, determine the loss between the first matching result and the corresponding label information, and the loss between the second matching result and the corresponding label information through a preset matching loss function, and then obtain gradient information according to the loss, and adjust the parameters of the initial cross-modal matching model by using the stochastic gradient descent method.

[0059] For the classification task of the first modal data, first, the execution subject clusters the first level feature information and the second level feature information of the first modal data respectively to obtain clustering results; then, for the first level feature information and the second level feature information, determine the loss between the classification results based on the feature information and the corresponding clustering results respectively through a preset classification loss function, and then obtain gradient information according to the loss, and adjust the parameters of the initial cross-modal matching model by using the stochastic gradient descent method.

[0060] For the classification task of the second modal data, the execution subject can refer to the operation process of the classification task of the first modal data, which is not described here.

[0061] It can be understood that the execution subject can obtain the total loss according to the loss corresponding to the matching task and the loss corresponding to the classification task, and then adjust the parameters of the initial cross-modal matching model according to the total loss. By repeatedly executing the parameter adjustment process, the cross-modal matching model is obtained in response to reaching a preset end condition.

[0062] The preset end condition may be, for example, that the training time exceeds a preset time threshold, the training frequency exceeds a preset frequency threshold, or the loss tends to converge.

[0063] In some optional implementations of the embodiment, the execution subject may guide the matching learning process of the initial cross-modal matching model in the following manner:

[0064] The first ranking loss function is used to constrain the matching result between the first-level feature information of the first modal data and the first-level feature information of the second modal data, and the second ranking loss function is used to constrain the matching result between the second-level feature information of the first modal data and the second-level feature information of the second modal data.

[0065] The first ranking loss function and the second ranking loss function may adopt a bidirectional triple ranking loss function.

[0066] Specifically, the first ranking loss function is represented by the following formula:

[0067]

[0068] wherein λ represents a margin parameter, which may be understood as a distance threshold, S() represents a cosine distance function, respectively represent the first-level feature information of different modal data, represent a positive sample pair corresponding to the first-level feature information, respectively represent a negative sample pair corresponding to the first-level feature information.

[0069] The second ranking loss function is represented by the following formula:

[0070]

[0071] wherein λ represents a margin parameter, which may be understood as a distance threshold, S() represents a cosine distance function, respectively represent the second-level feature information of different modal data, represent a positive sample pair corresponding to the second-level feature information, respectively represent a negative sample pair corresponding to the second-level feature information.

[0072] In the implementation, the ranking loss function is used to constrain the matching degree between the corresponding feature information of different levels of data of different modalities, and the learning efficiency and accuracy of the cross-modal matching model are improved through the constraint learning of the feature information of different levels.

[0073] In some optional implementations of the embodiment, the execution subject may guide the classification learning process of the initial cross-modal matching model in the following manner:

[0074] The classification results obtained based on the first-level feature information of the first modality data and the classification results obtained based on the first-level feature information of the second modality data are constrained by a self-supervised clustering loss function, and the classification results obtained based on the second-level feature information of the first modality data and the classification results obtained based on the second-level feature information of the second modality data are constrained by an instance loss function.

[0075] For learning of the concept-level feature information of the first modality data and the second modality data, the first modality data and the second modality data are taken as a multi-modal data pair, and then a single label id is assigned to each data pair h . Thus, we convert the cross-modal mapping learning process into a classification learning process of two modal data. For the first modality data, the mathematical expression of the classification learning process based on the instance loss function is as follows:

[0076] P i,h = softmax(W h v h )

[0077]

[0078] For the second modality data, the mathematical expression of the classification learning process based on the instance loss function is as follows:

[0079] P t,h = softmax(W h t h )

[0080]

[0081] Wherein, softmax() represents a softmax function for normalization, "1{}" is used to indicate true and false mapped to a class, P represents the probability of classification respectively, and c represents the number of concept-level classes. W h represents a common mapping matrix, v h , t h respectively represent the concept-level feature information of the first modality data and the concept-level feature information of the second modality data, represents an instance loss.

[0082] For learning appearance-level feature information from both the first and second modalities, the aforementioned classification constraints based on concept-level feature information are inapplicable. This is because the training sample set contains many samples with similar appearance-level feature information but drastically different concept-level feature information. For example, cats and tigers look similar and belong to the same family, the Felidae, but they are different species. Based on this consideration, we choose to cluster based on image features (when the first modality data is image data) and generate category labels to constrain the appearance-level feature information. It is worth noting that since image data contains more low-level detailed information than text data (when the second modality data is text data), and since the image encoder uses a pre-trained ResNet-50, we choose to cluster based on image features rather than text features.

[0083] Specifically, we use the k-means algorithm to divide the C data points in the training sample set into k data sets S = {S1, ..., S2}. k}. Here, C represents the amount of data in the training sample set. First, we use a ResNet-50 pre-trained on ImageNet to extract features from the image data in the training sample set, obtaining an image feature information set X = {x1, ..., x...}. c To initialize the clustering model, k cluster centers μ = {μ1, ..., μ2} are randomly initialized. k The clustering optimization objective function J is defined as follows:

[0084]

[0085] z i =argmin j ||x i -μ j || 2

[0086] Among them, z i Indicates training sample x i The index of the assigned cluster. When the iteration converges, we will obtain the cluster labels Z = {z1, ..., z2}. c The category label used as the first-level feature information to constrain the first modality data and the second modality data is called the appearance label id. m This means that images with the same appearance and their corresponding text will be defined as a new cross-modal data set, where members share the same appearance label. Similar to the classification constraints of concept-level feature information, the mathematical expression of the classification learning process based on the self-supervised clustering loss function for the first modality data is as follows:

[0087] P i，m= soft max(W m v m )

[0088]

[0089] For the first modality data, the mathematical representation of the classification learning process based on the self-supervised clustering loss function is as follows:

[0090] P t，m = soft max(W m t m )

[0091]

[0092] wherein softmax() represents a softmax function for normalization, "1{}" is used to indicate true and false mapped to a class, P represents the probability of classification respectively, and k represents the number of appearance-level classes. W m represents a common mapping matrix.

[0093] In the present implementation, in the training process of the cross-modality matching model, the cross-modality contrast learning task mainly focuses on the semantic correlation learning between modalities, while the classification task corresponds to the semantic correlation learning within modalities, thereby improving the learning efficiency and accuracy of the cross-modality matching model.

[0094] It should be noted that in the present embodiment, when the cross-modality data includes multiple types, the learning of the cross-modality matching model can be performed between each two types of cross-modality data by referring to the above process. Moreover, in the present embodiment, it is not limited to only learning the first-level feature information and the second-level feature information, and according to actual needs, learning between more levels of feature information can be performed. When the cross-modality matching model learns more levels of feature information, the feature information of multiple levels can be divided into low-level feature information and high-level feature information. For the low-level feature information, the learning of the cross-modality matching model can be guided by referring to the learning process of the first-level feature information; for the high-level feature information, the learning of the cross-modality matching model can be guided by referring to the learning process of the second-level feature information, which will not be described herein again.

[0095] Continuing to refer to Figure 5 , Figure 5 is one schematic diagram 500 of an application scenario of the training method of the cross-modality matching model according to the present embodiment. In Figure 5In the application scenario of the server 501, the server 501 first obtains a training sample set from the database 502. The training sample in the training sample set includes first modal data, second modal data, and a label representing whether the first modal data and the second modal data match; first-level feature information and second-level feature information of the first modal data and the second modal data in each training sample are extracted respectively; a preset matching loss function is used to constrain a matching result obtained based on the first-level feature information between the first modal data and the second modal data, and a matching result based on the second-level feature information, a preset classification loss function is used to constrain classification results obtained based on the first-level feature information and the second-level feature information of the first modal data respectively, and classification results obtained based on the first-level feature information and the second-level feature information of the second modal data respectively, and a cross-modal matching model is trained.

[0096] In the embodiment, a method for training a cross-modal matching model by combining multi-level feature information is provided. In the training process, a matching loss function is used to constrain the matching task between the feature information of the corresponding level of the first modal data and the second modal data, and a classification loss function is used to constrain the classification task of the feature information of each level of the first modal data and the classification task of the feature information of each level of the second modal data, thereby improving the matching speed and accuracy of the cross-modal matching model.

[0097] With reference to Figure 6 , a schematic flow 600 of one embodiment of a cross-modal data matching method according to the present application is shown, including the following steps:

[0098] In step 601, modal data to be matched is obtained.

[0099] In the embodiment, the execution subject (for example, a terminal device or a server in Figure 1 ) of the cross-modal data matching method can obtain the modal data to be matched from a remote or local modal data to be matched based on a wired network connection mode or a wireless network connection mode.

[0100] The modal data to be matched can be any modal data, including but not limited to voice, text, image, and video.

[0101] In step 602, a target modal data matched with the modal data to be matched is determined by using a cross-modal matching model.

[0102] In the embodiment, the execution subject can determine the target modal data matched with the modal data to be matched by using the cross-modal matching model.

[0103] The target modal data can be any modal data different from the modal data to be matched. As an example, the modal data to be matched is text data, and the target modal data is image data.

[0104] In the embodiment, the cross-modal matching model can be applied to cross-modal retrieval, and the retrieval modal data is input into the cross-modal matching model for retrieval to obtain the target modal data.

[0105] In the embodiment, the cross-modal matching model is used to determine the target modal data corresponding to the to-be-matched modal data, thereby improving the matching accuracy and speed between cross-modal data.

[0106] With reference to Figure 7 , to implement the method shown in the above figures, the disclosure provides an embodiment of a training device of a cross-modal matching model. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0107] As shown in Figure 7 , the training device of the cross-modal matching model includes: a first acquisition unit 701 configured to acquire a training sample set, wherein each training sample in the training sample set includes first modal data, second modal data, and a label representing whether the first modal data and the second modal data match; an extraction unit 702 configured to extract first-level feature information and second-level feature information of the first modal data and the second modal data in each training sample, respectively; and a training unit 703 configured to train a cross-modal matching model by using a preset matching loss function to constrain a matching result obtained based on the first-level feature information between the first modal data and the second modal data and a matching result based on the second-level feature information, and using a preset classification loss function to constrain classification results obtained based on the first-level feature information and the second-level feature information of the first modal data, respectively, and classification results obtained based on the first-level feature information and the second-level feature information of the second modal data, respectively.

[0108] In some optional implementations of the embodiment, the extraction unit 702 is further configured to: perform feature extraction on the first modal data by using a first modal feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the first modal data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; obtain the first-level feature information corresponding to the initial first-level feature information by using a first convolutional layer and a first mapping unit; and obtain the second-level feature information corresponding to the initial second-level feature information by using a second mapping unit.

[0109] In some optional implementation forms of the embodiment, the extraction unit 702 is further configured to: perform feature extraction on the second modality data by a second modality feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the second modality data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; obtain the first-level feature information corresponding to the initial first-level feature information by a second convolutional layer and a third mapping unit; and obtain the second-level feature information corresponding to the initial second-level feature information by a fourth mapping unit.

[0110] In some optional implementation forms of the embodiment, the training unit 703 is further configured to: constrain a matching result between the first-level feature information of the first modality data and the first-level feature information of the second modality data by using a first ranking loss function; and constrain a matching result between the second-level feature information of the first modality data and the second-level feature information of the second modality data by using a second ranking loss function.

[0111] In some optional implementation forms of the embodiment, the training unit 703 is further configured to: constrain a classification result based on the first-level feature information of the first modality data and a classification result based on the first-level feature information of the second modality data by using a self-supervised clustering loss function; and constrain a classification result based on the second-level feature information of the first modality data and a classification result based on the second-level feature information of the second modality data by using an instance loss function.

[0112] In some optional implementation forms of the embodiment, the first modality data is image data, and the second modality data is text data.

[0113] In the embodiment, an apparatus for training a cross-modality matching model by combining multi-level feature information is provided. In the training process, a matching loss function is used to constrain a matching task between the corresponding level feature information between the first modality data and the second modality data, and a classification loss function is used to constrain a classification task of each level feature information of the first modality data and a classification task of each level feature information of the second modality data, thereby improving the matching speed and accuracy of the cross-modality matching model.

[0114] With reference to Figure 8 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a cross-modality data matching apparatus, which corresponds to the method embodiment shown in Figure 6 , and the apparatus can be applied in various electronic devices.

[0115] As Figure 8As shown, the cross-modal data matching apparatus comprises: a second acquisition unit 801 configured to acquire to-be-matched modal data; and a determination unit 802 configured to determine, by a cross-modal matching model, target modal data matched with the to-be-matched modal data. The cross-modal matching model is trained by the apparatus described in any of the implementations of the embodiment 600.

[0116] In this embodiment, the target modal data corresponding to the to-be-matched modal data is determined by the cross-modal matching model, thereby improving the matching accuracy and speed between cross-modal data.

[0117] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the training method of the cross-modal matching model and the matching method of cross-modal data described in any of the above embodiments when executed.

[0118] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium storing computer instructions for enabling a computer to implement the training method of the cross-modal matching model and the matching method of cross-modal data described in any of the above embodiments when executed.

[0119] The present disclosure provides a computer program product, which, when executed by a processor, can implement the training method of the cross-modal matching model and the matching method of cross-modal data described in any of the above embodiments.

[0120] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0121] As Figure 9As shown, the device 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0122] A plurality of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, and the like; an output unit 907, such as various types of displays, speakers, and the like; a storage unit 908, such as a magnetic disk, an optical disk, and the like; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0123] The computing unit 901 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 901 performs various methods and processes described above, such as the training method of the cross-modal matching model, the matching method of cross-modal data. For example, in some embodiments, the training method of the cross-modal matching model, the matching method of cross-modal data can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the cross-modal matching model, the matching method of cross-modal data described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the training method of the cross-modal matching model, the matching method of cross-modal data by any other appropriate means (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0125] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0127] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0128] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services; or can be a server of a distributed system, or a server combined with a blockchain.

[0130] According to the technical scheme of the embodiment of the present disclosure, a method for training a cross-modal matching model combined with multi-level feature information is provided. In the training process, a matching loss function is used to constrain the matching task between the corresponding level feature information between the first modal data and the second modal data, and a classification loss function is used to constrain the classification task of the feature information of each level of the first modal data and the classification task of the feature information of each level of the second modal data, thereby improving the matching speed and accuracy of the cross-modal matching model.

[0131] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions provided by the present disclosure can be achieved, which is not limited herein.

[0132] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method of a cross-modal matching model, comprising: obtaining a training sample set, wherein each training sample in the training sample set comprises first modal data, second modal data, and a label representing whether the first modal data and the second modal data match, the data modalities of the first modal data and the second modal data are different, and the data modalities comprise text, speech, image, and video; extracting first-level feature information and second-level feature information of the first modal data and the second modal data in each training sample, respectively, wherein the first-level feature information is obtained through a first feature extraction layer in a feature extraction network, the second-level feature information is obtained through a second feature extraction layer in the feature extraction network, and the second feature extraction layer is located after the first feature extraction layer; constraining, by using a preset matching loss function, a matching result obtained based on the first-level feature information between the first modal data and the second modal data and a matching result based on the second-level feature information, determining a first classification label of the first modal data and the second modal data based on clustering of the first-level feature information, constraining, by using a self-supervised clustering loss function, a classification result obtained based on the first-level feature information of the first modal data and a classification result obtained based on the first-level feature information of the second modal data according to the first classification label, and constraining, by using an instance loss function, a classification result obtained based on the second-level feature information of the first modal data and a classification result obtained based on the second-level feature information of the second modal data according to a second classification label preset for the first modal data and the second modal data, to obtain the cross-modal matching model.

2. The method of claim 1, wherein, extracting the first-level feature information and the second-level feature information of the first modal data in each training sample comprises: extracting features of the first modal data by using a first modal feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the first modal data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; obtaining the first-level feature information corresponding to the initial first-level feature information by using a first convolutional layer and a first mapping unit; obtaining the second-level feature information corresponding to the initial second-level feature information by using a second mapping unit.

3. The method of claim 1, wherein, extracting the first-level feature information and the second-level feature information of the second modal data in each training sample comprises: extracting features of the second modal data by using a second modal feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the second modal data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; obtaining the first-level feature information corresponding to the initial first-level feature information by using a second convolutional layer and a third mapping unit; obtaining the second-level feature information corresponding to the initial second-level feature information by using a fourth mapping unit.

4. The method of claim 1, wherein, The matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data and the second modality data are constrained by a preset matching loss function, and the matching results based on the first-level feature information and the matching results based on the second-level feature information between the first modality data ​ ​ ​ ​ ​ ​ ​ ​ ​ 7. The apparatus of claim 6, wherein, ​ The first modality data is subjected to feature extraction by a first modality feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the first modality data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; the initial first-level feature information is subjected to a first convolutional layer and a first mapping unit to obtain first-level feature information corresponding to the initial first-level feature information; and the initial second-level feature information is subjected to a second mapping unit to obtain second-level feature information corresponding to the initial second-level feature information.

8. The apparatus of claim 6, wherein, The extraction unit is further configured to: The second modality data is subjected to feature extraction by a second modality feature extraction network to obtain initial first-level feature information and initial second-level feature information corresponding to the second modality data, wherein the initial second-level feature information is extracted based on the initial first-level feature information; the initial first-level feature information is subjected to a second convolutional layer and a third mapping unit to obtain first-level feature information corresponding to the initial first-level feature information; and the initial second-level feature information is subjected to a fourth mapping unit to obtain second-level feature information corresponding to the initial second-level feature information.

9. The apparatus of claim 6, wherein, The training unit is further configured to: The matching result between the first-level feature information of the first modality data and the first-level feature information of the second modality data is constrained by a first ranking loss function; and the matching result between the second-level feature information of the first modality data and the second-level feature information of the second modality data is constrained by a second ranking loss function.

10. A cross-modality data matching apparatus, comprising: a second acquisition unit configured to acquire modality data to be matched; a determination unit configured to determine target modality data matched with the modality data to be matched by a cross-modality matching model, wherein the cross-modality matching model is trained by any one of claims 6-9.

11. An electronic device, comprising: comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer perform the method of any one of claims 1-5.

13. A computer program product, comprising: A computer program, when executed by a processor, implements the method of any one of claims 1-5. A computer program, when executed by a processor, implements the method of any one of claims 1-5.

Images