Method for multimodal data matching degree calculation and calculation model training, related apparatus

Abstract

The present disclosure provides a multi-modal data matching degree calculation model training, a multi-modal data matching degree calculation method and device, an electronic device, a computer readable storage medium and a computer program product, relating to the technical fields of artificial intelligence such as deep learning, image processing and computer vision. The method comprises: obtaining first sample data and second sample data different in mode, determining a semantic confusion degree based on the semantic feature distance between the first sample data and the second sample data, then constructing a contrast learning loss function including the semantic confusion degree parameter, and then using the contrast learning loss function to train an initial multi-modal data matching degree calculation model in a contrast learning manner to obtain a target multi-modal data matching degree calculation model. The multi-modal data matching degree calculation model provided by the embodiment can more accurately calculate the matching degree between data of different modes.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence technology such as deep learning, image processing and computer vision, and in particular to a training method for a multimodal data matching degree calculation model and a method for calculating multimodal data matching degree, as well as corresponding devices, electronic devices, computer-readable storage media and computer program products. Background Technology

[0002] Cross-modal matching aims to establish semantic relationships between data from different modalities, where data from one modality is used as a query term to match other data from other modalities with the same or similar semantics. Among these, cross-modal retrieval between images and text, the two most prevalent types of information on the Internet, namely image-text matching, is widely considered to be the core task of cross-modal retrieval. Summary of the Invention

[0003] This disclosure provides embodiments of a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for a multimodal data matching degree calculation model.

[0004] In a first aspect, embodiments of this disclosure propose a training method for a multimodal data matching degree calculation model, comprising: acquiring first sample data and second sample data with different modalities; constructing a contrastive learning loss function including a semantic perplexity parameter, the semantic perplexity parameter being determined based on the semantic feature distance between the first sample data and the second sample data; and using the contrastive learning loss function to train an initial multimodal data matching degree calculation model in a contrastive learning manner to obtain a target multimodal data matching degree calculation model.

[0005] Secondly, embodiments of this disclosure propose a training apparatus for a multimodal data matching degree calculation model, comprising: a sample data acquisition unit configured to acquire first sample data and second sample data with different modalities; a loss function construction unit configured to construct a contrastive learning loss function including a semantic perplexity parameter, the semantic perplexity parameter being determined based on the semantic feature distance between the first sample data and the second sample data; and a training unit for the multimodal data matching degree calculation model configured to use the contrastive learning loss function to train an initial multimodal data matching degree calculation model in a contrastive learning manner to obtain a target multimodal data matching degree calculation model.

[0006] Thirdly, embodiments of this disclosure propose a method for calculating multimodal data matching degree, comprising: acquiring a test data pair consisting of first data and second data with different modalities; calling a preset multimodal data matching degree calculation model to process the test data pair and obtain the semantic matching degree of the test data pair; wherein, the multimodal data matching degree calculation model is trained in a contrastive learning manner based on a contrastive learning loss function, the contrastive learning loss function includes a semantic perplexity parameter, the semantic perplexity parameter being determined based on the semantic feature distance between the first sample data and the second sample data with different modalities.

[0007] Fourthly, embodiments of this disclosure propose a multimodal data matching degree calculation device, comprising: a data acquisition unit configured to acquire a test data pair consisting of first data and second data with different modalities; and a matching degree calculation unit configured to call a preset multimodal data matching degree calculation model to process the test data pair and obtain the semantic matching degree of the test data pair; wherein the multimodal data matching degree calculation model is trained in a contrastive learning manner based on a contrastive learning loss function, the contrastive learning loss function including a semantic perplexity parameter, the semantic perplexity parameter being determined based on the semantic feature distance between the first sample data and the second sample data with different modalities.

[0008] Fifthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement a training method for a multimodal data matching degree calculation model as described in any implementation of the first aspect or a multimodal data matching degree calculation method as described in any implementation of the third aspect.

[0009] In a sixth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer, when executed, to implement a training method for a multimodal data matching degree calculation model as described in any implementation of the first aspect or a multimodal data matching degree calculation method as described in any implementation of the third aspect.

[0010] In a seventh aspect, embodiments of this disclosure provide a computer program product including a computer program, which, when executed by a processor, can implement a training method for a multimodal data matching degree calculation model as described in any implementation of the first aspect or a multimodal data matching degree calculation method as described in any implementation of the third aspect.

[0011] The multimodal data matching degree calculation model training and multimodal data matching degree calculation method provided in this embodiment of the disclosure obtains first sample data and second sample data with different modalities. Then, a contrastive learning loss function including a semantic perplexity parameter is constructed. This contrastive learning loss function is used to train an initial multimodal data matching degree calculation model in a contrastive learning manner. The semantic perplexity parameter is determined based on the semantic feature distance between the first sample data and the second sample data. Finally, the initial multimodal data matching degree calculation model is trained using the first sample data, the second sample data, and the contrastive learning loss function to obtain a target multimodal data matching degree calculation model.

[0012] The training method and calculation method of the multimodal data matching degree calculation model provided in this disclosure add a semantic perplexity parameter to the loss function constructed based on the conventional contrastive learning idea. Since the semantic perplexity parameter is determined based on the semantic feature distance between the first sample data and the second sample data with different modalities, the contrastive learning loss function constructed with the semantic perplexity parameter can adjust the attention to cross-modal matching of samples according to the magnitude of semantic perplexity during the model training stage. This allows the matching accuracy to be improved by increasing the attention to input data with high semantic perplexity candidate data when using the multimodal data matching degree calculation model in the future.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0015] Figure 1 This is an exemplary system architecture to which this disclosure can be applied;

[0016] Figure 2 A flowchart illustrating a training method for a multimodal data matching degree calculation model provided in this disclosure embodiment;

[0017] Figure 3 A flowchart illustrating a method for extracting semantic features based on a memory bank, provided in this embodiment of the disclosure;

[0018] Figure 4 A schematic diagram of the network structure of the multimodal data matching degree calculation model obtained by the training method of the multimodal data matching degree calculation model in an application scenario provided in an embodiment of this disclosure;

[0019] Figure 5 This is a structural block diagram of a training device for a multimodal data matching degree calculation model provided in an embodiment of the present disclosure;

[0020] Figure 6 A structural block diagram of a multimodal data matching degree calculation device provided in this disclosure embodiment;

[0021] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for performing a training method and / or a multimodal data matching degree calculation method for executing a multimodal data matching degree calculation model, as provided in an embodiment of this disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0023] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information (such as the first sample data and the second sample data, which include user-related identity information, facial images, etc.) comply with the provisions of relevant laws and regulations, have taken necessary confidentiality measures, and do not violate public order and good morals.

[0024] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the methods, apparatus, electronic devices, and computer-readable storage media disclosed herein for training face recognition models and recognizing faces are applied.

[0025] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0026] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include cross-modal matching applications and cross-modal data matching degree calculation applications.

[0027] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0028] Server 105 can provide various services through its built-in applications. Taking a cross-modal data matching degree calculation application that can provide image-text matching degree calculation services as an example, when running this image-text matching application, server 105 can achieve the following effects: after obtaining the test data pair consisting of the first data represented as image data and the second data represented as text data, it calls the preset multimodal data matching degree calculation model to process the test data pair and obtain the semantic matching degree of the test data pair; wherein, the multimodal data matching degree calculation model is trained in a contrastive learning manner based on the contrastive learning loss function, which includes a semantic perplexity parameter, and the semantic perplexity parameter is determined based on the semantic feature distance between the first sample data and the second sample data of different modalities.

[0029] The multimodal data matching degree calculation model can be trained by the training class application of the multimodal data matching degree calculation model built into the server 105 according to the following steps: acquire first sample data and second sample data with different modalities; construct a contrastive learning loss function including a semantic perplexity parameter, which is determined based on the semantic feature distance between the first sample data and the second sample data; use the contrastive learning loss function to train the initial multimodal data matching degree calculation model in a contrastive learning manner to obtain the target multimodal data matching degree calculation model.

[0030] Because training a multimodal data matching degree calculation model requires significant computing resources and strong computing power, the training methods for the multimodal data matching degree calculation model provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the training device for the multimodal data matching degree calculation model is also generally located in the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also complete the aforementioned calculations performed by the server 105 through the multimodal data matching degree calculation model training application installed on them, thereby outputting the same results as the server 105. Accordingly, the training device for the multimodal data matching degree calculation model can also be located in terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0031] Of course, the server used to train the multimodal data matching degree calculation model can be different from the server used to call the trained multimodal data matching degree calculation model. Specifically, the multimodal data matching degree calculation model trained by server 105 can also be used to obtain a lightweight multimodal data matching degree calculation model suitable for being placed into terminal devices 101, 102, and 103 through model distillation. That is, the lightweight multimodal data matching degree calculation model in terminal devices 101, 102, and 103 can be flexibly selected according to the recognition accuracy required in actual needs, or the more complex multimodal data matching degree calculation model in server 105 can be selected.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of a training method for a multimodal data matching degree calculation model provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0034] Step 201: Obtain first sample data and second sample data with different modalities.

[0035] In this embodiment, the execution body of the training method for the multimodal data matching degree calculation model (e.g., Figure 1The server shown obtains first sample data and second sample data, wherein the first sample data and second sample data have different modalities. For example, the first sample data is image data and the second sample data is text data. Or, for example, the first sample data is video data and the second sample data is text data.

[0036] It should be noted that the first and second sample data can be obtained directly from the local storage device by the aforementioned executing entity, or from a non-local storage device (e.g., Figure 1 The data is obtained from the terminal devices 101, 102, and 103 shown. The local storage device can be a data storage module located within the execution entity, such as a server hard drive. In this case, the two original images and their sorting information can be quickly retrieved locally. The non-local storage device can also be any other electronic device configured to store data, such as user terminals. In this case, the execution entity can obtain the required first and second sample data by sending a retrieval command to the electronic device.

[0037] Step 202: Construct a contrastive learning loss function that includes a semantic perplexity parameter.

[0038] In this embodiment, after acquiring the first sample data and the second sample data, the semantic features of the first sample data and the semantic features of the output sample are acquired respectively. After establishing a semantic perplexity parameter to characterize the semantic feature distance between the first sample data and the second sample data, a contrastive learning loss function including the semantic perplexity parameter is constructed based on the semantic perplexity parameter to train the initial multimodal data matching degree calculation model in a contrastive learning manner.

[0039] Contrastive learning is essentially a self-supervised learning method used to learn general features of a dataset by having the model learn the similarities or differences of specific data points without labels. In this disclosure, it is mainly used to find semantic similarities or differences between different modalities of data.

[0040] The semantic perplexity parameter can be determined based on the calculation method of the feature distance between the semantic features corresponding to the second sample data corresponding to the first sample data, such as Euclidean distance, Hamadin distance, and Mahalanobis distance. In some embodiments, multiple numerical ranges can also be set in advance to identify the semantic perplexity, so that the semantic perplexity parameter can be determined according to the numerical range into which the generated feature distance falls.

[0041] Step 203: Using the contrastive learning loss function, train the initial multimodal data matching degree calculation model in a contrastive learning manner to obtain the target multimodal data matching degree calculation model.

[0042] In this embodiment, based on the contrastive learning loss function including semantic perplexity parameters constructed in step 202 above, the initial multimodal data matching degree calculation model is trained using the first sample data, the second sample data, and the contrastive learning loss function including semantic perplexity parameters to obtain the target multimodal data matching degree calculation model.

[0043] The training method for the multimodal data matching degree calculation model provided in this disclosure adds a semantic perplexity parameter determined by the semantic feature distance between first and second sample data with different modalities to the loss function constructed based on the conventional contrastive learning idea. The multimodal data matching degree calculation model trained by this contrastive learning loss function can adjust the attention to calculating the matching degree of different modal data according to the semantic perplexity between the multimodal data pairs used to calculate the matching degree, so as to obtain the matching degree of multimodal data more accurately.

[0044] In some optional implementations of this embodiment, constructing a contrastive learning loss function based on semantic perplexity for training the initial multimodal data matching degree calculation model in a contrastive learning manner includes: obtaining the initial contrastive learning loss function used to supervise model training in a contrastive learning manner; using the cosine relationship of semantic features between the first sample data and the second sample data to characterize the semantic perplexity parameter; and constructing a contrastive learning loss function based on the initial contrastive learning loss function and the semantic perplexity parameter.

[0045] Specifically, after obtaining the semantic features of the first sample data and the second sample data respectively, the semantic perplexity parameter can be characterized by the cosine relationship between the semantic features of the first sample data and the semantic features of the second sample data, which can be used to characterize the semantic feature distance between the semantic features of the sample data and the semantic features of the second sample data. The contrastive learning loss function is then constructed based on the initial contrastive learning loss function and the semantic perplexity parameter, so as to quickly and easily characterize the semantic features between the semantic features of the first sample data and the semantic features of the corresponding second sample data, thereby improving the labeling efficiency of semantic perplexity.

[0046] In some optional embodiments, when training the initial multimodal data matching degree calculation model using a first sample data set including multiple first sample data and a second sample data set including multiple second sample data as samples, i.e. when the first sample data and / or the second sample data are represented as data sets, the semantic feature distance between the first sample data (first sample data set) and each second sample data included in the second sample data set can be determined based on the above method, and the semantic perplexity between the first sample data as input data and the second sample data set can be statistically obtained. At this time, the semantic perplexity can be determined based on the following mathematical expression:

[0047]

[0048]

[0049] Where E() represents the expected value, σ() represents the activation function (Sigmoid) that normalizes the reciprocal of the SD value, per(i) represents the semantic perplexity of the first sample data i relative to the second sample data set; ε represents the hyperparameter used to adjust the smoothness of the function, and S ij This represents the feature distance between the first sample data i and the second sample data j.

[0050] Furthermore, obtain the initial contrastive learning loss function L for model training using the contrastive learning approach. PCL_I (V,T):

[0051]

[0052]

[0053] L PCL_I (T,V)=l PCL_I (T,V)+l PCL_I (V,T) ⑸

[0054] Where V represents the first sample data, T represents the second sample data, μ is the temperature parameter, γ is a boundary parameter, N and Q are the number of samples in the mini-batch, and S... nq =cos(V n ,T q ), S qn =cos(T) q V n ), S nn =cos(V n T n ) and S qq =cosine(T q V q Both ) represent cosine similarity.

[0055] Then, the semantic perplexity obtained based on formula (2) above is introduced into the L obtained based on formula (5) above. PCL_I In (T,V), a contrastive learning loss function L is generated based on semantic perplexity and used to train the initial multimodal data matching degree calculation model in a contrastive learning manner. PCL (T,V), specifically:

[0056]

[0057]

[0058] L PCL (T,V)=l PCL_I (T,V)+l PCL_I (V,T) ⑻

[0059] Where per(n) and per(q) represent V n and T q The semantic perplexity is used to adaptively assign weights to each negative sample. The remaining parameters are as described in formulas (3) to (5) above, and will not be repeated here.

[0060] Please refer to Figure 3 , Figure 3 A flowchart for obtaining semantic features of first sample data and second sample data is provided in this embodiment of the disclosure, that is, for... Figure 2 The semantic features involved in step 202 of the illustrated embodiment provide a specific acquisition method, wherein process 300 includes the following steps:

[0061] Step 301: Obtain multiple first sample data and multiple second sample data.

[0062] In this embodiment, in the scenario of training a multimodal data matching degree calculation model using multiple different first sample data and multiple different second sample data, multiple first sample data and multiple second sample data are obtained.

[0063] Step 302: Store the semantic features of each first sample data into the first memory bank and store the semantic features of each second sample data into the second memory bank.

[0064] In this embodiment, a feature extractor is used to extract the semantic features of each first sample data and the corresponding semantic features of each sample data. The semantic features of the first sample data are stored in the first memory bank, and the semantic features of the second sample data are stored in the second memory bank. For example, when the first sample data is image data and the second sample data is text data, a target detector can be used to extract the semantic features of the first sample data, and a text feature extractor can be used to extract the semantic features of the second sample data.

[0065] Among them, a memory bank is a sample storage device used in contrastive learning. Taking image-based sample data as an example, it can store the semantic features corresponding to the sample image data, so that the memory bank can be used to quickly extract the semantic features corresponding to the sample image data.

[0066] Among them, the preferred method is to use a queue (Queue) first-in-first-out method to continuously store the first sample data, the second sample data, and their corresponding semantic features into the memory bank during the training iteration process.

[0067] In practice, after extracting the semantic features of the first and second sample data respectively, global average pooling (GAP, used to minimize overfitting by reducing the number of model parameters) can be used to generate the first global feature of the first sample data and the second global feature of the second sample data, so as to improve the quality of the semantic features stored in the first and second memory banks.

[0068] In step 303, the encoders of the first memory bank and the second memory bank are updated with momentum respectively, and the semantic features of the first sample data and the second sample data are extracted from the first memory bank and the second memory bank that have completed the momentum update.

[0069] Momentum update is a model parameter update technique that differs from gradient backpropagation optimization. With momentum update, the updated key encoder parameters slowly change in sync with the query encoder parameters, allowing the key encoder to maintain good consistency with the semantic features output by the query encoder while avoiding the significant computational and storage overhead required by gradient backpropagation optimization. Subsequently, semantic features of the first sample data can be extracted from the first memory bank (after momentum update) and the second memory bank (after momentum update), respectively. Semantic features of the second sample data

[0070] In this embodiment, the encoder of the first memory bank The encoder of the second memory bank respectively with g V () and g T () is used as the reference encoder for parameter updates, where the reference encoder g V () and g T () can also be called a key encoder, the encoder to be updated. and Also known as a query encoder, we use Query Encoder Model parameters, using Indicates query encoder g V ()(g T The mathematical form of the momentum update process can be expressed as follows: ()) model parameters.

[0071]

[0072]

[0073] Where m = 0.995 is called the momentum renewal coefficient.

[0074] Step 304: Determine the semantic perplexity parameter based on the semantic feature distance between the first semantic feature and the second semantic feature.

[0075] In this embodiment, with Figure 2 The embodiment shown, as well as the method of semantic feature distance based on the first semantic feature and the second semantic feature described in some optional embodiments later, are the same and will not be repeated here.

[0076] This embodiment is based on the above. Figure 2 Based on the illustrated embodiment, the memory bank method is further used to process each first sample data and each second sample data to avoid limiting the model performance during the training process of the initial multimodal data matching degree calculation model due to the small batch size of the sample data including the first sample data and the second sample data, so as to further improve the performance of the trained multimodal data matching degree calculation model.

[0077] In some optional implementations of this embodiment, storing the semantic features of each of the first sample data into the first memory bank and storing the semantic features of each of the second sample data into the second memory bank includes: storing at least two of the first sample data into the first memory bank in the form of a set and storing at least two of the second sample data into the second memory bank in the form of a set.

[0078] Specifically, in the process of storing the semantic features of the first sample data into the first memory bank and the semantic features of the second sample data into the second memory bank, multiple sets of first sample data and sets of second sample data can be constructed respectively. Each set of first sample data contains at least two sets of first sample data and each set of second sample data contains at least two sets of second sample data. Subsequently, the semantic features corresponding to the first sample data and the semantic features of the first sample data are stored into the first memory bank and the second memory bank respectively in the form of sets, so as to improve the training efficiency of the multimodal data matching degree calculation model through batch training.

[0079] In some optional implementations of this embodiment, at least one of the following constraints is imposed on the contrastive learning loss function: first sample data, second sample data, semantic features of the first sample data obtained based on the first memory bank, and semantic features of the second sample data obtained based on the second memory bank.

[0080] Specifically, depending on specific needs, constraints can be imposed on at least one of the following in the determined contrastive learning loss function: the first sample data, the second sample data, the semantic features of the first sample data obtained from the first memory bank, and the semantic features of the second sample data obtained from the second memory bank. This can improve the training effect of the initial multimodal data matching degree calculation model. For example, constraints can be imposed on mini-batch data pairs (T,V), denoted as... Constraints are imposed on the dual first and second memory banks, denoted as . At this point, the final overall contrastive learning loss function can be:

[0081]

[0082] The above embodiments illustrate from various aspects how to train a multimodal data matching degree calculation model. To highlight the effectiveness of the trained multimodal data matching degree calculation model in practical applications, this disclosure also provides a specific implementation method for using this model to calculate the matching degree between multimodal data:

[0083] First, obtain test data pairs consisting of first and second data with different modalities.

[0084] Specifically, test data pairs consisting of first data and second data with different modalities are acquired, such as first data representing image data and second data representing text data.

[0085] Furthermore, the preset multimodal data matching degree calculation model is invoked to process the test data pairs to obtain the semantic matching degree of the test data pairs.

[0086] Specifically, call the above Figure 2 In the embodiment shown, after training a multimodal data matching degree calculation model based on a contrastive learning loss function constructed using semantic perplexity, the multimodal data matching degree calculation model is used to process the test data pair to obtain the multimodal data matching degree between the first data and the second data.

[0087] The multimodal data matching degree calculation method provided in this embodiment, after obtaining a test data pair consisting of first data and second data with different modalities, calls a multimodal data matching degree calculation model trained with a contrastive learning loss including semantic perplexity parameters to process the test data pair and obtain the matching degree between the first data and the second data included in the test data pair. The multimodal data matching degree calculation model can determine the corresponding matching degree calculation strategy and matching resources based on the semantic perplexity parameter between the first data and the second data. That is, it uses a matching degree calculation strategy and matching resources that match the size of the semantic perplexity parameter to calculate the matching degree between the first data and the second data, rather than the method of uniformly or fixedly allocating computing resources in the prior art to perform matching degree calculation. Therefore, matching degree calculation resources can be tilted to the first data and the second data with high semantic perplexity, thereby improving the accuracy of matching degree calculation.

[0088] To enhance understanding, this disclosure also provides a specific implementation scheme, taking into account the particular application scenario of calculating the matching degree between text and image data:

[0089] Please see Figure 4 , Figure 4 A schematic diagram of a network structure for calculating the degree of matching between images and text is shown. Figure 4 In the process, the sample image input on the left is processed by an object detector such as Faster-RCNN (a model that combines RCNN and FastRCNN, RCNN, Region-CNN, Fast RCNN, and CNN, Convolutional Neural Network) to extract local image features (it should be noted that all features in this embodiment are semantic features). After global average pooling, global image features are obtained. The sample text input on the right is processed by a text feature extractor such as BERT (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) to extract local text features. After global average pooling, global text features are obtained.

[0090] Regarding the potential issue of insufficient batch size... Figure 4A momentum contrastive learning framework was also introduced, in which memory banks were established for image and text data respectively. During the training iteration, small batches of data were continuously stored into the memory banks in a first-in-first-out queue manner. Simultaneously, the encoders of the image and text memory banks were updated with momentum updates using the target detector (FastCNN if FastCNN is used for local feature extraction of sample images, here it is FastCNN) and the text feature extractor (BERT if BERT is used for local feature extraction of sample text, here it is BERT) as baseline models. Finally, the target image features and target text features output by the memory banks after momentum updates were obtained.

[0091] Next, using the target image features, global image features, target text features, and global text features output from the image memory bank, and based on the pre-constructed contrastive learning loss function (please refer to Equation 11 above and the related pre-formulas for deriving Equation 11), a model for calculating the matching degree of image and text data is trained in a contrastive learning manner. Furthermore, global average pooling can also be performed on the target image features and target text features to ensure consistency in feature characteristics with the global image features and global text features.

[0092] In other words, the trained image-text matching degree calculation model, when used, after inputting a pair of image and text data, uses the semantic feature extraction method extracted during training based on the contrastive learning loss function to first extract the semantic features of the image data and text data respectively, and calculates the distance of the semantic features in a cosine manner. Based on the calculated distance, the image-text matching degree is generated and finally output as the model output.

[0093] This embodiment is illustrated by... Figure 4 This paper presents a specific model training and usage scheme for calculating the matching degree of image-text data. It should be noted that, in addition to calculating the matching degree of image-text data, the scheme provided in this application can also be applied to calculating the matching degree between different modal data combinations such as image-speech and text-speech. This provides a model for calculating the matching degree between different modal data. It only requires inputting sample data of two modalities matching the final application into the model training results and using a feature extractor corresponding to each modality. Therefore, similar to a target detector extracting local features from image data, when extracting local features from speech data, a CNN or DNN can be used as its feature extractor.

[0094] Further reference Figure 5 and Figure 6As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a training device for a multimodal data matching degree calculation model and an embodiment of a multimodal data matching degree calculation device. The embodiment of the training device for the multimodal data matching degree calculation model and... Figure 2 The training method embodiment for the multimodal data matching degree calculation model shown corresponds to the multimodal data matching degree calculation device embodiment, which corresponds to the multimodal data matching degree calculation method embodiment. The above-described device can be specifically applied to various electronic devices.

[0095] like Figure 5 As shown, the training device 500 for the multimodal data matching degree calculation model in this embodiment may include: a sample data acquisition unit 501, a loss function construction unit 502, and a training unit 503 for the multimodal data matching degree calculation model. The sample data acquisition unit 501 is configured to acquire first sample data and second sample data with different modalities; the loss function construction unit 502 is configured to construct a contrastive learning loss function including a semantic perplexity parameter, which is determined based on the semantic feature distance between the first sample data and the second sample data; the training unit 503 for the multimodal data matching degree calculation model is configured to use the contrastive learning loss function to train an initial multimodal data matching degree calculation model in a contrastive learning manner to obtain a target multimodal data matching degree calculation model.

[0096] In this embodiment, the specific processing of the sample data acquisition unit 501, the loss function construction unit 502, and the training unit 503 of the multimodal data matching degree calculation model in the training device 500, and the resulting technical effects, can be found in the following references: Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0097] In some optional implementations of this embodiment, the training apparatus for the multimodal data matching degree calculation model further includes: a sample data batch acquisition unit configured to acquire multiple first sample data and multiple second sample data; a memory bank storage unit configured to store the semantic features of each first sample data into a first memory bank and the semantic features of each second sample data into a second memory bank; a momentum update unit configured to perform momentum updates on the encoders of the first memory bank and the second memory bank respectively, and extract a first semantic feature from the first memory bank after momentum update and extract a second semantic feature from the second memory bank after momentum update; and a semantic feature extraction unit configured to determine the semantic perplexity parameter based on the semantic feature distance between the first semantic feature and the second semantic feature.

[0098] In some optional implementations of this embodiment, the memory bank storage unit is further configured to store at least two of the first sample data in the form of a set in the first memory bank and at least two of the second sample data in the form of a set in the second memory bank.

[0099] In some optional implementations of this embodiment, the loss function construction unit 502 includes: an initial loss function acquisition subunit configured to acquire an initial contrastive learning loss function used to supervise model training in the contrastive learning method; a semantic perplexity representation subunit configured to characterize the semantic perplexity parameter using the cosine relationship of the semantic features between the first sample data and the second sample data; and a contrastive learning loss function construction subunit configured to construct the contrastive learning loss function based on the initial contrastive learning loss function and the semantic perplexity parameter.

[0100] In some optional implementations of this embodiment, the training apparatus for the multimodal data matching degree calculation model further includes: a constraint application unit configured to apply constraints to at least one of the following in the contrastive learning loss function: the first sample data, the second sample data, the semantic features of the first sample data obtained based on the first memory bank, and the semantic features of the second sample data obtained based on the second memory bank.

[0101] In some optional implementations of this embodiment, the first sample data includes sample image data, and the second sample data includes sample text data.

[0102] This embodiment exists as a device embodiment corresponding to the above method embodiment. The training device and training method of the multimodal data matching degree calculation model provided in this embodiment add a semantic perplexity parameter determined by the semantic feature distance between the first sample data and the second sample data with different modalities to the loss function constructed based on the conventional contrastive learning idea. The multimodal data matching degree calculation model trained by this contrastive learning loss function can adjust the attention to calculate the matching degree of different modal data according to the semantic perplexity between the multimodal data pairs used to calculate the matching degree, so as to obtain the matching degree of multimodal data more accurately.

[0103] like Figure 6As shown, the multimodal data matching degree calculation device 600 of this embodiment may include: a data acquisition unit 601 to be matched and a matching degree calculation unit 602. The data acquisition unit 601 is configured to acquire a test data pair consisting of first data and second data with different modalities. The matching degree calculation unit 602 is configured to call a preset multimodal data matching degree calculation model to process the test data pair and obtain the semantic matching degree of the test data pair. The multimodal data matching degree calculation model is trained using a contrastive learning loss function, which includes a semantic perplexity parameter. This semantic perplexity parameter is determined based on the semantic feature distance between the first sample data and the second sample data with different modalities.

[0104] In this embodiment, the specific processing of the data acquisition unit 601 and the matching degree calculation unit 602 in the multimodal data matching degree calculation device 600 and the resulting technical effects can be described in the relevant descriptions in the method embodiment, and will not be repeated here.

[0105] This embodiment exists as a device embodiment corresponding to the above method embodiment. The multimodal data matching degree calculation device provided in this embodiment, after acquiring a test data pair consisting of first data and second data with different modalities, calls a multimodal data matching degree calculation model trained with a contrastive learning loss including semantic perplexity parameters to process the test data pair and obtain the matching degree between the first data and the second data included in the test data pair. The multimodal data matching degree calculation model can determine the corresponding matching degree calculation strategy and matching resources based on the semantic perplexity parameter between the first data and the second data. That is, it uses a matching degree calculation strategy and matching resources that match the size of the semantic perplexity parameter to calculate the matching degree between the first data and the second data, rather than the method of uniformly or fixedly allocating computing resources in the prior art to perform matching degree calculation. Therefore, the matching degree calculation resources can be tilted to the first data and the second data with high semantic perplexity, thereby improving the accuracy of matching degree calculation.

[0106] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the training method and / or the multimodal data matching degree calculation method of any of the above embodiments when executed.

[0107] According to embodiments of the present disclosure, the present disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the training method and / or the multimodal data matching degree calculation method of the multimodal data matching degree calculation model described in any of the above embodiments when executed.

[0108] This disclosure provides a computer program product that, when executed by a processor, can implement the training method and / or multimodal data matching degree calculation method of the multimodal data matching degree calculation model described in any of the above embodiments.

[0109] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0111] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as training methods and / or multimodal data matching degree calculation methods for multimodal data matching degree calculation models. For example, in some embodiments, the training methods and / or multimodal data matching degree calculation methods for multimodal data matching degree calculation models can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the training method and / or multimodal data matching degree calculation method of the multimodal data matching degree calculation model described above can be performed. Alternatively, in other embodiments, computing unit 701 can be configured to execute the training method and / or multimodal data matching degree calculation method of the multimodal data matching degree calculation model by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0118] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0119] The technical solution of this disclosure, a training method for a multimodal data matching degree calculation model, adds a semantic perplexity parameter determined by the semantic feature distance between first and second sample data of different modalities to the loss function constructed based on the conventional contrastive learning idea. The multimodal data matching degree calculation model trained by this contrastive learning loss function can adjust the attention to calculating the matching degree of different modal data according to the semantic perplexity between the multimodal data pairs used to calculate the matching degree, so as to obtain the matching degree of multimodal data more accurately.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a multimodal data matching degree calculation model, comprising: Acquire first sample data and second sample data with different modalities. The first sample data includes sample image data, and the second sample data includes sample text data. Obtain the initial contrastive learning loss function used to supervise model training in the contrastive learning approach; characterize the semantic perplexity parameter using the cosine relationship of semantic features between the first sample data and the second sample data; construct the contrastive learning loss function based on the initial contrastive learning loss function and the semantic perplexity parameter; Using the contrastive learning loss function, an initial multimodal data matching degree calculation model is trained in a contrastive learning manner to obtain a target multimodal data matching degree calculation model; Also includes: Acquire multiple sets of the first sample data and multiple sets of the second sample data; The semantic features of each of the first sample data are stored in the first memory bank, and the semantic features of each of the second sample data are stored in the second memory bank; Momentum updates are performed on the encoders of the first memory bank and the second memory bank respectively, and first semantic features are extracted from the first memory bank after momentum update and second semantic features are extracted from the second memory bank after momentum update. The semantic perplexity parameter is determined based on the semantic feature distance between the first semantic feature and the second semantic feature.

2. The method according to claim 1, wherein, The step of storing the semantic features of each of the first sample data into the first memory bank and storing the semantic features of each of the second sample data into the second memory bank includes: At least two sets of the first sample data are stored in the first memory bank, and at least two sets of the second sample data are stored in the second memory bank.

3. The method according to claim 1 or 2, further comprising imposing a constraint on at least one of the following in the contrastive learning loss function: The first sample data, the second sample data, the semantic features of the first sample data obtained based on the first memory bank, and the semantic features of the second sample data obtained based on the second memory bank.

4. A method for calculating the matching degree of multimodal data, comprising: Acquire test data pairs consisting of first data and second data with different modalities, wherein the first data includes image data and the second data includes text data; The test data pair is processed by a preset multimodal data matching degree calculation model to obtain the semantic matching degree of the test data pair. The multimodal data matching degree calculation model is trained using a contrastive learning loss function, which includes a semantic perplexity parameter. This parameter is determined based on the semantic feature distance between first and second sample data of different modalities. The semantic perplexity parameter is determined as follows: multiple first sample data and multiple second sample data are acquired; the semantic features of each first sample data are stored in a first memory bank, and the semantic features of each second sample data are stored in a second memory bank; momentum updates are performed on the encoders of the first and second memory banks, and a first semantic feature is extracted from the first memory bank after momentum update, and a second semantic feature is extracted from the second memory bank after momentum update; the semantic perplexity parameter is determined based on the semantic feature distance between the first and second semantic features.

5. A training device for a multimodal data matching degree calculation model, comprising: The sample data acquisition unit is configured to acquire first sample data and second sample data with different modalities, wherein the first sample data includes sample image data and the second sample data includes sample text data. The loss function construction unit includes: an initial loss function acquisition subunit, configured to acquire an initial contrastive learning loss function used to supervise model training in the contrastive learning approach; a semantic perplexity representation subunit, configured to represent a semantic perplexity parameter using the cosine relationship of semantic features between the first sample data and the second sample data; and a contrastive learning loss function construction subunit, configured to construct the contrastive learning loss function based on the initial contrastive learning loss function and the semantic perplexity parameter. The training unit of the multimodal data matching degree calculation model is configured to train the initial multimodal data matching degree calculation model in a contrastive learning manner using the contrastive learning loss function to obtain the target multimodal data matching degree calculation model. Also includes: The sample data batch acquisition unit is configured to acquire multiple sets of the first sample data and multiple sets of the second sample data; The memory bank storage unit is configured to store the semantic features of each of the first sample data into the first memory bank and the semantic features of each of the second sample data into the second memory bank. The momentum update unit is configured to perform momentum updates on the encoders of the first memory bank and the second memory bank respectively, and extract a first semantic feature from the first memory bank after the momentum update and extract a second semantic feature from the second memory bank after the momentum update. The semantic feature extraction unit is configured to determine the semantic perplexity parameter based on the semantic feature distance between the first semantic feature and the second semantic feature.

6. The apparatus according to claim 5, wherein, The memory bank storage unit is further configured to store at least two sets of the first sample data into the first memory bank and at least two sets of the second sample data into the second memory bank.

7. The apparatus according to claim 5 or 6, further comprising: The constraint applying unit is configured to apply constraints to at least one of the following in the contrastive learning loss function: The first sample data, the second sample data, the semantic features of the first sample data obtained based on the first memory bank, and the semantic features of the second sample data obtained based on the second memory bank.

8. A multimodal data matching degree calculation device, comprising: The data acquisition unit is configured to acquire a pair of test data consisting of first data and second data with different modalities, wherein the first data includes image data and the second data includes text data; A matching degree calculation unit is configured to call a preset multimodal data matching degree calculation model to process the test data pair and obtain the semantic matching degree of the test data pair. The multimodal data matching degree calculation model is trained using a contrastive learning loss function, which includes a semantic perplexity parameter. This parameter is determined based on the semantic feature distance between first and second sample data of different modalities. The semantic perplexity parameter is determined as follows: multiple first sample data and multiple second sample data are acquired; the semantic features of each first sample data are stored in a first memory bank, and the semantic features of each second sample data are stored in a second memory bank; momentum updates are performed on the encoders of the first and second memory banks, and a first semantic feature is extracted from the first memory bank after momentum update, and a second semantic feature is extracted from the second memory bank after momentum update; the semantic perplexity parameter is determined based on the semantic feature distance between the first and second semantic features.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the training method of the multimodal data matching degree calculation model according to any one of claims 1-4 and / or the multimodal data matching degree calculation method according to any one of claims 5-6.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the training method of the multimodal data matching degree calculation model according to any one of claims 1-3 and / or the multimodal data matching degree calculation method according to claim 4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the training method for a multimodal data matching degree calculation model according to any one of claims 1-3 and / or the multimodal data matching degree calculation method according to claim 4.

Images