Data matching method and device, multimodal model processing method and device

Abstract

The present application relates to a data matching method, apparatus, computer device, storage medium and computer program product. The method comprises: encoding first data and second data respectively to obtain a first modal representation and a second modal representation, wherein the first data and the second data are data of different modalities; combining the first modal representation and the second modal representation to obtain a mixed modal representation, and determining adaptive gating weights according to the mixed modal representation; according to the gating weights, fusing the first modal representation and the mixed modal representation to obtain a first fused representation, and fusing the second modal representation and the mixed modal representation to obtain a second fused representation; and determining the matching degree between the first data and the second data according to the first fused representation and the second fused representation. The use of this method can improve data processing performance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data matching method, apparatus, computer equipment, storage medium and computer program product, as well as a multimodal model processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, machine learning has emerged. By building and training machine learning models, these models can be equipped to perform tasks. In the field of vision-language learning, to improve the practicality of the model, a general vision-language pre-training model, also known as a VLP (Vision-Language Pre-training) model, is often pre-trained for image and text processing.

[0003] Traditional VLP models typically use a dual encoder architecture, with one encoder performing unimodal processing on image data and the other on text data. However, in real-world scenarios, paired image and text data may be correlated. This unimodal processing approach fails to effectively leverage the interactive information between these modalities, resulting in poor data processing performance. Summary of the Invention

[0004] Based on this, it is necessary to provide a data matching method, apparatus, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.

[0005] In one aspect, the present application provides a data matching method, comprising:

[0006] Encoding the first data and the second data respectively to obtain a first modal representation and a second modal representation, wherein the first data and the second data are data of different modalities;

[0007] combining the first modal representation and the second modal representation to obtain a hybrid modal representation, and determining adaptive gating weights according to the hybrid modal representation;

[0008] According to the gating weights, fusing the first modality representation and the mixed modality representation to obtain a first fused representation, and fusing the second modality representation and the mixed modality representation to obtain a second fused representation;

[0009] A degree of matching between the first data and the second data is determined according to the first fused representation and the second fused representation.

[0010] In some embodiments, fusing the first intermediate mixed representation into the second intermediate mixed representation to obtain the (i+1)th interactively fused second intermediate mixed representation includes:

[0011] Perform an attention operation on the second intermediate mixed representation output by the i-th interactive fusion and the first intermediate mixed representation, combine the attention operation result with the second intermediate mixed representation output by the i-th interactive fusion, and then perform a layer regularization operation to obtain the second intermediate processed feature;

[0012] The second intermediate processing feature is processed by a forward neural network, and the forward neural network processing result and the second intermediate processing feature are combined and then a layer regularization operation is performed to obtain a second intermediate mixed representation of the i+1th interactive fusion.

[0013] In some embodiments, fusing the first modality representation and the mixed modality representation according to the gating weight to obtain a first fused representation includes:

[0014] Using the gating weight as a coefficient of the mixed modal representation, and using the difference between the value one and the gating weight as a coefficient of the first modal representation;

[0015] The mixed modal representation and the first modal representation are weighted according to the coefficients of the mixed modal representation and the coefficients of the first modal representation to obtain a first fused representation.

[0016] In some embodiments, fusing the second modality representation and the mixed modality representation to obtain a second fused representation according to the gating weights includes:

[0017] Using the gating weight as a coefficient of the mixed modal representation, and using the difference between the value one and the gating weight as a coefficient of the second modal representation;

[0018] The mixed modal representation and the second modal representation are weighted according to the coefficients of the mixed modal representation and the coefficients of the second modal representation to obtain a first fused representation.

[0019] On the other hand, the present application also provides a data matching device, comprising:

[0020] an encoding module, configured to encode the first data and the second data respectively to obtain a first modal representation and a second modal representation, wherein the first data and the second data are data of different modalities;

[0021] a determination module, configured to combine the first modal representation and the second modal representation to obtain a hybrid modal representation, and determine an adaptive gating weight according to the hybrid modal representation;

[0022] a fusion module, configured to fuse the first modality representation and the mixed modality representation to obtain a first fused representation, and to fuse the second modality representation and the mixed modality representation to obtain a second fused representation, according to the gating weights;

[0023] A matching module is used to determine a matching degree between the first data and the second data according to the first fusion representation and the second fusion representation.

[0024] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0025] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0026] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0027] The above-mentioned data matching method, device, computer equipment, storage medium and computer program product encode the first data and the second data belonging to different modalities respectively to obtain the first modal representation and the second modal representation. Then, the first modal representation and the second modal representation are combined to obtain a hybrid modal representation. According to the hybrid modal representation, an adaptive gating weight can be obtained. The gating weight is used to control the proportion of the first modal representation and the second modal representation when they are fused with the hybrid modal representation. That is, information can be intelligently extracted from the single modality / hybrid modality to obtain the first fusion representation and the second fusion representation. The first fusion representation and the second fusion representation obtained in this way can better highlight the relevant information of the first data and the second data, so they can be used to more accurately determine the matching degree between the first data and the second data, greatly improving the data processing effect. In particular, when there is a lack of the first data and the second data, by intelligently extracting information from the hybrid modality, intelligently selecting the proportion of the extracted hybrid modal information, and reconstructing the first fusion representation of the first data and the second fusion representation of the second data through multimodal information, the missing information is compensated, the matching accuracy can be greatly improved, and the data processing performance is improved.

[0028] On the other hand, the present application provides a multimodal model processing method, the method comprising:

[0029] Encoding the first sample data and the second sample data in the training sample pair respectively using the initial model to be trained to obtain a first sample modal representation and a second sample modal representation; the first sample data and the second sample data are sample data of different modalities;

[0030] Obtaining a mixed sample modality representation by combining the first sample modality representation and the second sample modality representation through the initial model, and determining a prediction weight according to the mixed sample modality representation;

[0031] fusing the first sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and performing decoding based on the fused result to obtain a first prediction feature representation;

[0032] fusing the second sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and performing decoding based on the fused result to obtain a second prediction feature representation;

[0033] Determining a first loss based on the true feature representation of the first sample data and the first predicted feature representation, and the true feature representation of the second sample data and the second predicted feature representation;

[0034] Determining a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weight;

[0035] A target loss function is constructed according to the first loss and the second loss, and the initial model to be trained is trained based on the target loss function, so as to obtain a multimodal model after the training is completed.

[0036] In some embodiments, obtaining a mixed sample modality representation by combining the initial model with the first sample modality representation and the second sample modality representation includes:

[0037] Performing multiple interactive fusions on the first sample modality representation and the second sample modality representation through the initial model to obtain a first mixed sample representation biased towards the first modality side and a second mixed sample representation biased towards the second modality side;

[0038] The first mixed sample representation and the second mixed sample representation are combined to obtain a mixed sample modality representation.

[0039] In some embodiments, the mixed sample modality representation includes a first mixed sample representation biased toward a first modality and a second mixed sample representation biased toward a second modality, and the prediction weight includes a first prediction weight determined based on the first mixed sample representation and a second prediction weight determined based on the second mixed sample representation;

[0040] The determining a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weight, includes:

[0041] constructing a first vector based on the alignment labels of the first sample data and the second sample data;

[0042] Determining a target weight according to an average of the first prediction weight and the second prediction weight, and constructing a third vector based on the target weight and a difference between the value one and the target weight;

[0043] A second loss is determined based on the first vector and the third vector.

[0044] On the other hand, the present application also provides a multimodal model processing device, comprising:

[0045] an encoding module, configured to encode, using an initial model to be trained, first sample data and second sample data in a training sample pair to obtain a first sample modal representation and a second sample modal representation; the first sample data and the second sample data are sample data of different modalities;

[0046] a determination module, configured to obtain a mixed sample modality representation by combining the first sample modality representation and the second sample modality representation through the initial model, and determine a prediction weight according to the mixed sample modality representation;

[0047] a fusion module, configured to fuse the first sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and perform decoding based on the fused result to obtain a first prediction feature representation;

[0048] The fusion module is further configured to fuse the second sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and perform decoding based on the fused result to obtain a second prediction feature representation;

[0049] A construction module is configured to determine a first loss based on the true feature representation of the first sample data and the first predicted feature representation, and the true feature representation of the second sample data and the second predicted feature representation; determine a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weight; and construct a target loss function based on the first loss and the second loss;

[0050] A training module is used to train the initial model to be trained based on the target loss function, and obtain a multimodal model after the training is completed.

[0051] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0052] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0053] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0054] The multimodal model processing method, apparatus, computer device, storage medium, and computer program product described above encode first sample data and second sample data belonging to different modalities, respectively, to obtain a first sample modal representation and a second sample modal representation. The first sample modal representation and the second sample modal representation are then combined to obtain a mixed sample modal representation. Prediction weights can be derived from the mixed sample modal representation. These prediction weights are used to control the proportions of the first sample modal representation and the second sample modal representation when they are fused with the mixed sample modal representation. This allows for intelligent extraction of information from single / mixed modalities for subsequent decoding. During training, on the one hand, the loss between the predicted feature representation and the true feature representation is taken into account, and on the other hand, the matching loss between sample data of different modalities (that is, the second loss) is taken into account. This allows the model to learn encoding and decoding capabilities that are closer to the true feature representation. It can also intelligently control the ratio of the referenced single-modal representation and mixed-modal representation during the encoding and decoding process, so that when the alignment quality of the training sample pairs is high, the mixed-modal information can be considered more during the decoding of the single modality; when the training sample pairs are not aligned, the single-modal information can be considered more during the decoding of the single modality. This can better handle the problem of modal content mismatch in the training data set and improve the processing performance of the multimodal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A diagram illustrating an application environment of a data matching method and / or a multimodal model processing method in one embodiment;

[0057] Figure 2 1 is a flow chart of a multimodal model processing method in one embodiment;

[0058] Figure 3 Schematic diagram of the principle of mask processing in one embodiment;

[0059] Figure 4 is a flowchart of a multimodal model processing method in another embodiment;

[0060] Figure 5 1 is a flow chart of a data matching method in one embodiment;

[0061] Figure 6 is a flow chart of a data matching method in another embodiment;

[0062] Figure 7 is a structural block diagram of a multimodal model in one embodiment;

[0063] Figure 8 is a structural block diagram of a data matching device in one embodiment;

[0064] Figure 9 is a structural block diagram of a multimodal model processing device in one embodiment;

[0065] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0067] The data matching method and / or multimodal model processing method provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal and the server can respectively execute the data matching method and / or the multimodal model processing method provided in the embodiment of the present application separately, or they can collaboratively execute the data matching method and / or the multimodal model processing method provided in the embodiment of the present application. Take the server executing the data matching method provided in the present application alone as an example for explanation: the server receives the first data and the second data sent by the terminal, and encodes the first data and the second data respectively to obtain a first modal representation and a second modal representation, and the first data and the second data are data of different modalities; the first modal representation and the second modal representation are combined to obtain a mixed modal representation, and an adaptive gating weight is determined according to the mixed modal representation; according to the gating weight, the first modal representation and the mixed modal representation are fused to obtain a first fused representation, and the second modal representation and the mixed modal representation are fused to obtain a second fused representation; according to the first fused representation and the second fused representation, the matching degree of the first data and the second data is determined. The server feeds back the matching degree to the terminal.

[0068] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0069] It should be noted that this application involves artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, giving them the capabilities of perception, reasoning, and decision-making.

[0070] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0071] This application relates to machine learning, which is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. The pre-trained model is the latest development in deep learning, which integrates the above technologies.

[0072] More specifically, the multimodal models mentioned in the embodiments of this application can be pre-trained models. A pre-training model, also known as a cornerstone model or a large model, refers to a deep neural network (DNN) with a large number of parameters. This model is trained on massive amounts of unlabeled data. The function approximation capabilities of the large-parameter DNN are leveraged to enable the pretrained training model (PTM) to extract common features from the data. Through techniques such as fine tuning, efficient parameter fine tuning (PEFT), and prompt-tuning, the pre-trained model is then adapted for downstream tasks. Therefore, the pre-trained model can achieve ideal results in few-shot or zero-shot scenarios. Based on the data modality processed, PTMs can be categorized into language models (ELMO, BERT, GPT), vision models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato). A multimodal model refers to a model that establishes feature representations of two or more data modalities. Pre-trained models are important tools for outputting artificial intelligence generated content (AIGC) and can also serve as a universal interface for connecting multiple specific task models.

[0073] The solutions provided in the embodiments of this application involve artificial intelligence machine learning, large models and other technologies, which are specifically introduced in detail through the following embodiments.

[0074] Before introducing the data matching method and / or multimodal model processing method provided in this application, the relevant English abbreviations are explained:

[0075] VLP: Vision-Language Pre-training, vision-language pre-training.

[0076] GIAE: Gated Interactive Masked AutoEncoders, gated interactive masked autoencoders.

[0077] GIM: Gated Interactive Mechanism, gated interactive mechanism.

[0078] MAE: Masked AutoEncoders, masked autoencoder.

[0079] Image-text mismatching issue: Image-text mismatching issue.

[0080] Transformer: A neural network architecture based on the attention mechanism that can effectively encode temporal information in the encoder part. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0081] LightVLP: a lightweight Vision-Language Pre-training framework, a lightweight vision-language pre-training model framework, which is also the multimodal model proposed in this application.

[0082] In the field of vision-language learning, to improve model practicality, a general vision-language pre-training model, also known as a VLP (Vision-Language Pre-training) model, is typically pre-trained for image and text processing. Furthermore, in actual business processing, to adapt to diverse downstream tasks, the VLP model can be retrained to obtain a task model suitable for specific business scenarios. Therefore, training a high-performance VLP model is a highly practical project.

[0083] Traditional VLP methods often face the following challenges: (1) The training process is very time-consuming because the text and image inputs are often very long; (2) The embedded information of different modalities is not effectively fused; (3) In the real world, many training image-text alignment data contain a lot of noise and need further processing.

[0084] In view of this, this application proposes a multimodal model processing method that can solve the above-mentioned technical problems, optimize the multimodal information fusion method on the basis of improving the efficiency of the VLP model, and obtain better results. In addition, the multimodal model of this application can better handle the modal data mismatch problem in the training data set during the training process, such as the image-text mismatch problem, thereby improving the processing performance of the model.

[0085] The following is an introduction to the training method of the multimodal model involved in this application:

[0086] In an exemplary embodiment, Figure 2 As shown, a multimodal model processing method is provided, which is applied to computer devices (such as Figure 1 The terminal or server in FIG) is taken as an example to illustrate, including the following steps 202 to 214. Among them:

[0087] In step 202 , the first sample data and the second sample data in the training sample pair are respectively encoded using the initial model to be trained to obtain a first sample modality representation and a second sample modality representation; the first sample data and the second sample data are sample data of different modalities.

[0088] The initial model is a pre-built neural network model, which becomes a multimodal model after training. The initial model in this application may include two encoders and decoders with a symmetrical relationship, two decoders with a symmetrical relationship, and a cross transformer for processing interactive information.

[0089] The training sample pair is a pre-constructed data pair for model training. The training sample pair includes first sample data and second sample data. The first sample data is data of the first modality, and the second sample data is data of the second modality, and the first modality is different from the second modality. Among them, the modality is a way of carrying information in a perceptual channel, such as visual, auditory, olfactory or tactile channels. The modality in this application can be at least one of a visual modality, a text modality, or an audio modality. For example, the first sample data can be image data, and the second sample data can be text data; or, the first sample data can be image data, and the second sample data can be audio data, etc., which is not limited in the embodiments of this application.

[0090] It should be noted that, for ease of understanding, the data used and generated in the model training phase are distinguished from the data used and generated in the model application phase. The word "sample" is included in the relevant data names, such as the first sample data, the second sample data, etc.

[0091] It should be understood that the terms "first," "second," and the like used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are merely used to distinguish a first element from another element. For example, first sample data may be referred to as second sample data, and similarly, first data may be referred to as second data without departing from the scope of this application.

[0092] Encoding is the process of mapping plaintext information into a latent space through calculations. This application can encode the first sample data using a first encoder in the initial model to be trained to obtain a first sample modal representation; and encode the second sample data using a second encoder in the initial model to obtain a second sample modal representation.

[0093] In some embodiments, the computer device may directly encode the first sample data to obtain the first sample modal representation, and directly encode the second sample data to obtain the second sample modal representation. In other embodiments, the computer device may mask the first sample data, and then encode the masked first sample data using the first encoder in the initial model to obtain the first sample modal representation. The computer device may mask the second sample data, and then encode the masked second sample data using the second encoder in the initial model to obtain the second sample modal representation.

[0094] For ease of description, the data input to the first encoder is referred to as the first input sequence, and the data input to the second encoder is referred to as the second input sequence. That is, the first input sequence can be the first sample data or the masked first sample data; the second input sequence can be the second sample data or the masked second sample data.

[0095] In some embodiments, the initial model to be trained is encoded multiple times for the first input sequence to obtain a first sample modality representation. The initial model to be trained is encoded multiple times for the second input sequence to obtain a second sample modality representation.

[0096] In some embodiments, the first encoder and the second encoder have the same network structure, so the encoding process performed on the first input sequence is the same as the encoding process performed on the second input sequence. Of course, in other embodiments, the first encoder and the second encoder can have different network structures, and the encoding process performed on the first input sequence is different from the encoding process performed on the second input sequence. This embodiment of the present application is not limited to this.

[0097] Taking the example of the first encoder and the second encoder having the same network structure, the following is explained: for any encoding, such as the l+1th encoding, the output of the lth encoding can be obtained first. A multi-head self-attention layer operation can be performed based on the output of the lth encoding. The result of the multi-head self-attention layer operation is combined with the output of the lth encoding and then a layer regularization operation is performed to obtain the intermediate processing features in the encoding process; the intermediate processing features in the encoding process are processed by a forward neural network, and the result of the forward neural network processing is combined with the intermediate processing features in the encoding process and then a layer regularization operation is performed to obtain the output of the l+1th encoding; l+1 is used as the new l, and the output of the lth encoding is returned to continue execution until the last encoding process is completed, and the output of the last encoding is used as the first sample modal representation or the second sample modal representation. Where l is a natural number greater than or equal to 0. When l is 0, the output of the lth encoding obtained is the first input sequence or the second input sequence.

[0098] In some embodiments, the computer device may encode each input data in the first input sequence (when the first input sequence includes multiple data blocks, each input data is referred to as each data block) multiple times to obtain an output corresponding to each input data, and then concatenate the outputs corresponding to each input data in the last encoding to obtain a first sample modal representation. Similarly, the computer device may encode each input data in the second input sequence multiple times to obtain an output corresponding to each input data, and then concatenate the outputs corresponding to each input data in the last encoding to obtain a second sample modal representation.

[0099] For example, the computer device may express the output of the (l+1)th encoding corresponding to any input data by Formula 1:

[0100] ; (Formula 1)

[0101] in, Represents the output of the l-th encoding, which can also be considered as the hidden feature of the l-th layer; Representation layer regularization operation, Indicates the multi-head self-attention layer operation through the encoder layer l network, Indicates processing by the forward neural network of the encoder's lth layer (specifically, a two-layer fully connected network); That is, it is an intermediate processing feature in the encoding process. The encoder can be the first encoder or the second encoder.

[0102] Furthermore, the computer device may express the first sample modality representation by the following formula 2:

[0103] ;(Formula 2)

[0104] in, represents the first data block in the first input sequence, represents the second data block in the first input sequence, represents the Mth data block (that is, the last data block) in the first input sequence; represents the latent features corresponding to the first data block output by the last encoding layer of the first encoder, represents the latent features corresponding to the second data block output by the last encoding layer of the first encoder, Represents the latent features corresponding to the Mth data block output by the last encoding layer of the first encoder.

[0105] Furthermore, the computer device may express the second sample modality representation by the following formula 3:

[0106] ;(Formula 3)

[0107] in, represents the first data block in the second input sequence, represents the second data block in the second input sequence, Represents the Nth data block (that is, the last data block) in the second input sequence; represents the latent features corresponding to the first data block output by the last encoding layer of the second encoder, represents the latent features corresponding to the second data block output by the last encoding layer of the second encoder, Represents the latent features corresponding to the Nth data block output by the last encoding layer of the second encoder.

[0108] In some embodiments, the first encoder and / or the second encoder are composed of a recurrent neural network, a convolutional neural network, or a graph neural network. In practical applications, more or fewer network layers can be designed according to actual needs, and this is not limited in the embodiments of the present application. In some embodiments, the first encoder and / or the second encoder can be a Transformers structure.

[0109] Step 204 : A mixed sample modality representation is obtained by combining the first sample modality representation and the second sample modality representation through the initial model, and a prediction weight is determined according to the mixed sample modality representation.

[0110] It should be noted that when the alignment quality of the first sample data and the second sample data in a training sample pair is high, more mixed-modal information can be considered during unimodal decoding; when the first sample data and the second sample data have relatively large noise (such as image-text mismatch), more unimodal information can be considered during unimodal decoding. Based on this consideration, this application designs a gated interaction mechanism to control the ratio of reference unimodal representation and mixed-modal representation during decoding, that is, dynamically adjust the reference ratio through gating weights.

[0111] Specifically, the initial model can fuse the first sample modal representation and the second sample modal representation through the interactive transformation structure (Cross Transformers) to obtain a mixed sample modal representation, and then perform a numerical transformation on the mixed modal sample representation to obtain the prediction weight.

[0112] In some embodiments, the computer device may directly perform addition or multiplication operations on the first sample modal representation and the second sample modal representation to obtain a mixed sample modal representation. In other embodiments, the computer device may also perform an attention operation on the first sample modal representation and the second sample modal representation to obtain a mixed sample modal representation. In other embodiments, the computer device may also use more complex operations, such as performing attention mechanism processing, matrix addition or multiplication, and then performing linear / nonlinear changes to obtain a mixed sample modal representation, etc. This application does not limit the specific fusion method.

[0113] Furthermore, after the initial model combines the first sample modal representation and the second sample modal representation to obtain a mixed sample modal representation, it can perform a linear transformation on the mixed sample modal representation and then perform activation processing to obtain prediction weights. In other words, the mixed sample modal representation is converted into a numerical value, which is also the prediction weight.

[0114] In step 206 , the first sample modality representation and the mixed sample modality representation are fused according to the prediction weights through the initial model, and decoding is performed based on the fused result to obtain a first prediction feature representation.

[0115] Specifically, the initial model may perform weighted processing on the first sample modality representation and the mixed sample modality representation according to the prediction weight, and decode based on the result obtained by the weighted processing to obtain the first prediction feature representation.

[0116] In some embodiments, the computer device may use the predicted weight as the weight of any one of the first sample modality representation and the mixed sample modality representation, and perform weighted sum processing on the first sample modality representation and the mixed sample modality representation.

[0117] In some embodiments, the computer device may use the predicted weight as the weight of any one of the first sample modal representation and the mixed sample modal representation, and use the difference between the value one and the predicted weight as the weight of the other representation, and perform weighted summation processing on the first sample modal representation and the mixed sample modal representation.

[0118] In some embodiments, the first sample modal representation and the mixed sample modal representation are fused according to the predicted weight, including: using the predicted weight as the coefficient of the mixed sample modal representation, and using the difference between the value one and the predicted weight as the coefficient of the first sample modal representation; according to the coefficient of the mixed sample modal representation and the coefficient of the first sample modal representation, the mixed sample modal representation and the first sample modal representation are weighted to obtain a first fused sample representation.

[0119] Specifically, the initial model may perform a weighted summation of the mixed sample modal representation and the first sample modal representation according to the coefficients of the mixed sample modal representation and the coefficients of the first sample modal representation to obtain a first fused sample representation. For example, the computer device may calculate the first fused sample representation using the following formula 4:

[0120] ;(Formula 4)

[0121] Among them, p is the prediction weight, is the first sample modality representation, and S is the mixed sample modality representation.

[0122] In the above embodiment, by fusing the mixed sample modal representation and the first sample modality through prediction weights, the ratio of the single modal representation and the mixed modal representation can be flexibly referenced according to the alignment quality of the two modal data, so that the fused representation can better express the information of the single modal data.

[0123] Furthermore, the initial model may be decoded based on the first fused sample representation to obtain a first prediction feature representation. The specific decoding method may be a traditional decoding method.

[0124] In some embodiments, the initial model encodes masked first sample data, and during the masking process, some data blocks are randomly discarded. Therefore, during decoding, a mask embedding can be inserted into the first fused sample representation based on the position of the discarded data blocks in the first sample data. The mask embedding can be a preset feature representation, such as a feature representation composed of preset values (such as 0 or 1). The first decoder in the initial model can then decode the first fused sample representation after the mask embedding is inserted to obtain a first predicted feature representation. This first predicted feature representation represents the latent features of the discarded data blocks.

[0125] In step 208 , the second sample modality representation and the mixed sample modality representation are fused according to the prediction weights through the initial model, and decoding is performed based on the fused result to obtain a second prediction feature representation.

[0126] Specifically, the initial model may perform weighted processing on the second sample modality representation and the mixed sample modality representation according to the prediction weight, and decode based on the result obtained by the weighted processing to obtain the second prediction feature representation.

[0127] In some embodiments, the computer device may use the predicted weight as the weight of any one of the second sample modality representation and the mixed sample modality representation, and perform weighted sum processing on the second sample modality representation and the mixed sample modality representation.

[0128] In some embodiments, the computer device may use the predicted weight as the weight of any one of the second sample modal representation and the mixed sample modal representation, and use the difference between the value one and the predicted weight as the weight of the other representation, and perform weighted summation processing on the second sample modal representation and the mixed sample modal representation.

[0129] In some embodiments, the second sample modal representation and the mixed sample modal representation are fused according to the predicted weight, including: using the predicted weight as the coefficient of the mixed sample modal representation, and using the difference between the value one and the predicted weight as the coefficient of the second sample modal representation; according to the coefficient of the mixed sample modal representation and the coefficient of the second sample modal representation, the mixed sample modal representation and the second sample modal representation are weighted to obtain a second fused sample representation.

[0130] Specifically, the initial model may perform a weighted summation of the mixed sample modal representation and the second sample modal representation according to the coefficients of the mixed sample modal representation and the coefficients of the second sample modal representation to obtain a second fused sample representation. For example, the computer device may calculate the second fused sample representation using the following formula 5:

[0131] ;(Formula 5)

[0132] Among them, p is the prediction weight, is the second sample modality representation, and S is the mixed sample modality representation.

[0133] In the above embodiment, by fusing the mixed sample modality representation and the second sample modality through prediction weights, the ratio of the single modality representation and the mixed modality representation can be flexibly referenced according to the alignment quality of the two modal data, so that the fused representation can better express the information of the single modality data.

[0134] Furthermore, the initial model can be decoded based on the second fused sample representation to obtain a second prediction feature representation. The specific decoding method can be a traditional decoding method.

[0135] In some embodiments, the initial model encodes the masked second sample data, and during the masking process, some data blocks are randomly discarded. Therefore, during decoding, a mask embedding can be inserted into the second fused sample representation based on the position of the discarded data blocks in the second sample data. The mask embedding can be a preset feature representation, such as a feature representation composed of preset values (such as 0 or 1). The second decoder in the initial model can then decode the second fused sample representation after the mask embedding is inserted to obtain a second predicted feature representation. This second predicted feature representation represents the latent features of the discarded data blocks.

[0136] Step 210 : Determine a first loss based on the true feature representation and the first predicted feature representation of the first sample data, and the true feature representation and the second predicted feature representation of the second sample data.

[0137] Specifically, the computer device may extract the true feature representation of the first sample data, and then calculate the difference between the true feature representation of the first sample data and the first predicted feature representation, using this difference as the first modal loss. The computer device may also extract the true feature representation of the second sample data, and then calculate the difference between the true feature representation of the second sample data and the second predicted feature representation, using this difference as the second modal loss. The first loss is determined based on the first modal loss and the second modal loss. The difference may be represented by a difference, a comparison value, or a quotient, etc.

[0138] In some embodiments, the true feature representation of the first sample data involved in the loss calculation may specifically be the true feature representation of the partial data blocks discarded from the first sample data before encoding; the true feature representation of the second sample data involved in the loss calculation may specifically be the true feature representation of the partial data blocks discarded from the second sample data before encoding.

[0139] In other embodiments, the true feature representation of the first sample data involved in the loss calculation may also be the true feature representation of all data blocks in the first sample data; the true feature representation of the second sample data involved in the loss calculation may also be the true feature representation of all data blocks in the second sample data.

[0140] In some embodiments, the computer device may extract the true feature representation of the first sample data and the true feature representation of the second sample data through other trained codecs.

[0141] In some embodiments, when the first sample data is image data, the difference between the true feature representation of the first sample data and the first predicted feature representation can be measured by MSE (Mean Square Error) loss; when the second sample data is text data, the difference between the true feature representation of the second sample data and the second predicted feature representation can be measured by CE (Cross-Entropy) loss.

[0142] In some embodiments, the computer device may use the mean of the differences between the true feature representations of multiple first sample data and the first predicted feature representation as the first modal loss; and use the mean of the differences between the true feature representations of multiple second sample data and the second predicted feature representation as the second modal loss.

[0143] Exemplarily, the computer device may calculate the first modal loss by the following formula:

[0144] ;(Formula 6)

[0145] Among them, among them, represents the set consisting of the first sample data in the training sample set, is the number of first sample data in the set consisting of the first sample data, represents the average error function, Represents the first prediction feature representation of the mth first sample data, Represents the true feature representation of the mth first sample data.

[0146] Exemplarily, the computer device may calculate the second modal loss by the following formula:

[0147] ;(Formula 7)

[0148] in, represents the set consisting of the second sample data in the training sample set, is the number of second sample data in the set consisting of the second sample data, represents the cross entropy function, Represents the second prediction feature representation of the mth second sample data, Represents the true feature representation of the m-th second sample data.

[0149] In some embodiments, the computer device may use the sum of the first modal loss and the second modal loss as the first loss, or use the average of the first modal loss and the second modal loss as the first loss, or use the product of the first modal loss and the second modal loss as the first loss, etc. The embodiments of the present application are not limited to this.

[0150] Step 212: Determine a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weights.

[0151] The alignment tag indicates whether the first sample data and the second sample data are aligned. Alignment indicates that the content of the first and second sample data matches. Misalignment indicates that the content of the first and second sample data does not match. For example, if the first sample data is image data and the second sample data is text data, alignment indicates that the image and text match, while misalignment indicates that the image and text do not match.

[0152] It can be understood that when the first sample data and the second sample data are aligned data, it means that the contents of the first sample data and the second sample data are matched. Then, when decoding, more reference can be made to the mixed sample modal representation, that is, the proportion of the mixed sample modal representation is controlled to be larger.

[0153] In some embodiments, the computer device may set the alignment label to 1 or 0, where 1 indicates alignment and 0 indicates misalignment. A second loss may be determined based on the difference between the alignment label and the predicted weight. For example, the second weight may be determined based on the difference between the alignment label and the predicted weight.

[0154] In some embodiments, the second loss is determined based on the alignment labels of the first sample data and the second sample data, and the prediction weights, including: constructing a first vector based on the alignment labels of the first sample data and the second sample data; constructing a second vector based on the prediction weights and the difference between the value one and the prediction weights; and determining the second loss based on the first vector and the second vector.

[0155] Specifically, the computer device may construct a first vector based on the alignment tags of the first sample data and the second sample data. Specifically, the value of the alignment tag and the difference between 1 and the alignment tag may be combined into a two-dimensional vector, namely, the first vector. For example, when the alignment tag is 1, the constructed first vector is [0, 1]; when the alignment tag is 0, the constructed first vector is [1, 0].

[0156] The computer device may construct a second vector using the predicted weight and the difference between the value 1 and the predicted weight, such as [1-p, p], where p is the predicted weight.

[0157] Furthermore, the computer device may determine the second loss based on the cross entropy of the first vector and the second vector. For example, the computer device may determine the second loss using the following formula:

[0158] L ITM = f CE ( y itm ,[1-p,p]) ;(Formula 8)

[0159] in, represents the first vector, p is the prediction weight, represents the cross entropy function.

[0160] In the above embodiment, determining the second loss based on the alignment labels of the first and second sample data and the predicted weights ensures that the predicted weights learned by the model during training are positively correlated with the degree of alignment of the input data. In other words, the higher the quality of input data alignment, the greater the predicted weights, which can dynamically control the increase in the proportion of mixed-modal representations.

[0161] In step 214 , a target loss function is constructed based on the first loss and the second loss, and the initial model to be trained is trained based on the target loss function, and a multimodal model is obtained after the training is completed.

[0162] Specifically, the computer device may perform a weighted summation process on the first loss and the second loss to construct a target loss function, wherein the weighting coefficient may be 1 or another pre-set value, which is not limited in the embodiment of the present application.

[0163] Exemplarily, the computer device may construct the target loss function using the following formula:

[0164] ;(Formula 9)

[0165] in, represents the first modal loss in the first loss, represents the second modal loss in the first loss, Indicates the second loss.

[0166] Furthermore, the computer device can input multiple training sample pairs into the initial model in batches for processing, and adjust the model weights of the initial model through the target loss function to train the initial model. The training stops when the training stop condition is reached, and a multimodal model with training is obtained. Among them, the training stop condition can specifically be reaching a preset number of iterations, reaching a preset training time, or the model performance reaches a preset performance, or the change in model prediction accuracy is less than a preset change, etc., which is not limited in the embodiments of the present application.

[0167] The multimodal model processing method described above encodes the first sample data and the second sample data belonging to different modalities, respectively, to obtain a first sample modal representation and a second sample modal representation. The first sample modal representation and the second sample modal representation are then combined to obtain a mixed sample modal representation. Based on this mixed sample modal representation, a prediction weight can be obtained. This prediction weight is used to control the proportion of the first sample modal representation and the second sample modal representation when they are fused with the mixed sample modal representation. This allows for intelligent extraction of information from single modalities / mixed modalities for subsequent decoding. During training, on the one hand, the loss between the predicted feature representation and the true feature representation is taken into account, and on the other hand, the matching loss between sample data of different modalities (that is, the second loss) is taken into account. This allows the model to learn encoding and decoding capabilities that are closer to the true feature representation. In addition, it can intelligently control the ratio of the referenced single-modal representation and mixed-modal representation during the encoding and decoding process, so that when the alignment quality of the training sample pairs is high, more mixed-modal information can be considered during single-modal decoding; when the training sample pairs are not aligned, more single-modal information can be considered during single-modal decoding. This can better handle the problem of modal content mismatch in the training dataset and improve the processing performance of the multimodal model.

[0168] In some embodiments, before encoding the first sample data and the second sample data in the training sample pair, the method further includes: obtaining a training sample pair, the training sample pair including the first sample data and the second sample data; splitting the first sample data into multiple first data blocks, splitting the second sample data into multiple second data blocks, extracting some of the multiple first data blocks to form a first input sequence, and extracting some of the multiple second data blocks to form a second input sequence. Encoding the first sample data and the second sample data in the training sample pair to obtain a first sample modal representation and a second sample modal representation includes: encoding the first input sequence multiple times to obtain the first modal representation, and encoding the second input sequence multiple times to obtain the second modal representation.

[0169] Specifically, the computer device can train the initial model through the training sample pairs. During the training process, the first sample data and the second sample data in the training sample pairs can be directly processed, or the first sample data and the second sample data can be masked before being input into the initial model.

[0170] In some embodiments, the computer device may split the first sample data into multiple first data blocks and split the second sample data into multiple second data blocks. Then, a portion of the first data blocks is extracted according to a certain ratio to form a first input sequence, and a portion of the second data blocks is extracted to form a second input sequence. The first input sequence is input into one encoder of the initial model, and the second input sequence is input into another encoder of the initial model.

[0171] In some embodiments, for multiple first data blocks of the first sample data, the computer device may randomly mask a certain proportion of the first data blocks, and form the remaining first data blocks into a first input sequence. For multiple second data blocks of the second sample data, the computer device may randomly mask a certain proportion of the second data blocks, and form the remaining second data blocks into a second input sequence.

[0172] For example, the initial model uses two separate autoencoders to model different modal information. For one modality, such as image data, the computer device can randomly mask a certain percentage (e.g., 50%) of the patches. The remaining patches then form an input sequence and are fed into the autoencoder corresponding to the image data to obtain an image representation.

[0173] In some embodiments, taking the example of an image modality as the first modality and a text modality as the second modality, LightVLP (i.e., the multimodal model of this application) includes two autoencoders, one for processing image information and the other for processing text information. To improve the efficiency of the model, this application introduces a masked autoencoder strategy, which removes the masked tokens (masked features) from the resulting sequences of sample data from the two modalities before inputting them into their respective autoencoders.

[0174] For easier understanding, please refer to Figure 3 , Figure 3 A schematic diagram of a masking strategy in an embodiment is shown. Taking the sample data (x1, x2, x3, x4, x5, x6) as an example, in the traditional masking strategy, reference Figure 3 In the A part, some data blocks, such as x2, x4 and x5, are usually randomly masked to become mask features, such as replacing them with 0 or 1. Then the sequence including the mask features is input as the input sequence to the encoder for processing. The masking strategy in this application is to refer to Figure 3In the B part, some of the data blocks therein, such as x2, x4 and x5, are randomly masked and converted into mask features, such as by replacing them with 0 or 1. Then the sequence after removing the mask features is input as the input sequence to the encoder for processing. Furthermore, after encoding the input sequence by the encoder, the corresponding modal representation (e1, e3, e6) is obtained, and the mask features are inserted into the modal representation and then decoded. In the above embodiment, in the process of processing different modal data, a mask encoding strategy is introduced. For each modal data, part of the data blocks are extracted to form the input sequence, which can greatly reduce the amount of data in the encoding process of the model and improve the model processing efficiency. Especially in the training stage, the model training efficiency can be effectively improved.

[0175] In some embodiments, a mixed sample modal representation is obtained by combining the first sample modal representation and the second sample modal representation through an initial model, including: performing multiple interactive fusions of the first sample modal representation and the second sample modal representation through the initial model to obtain a first mixed sample representation biased towards the first modal side and a second mixed sample representation biased towards the second modal side; combining the first mixed sample representation and the second mixed sample representation to obtain a mixed sample modal representation.

[0176] Specifically, in the process of fusing the first sample modal representation and the second sample modal representation, the initial model can perform multiple fusions through a multi-layer neural network to obtain a first mixed sample representation biased towards the first modal side and a second mixed sample representation biased towards the second modal side. Each time the interactive fusion is performed, the second intermediate mixed sample representation can be fused into the first intermediate mixed sample representation based on the attention mechanism to obtain a first intermediate mixed sample representation biased towards the first modal side. The first intermediate mixed sample representation is fused into the second intermediate mixed sample representation based on the attention mechanism to obtain a second intermediate mixed sample representation biased towards the second modal side. In this way, the first intermediate mixed sample representation output by the last layer of the neural network is the first mixed sample representation, and the second intermediate mixed sample representation output by the last layer of the neural network is the second mixed sample representation.

[0177] Furthermore, the initial model may fuse the first mixed sample representation and the second mixed sample representation to obtain a mixed sample modality representation. For example, the fusion may be performed by weighted summation, concatenation, or splicing, which is not limited in this embodiment of the present application.

[0178] In the above embodiment, the first sample modal representation and the second sample modal representation are interactively fused multiple times through the initial model, so that a first mixed sample representation biased towards the first modal side and a second mixed sample representation biased towards the second modal side can be obtained, and the relationship information between the first sample data and the second sample data can be fully extracted. Then, the first mixed sample representation and the second mixed sample representation are combined to obtain a mixed sample modal representation.

[0179] In some embodiments, the first sample modal representation and the second sample modal representation are interactively fused multiple times through the initial model to obtain a first mixed sample representation biased towards the first modal side and a second mixed sample representation biased towards the second modal side, including: when performing the i+1th interactive fusion, obtaining the first intermediate mixed sample representation and the second intermediate mixed sample representation output by the i-th interactive fusion; fusing the second intermediate mixed sample representation into the first intermediate mixed sample representation to obtain the first intermediate mixed sample representation of the i+1th interactive fusion; fusing the first intermediate mixed sample representation into the second intermediate mixed sample representation to obtain the second intermediate mixed sample representation of the i+1th interactive fusion. ; Take i+1 as the new i, and return to the first intermediate mixed sample representation and the second intermediate mixed sample representation obtained by the i-th interactive fusion during the i+1-th interactive fusion, and continue to execute until the stopping condition is reached; take the first intermediate mixed sample representation output by the last interactive fusion as the first mixed sample representation, and take the second intermediate mixed sample representation output by the last interactive fusion as the second mixed sample representation; wherein, i is a natural number greater than or equal to 0. When i is 0, the first intermediate mixed representation obtained by the i-th interactive fusion output is the first sample modal representation, and the second intermediate mixed representation obtained by the i-th interactive fusion output is the second sample modal representation.

[0180] Specifically, for the first interactive fusion, the initial model can fuse the second sample modality representation into the first sample modality representation based on the attention mechanism to obtain a first intermediate mixed sample representation, and fuse the first sample modality representation into the second sample modality representation based on the attention mechanism to obtain a second intermediate mixed sample representation.

[0181] For the second interactive fusion, the initial model can fuse the second intermediate mixed sample representation obtained from the first interactive fusion into the first intermediate mixed sample representation based on the attention mechanism to obtain a new first intermediate mixed sample representation. The first intermediate mixed sample representation obtained from the first interactive fusion is fused into the second intermediate mixed sample representation based on the attention mechanism to obtain a new second intermediate mixed sample representation.

[0182] In this way, during each interactive fusion process, the current interactive fusion is continuously performed based on the output of the previous interactive fusion, until the final interactive fusion. The initial model can directly output the first intermediate mixed sample representation and the second intermediate mixed sample representation obtained from the last interactive fusion. The first intermediate mixed sample representation outputted this time is the first mixed sample representation, and the second intermediate mixed sample representation outputted this time is the second mixed sample representation.

[0183] In the above embodiment, after the first sample modal representation and the second sample modal representation are fused, the next interactive fusion is continuously performed based on the previous output through an iterative cycle, which can better utilize the information obtained from each fusion and make the information fusion more complete.

[0184] In some embodiments, the second intermediate mixed sample representation is fused into the first intermediate mixed sample representation to obtain the first intermediate mixed sample representation of the i+1th interactive fusion, including: performing an attention operation on the first intermediate mixed sample representation and the second intermediate mixed sample representation output by the i-th interactive fusion, combining the attention operation result with the first intermediate mixed sample representation output by the i-th interactive fusion, and then performing a layer regularization operation to obtain a first intermediate sample processing feature; performing a forward neural network processing on the first intermediate sample processing feature, combining the forward neural network processing result with the first intermediate sample processing feature, and then performing a layer regularization operation to obtain the first intermediate mixed sample representation of the i+1th interactive fusion.

[0185] Specifically, when performing a layer regularization operation after combining the attention operation result and the first intermediate mixed sample representation output by the i-th interactive fusion, the way of combining the attention operation result and the first intermediate mixed sample representation output by the i-th interactive fusion can be addition, multiplication, or splicing, etc., which is not limited in this embodiment of the present application. When performing a layer regularization operation after combining the forward neural network processing result and the first intermediate sample processing feature, the way of combining the forward neural network processing result and the first intermediate sample processing feature can be addition, multiplication, or splicing, etc., which is not limited in this embodiment of the present application.

[0186] Exemplarily, the computer device may determine the first intermediate mixed sample representation of the (i+1)th interactive fusion by the following formula:

[0187] ;(Formula 10)

[0188] ;(Formula 11)

[0189] in, is the first intermediate mixed sample representation obtained by the i-th interactive fusion, is the second intermediate mixed sample representation obtained by the i-th interactive fusion, Is the classic attention calculation formula, which is based on is the Q value, Attention processing for K and V values, is the normalization function. Representation layer regularization operation, It represents the processing of the i-th layer of the feedforward neural network (specifically, a two-layer fully connected network) through the interactive transformation structure.

[0190] In the above embodiment, the interactive fusion processing method based on the attention mechanism can better integrate the second intermediate mixed representation into the first intermediate mixed representation. Moreover, this stacked fusion design can prevent the gradient explosion problem during model training.

[0191] In some embodiments, the first intermediate mixed sample representation is fused into the second intermediate mixed sample representation to obtain the second intermediate mixed sample representation of the i+1th interactive fusion, including: performing an attention operation on the second intermediate mixed sample representation output by the i-th interactive fusion and the first intermediate mixed sample representation, combining the attention operation result with the second intermediate mixed sample representation output by the i-th interactive fusion and performing a layer regularization operation to obtain a second intermediate sample processing feature; performing a forward neural network processing on the second intermediate sample processing feature, combining the forward neural network processing result with the second intermediate sample processing feature and performing a layer regularization operation to obtain the second intermediate mixed sample representation of the i+1th interactive fusion.

[0192] Specifically, when performing a layer regularization operation after combining the attention operation result and the second intermediate mixed sample representation output by the i-th interactive fusion, the way of combining the attention operation result and the second intermediate mixed sample representation output by the i-th interactive fusion can be addition, multiplication, or splicing, etc., which is not limited in the embodiments of the present application. When performing a layer regularization operation after combining the forward neural network processing result and the second intermediate sample processing feature, the way of combining the forward neural network processing result and the second intermediate sample processing feature can be addition, multiplication, or splicing, etc., which is not limited in the embodiments of the present application.

[0193] Exemplarily, the computer device may determine the second intermediate mixed sample representation of the (i+1)th interactive fusion by the following formula:

[0194] ;(Formula 12)

[0195] ;(Formula 13)

[0196] in, is the first intermediate mixed sample representation obtained by the i-th interactive fusion, is the second intermediate mixed sample representation of the i-th interactive fusion, Is the classic attention calculation formula, which is based on is the Q value, Attention processing for K and V values, is the normalization function. Representation layer regularization operation, It represents the processing of the i-th layer of the feedforward neural network (specifically, a two-layer fully connected network) through the interactive transformation structure.

[0197] In the above embodiment, the interactive fusion processing method based on the attention mechanism can better fuse the first intermediate mixed representation into the second intermediate mixed representation. Moreover, through this stacked fusion design, the gradient explosion problem can be prevented during model training.

[0198] In some embodiments, reference Figure 4 , the multimodal model processing method includes the following steps:

[0199] In step 402 , the first sample data and the second sample data in the training sample pair are respectively encoded using the initial model to be trained to obtain a first sample modality representation and a second sample modality representation; the first sample data and the second sample data are sample data of different modalities.

[0200] Step 404 : The first sample modality representation and the second sample modality representation are combined with the initial model to obtain a first mixed sample representation biased toward the first modality side and a second mixed sample representation biased toward the second modality side.

[0201] Specifically, the initial model performs multiple interactive fusions on the first sample modality representation and the second sample modality representation to obtain a first mixed sample representation biased towards the first modality and a second mixed sample representation biased towards the second modality. For details on the manner of performing multiple interactive fusions, please refer to the description of the aforementioned related embodiments.

[0202] Step 406: Determine a first prediction weight based on the first mixed sample representation, and determine a second prediction weight based on the second mixed sample representation.

[0203] Specifically, the initial model may perform a linear transformation on the first mixed sample representation and then perform activation processing to obtain a first prediction weight; and perform a linear transformation on the second mixed sample representation and then perform activation processing to obtain a second prediction weight.

[0204] Step 408: The first sample modality representation and the first mixed sample representation are fused according to the first prediction weight through the initial model, and decoding is performed based on the fused result to obtain a first prediction feature representation.

[0205] Specifically, the initial model can use the first prediction weight as the coefficient of the first mixed sample representation, and the difference between the value one and the first prediction weight as the coefficient of the first sample modal representation; according to the coefficient of the first mixed sample representation and the coefficient of the first sample modal representation, the first mixed sample representation and the first sample modal representation are weightedly summed, and then decoded based on the result of the weighted summation to obtain the first prediction feature representation.

[0206] Step 410: The second sample modality representation and the second mixed sample representation are fused according to the second prediction weight through the initial model, and decoding is performed based on the fused result to obtain a second prediction feature representation.

[0207] Specifically, the initial model can use the second prediction weight as the coefficient of the second mixed sample representation, and the difference between the value one and the second prediction weight as the coefficient of the second sample modal representation; according to the coefficient of the second mixed sample representation and the coefficient of the second sample modal representation, the second mixed sample representation and the second sample modal representation are weightedly summed, and then decoded based on the result of the weighted summation to obtain the second prediction feature representation.

[0208] Step 412: Determine a first loss based on the true feature representation and the first predicted feature representation of the first sample data, and the true feature representation and the second predicted feature representation of the second sample data.

[0209] Step 414 : Determine a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weights.

[0210] In step 416 , a target loss function is constructed based on the first loss and the second loss, and the initial model to be trained is trained based on the target loss function, and a multimodal model is obtained after the training is completed.

[0211] In the above embodiment, for the first modality, the initial model will use the first sample modality as the main information, and at the same time integrate the information represented by the second sample modality to obtain a first mixed sample representation, and then determine the prediction weight of the first modality side based on the first mixed sample representation, that is, the first prediction weight. In this way, the proportion of single modality information and mixed modality information referenced during decoding on the first modality side can be controlled based on the first prediction weight. For the second modality, the initial model will use the second sample modality as the main information, and at the same time integrate the information represented by the first sample modality to obtain a second mixed sample representation, and then determine the prediction weight of the first modality side based on the second mixed sample representation, that is, the second prediction weight. In this way, the proportion of single modality information and mixed modality information referenced during decoding on the second modality side can be controlled based on the second prediction weight. In this way, different modality sides can be more flexible and adaptive when fusing mixed modality information.

[0212] In some embodiments, the mixed sample modality representation includes a first mixed sample representation biased toward the first modality and a second mixed sample representation biased toward the second modality, and the prediction weights include a first prediction weight determined based on the first mixed sample representation and a second prediction weight determined based on the second mixed sample representation. Determining a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weights, includes: constructing a first vector based on the alignment labels of the first sample data and the second sample data; determining a target weight based on the mean of the first prediction weight and the second prediction weight, constructing a third vector based on the target weight and the difference between the value one and the target weight; and determining a second loss based on the first vector and the third vector.

[0213] As mentioned above, for different modalities, the initial model will generate a prediction loss that is adapted to the modality. During the model training process, the average of the first prediction loss and the second prediction loss can be used as the target loss, and the second loss can be determined based on the target loss and the alignment label.

[0214] Specifically, the computer device may construct a first vector based on the alignment tags of the first sample data and the second sample data. Specifically, the value of the alignment tag and the difference between 1 and the alignment tag may be combined into a two-dimensional vector, namely, the first vector. For example, when the alignment tag is 1, the constructed first vector is [0, 1]; when the alignment tag is 0, the constructed first vector is [1, 0].

[0215] The computing device may construct a second vector using the target weight and the difference between the value 1 and the target weight. For example, [1-p, p], where p is the target weight. Furthermore, the computing device may determine a second loss based on the cross entropy between the first and second vectors.

[0216] In the above embodiment, determining the second loss based on the alignment labels of the first and second sample data and the mean of the first and second prediction weights ensures that the two prediction weights learned by the model during training are positively correlated with the degree of alignment of the input data. In other words, the higher the quality of input data alignment, the larger the two predicted weights, which can dynamically control the increase in the proportion of mixed-modal representation.

[0217] In some embodiments, the method further includes constructing a third loss, wherein the step of constructing the third loss includes: determining the correlation between the first sample data and the second sample data based on the first sample modality representation and the second sample modality representation; and determining the third loss based on the alignment labels of the first sample data and the second sample data and the correlation between the first sample data and the second sample data. Constructing a target loss function based on the first loss and the second loss includes: constructing the target loss function based on the first loss, the second loss, and the third loss.

[0218] In some embodiments, in order to align the representations of different modalities, the present application introduces contrastive loss, that is, the third loss. By adding the third loss to the target loss for learning, the model can compensate for the performance impact caused by the data misalignment of training sample pairs during the training process.

[0219] Specifically, the computer device may obtain a first sample modality representation output by the first encoder and a second sample modality representation output by the second encoder. The first sample modality representation is then input into a linear layer and a regularization layer to obtain a first transformed representation, and the second sample modality representation is then input into a linear layer and a regularization layer to obtain a second transformed representation.

[0220] Exemplarily, the computer device may calculate the first conversion representation using the following formula:

[0221] ;(Formula 14)

[0222] in, represents regularization processing, represents linearization processing, represents the first sample modal representation.

[0223] Exemplarily, the computer device may calculate the second conversion representation using the following formula:

[0224] ;(Formula 15)

[0225] in, represents regularization processing, represents linearization processing, represents the second sample modality representation.

[0226] Furthermore, the computer device may determine the correlation between the first sample data and the second sample data based on the first transformed representation and the second transformed representation. For example, the computer device may calculate the similarity between the first transformed representation and the second transformed representation, and use the similarity as the correlation between the first sample data and the second sample data. The similarity may specifically be cosine similarity or distance similarity, etc., which is not limited in this embodiment of the present application.

[0227] Next, the computer device may determine a third loss based on the alignment labels of the first sample data and the second sample data, and the difference between the correlations between the first sample data and the second sample data. Specifically, the computer device may calculate the cross entropy between the alignment labels and the correlations, and use the cross entropy as the third loss.

[0228] Furthermore, the computer device may construct a target loss function based on the first loss, the second loss, and the third loss. For example, the computer device may construct the target loss function by summing the first loss, the second loss, and the third loss. For example, the computer device may construct the target loss function by the following formula:

[0229] ;(Formula 16)

[0230] in, represents the first modal loss in the first loss, represents the second modal loss in the first loss, represents the second loss, Indicates the third loss.

[0231] In the above embodiment, the contrast loss is introduced during the model training process, which can guide the model to learn the ability to distinguish whether the representations of different modalities are aligned.

[0232] In some embodiments, the correlation between the first sample data and the second sample data includes: the correlation of the first sample data with respect to the second sample data, and the correlation of the second sample data with respect to the first sample data. Determining the correlation between the first sample data and the second sample data based on the first sample modal representation and the second sample modal representation includes: determining a preset number of third sample data having the same modality as the first sample data, and a preset number of fourth sample data having the same modality as the second sample data; calculating the correlation of the first sample data with respect to the second sample data based on the first sample modal representation, the second sample modal representation, and the sample modal representations of each of the fourth sample data; and calculating the correlation of the second sample data with respect to the first sample data based on the first sample modal representation, the second sample modal representation, and the sample modal representations of each of the third sample data.

[0233] When calculating the correlation between the first sample data and the second sample data, the correlation of the first sample data with respect to the second sample data and the correlation of the second sample data with respect to the first sample data may be calculated respectively.

[0234] Specifically, the computer device performs linear processing and regularization on the first sample modal representation to obtain a first transformed representation, performs linear processing and regularization on the second sample modal representation to obtain a second transformed representation, performs linear processing and regularization on the third sample modal representation to obtain a third transformed representation, and performs linear processing and regularization on the fourth sample modal representation to obtain a fourth transformed representation.

[0235] Furthermore, the computer device can calculate the correlation of the first sample data relative to the second sample data based on the first conversion representation, the second conversion representation, and each fourth conversion representation; and calculate the correlation of the second sample data relative to the first sample data based on the first conversion representation, the second conversion representation, and each third conversion representation.

[0236] In some embodiments, the correlation of the first sample data with respect to the second sample data is calculated based on the first sample modal representation, the second sample modal representation, and the sample modal representation of each fourth sample data, including: determining a first value based on the first sample modal representation and the second sample modal representation; determining multiple second values based on the first sample modal representation and the sample modal representation of each fourth sample data; and using the comparison value of the first value and the sum of the multiple second values as the correlation of the first sample data with respect to the second sample data.

[0237] Specifically, the computer device may determine the first value based on the product of the first converted representation and the second converted representation, for example, by directly using the product as the first value, or by using the quotient of the product and the temperature coefficient as the exponent, and raising the result of the power operation with the natural constant e as the base as the first value. Similarly, the computer device may determine the second value based on the product of the first converted representation and each fourth converted representation. Furthermore, the computer device uses a comparison value between the first value and the sum of the multiple second values as the correlation between the first sample data and the second sample data. The comparison value may specifically be a quotient value, a difference value, or other value that can reflect the difference.

[0238] For example, the computer device may calculate the correlation between the first sample data and the second sample data using the following formula:

[0239] ; (Formula 17)

[0240] in, Represents the correlation between the first sample data and the second sample data; I represents the first conversion representation of the first sample data in the training sample pair. A second transformed representation of the second sample data corresponding to the first sample data in the training sample pair; is the preset quantity, A fourth transformed representation representing the j-th fourth sample data; Represents the temperature coefficient.

[0241] In the above embodiment, when calculating the correlation of the first sample data with respect to the second sample data, the fourth sample data having different content from the second sample data but the same modality is utilized, and the degree of alignment of the sample data pair with respect to other data pairs composed of the first sample data can be calculated, thereby accurately obtaining the correlation of the first sample data with respect to the second sample data.

[0242] In some embodiments, the correlation of the second sample data with respect to the first sample data is calculated based on the first sample modal representation, the second sample modal representation, and the sample modal representation of each third sample data, including: determining a first value based on the first sample modal representation and the second sample modal representation; determining multiple third values based on the second sample modal representation and the sample modal representation of each third sample data; and using the comparison value of the first value and the sum of the multiple third values as the correlation of the second sample data with respect to the first sample data.

[0243] Specifically, the computer device may determine the first value based on the product of the first converted representation and the second converted representation, for example, by directly using the product as the first value, or by using the quotient of the product and the temperature coefficient as the exponent, with the natural constant e as the base, as the result of the power operation. Similarly, the computer device may determine the third value based on the product of the second converted representation and each third converted representation. Furthermore, the computer device uses a comparison value between the first value and the sum of the multiple third values as the correlation of the second sample data with the first sample data. The comparison value may specifically be a quotient value, a difference value, or other numerical value that can reflect the difference.

[0244] For example, the computer device may calculate the correlation between the first sample data and the second sample data using the following formula:

[0245] ; (Formula 18)

[0246] in, represents the correlation between the second sample data and the first sample data; T represents the second transformed representation of the second sample data in the training sample pair. A first transformed representation of the first sample data corresponding to the second sample data in the training sample pair; is the preset quantity, A third transformed representation representing the j-th third sample data; Represents the temperature coefficient.

[0247] In the above embodiment, when calculating the correlation of the second sample data with respect to the first sample data, third sample data having different content from the first sample data but the same modality as the first sample data is utilized, and the degree of alignment of the sample data pair with respect to other data pairs composed of the second sample data can be calculated, thereby accurately obtaining the correlation of the second sample data with respect to the first sample data.

[0248] Furthermore, the computer device can determine a loss based on the alignment labels of the first sample data and the second sample data, and the difference between the correlation of the first sample data and the second sample data, and determine another loss based on the alignment labels and the difference between the correlation of the second sample data and the first sample data, and then average the two losses to obtain a third loss.

[0249] Exemplarily, the computer device may calculate the third loss by the following formula:

[0250] ; (Formula 19)

[0251] Among them, for a set of training sample pairs, and The same is the alignment label representing the first sample data and the second sample data in the training sample pair. represents the correlation between the first sample data and the second sample data, Indicates the correlation between the second sample data and the first sample data. is the cross entropy loss function.

[0252] In the above embodiment, the fourth sample data having different content from the second sample data but the same modality can be used to accurately calculate the correlation of the first sample data with respect to the second sample data; the third sample data having different content from the first sample data but the same modality can be used to accurately obtain the correlation of the second sample data with respect to the first sample data, thereby obtaining the correlation in two dimensions, which can more accurately measure the degree of alignment between the representations of the two modalities after processing by the encoder in the multimodal model.

[0253] In some embodiments, the modality includes an image modality and a text modality, and the method further includes a text retrieval step, specifically including: obtaining a text retrieval task, extracting the image data specified in the text retrieval task, and forming a first data pair to be processed by combining each text data in the text library with the specified image data; processing the first data pair through a multimodal model to obtain two feature representations, and determining the matching degree of the first data pair based on the two feature representations; and determining the target text that matches the specified image data according to the matching degree of each first data pair.

[0254] The multimodal model obtained through training in the above embodiment can be applied in image-text retrieval scenarios, for example, it can be used to retrieve text that matches an image. In actual applications, if a text retrieval task is obtained, the computer device can extract the image data specified in the text retrieval task, and then form multiple first data pairs with each text data in the text library and the specified image data. Thus, any first data pair is processed by the trained multimodal model, and two feature representations corresponding to the first data pair are output. Thus, the computer device can determine the matching degree of the first data pair based on the two feature representations, and according to the matching degree of each first data pair, the text data in the first data whose matching degree meets the push condition is used as the target text that matches the specified image data and is pushed to the terminal for display, that is, the text retrieval result of the specified image data is obtained. Among them, the matching degree meeting the push condition can specifically be the highest matching degree, or the matching degree is greater than a preset threshold, or the top N with the highest matching degree, etc., and this embodiment of the present application does not limit this.

[0255] In some embodiments, the specific method for determining the matching degree of the first data pair can be any of the following methods: calculating the feature similarity of the two feature representations and using the feature similarity as the matching degree; inputting the two feature representations into a classification layer, having the classification layer output a classification result, and using the classification result as the matching degree. The feature similarity can be represented by cosine similarity, Euclidean distance, Manhattan distance, etc.

[0256] In the above embodiment, the trained multimodal model can be used to perform image and text matching calculations, thereby helping to improve the quality and efficiency of text retrieval.

[0257] In some embodiments, the modality includes images and text, and the method also includes an image retrieval step, specifically including: obtaining an image retrieval task, extracting the text data specified in the image retrieval task, and forming a second data pair to be processed by respectively combining each image data in the image library with the specified text data; processing the second data pair through a multimodal model to obtain two feature representations, and determining the matching degree of the second data pair based on the two feature representations; and determining the target image that matches the specified text data according to the matching degree of each second data pair.

[0258] The multimodal model obtained through training in the above embodiment can be applied in image and text retrieval scenarios, for example, it can be used to retrieve images that match text. In actual applications, if an image retrieval task is obtained, the computer device can extract the text data specified in the image retrieval task, and then form multiple second data pairs with each image data in the image library and the specified text data. Thus, any second data pair is processed by the trained multimodal model, and two feature representations corresponding to the second data pair are output. Thus, the computer device can determine the matching degree of the second data pair based on the two feature representations, and according to the matching degree of each second data pair, the image data in the second data whose matching degree meets the push condition is used as the target image that matches the specified text data and is pushed to the terminal for display, that is, the image retrieval result of the specified text data is obtained. Among them, the matching degree meeting the push condition can specifically be the highest matching degree, or the matching degree is greater than a preset threshold, or the top N with the highest matching degree, etc., and the embodiment of the present application does not limit this.

[0259] In some embodiments, the specific method for determining the matching degree of the second data pair can be any of the following methods: calculating the feature similarity of the two feature representations and using the feature similarity as the matching degree; inputting the two feature representations into a classification layer, having the classification layer output a classification result, and using the classification result as the matching degree. The feature similarity can be represented by cosine similarity, Euclidean distance, Manhattan distance, etc.

[0260] In the above embodiment, the trained multimodal model can be used to perform image and text matching calculations, thereby helping to improve the quality and efficiency of image retrieval.

[0261] In an exemplary embodiment, Figure 5 As shown, a data matching method is provided, which is applied to a computer device (such as Figure 1 The following steps are taken as an example to illustrate the terminal or server in the example.

[0262] In step 502 , the first data and the second data are respectively encoded to obtain a first modal representation and a second modal representation, where the first data and the second data are data of different modalities.

[0263] The present application can encode based on the first data through a first encoder in a multimodal model to obtain a first modal representation; and encode based on the second data through a second encoder in the multimodal model to obtain a second modal representation.

[0264] In some embodiments, the computer device may directly encode the first data to obtain the first modal representation, and directly encode the second data to obtain the second modal representation. In other embodiments, the computer device may mask the first data, then encode the masked first data using a first encoder in a multimodal model to obtain the first modal representation. The computer device may mask the second data, then encode the masked second data using a second encoder in the multimodal model to obtain the second modal representation.

[0265] For ease of description, the data input to the first encoder is referred to as the first input sequence, and the data input to the second encoder is referred to as the second input sequence. That is, the first input sequence can be the first data or the masked first data; the second input sequence can be the second data or the masked second data.

[0266] In some embodiments, the multimodal model encodes the first input sequence multiple times to obtain a first modal representation, and the multimodal model encodes the second input sequence multiple times to obtain a second modal representation.

[0267] In some embodiments, the first encoder and the second encoder have the same network structure, so the encoding process performed on the first input sequence is the same as the encoding process performed on the second input sequence. Of course, in other embodiments, the first encoder and the second encoder can have different network structures, and the encoding process performed on the first input sequence is different from the encoding process performed on the second input sequence. This embodiment of the present application is not limited to this.

[0268] Taking the example of the first encoder and the second encoder having the same network structure, the following is explained: for any encoding, such as the l+1th encoding, the output of the lth encoding can be obtained first. A multi-head self-attention layer operation can be performed based on the output of the lth encoding. The result of the multi-head self-attention layer operation and the output of the lth encoding are combined and layer regularization is performed to obtain intermediate processing features; the intermediate processing features are processed by a forward neural network, and the result of the forward neural network processing and the intermediate processing features are combined and layer regularization is performed to obtain the output of the l+1th encoding; l+1 is used as the new l, and the output of the lth encoding is returned to continue execution until the last encoding process is completed, and the output of the last encoding is used as the first modal representation or the second modal representation. Where l is a natural number greater than or equal to 0. When l is 0, the output of the lth encoding obtained is the first input sequence or the second input sequence.

[0269] In some embodiments, the computer device may encode each input data in the first input sequence multiple times to obtain an output corresponding to each input data, and then concatenate the outputs corresponding to each input data in the last encoding to obtain the first modal representation. Similarly, the computer device may encode each input data in the second input sequence multiple times to obtain an output corresponding to each input data, and then concatenate the outputs corresponding to each input data in the last encoding to obtain the second modal representation.

[0270] For example, the computer device may determine the output of the (l+1)th encoding by using formula (1) in step 202.

[0271] Step 504: Combine the first modality representation and the second modality representation to obtain a hybrid modality representation, and determine an adaptive gating weight based on the hybrid modality representation.

[0272] This application designs a gated interaction mechanism to control the ratio of reference single-modal representation and mixed-modal representation during decoding, that is, dynamically adjust the reference ratio through gating weights.

[0273] Specifically, the multimodal model can fuse the first modal representation and the second modal representation through the interactive transformation structure (Cross Transformers) to obtain a hybrid modal representation, and then perform numerical conversion on the hybrid modal representation to obtain the gating weight.

[0274] In some embodiments, the computer device may directly perform addition or multiplication operations on the first modal representation and the second modal representation to obtain a mixed modal representation. In other embodiments, the computer device may also perform an attention operation on the first modal representation and the second modal representation to obtain a mixed modal representation. In other embodiments, the computer device may also use more complex operations, such as performing attention mechanism processing, then performing matrix addition or multiplication, and then performing linear / nonlinear transformations to obtain a mixed modal representation. This application does not limit the specific fusion method.

[0275] Furthermore, after combining the first modal representation and the second modal representation to obtain a hybrid modal representation, the computer device can perform a linear transformation on the hybrid modal representation before performing activation processing to obtain adaptive gating weights. In other words, the hybrid modal representation is converted into a numerical value, which is also the gating weight. In this way, the adaptive gating weight for the current time can be obtained based on the relationship between the first data and the second data at that time.

[0276] Step 506: According to the gating weights, the first modality representation and the mixed modality representation are fused to obtain a first fused representation, and the second modality representation and the mixed modality representation are fused to obtain a second fused representation.

[0277] Specifically, the computer device can perform weighted processing on the first modality representation and the mixed modality representation according to the gating weight to obtain a first fusion representation; and perform weighted processing on the second modality representation and the mixed modality representation according to the gating weight to obtain a second fusion representation.

[0278] In some embodiments, the computer device may use the gating weight as the weight of any one of the first modal representation and the mixed modal representation, and use the difference between the value one and the gating weight as the weight of the other representation, and perform weighted summation processing on the first modal representation and the mixed modal representation.

[0279] In some embodiments, the computer device may use the gating weight as the weight of any one of the second modal representation and the mixed modal representation, and use the difference between the value one and the gating weight as the weight of the other representation, and perform weighted summation processing on the second modal representation and the mixed modal representation.

[0280] Step 508: Determine the matching degree between the first data and the second data according to the first fused representation and the second fused representation.

[0281] In some embodiments, the computer device can directly calculate the matching degree between the first fused representation and the second fused representation, and use the matching degree as the matching degree between the first data and the second data. In other words, in the use scenario of the multimodal model, the first encoder, the second encoder, and the interactive transformation structure can be directly used.

[0282] In other embodiments, determining the degree of matching between the first data and the second data based on the first fused representation and the second fused representation includes: decoding according to the first fused representation to obtain a first feature representation of the first data; decoding according to the second fused representation to obtain a second feature representation of the second data; and determining the degree of matching between the first data and the second data based on the first feature representation and the second feature representation.

[0283] For the specific decoding method, please refer to the decoding method in the processing of the multimodal model in the aforementioned embodiment. The specific method for determining the matching degree between the first data and the second data can be any of the following methods: calculating the feature similarity between the two representations (the first fused representation and the second fused representation, or the first feature representation and the second feature representation), and using the feature similarity as the matching degree; inputting the two representations (the first fused representation and the second fused representation, or the first feature representation and the second feature representation) into the classification layer, and outputting the classification result through the classification layer, and using the classification result as the matching degree. The feature similarity can be represented by cosine similarity, Euclidean distance, Manhattan distance, etc.

[0284] The above-mentioned data matching method encodes the first data and the second data belonging to different modalities respectively to obtain the first modal representation and the second modal representation. Then, the first modal representation and the second modal representation are combined to obtain a hybrid modal representation. According to the hybrid modal representation, an adaptive gating weight can be obtained. The gating weight is used to control the proportion of the first modal representation and the second modal representation when they are fused with the hybrid modal representation. That is, information can be intelligently extracted from the single modality / hybrid modality to obtain the first fusion representation and the second fusion representation. The first fusion representation and the second fusion representation obtained in this way can better highlight the relevant information of the first data and the second data, so they can be used to more accurately determine the matching degree between the first data and the second data, greatly improving the data processing effect. In particular, in the case where the first data and the second data are missing, by intelligently extracting information from the hybrid modality, intelligently selecting the proportion of the extracted hybrid modal information, and reconstructing the first fusion representation of the first data and the second fusion representation of the second data through multimodal information, the missing information is compensated, the matching accuracy can be greatly improved, and the data processing performance is improved.

[0285] In some embodiments, the first modal representation and the second modal representation are combined to obtain a mixed modal representation, including: performing multiple interactive fusions based on the first modal representation and the second modal representation to obtain a first mixed representation biased towards the first modal side and a second mixed representation biased towards the second modal side; combining the first mixed representation and the second mixed representation to obtain a mixed modal representation.

[0286] Specifically, in the process of fusing the first modal representation and the second modal representation, the initial model can perform multiple fusions through a multi-layer neural network to obtain a first mixed representation biased towards the first modal side and a second mixed representation biased towards the second modal side. Each time the interactive fusion is performed, the second intermediate mixed representation can be fused into the first intermediate mixed representation based on the attention mechanism to obtain a first intermediate mixed representation biased towards the first modal side. The first intermediate mixed representation can be fused into the second intermediate mixed representation based on the attention mechanism to obtain a second intermediate mixed representation biased towards the second modal side. In this way, the first intermediate mixed representation output by the last layer of the neural network is the first mixed representation, and the second intermediate mixed representation output by the last layer of the neural network is the second mixed representation.

[0287] Furthermore, the initial model may fuse the first mixed representation and the second mixed representation to obtain a mixed modality representation. For example, the fusion may be performed by weighted summation, connection, or concatenation, which is not limited in this embodiment of the present application.

[0288] In the above embodiment, the first modal representation and the second modal representation are interactively fused multiple times through the initial model, so that a first mixed representation biased towards the first modal side and a second mixed representation biased towards the second modal side can be obtained, and the relationship information between the first data and the second data can be fully extracted. Then, the first mixed representation and the second mixed representation are combined to obtain a mixed modal representation.

[0289] In some embodiments, multiple interactive fusions are performed based on the first modal representation and the second modal representation to obtain a first mixed representation biased towards the first modal side and a second mixed representation biased towards the second modal side, including: when performing the i+1th interactive fusion, obtaining the first intermediate mixed representation and the second intermediate mixed representation output by the i-th interactive fusion; fusing the second intermediate mixed representation into the first intermediate mixed representation to obtain the first intermediate mixed representation of the i+1th interactive fusion; fusing the first intermediate mixed representation into the second intermediate mixed representation to obtain the second intermediate mixed representation of the i+1th interactive fusion; taking i+1 as the new i, and returning to the first intermediate mixed representation and the second intermediate mixed representation obtained by the i-th interactive fusion when performing the i+1th interactive fusion, and continuing to execute until the stopping condition is reached; taking the first intermediate mixed representation output by the last interactive fusion as the first mixed representation, and taking the second intermediate mixed representation output by the last interactive fusion as the second mixed representation; wherein i is a natural number greater than or equal to 0, when i is 0, the first intermediate mixed representation output by the i-th interactive fusion obtained is the first modal representation, and the second intermediate mixed representation output by the i-th interactive fusion obtained is the second modal representation.

[0290] Specifically, for the first interactive fusion, the initial model can fuse the second modal representation into the first modal representation based on the attention mechanism to obtain a first intermediate mixed representation, and fuse the first modal representation into the second modal representation based on the attention mechanism to obtain a second intermediate mixed representation.

[0291] For the second interactive fusion, the initial model can fuse the second intermediate mixed representation obtained from the first interactive fusion into the first intermediate mixed representation based on the attention mechanism to obtain a new first intermediate mixed representation. The initial model can fuse the first intermediate mixed representation obtained from the first interactive fusion into the second intermediate mixed representation based on the attention mechanism to obtain a new second intermediate mixed representation.

[0292] In this way, each interactive fusion process continuously proceeds based on the output of the previous interactive fusion, until the final interactive fusion. The initial model can directly output the first intermediate mixed representation and the second intermediate mixed representation obtained from the last interactive fusion. The first intermediate mixed representation outputted this time is the first mixed representation, and the second intermediate mixed representation outputted this time is the second mixed representation.

[0293] In the above embodiment, after the first modality representation and the second modality representation are fused, the next interactive fusion is continuously performed based on the previous output through an iterative cycle, which can better utilize the information obtained from each fusion and make the information fusion more complete.

[0294] In some embodiments, the second intermediate mixed representation is fused into the first intermediate mixed representation to obtain the first intermediate mixed representation of the i+1th interactive fusion, including: performing an attention operation based on the first intermediate mixed representation and the second intermediate mixed representation output by the i-th interactive fusion, combining the result of the attention operation with the first intermediate mixed representation output by the i-th interactive fusion, and then performing a layer regularization operation to obtain a first intermediate processing feature; performing a forward neural network processing on the first intermediate processing feature, combining the forward neural network processing result with the first intermediate processing feature, and then performing a layer regularization operation to obtain the first intermediate mixed representation of the i+1th interactive fusion.

[0295] Specifically, when performing a layer regularization operation after combining the result of the attention operation with the first intermediate mixed representation of the output of the i-th interactive fusion, the way in which the result of the attention operation and the first intermediate mixed representation of the output of the i-th interactive fusion are combined can be addition, multiplication, or splicing, etc., and this embodiment of the present application does not limit this. When performing a layer regularization operation after combining the result of the forward neural network processing with the first intermediate processing feature, the way in which the forward neural network processing result and the first intermediate processing feature are combined can be addition, multiplication, or splicing, etc., and this embodiment of the present application does not limit this.

[0296] Similar to the processing method in the model training process, the computer device can determine the first intermediate mixed representation of the (i+1)th interactive fusion through the above formulas 10 and 11.

[0297] In the above embodiment, the interactive fusion processing method based on the attention mechanism can better integrate the second intermediate mixed representation into the first intermediate mixed representation. Moreover, this stacked fusion design can prevent the gradient explosion problem during model training.

[0298] In some embodiments, the first intermediate mixed representation is fused into the second intermediate mixed representation to obtain the second intermediate mixed representation of the i+1th interactive fusion, including: performing an attention operation based on the second intermediate mixed representation output by the i-th interactive fusion and the first intermediate mixed representation, combining the result of the attention operation with the second intermediate mixed representation output by the i-th interactive fusion and performing a layer regularization operation to obtain a second intermediate processing feature; performing a forward neural network processing on the second intermediate processing feature, combining the forward neural network processing result with the second intermediate processing feature and performing a layer regularization operation to obtain the second intermediate mixed representation of the i+1th interactive fusion.

[0299] Specifically, when performing a layer regularization operation after combining the result of the attention operation with the second intermediate mixed representation output by the i-th interactive fusion, the way in which the result of the attention operation and the second intermediate mixed representation output by the i-th interactive fusion are combined can be addition, multiplication, or splicing, etc., which is not limited in this embodiment of the present application. When performing a layer regularization operation after combining the result of the feedforward neural network processing with the second intermediate processing feature, the way in which the result of the feedforward neural network processing with the second intermediate processing feature are combined can be addition, multiplication, or splicing, etc., which is not limited in this embodiment of the present application.

[0300] Similar to the processing method in the model training process, the computer device can determine the second intermediate mixed representation of the (i+1)th interactive fusion through Formula 12 and Formula 13 in the aforementioned embodiment.

[0301] In the above embodiment, the interactive fusion processing method based on the attention mechanism can better fuse the first intermediate mixed representation into the second intermediate mixed representation. Moreover, through this stacked fusion design, the gradient explosion problem can be prevented during model training.

[0302] In some embodiments, the first modal representation and the mixed modal representation are fused according to the gating weight to obtain a first fused representation, including: taking the gating weight as the coefficient of the mixed modal representation, and taking the difference between the value one and the gating weight as the coefficient of the first modal representation; and performing weighted processing on the mixed modal representation and the first modal representation according to the coefficient of the mixed modal representation and the coefficient of the first modal representation to obtain the first fused representation.

[0303] Specifically, the initial model may perform weighted summation of the mixed modal representation and the first modal representation according to the coefficients of the mixed modal representation and the coefficients of the first modal representation to obtain the first fusion representation.

[0304] In the above embodiment, by fusing the hybrid modal representation and the first modality through gated weights, the ratio of the single modal representation and the hybrid modal representation can be flexibly referenced according to the alignment quality of the two modal data, so that the fused representation can better express the information of the single modal data.

[0305] In some embodiments, the second modal representation and the mixed modal representation are fused according to the gating weight to obtain the second fused representation, including: taking the gating weight as the coefficient of the mixed modal representation, and taking the difference between the value one and the gating weight as the coefficient of the second modal representation; according to the coefficient of the mixed modal representation and the coefficient of the second modal representation, the mixed modal representation and the second modal representation are weighted to obtain the first fused representation.

[0306] Specifically, the initial model may perform weighted summation on the mixed modal representation and the second modal representation according to the coefficients of the mixed modal representation and the coefficients of the second modal representation to obtain the second fused representation.

[0307] In the above embodiment, by fusing the hybrid modal representation and the second modality through gated weights, the ratio of the single modal representation and the hybrid modal representation can be flexibly referenced according to the alignment quality of the two modal data, so that the fused representation can better express the information of the single modal data.

[0308] In some embodiments, reference Figure 6 , the data matching method includes the following steps:

[0309] In step 602 , the first data and the second data are respectively encoded to obtain a first modal representation and a second modal representation, where the first data and the second data are data of different modalities.

[0310] Step 604 : Combine the first modality representation and the second modality representation to obtain a first mixed representation biased toward the first modality and a second mixed representation biased toward the second modality.

[0311] Step 606 : Determine an adaptive first gating weight according to the first mixed modality representation, and determine an adaptive second gating weight according to the second mixed modality representation.

[0312] Specifically, the multimodal model may perform a linear transformation on the first mixed modal representation and then perform activation processing to obtain a first gating weight; and perform a linear transformation on the second mixed modal representation and then perform activation processing to obtain a second gating weight.

[0313] Step 608: According to the first gating weight, the first modal representation and the first mixed representation are fused to obtain a first fused representation; according to the second gating weight, the second modal representation and the second mixed representation are fused to obtain a second fused representation.

[0314] Specifically, the multimodal model may use the first gating weight as the coefficient of the first mixed representation, and the difference between the value one and the first gating weight as the coefficient of the first modal representation; according to the coefficient of the first mixed representation and the coefficient of the first modal representation, the first mixed representation and the first modal representation are weightedly summed, and then decoded based on the result of the weighted summation to obtain the first predicted feature representation.

[0315] The multimodal model can use the second gating weight as the coefficient of the second mixed representation, and the difference between the value one and the second gating weight as the coefficient of the second modal representation; according to the coefficient of the second mixed representation and the coefficient of the second modal representation, the second mixed representation and the second modal representation are weightedly summed, and then decoded based on the result of the weighted summation to obtain a second predicted feature representation.

[0316] Step 610: Determine the matching degree between the first data and the second data according to the first fused representation and the second fused representation.

[0317] In the above embodiment, for the first modality, the multimodal model will use the first modality as the main information, and at the same time integrate the information represented by the second modality to obtain a first mixed representation, and then determine the gating weight of the first modality side according to the first mixed representation, that is, the first gating weight. In this way, the proportion of the single modality information and the mixed modality information referenced when decoding the first modality side can be controlled according to the first gating weight. For the second modality, the initial model will use the second modality as the main information, and at the same time integrate the information represented by the first modality to obtain a second mixed representation, and then determine the gating weight of the first modality side according to the second mixed representation, that is, the second gating weight. In this way, the proportion of the single modality information and the mixed modality information referenced when decoding the second modality side can be controlled according to the second gating weight. In this way, different modal sides can be more flexible and adaptive when fusing mixed modality information.

[0318] In some embodiments, it is characterized in that the first data includes image data and the second data includes text data, and the method further includes: determining the text retrieval result of the specified image data based on the matching degree between the specified image data and multiple different text data; and / or determining the image retrieval result of the specified text data based on the matching degree between the specified text data and multiple different image data.

[0319] Specifically, the specific method of determining text retrieval results based on matching can be to use text data whose matching degree meets the push conditions as text retrieval results. The specific method of determining image retrieval results based on matching can be to use image data whose matching degree meets the push conditions as image retrieval results.

[0320] Among them, the matching degree that meets the push conditions can specifically be the highest matching degree, or the matching degree is greater than a preset threshold, or the top N with the highest matching degree, etc., and this embodiment of the application does not limit this.

[0321] In the above embodiment, the image and text matching calculation can be realized through the trained multimodal model, which can help improve the quality and efficiency of image and text retrieval.

[0322] In a specific embodiment, the first mode is an image mode and the second mode is a text mode. Figure 7 , Figure 7 4 is a structural block diagram of a multimodal model in one embodiment.

[0323] Take the model training stage as an example: Figure 7As shown, LightVLP uses two separate encoders to model different modal information. For one of the modalities, such as image input, the model randomly masks a certain proportion (such as 50%) of patches. Then, the remaining patches form a sequence and are input into the image transformer to obtain an image representation (in the training phase, this corresponds to the first sample modality representation in the aforementioned embodiment). Correspondingly, for text input, the model randomly masks a certain proportion of characters. Then, the remaining characters form a sequence and are input into the text transformer to obtain a text representation (in the training phase, this corresponds to the second sample modality representation in the aforementioned embodiment).

[0324] The image and text representations are fed into Cross Transformers, which process them to produce a mixed-modal sample representation. A gating weight p is then determined based on this mixed-modal representation. The image and mixed-modal sample representations are fused using the gating weight p to produce a first fused sample representation. The text and mixed-modal sample representations are also fused using the gating weight p to produce a second fused sample representation.

[0325] Furthermore, based on the first fused sample representation and the mask features on the image side, decoding is performed to obtain a representation of the masked block, and based on the second fused sample representation and the mask features on the text side, decoding is performed to obtain a representation of the masked character. This application introduces two traditional image / text unsupervised training tasks, namely the image reconstruction task and the text reconstruction task, to construct the first modality loss and the second modality loss. An image-text matching task is also introduced to construct the second loss. A contrastive learning task is also introduced to construct the third loss. Model training is achieved through joint task learning.

[0326] In some application scenarios, a trained multimodal model can be used as a vision-language pre-training model, or a large model. Based on the requirements of specific downstream tasks, the multimodal model can be fine-tuned to obtain a task model suitable for the task. Downstream tasks include generating text from images, generating images from text, generating image captions, and so on.

[0327] In some application scenarios, this multimodal model can be used to recover original data. For example, the input data is a first data and a second data in two modalities, where the first data is an image with missing data, and the second data is text with missing data. The multimodal model can then output the missing image and text.

[0328] In some embodiments, the present application also conducts an effect test on the multimodal model obtained by the above training. Please refer to Table 1. From the results presented in Table 1, it can be seen that the model of the present application has a good improvement in efficiency and effect compared with the baseline model.

[0329] Table 1 Experimental results of different models on computing speed and recall rate

[0330]

[0331] Among them, TR refers to text retrieval tasks and IR refers to image retrieval tasks.

[0332] In addition, an ablation experiment was conducted to verify the effectiveness of the gated interaction strategy (gate part) in the multimodal model in this application. Please refer to Table 2 below for the experimental data:

[0333] Table 2 Ablation experiment results

[0334]

[0335] It can be clearly seen from Table 2 above that the model of this application with the gated interaction strategy has a significant improvement in recall rate.

[0336] Finally, we conducted detailed tests on different mask ratios, and the conclusions are shown in Table 3:

[0337] Table 3 Test results of different mask ratios

[0338]

[0339] It can be clearly seen from the above test results that the multimodal model provided in this application has advantages in flexibility and efficiency compared with the traditional VLP model, and the dual-coding structure is more suitable for image-text retrieval tasks.

[0340] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0341] Based on the same inventive concept, the present application also provides a data matching device for implementing the aforementioned data matching method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more data matching device embodiments provided below can be found in the above-mentioned limitations on the data matching method and will not be further elaborated here.

[0342] In an exemplary embodiment, Figure 8 As shown, a data matching device 800 is provided, comprising: an encoding module 801, a determination module 802, a fusion module 803 and a matching module 804, wherein:

[0343] The encoding module is used to encode the first data and the second data respectively to obtain a first modal representation and a second modal representation, where the first data and the second data are data of different modalities.

[0344] A determination module is configured to combine the first modality representation and the second modality representation to obtain a hybrid modality representation, and determine an adaptive gating weight according to the hybrid modality representation.

[0345] A fusion module is used to fuse the first modality representation and the mixed modality representation according to the gating weight to obtain a first fused representation, and to fuse the second modality representation and the mixed modality representation to obtain a second fused representation.

[0346] The matching module is used to determine the matching degree between the first data and the second data according to the first fused representation and the second fused representation.

[0347] In some embodiments, the determination module is further used to perform multiple interactive fusions based on the first modal representation and the second modal representation to obtain a first mixed representation biased towards the first modal side and a second mixed representation biased towards the second modal side; and combine the first mixed representation and the second mixed representation to obtain a mixed modal representation.

[0348] In some embodiments, the determination module is also used to obtain the first intermediate mixed representation and the second intermediate mixed representation output of the i-th interactive fusion when performing the i+1-th interactive fusion; fuse the second intermediate mixed representation into the first intermediate mixed representation to obtain the first intermediate mixed representation of the i+1-th interactive fusion; fuse the first intermediate mixed representation into the second intermediate mixed representation to obtain the second intermediate mixed representation of the i+1-th interactive fusion; use i+1 as the new i, and return to the first intermediate mixed representation and the second intermediate mixed representation obtained by the i-th interactive fusion when performing the i+1-th interactive fusion, and continue to execute until the stopping condition is reached; use the first intermediate mixed representation output of the last interactive fusion as the first mixed representation, and use the second intermediate mixed representation output of the last interactive fusion as the second mixed representation; wherein, i is a natural number greater than or equal to 0, when i is 0, the first intermediate mixed representation output of the i-th interactive fusion obtained is the first modal representation, and the second intermediate mixed representation output of the i-th interactive fusion obtained is the second modal representation.

[0349] In some embodiments, the determination module is further used to perform an attention operation based on the first intermediate mixed representation and the second intermediate mixed representation output by the i-th interactive fusion, and perform a layer regularization operation after combining the attention operation result with the first intermediate mixed representation output by the i-th interactive fusion to obtain a first intermediate processing feature; perform a forward neural network processing on the first intermediate processing feature, and perform a layer regularization operation after combining the forward neural network processing result with the first intermediate processing feature to obtain the first intermediate mixed representation of the i+1-th interactive fusion.

[0350] In some embodiments, the determination module is further used to perform an attention operation based on the second intermediate mixed representation output by the i-th interactive fusion and the first intermediate mixed representation, and perform a layer regularization operation after combining the attention operation result and the second intermediate mixed representation output by the i-th interactive fusion to obtain a second intermediate processing feature; perform a forward neural network processing on the second intermediate processing feature, and perform a layer regularization operation after combining the forward neural network processing result and the second intermediate processing feature to obtain a second intermediate mixed representation of the i+1-th interactive fusion.

[0351] In some embodiments, the determination module is further configured to perform linear transformation on the mixed modality representation and then perform activation processing to obtain adaptive gating weights.

[0352] In some embodiments, the fusion module is also used to use the gating weight as the coefficient of the mixed modal representation, and the difference between the value one and the gating weight as the coefficient of the first modal representation; according to the coefficient of the mixed modal representation and the coefficient of the first modal representation, the mixed modal representation and the first modal representation are weighted to obtain a first fused representation.

[0353] In some embodiments, the fusion module is also used to use the gating weight as the coefficient of the mixed modal representation, and the difference between the value one and the gating weight as the coefficient of the second modal representation; according to the coefficient of the mixed modal representation and the coefficient of the second modal representation, the mixed modal representation and the second modal representation are weighted to obtain a first fused representation.

[0354] In some embodiments, the mixed modal representation includes a first mixed representation biased toward the first modality and a second mixed representation biased toward the second modality, and the gating weights include a first gating weight determined according to the first mixed modality representation and a second gating weight determined according to the second mixed modality representation; the fusion module is further used to fuse the first modality representation and the first mixed representation according to the first gating weight to obtain a first fused representation; and to fuse the second modality representation and the second mixed representation according to the second gating weight to obtain a second fused representation.

[0355] In some embodiments, the matching module is further used to decode according to the first fused representation to obtain a first feature representation of the first data; decode according to the second fused representation to obtain a second feature representation of the second data; and determine the degree of matching between the first data and the second data based on the first feature representation and the second feature representation.

[0356] In some embodiments, the first data includes image data, the second data includes text data, and the device also includes a retrieval module for determining the text retrieval results of the specified image data based on the matching degree between the specified image data and multiple different text data; and / or determining the image retrieval results of the specified text data based on the matching degree between the specified text data and multiple different image data.

[0357] Based on the same inventive concept, the present application also provides a multimodal model processing device for implementing the multimodal model processing method mentioned above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more multimodal model processing device embodiments provided below can be found in the above-mentioned limitations of the multimodal model processing method and will not be repeated here.

[0358] In an exemplary embodiment, Figure 9 As shown, a multimodal model processing device 900 is provided, including an encoding module 901, a determination module 902, a fusion module 903, a construction module 904 and a training module 905, wherein:

[0359] The encoding module is used to encode the initial model to be trained and the first sample data and the second sample data in the training sample pair respectively to obtain the first sample modal representation and the second sample modal representation; the first sample data and the second sample data are sample data of different modalities.

[0360] The determination module is used to obtain a mixed sample modality representation by combining the first sample modality representation and the second sample modality representation through an initial model, and determine a prediction weight according to the mixed sample modality representation.

[0361] The fusion module is used to fuse the first sample modality representation and the mixed sample modality representation according to the prediction weight through the initial model, and decode based on the fused result to obtain the first prediction feature representation.

[0362] The fusion module is also used to fuse the second sample modality representation and the mixed sample modality representation according to the prediction weight through the initial model, and decode based on the fused result to obtain the second prediction feature representation.

[0363] A construction module is used to determine a first loss based on the true feature representation and the first predicted feature representation of the first sample data, and the true feature representation and the second predicted feature representation of the second sample data; determine a second loss based on the alignment labels of the first sample data and the second sample data, and the prediction weights; and construct a target loss function based on the first loss and the second loss.

[0364] The training module is used to train the initial model to be trained based on the target loss function, and obtain the multimodal model after the training is completed.

[0365] In some embodiments, the apparatus further includes a masking module configured to obtain a training sample pair comprising first sample data and second sample data; split the first sample data into a plurality of first data blocks, split the second sample data into a plurality of second data blocks, extract a portion of the plurality of first data blocks to form a first input sequence, and extract a portion of the plurality of second data blocks to form a second input sequence. An encoding module is configured to perform multiple encoding operations on the first input sequence to obtain a first modal representation, and to perform multiple encoding operations on the second input sequence to obtain a second modal representation.

[0366] In some embodiments, the determination module is further used to perform multiple interactive fusions of the first sample modal representation and the second sample modal representation through the initial model to obtain a first mixed sample representation biased towards the first modal side and a second mixed sample representation biased towards the second modal side; and combine the first mixed sample representation and the second mixed sample representation to obtain a mixed sample modal representation.

[0367] In some embodiments, the construction module is further used to construct a first vector based on the alignment labels of the first sample data and the second sample data; construct a second vector based on the predicted weight and the difference between the numerical value one and the predicted weight; and determine the second loss based on the first vector and the second vector.

[0368] In some embodiments, the mixed sample modality representation includes a first mixed sample representation biased toward the first modality and a second mixed sample representation biased toward the second modality, and the prediction weights include a first prediction weight determined based on the first mixed sample representation and a second prediction weight determined based on the second mixed sample representation. The fusion module is further configured to fuse the first sample modality representation and the first mixed sample representation using the initial model according to the first prediction weight; and to fuse the second sample modality representation and the second mixed sample representation using the initial model according to the second prediction weight.

[0369] In some embodiments, the mixed sample modality representation includes a first mixed sample representation biased toward the first modality side and a second mixed sample representation biased toward the second modality side, and the prediction weight includes a first prediction weight determined according to the first mixed sample representation and a second prediction weight determined according to the second mixed sample representation; the construction module is also used to construct a first vector based on the alignment labels of the first sample data and the second sample data; determine the target weight according to the mean of the first prediction weight and the second prediction weight, and construct a third vector based on the target weight and the difference between the numerical value one and the target weight; determine the second loss according to the first vector and the third vector.

[0370] In some embodiments, the construction module is further used to determine the correlation between the first sample data and the second sample data based on the first sample modal representation and the second sample modal representation; determine the third loss based on the alignment labels of the first sample data and the second sample data, and the correlation between the first sample data and the second sample data; and construct a target loss function based on the first loss, the second loss, and the third loss.

[0371] In some embodiments, the correlation between the first sample data and the second sample data includes: the correlation of the first sample data relative to the second sample data, and the correlation of the second sample data relative to the first sample data; the construction module is also used to determine a preset number of third sample data with the same modality as the first sample data, and a preset number of fourth sample data with the same modality as the second sample data; calculate the correlation of the first sample data with respect to the second sample data based on the first sample modal representation, the second sample modal representation, and the sample modal representation of each fourth sample data; calculate the correlation of the second sample data with respect to the first sample data based on the first sample modal representation, the second sample modal representation, and the sample modal representation of each third sample data.

[0372] In some embodiments, the construction module is further used to determine a first value based on the first sample modal representation and the second sample modal representation; determine multiple second values based on the first sample modal representation and the sample modal representation of each fourth sample data; and use the comparison value of the first value and the sum of the multiple second values as the correlation of the second sample data relative to the first sample data.

[0373] In some embodiments, the device also includes a retrieval module for obtaining a text retrieval task, extracting the image data specified in the text retrieval task, and forming a first data pair to be processed by combining each text data in the text library with the specified image data; processing the first data pair through a multimodal model to obtain two feature representations, and determining the matching degree of the first data pair based on the two feature representations; and determining the target text that matches the specified image data based on the matching degree of each first data pair.

[0374] In some embodiments, the device also includes a retrieval module for obtaining an image retrieval task, extracting the text data specified in the image retrieval task, and forming a second data pair to be processed by respectively combining each image data in the image library with the specified text data; processing the second data pair through a multimodal model to obtain two feature representations, and determining the matching degree of the second data pair based on the two feature representations; and determining the target image that matches the specified text data according to the matching degree of each second data pair.

[0375] Each module in the above-mentioned apparatus may be implemented in whole or in part by software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0376] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store graphic data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data matching method and / or a multimodal model processing method is implemented.

[0377] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0378] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0379] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0380] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0381] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0382] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0383] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0384] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data matching method, characterized in that: The method comprises: Encoding the first data and the second data respectively to obtain a first modal representation and a second modal representation, wherein the first data and the second data are data of different modalities; the modalities are visual modalities, text modalities, or audio modalities; Performing multiple interactive fusions based on the first modal representation and the second modal representation, and in each interactive fusion, fusing the second intermediate mixed representation into the first intermediate mixed representation based on the attention mechanism to obtain a first intermediate mixed representation biased towards the first modality, and fusing the first intermediate mixed representation into the second intermediate mixed representation based on the attention mechanism to obtain a second intermediate mixed representation biased towards the second modality, until the last interactive fusion outputs the first mixed representation biased towards the first modality and the second mixed representation biased towards the second modality; combining the first mixed representation and the second mixed representation to obtain a mixed modal representation; determining adaptive gating weights based on the mixed modality representation; According to the gating weights, fusing the first modality representation and the mixed modality representation to obtain a first fused representation, and fusing the second modality representation and the mixed modality representation to obtain a second fused representation; A degree of matching between the first data and the second data is determined according to the first fused representation and the second fused representation.

2. The method according to claim 1, characterized in that The interactive fusion is performed multiple times based on the first modal representation and the second modal representation. In each interactive fusion, the second intermediate mixed representation is fused into the first intermediate mixed representation based on the attention mechanism to obtain a first intermediate mixed representation biased towards the first modal side. The first intermediate mixed representation is fused into the second intermediate mixed representation based on the attention mechanism to obtain a second intermediate mixed representation biased towards the second modal side, until the last interactive fusion outputs the first mixed representation biased towards the first modal side and the second mixed representation biased towards the second modal side, including: When performing the (i+1)th interactive fusion, obtaining a first intermediate mixed representation and a second intermediate mixed representation of the output of the i-th interactive fusion; Fusing the second intermediate mixed representation into the first intermediate mixed representation to obtain an (i+1)th interactively fused first intermediate mixed representation; Fusing the first intermediate mixed representation into the second intermediate mixed representation to obtain an (i+1)th interactively fused second intermediate mixed representation; Set i+1 as the new i, and return the first intermediate mixed representation and the second intermediate mixed representation obtained by the i-th interactive fusion during the i+1-th interactive fusion, and continue executing until the stopping condition is met; The first intermediate mixed representation output by the last interactive fusion is used as the first mixed representation, and the second intermediate mixed representation output by the last interactive fusion is used as the second mixed representation; Wherein, i is a natural number greater than or equal to 0. When i is 0, the first intermediate mixed representation obtained from the i-th interactive fusion output is the first modal representation, and the second intermediate mixed representation obtained from the i-th interactive fusion output is the second modal representation.

3. The method according to claim 2, characterized in that The step of fusing the second intermediate mixed representation into the first intermediate mixed representation to obtain the (i+1)th interactively fused first intermediate mixed representation includes: Perform an attention operation on the first intermediate mixed representation and the second intermediate mixed representation output by the i-th interactive fusion, combine the attention operation result with the first intermediate mixed representation output by the i-th interactive fusion, and then perform a layer regularization operation to obtain a first intermediate processed feature; Perform feedforward neural network processing on the first intermediate processing feature, combine the feedforward neural network processing result and the first intermediate processing feature, and then perform a layer regularization operation to obtain a first intermediate mixed representation of the i+1th interactive fusion.

4. The method according to claim 1, wherein Determining an adaptive gating weight according to the mixed modality representation includes: The mixed modality representation is linearly transformed and then activated to obtain adaptive gating weights.

5. The method according to claim 1, wherein The mixed modality representation includes a first mixed representation biased toward the first modality and a second mixed representation biased toward the second modality, and the gating weight includes a first gating weight determined according to the first mixed representation and a second gating weight determined according to the second mixed representation; The fusing the first modality representation and the mixed modality representation to obtain a first fused representation, and fusing the second modality representation and the mixed modality representation to obtain a second fused representation according to the gating weights, includes: fusing the first modal representation and the first hybrid representation according to the first gating weight to obtain a first fused representation; According to the second gating weight, the second modality representation and the second mixed representation are fused to obtain a second fused representation.

6. The method according to claim 1, characterized in that The determining, based on the first fused representation and the second fused representation, a degree of matching between the first data and the second data includes: Decoding the first fused representation to obtain a first feature representation of the first data; Decoding the second fused representation to obtain a second feature representation of the second data; A degree of matching between the first data and the second data is determined based on the first feature representation and the second feature representation.

7. The method according to any one of claims 1 to 6, characterized in that The first data includes image data, the second data includes text data, and the method further includes: Determining a text retrieval result of the specified image data based on the matching degrees between the specified image data and a plurality of different text data; and / or, An image retrieval result of the specified text data is determined based on the matching degrees between the specified text data and a plurality of different image data.

8. A multimodal model processing method, characterized in that: The method comprises: Encoding first sample data and second sample data in a training sample pair respectively using an initial model to be trained to obtain a first sample modality representation and a second sample modality representation; the first sample data and the second sample data are sample data of different modalities; the modalities are visual modalities, textual modalities, or audio modalities; Obtaining a mixed sample modality representation by combining the first sample modality representation and the second sample modality representation through the initial model, and determining a prediction weight according to the mixed sample modality representation; fusing the first sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and performing decoding based on the fused result to obtain a first prediction feature representation; fusing the second sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and performing decoding based on the fused result to obtain a second prediction feature representation; Determining a first loss based on the true feature representation of the first sample data and the first predicted feature representation, and the true feature representation of the second sample data and the second predicted feature representation; constructing a first vector based on the alignment labels of the first sample data and the second sample data; constructing a second vector according to the predicted weight and the difference between the value 1 and the predicted weight; determining a second loss based on the first vector and the second vector; A target loss function is constructed according to the first loss and the second loss, and the initial model to be trained is trained based on the target loss function, so as to obtain a multimodal model after the training is completed.

9. The method according to claim 8, characterized in that Before encoding the first sample data and the second sample data in the training sample pair, the method further includes: Acquire a training sample pair, wherein the training sample pair includes first sample data and second sample data; Splitting the first sample data into a plurality of first data blocks, and splitting the second sample data into a plurality of second data blocks, Extracting some data blocks from the plurality of first data blocks to form a first input sequence, and extracting some data blocks from the plurality of second data blocks to form a second input sequence; The encoding of the first sample data and the second sample data in the training sample pair to obtain the first sample modal representation and the second sample modal representation includes: The first input sequence is encoded multiple times to obtain a first modal representation, and the second input sequence is encoded multiple times to obtain a second modal representation.

10. The method according to claim 8, characterized in that The mixed sample modality representation includes a first mixed sample representation biased towards the first modality and a second mixed sample representation biased towards the second modality, and the prediction weight includes a first prediction weight determined according to the first mixed sample representation and a second prediction weight determined according to the second mixed sample representation; The fusing the first sample modality representation and the mixed sample modality representation according to the prediction weight through the initial model includes: fusing the first sample modality representation and the first mixed sample representation using the initial model according to the first prediction weight; The fusing the second sample modality representation and the mixed sample modality representation according to the prediction weight by the initial model includes: The second sample modality representation and the second mixed sample representation are fused through the initial model according to the second prediction weight.

11. The method according to claim 8, characterized in that The method further comprises: determining a correlation between the first sample data and the second sample data according to the first sample modality representation and the second sample modality representation; Calculating the cross entropy between the alignment labels of the first sample data and the second sample data, and the correlation between the first sample data and the second sample data, and using the cross entropy as a third loss; The constructing a target loss function according to the first loss and the second loss includes: An objective loss function is constructed according to the first loss, the second loss, and the third loss.

12. The method according to claim 11, characterized in that The correlation between the first sample data and the second sample data includes: the correlation between the first sample data and the second sample data, and the correlation between the second sample data and the first sample data; The determining, based on the first sample modality representation and the second sample modality representation, the correlation between the first sample data and the second sample data includes: Determining a preset number of third sample data having the same modality as the first sample data, and a preset number of fourth sample data having the same modality as the second sample data; Calculating a correlation between the first sample data and the second sample data based on the first sample modality representation, the second sample modality representation, and the sample modality representation of each fourth sample data; The correlation of the second sample data with respect to the first sample data is calculated based on the first sample modality representation, the second sample modality representation, and the sample modality representation of each third sample data.

13. The method according to claim 12, characterized in that Calculating the correlation between the first sample data and the second sample data based on the first sample modality representation, the second sample modality representation, and the sample modality representation of each fourth sample data includes: determining a first value based on the first sample modality representation and the second sample modality representation; determining a plurality of second values based on the first sample modal representation and the sample modal representation of each fourth sample data; A comparison value between the first value and the sum of the plurality of second values is used as a correlation between the first sample data and the second sample data.

14. The method according to any one of claims 8 to 13, characterized in that The modality includes an image modality and a text modality, and the method further includes: Acquire a text retrieval task, extract the image data specified in the text retrieval task, and form a first data pair to be processed with each text data in the text library and the specified image data; Processing the first data pair using the multimodal model to obtain two feature representations, and determining a matching degree of the first data pair based on the two feature representations; According to the matching degree of each first data pair, a target text matching the specified image data is determined.

15. The method according to any one of claims 8 to 13, characterized in that The modality includes an image and text, and the method further includes: Obtaining an image retrieval task, extracting text data specified in the image retrieval task, and forming a second data pair to be processed with each image data in the image library and the specified text data; Processing the second data pair using the multimodal model to obtain two feature representations, and determining a matching degree of the second data pair based on the two feature representations; A target image matching the specified text data is determined according to the matching degree of each second data pair.

16. A data matching device, characterized in that: The device comprises: an encoding module, configured to encode the first data and the second data respectively to obtain a first modal representation and a second modal representation, wherein the first data and the second data are data of different modalities; the modalities are visual modality, text modality, or audio modality; A determination module is used to perform multiple interactive fusions based on the first modal representation and the second modal representation, and in each interactive fusion, the second intermediate mixed representation is fused into the first intermediate mixed representation based on the attention mechanism to obtain a first intermediate mixed representation biased towards the first modal side, and the first intermediate mixed representation is fused into the second intermediate mixed representation based on the attention mechanism to obtain a second intermediate mixed representation biased towards the second modal side, until the last interactive fusion outputs the first mixed representation biased towards the first modal side and the second mixed representation biased towards the second modal side; the first mixed representation and the second mixed representation are combined to obtain a mixed modal representation; and an adaptive gating weight is determined according to the mixed modal representation; a fusion module, configured to fuse the first modality representation and the mixed modality representation to obtain a first fused representation, and to fuse the second modality representation and the mixed modality representation to obtain a second fused representation, according to the gating weights; A matching module is used to determine a matching degree between the first data and the second data according to the first fusion representation and the second fusion representation.

17. The device according to claim 16, characterized in that The determining module is further configured to, when performing the (i+1)th interactive fusion, obtain a first intermediate mixed representation and a second intermediate mixed representation outputted by the (i+1)th interactive fusion; fuse the second intermediate mixed representation into the first intermediate mixed representation to obtain the first intermediate mixed representation outputted by the (i+1)th interactive fusion; and fuse the first intermediate mixed representation into the second intermediate mixed representation to obtain the second intermediate mixed representation outputted by the (i+1)th interactive fusion; Set i+1 as the new i, and return to the first intermediate mixed representation and the second intermediate mixed representation obtained by the i-th interactive fusion during the i+1-th interactive fusion, and continue executing until the stopping condition is met; use the first intermediate mixed representation output by the last interactive fusion as the first mixed representation, and use the second intermediate mixed representation output by the last interactive fusion as the second mixed representation; Wherein, i is a natural number greater than or equal to 0. When i is 0, the first intermediate mixed representation obtained from the i-th interactive fusion output is the first modal representation, and the second intermediate mixed representation obtained from the i-th interactive fusion output is the second modal representation.

18. The device according to claim 17, characterized in that The determination module is further configured to perform an attention operation based on the first intermediate mixed representation and the second intermediate mixed representation output by the i-th interactive fusion, and combine the attention operation result with the first intermediate mixed representation output by the i-th interactive fusion, and then perform a layer regularization operation to obtain a first intermediate processed feature; Perform feedforward neural network processing on the first intermediate processing feature, combine the feedforward neural network processing result and the first intermediate processing feature, and then perform a layer regularization operation to obtain a first intermediate mixed representation of the i+1th interactive fusion.

19. The device according to claim 16, characterized in that The determination module is further configured to perform linear transformation on the mixed modal representation and then perform activation processing to obtain adaptive gating weights.

20. The device according to claim 16, wherein The mixed modality representation includes a first mixed representation biased toward the first modality and a second mixed representation biased toward the second modality, and the gating weight includes a first gating weight determined according to the first mixed representation and a second gating weight determined according to the second mixed representation; The fusion module is further configured to fuse the first modal representation and the first mixed representation according to the first gating weight to obtain a first fused representation; and to fuse the second modal representation and the second mixed representation according to the second gating weight to obtain a second fused representation.

21. The device according to claim 16, characterized in that The matching module is further configured to decode according to the first fused representation to obtain a first feature representation of the first data; decode according to the second fused representation to obtain a second feature representation of the second data; and determine a degree of matching between the first data and the second data based on the first feature representation and the second feature representation.

22. The device according to any one of claims 16 to 21, characterized in that The first data includes image data, the second data includes text data, and the apparatus further includes a retrieval module for determining a text retrieval result for the specified image data based on a matching degree between the specified image data and a plurality of different text data; And / or, based on the matching degrees between the specified text data and a plurality of different image data, an image retrieval result of the specified text data is determined.

23. A multimodal model processing device, characterized in that: The device comprises: an encoding module, configured to encode, using an initial model to be trained, first sample data and second sample data in a training sample pair, respectively, to obtain a first sample modality representation and a second sample modality representation; the first sample data and the second sample data are sample data of different modalities; the modalities are visual modality, textual modality, or audio modality; a determination module, configured to obtain a mixed sample modality representation by combining the first sample modality representation and the second sample modality representation through the initial model, and determine a prediction weight according to the mixed sample modality representation; a fusion module, configured to fuse the first sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and perform decoding based on the fused result to obtain a first prediction feature representation; The fusion module is further configured to fuse the second sample modality representation and the mixed sample modality representation using the initial model according to the prediction weight, and perform decoding based on the fused result to obtain a second prediction feature representation; A construction module is configured to determine a first loss based on the true feature representation and the first predicted feature representation of the first sample data, and the true feature representation and the second predicted feature representation of the second sample data; construct a first vector based on the alignment labels of the first sample data and the second sample data; construct a second vector based on the predicted weight and the difference between a value of 1 and the predicted weight; determine a second loss based on the first vector and the second vector; and construct a target loss function based on the first loss and the second loss; A training module is used to train the initial model to be trained based on the target loss function, and obtain a multimodal model after the training is completed.

24. The device according to claim 23, characterized in that The apparatus further includes a masking module configured to obtain a training sample pair, the training sample pair including first sample data and second sample data; split the first sample data into a plurality of first data blocks, split the second sample data into a plurality of second data blocks, extract a portion of the plurality of first data blocks to form a first input sequence, and extract a portion of the plurality of second data blocks to form a second input sequence; The encoding module is further configured to perform multiple encodings on the first input sequence to obtain a first modal representation, and perform multiple encodings on the second input sequence to obtain a second modal representation.

25. The device according to claim 23, characterized in that The mixed sample modality representation includes a first mixed sample representation biased towards the first modality and a second mixed sample representation biased towards the second modality, and the prediction weight includes a first prediction weight determined according to the first mixed sample representation and a second prediction weight determined according to the second mixed sample representation; The fusion module is also used to fuse the first sample modality representation and the first mixed sample representation through the initial model according to the first prediction weight; and to fuse the second sample modality representation and the second mixed sample representation through the initial model according to the second prediction weight.

26. The device according to claim 23, characterized in that The building blocks are also used to: determining a correlation between the first sample data and the second sample data according to the first sample modality representation and the second sample modality representation; Calculating the cross entropy between the alignment labels of the first sample data and the second sample data, and the correlation between the first sample data and the second sample data, and using the cross entropy as a third loss; An objective loss function is constructed according to the first loss, the second loss, and the third loss.

27. The device according to claim 26, characterized in that The correlation between the first sample data and the second sample data includes: the correlation between the first sample data and the second sample data, and the correlation between the second sample data and the first sample data; The building blocks are also used to: Determining a preset number of third sample data having the same modality as the first sample data, and a preset number of fourth sample data having the same modality as the second sample data; Calculating a correlation between the first sample data and the second sample data based on the first sample modality representation, the second sample modality representation, and the sample modality representation of each fourth sample data; The correlation of the second sample data with respect to the first sample data is calculated based on the first sample modality representation, the second sample modality representation, and the sample modality representation of each third sample data.

28. The device according to claim 27, characterized in that The construction module is also used to determine a first value based on the first sample modal representation and the second sample modal representation; determine multiple second values based on the first sample modal representation and the sample modal representation of each fourth sample data; and use the comparison value of the first value and the sum of the multiple second values as the correlation between the first sample data and the second sample data.

29. The device according to any one of claims 23 to 28, characterized in that The modalities include image modality and text modality. The device also includes a retrieval module, which is used to obtain a text retrieval task, extract the image data specified in the text retrieval task, and form a first data pair to be processed by combining each text data in the text library with the specified image data; process the first data pair through the multimodal model to obtain two feature representations, and determine the matching degree of the first data pair based on the two feature representations; and determine the target text that matches the specified image data based on the matching degree of each first data pair.

30. The device according to any one of claims 23 to 28, characterized in that The modality includes images and text, and the device also includes a retrieval module, which is used to obtain an image retrieval task, extract the text data specified in the image retrieval task, and form a second data pair to be processed by combining each image data in the image library with the specified text data; process the second data pair through the multimodal model to obtain two feature representations, and determine the matching degree of the second data pair based on the two feature representations; and determine the target image that matches the specified text data according to the matching degree of each second data pair.

31. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

32. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

33. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images