Data processing model training method and data processing method

Abstract

The embodiment of the invention provides a data processing model training method and a data processing method.The data processing model training method comprises the steps that a data sample pair is determined, and the data sample pair comprises a text data sample and a corresponding target modal data sample; determining a first type of text data and a second type of text data from the text data sample, and calculating to obtain a first attribution score of the first type of text data and a second attribution score of the second type of text data; according to the first attribution score, the second attribution score, the text data sample and the target modal data sample, a data processing model is obtained through training, and the data processing model is used for matching the text data and the target modal data; the first attribution score and the second attribution score are utilized to guide the training process of the data processing model, and the accuracy of the output result of the data processing model is improved by effectively adjusting the importance of different types of text data.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing model training method and a data processing method. Background Technology

[0002] Research on large-scale visual-language multimodal models has attracted much attention in the fields of natural language processing and computer vision in recent years. These data processing models possess powerful representational capabilities and zero-shot generalization ability, and can be applied to multiple fields such as image description generation, visual question answering, and video understanding. However, although large-scale visual-language multimodal models are widely used in many downstream applications, their ability to understand combinatorial relationships still faces significant challenges.

[0003] For example, existing multimodal large models such as CLIP (Contrastive Language-Image Pre-training, a multimodal pre-trained model that combines visual and linguistic information and can match images and text without labeled data) are insufficient in their ability to perceive visual and linguistic combination relationships. That is, the CLIP model can identify objects or nouns in images and text very well, but it has a poor understanding of the relationships between objects in images or text, the binding relationships between attributes and objects, and is not sensitive to the order of words in text. For example, the CLIP model cannot even distinguish between "the horse is eating grass" and "the grass is eating the horse". Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data processing model training method and a data processing method. One or more embodiments of this specification also relate to a data processing model training apparatus, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies in the prior art where the data processing model's insufficient understanding of the combinatorial relationships in the data leads to low accuracy of the output results.

[0005] According to a first aspect of the embodiments of this specification, a data processing model training method is provided, comprising:

[0006] Determine data sample pairs, wherein the data sample pairs include text data samples and corresponding target modality data samples;

[0007] From the text data sample, a first type of text data and a second type of text data are determined, and a first attribution score for the first type of text data and a second attribution score for the second type of text data are calculated.

[0008] A data processing model is trained based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample, wherein the data processing model is used to match the text data and the target modality data.

[0009] According to a second aspect of the embodiments of this specification, a data processing model training apparatus is provided, comprising:

[0010] The sample determination module is configured to determine data sample pairs, wherein the data sample pairs include text data samples and corresponding target modality data samples;

[0011] The score calculation module is configured to determine a first type of text data and a second type of text data from the text data sample, and calculate a first attribution score for the first type of text data and a second attribution score for the second type of text data.

[0012] The model training module is configured to train a data processing model based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample, wherein the data processing model is used to match the text data and the target modality data.

[0013] According to a third aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0014] The target text data is determined and input into the data processing model, wherein the data processing model is trained using the data processing model training method described above.

[0015] Using the data processing model, target modal data corresponding to the target text data is obtained.

[0016] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0017] The first determining module is configured to determine target text data and input the target text data into a data processing model, wherein the data processing model is obtained by training using the data processing model training method described above.

[0018] The first obtaining module is configured to use the data processing model to obtain target modal data corresponding to the target text data.

[0019] According to a fifth aspect of the embodiments of this specification, a cloud training platform is provided, including a request interface and a response unit;

[0020] The request interface is used to receive a task generation request sent by a terminal device, wherein the task generation request includes request information;

[0021] The response unit is used to determine the data sample pair and the initial data processing model according to the request information, and to train the data processing model using the above-mentioned data processing model training method.

[0022] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0023] Memory and processor;

[0024] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned data processing model training method and data processing method.

[0025] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0026] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0027] This specification provides a data processing model training method in one embodiment. By identifying a first type of text data and a second type of text data from text data samples in a data sample pair, the method determines the type of each text data in the text data sample. It calculates a first attribution score for the first type of text data and a second attribution score for the second type of text data to determine the importance of different types of text data to the output of the data processing model. Based on training the data processing model using the first and second attribution scores, the method dynamically adjusts the attribution scores of different types of text data to achieve fine-tuning of the importance of different types of text data in the data processing model. Furthermore, the text data samples and target modality data samples in the data sample pair can be combined with the first and second attribution scores to jointly guide the training process of the data processing model. This ensures that the data processing model can correctly match text data and target modality data, and by effectively adjusting the importance of different types of text data, it addresses the problem of insufficient understanding of combinatorial relationships in the data processing model. This enables the data processing model to obtain more comprehensive and rich semantic information from the text data, improving the accuracy of the data processing model's output. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification.

[0029] Figure 2 This is a flowchart illustrating a data processing model training method provided in one embodiment of this specification;

[0030] Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0031] Figure 4 This is a flowchart illustrating the processing procedure of a data processing model training method provided in one embodiment of this specification.

[0032] Figure 5 This is a schematic diagram of the structure of a cloud training platform provided in one embodiment of this specification;

[0033] Figure 6 This is a schematic diagram of the structure of a data processing model training device provided in one embodiment of this specification;

[0034] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0035] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0036] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0037] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0039] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0040] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0041] Deep learning is a subfield of machine learning that is based on the concept of artificial neural networks and aims to simulate the way the human brain processes data and makes decisions. Deep learning uses multi-layered neural network models to learn and understand complex patterns and relationships through layers of abstract representations. The core concept of deep learning is the neural network, which consists of many nodes (called neurons) interconnected by connections (called weights). Each node receives input from the nodes in the previous layer and passes its output to the nodes in the next layer. Through multi-layered node connections, neural networks can learn higher-level features and abstract representations. Deep learning has achieved significant success in various fields, including image recognition, speech recognition, natural language processing, and recommender systems; it can automatically learn from large amounts of data and has the ability to handle unstructured and highly complex data.

[0042] Visual language multimodal large models aim to associate and understand visual information with natural language. These models are trained by learning the relationships between a large number of images and their corresponding language descriptions. Given a new image, they can generate accurate and coherent natural language descriptions (using generative large models) or match images and text (using discriminative or comparisonal large models). The applications of visual language multimodal large models are very wide. For example, in image description tasks, they can receive an image and automatically generate a text description of it; in visual question answering tasks, they can answer questions related to images; in image retrieval tasks, they can retrieve the corresponding image based on a given natural language description. Furthermore, visual language models can also be used in image search, automatic image annotation, and scene understanding.

[0043] Attribution score: This refers to the factors an analytical model relies on when making decisions. It can be understood as an interpretability method; for example, when classifying an image, it determines which regions of the image the model depends on to arrive at its prediction. In the embodiments of this specification, the attribution score refers to the text-side attribution score. That is, when the model performs image-text matching (the principle of image-text matching is that the model calculates a similarity score between an image and text, also called a matching score, and performs image-text matching based on this similarity score; similar to how image classification models achieve image classification by obtaining the predicted probability of a certain category), this attribution score can be understood as the importance of each word in the input text to the final output result (i.e., the model's decision). Typically, attribution methods can be used to obtain the importance score (i.e., the attribution score) of each word in the input text to the model's output result when the model calculates the image-text matching score (i.e., the model's decision or output result).

[0044] Combinatorial understanding in visual language multimodal models refers to the understanding of the conceptual relationships between different elements in images and text. Combinatorial concepts generally describe the relationships between objects (e.g., "riding" in "a person riding a horse") or the attributes of objects (e.g., "red" in "a red shirt"). For example, given an image and two texts, one correctly describing the image (e.g., "a person carrying a red backpack"), and the other subtly different in that certain key terms representing the combined concepts differ (e.g., incorrect text such as "a person carrying a red backpack" or "a person carrying a black backpack"), the visual language multimodal model is required to distinguish between the correct and incorrect texts based on the image content. Alternatively, given a text and two images, one of which matches the text description (i.e., the correct image contains "grass inside a cup"), and the other image contains the objects described in the text, but the arrangement does not match the text description (as an incorrect image, the incorrect image contains "cup inside grass"). The model is required to distinguish between the correct and incorrect images based on the text description.

[0045] This specification provides a data processing model training method, a data processing method, a data processing model training device, a data processing device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0046] See Figure 1 , Figure 1 A schematic diagram of a data processing method according to an embodiment of this specification is shown.

[0047] Specifically, this data processing method is implemented using a terminal device 102 and a server 104. The terminal device 102 is used to send target text data to the server 104, such as "an image of a person riding a horse". In practical applications, users can input target text data into the terminal device 102 via text or voice. If voice is used, the terminal device 102 will also include a corresponding voice processing component, such as a voice parsing, voice-to-text, and voice synthesis module, to convert the user's voice into text. This specification does not limit this.

[0048] A data processing model is trained in server 104. The data processing model is trained in the following manner: Determine data sample pairs, where each data sample pair includes a text data sample and a corresponding target modality data sample; From the text data samples, determine first-type text data and second-type text data. The first-type text data can be understood as the text data in the text data sample that represents a combined concept, and the second-type text data can be understood as the text data in the text data sample that represents an object concept. For example, when the text data sample is "a person riding a horse", the verb "riding" is the second-type text data, and the nouns "person" and "horse" are the first-type text data; Calculate the first attribution score for the first-type text data and the second attribution score for the second-type text data; Train a data processing model based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample. The data processing model is used to match text data and target modality data; When the target modality data sample is image modality data, by inputting the target text data into the data processing model, the target image data corresponding to the target text data is obtained, and the target image data is returned to the end-side device 102.

[0049] The end-side device 102 can include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5) application, or a light application (also known as a mini-program, a lightweight application program), or a cloud application, etc. The end-side device can be developed based on the software development kit (SDK) of the corresponding service provided by the server, such as developed based on the real-time communication (RTC) SDK. The end-side device can be deployed in an electronic device and needs to rely on the device or certain APPs in the device to run, etc. The electronic device can have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can usually be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0050] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0051] It is worth noting that the data processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the data processing model can be deployed in the end-side device 102, so that the end-side device 102 can also have similar functions to the server 104, thereby executing the data processing method provided in the embodiments of this specification. In other embodiments, the data processing method provided in the embodiments of this specification can also be jointly executed by the end-side device 102 and the server 104.

[0052] This specification provides a data processing model training method in one embodiment. By identifying a first type of text data and a second type of text data from text data samples in a data sample pair, the method determines the type of each text data in the text data sample. It calculates a first attribution score for the first type of text data and a second attribution score for the second type of text data to determine the importance of different types of text data to the output of the data processing model. Based on training the data processing model using the first and second attribution scores, the method dynamically adjusts the attribution scores of different types of text data to achieve fine-tuning of the importance of different types of text data in the data processing model. Furthermore, the text data samples and target modality data samples in the data sample pair can be combined with the first and second attribution scores to jointly guide the training process of the data processing model. This ensures that the data processing model can correctly match text data and target modality data, and by effectively adjusting the importance of different types of text data, it addresses the problem of insufficient understanding of combinatorial relationships in the data processing model. This enables the data processing model to obtain more comprehensive and rich semantic information from the text data, improving the accuracy of the data processing model's output.

[0053] See Figure 2 , Figure 2 A flowchart of a data processing model training method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0054] Step 202: Determine data sample pairs, wherein the data sample pairs include text data samples and corresponding target modality data samples.

[0055] The target modality data samples may include text modality data samples, visual modality data samples, and audio modality data samples, without limitation. Furthermore, data sample pairs may be text-text pairs, text-image pairs, or text-video pairs. When the data sample pair is a text-text pair, it may include the initial language text and the translated target language text, or it may include the summary text and the corresponding long text, or it may include the question text and the corresponding answer text, without limitation.

[0056] In practical applications, when the data sample pairs are sample pairs of different modal data, this data processing model can be understood as a multimodal data processing model.

[0057] Specifically, the application scenarios of multimodal data processing models trained using data sample pairs differ depending on the data modality of the data sample pairs. For example, when the data sample pairs are text-to-text pairs, the data processing model can be applied to text question answering scenarios; when the data sample pairs are text-to-image pairs, the multimodal data processing model can be applied to image search scenarios, visual question answering scenarios, etc.

[0058] Step 204: From the text data sample, determine the first type of text data and the second type of text data, and calculate the first attribution score of the first type of text data and the second attribution score of the second type of text data.

[0059] The first type of text data can be understood as text data in the text data sample that represents a combined concept, usually verbs or adjectives; the second type of text data can be understood as text data in the text data sample that represents an object concept, usually nouns. For example, in the case of the text data sample "a person riding a horse", "person" and "horse" are text data in the text data sample that represent object concepts, and "riding" is text data in the text data sample that represents a combined concept.

[0060] The first attribution score is used to represent the importance of the first type of text data to the output of the data processing model. That is, the first attribution score can be understood as the composite concept attribution score, which is used to represent the importance of the composite concept text data to the output of the data processing model. The second attribution score is used to represent the importance of the second type of text data to the output of the data processing model. That is, the second attribution score can be understood as the object concept attribution score, which is used to represent the importance of the object concept text data to the output of the data processing model.

[0061] Specifically, by analyzing the text data samples, the text data representing combined concepts and the text data representing object concepts in the text data samples are identified, and the combined concept attribution score of the text data representing combined concepts and the object concept attribution score of the text data representing object concepts are calculated respectively.

[0062] In one or more embodiments of this specification, a text scene graph corresponding to a text data sample is obtained through a text scene graph model, thereby determining a first type of text data and a second type of text data based on the text scene graph. Specific implementation methods are as follows:

[0063] The step of determining the first type of text data and the second type of text data from the text data sample includes:

[0064] The text data sample is input into a text scene graph model to obtain a text scene graph corresponding to the text data sample. The text scene graph model is a machine learning model. The text scene graph includes nodes and edges. The nodes are used to determine the first type of text data in the text data sample, and the edges are used to determine the second type of text data in the text data sample.

[0065] Based on the text scene diagram, the first type of text data and the second type of text data are determined.

[0066] The text scene graph model can be understood as a machine learning model used to extract text scene graphs based on text data samples. For example, the text scene graph model is implemented using SceneGraphParser (a scene graph parser used to parse natural language into scene graphs based on dependency parsing). This text scene graph can be understood as a structured representation of entities in text (i.e., text data representing object concepts in the above embodiments) and the relationships between them (i.e., text data representing combined concepts in the above embodiments). The text scene graph includes nodes and edges, where nodes represent entities, i.e., text data representing object concepts, and edges represent relationships between entities, i.e., text data representing combined concepts.

[0067] For example, if the text data sample is "a person riding a bicycle", by inputting the text data sample into the text scene graph model, the corresponding text scene graph is obtained. The nodes in the text scene graph are "person" and "bicycle", and the edges in the text scene graph are "riding". Therefore, the second type of text data in the text data sample can be determined as "person" and "bicycle", and the first type of text data is "riding". That is, the text scene model can determine the classification result of each text data in the text data sample. The classification result includes the first type of text data and the second type of text data.

[0068] In practical applications, image modality data samples of visual modalities can also be input into the image scene graph model. Using the image scene graph model, the image scene graph can be extracted from the image modality data samples to determine the image regions representing combined concepts and the image regions representing object concepts. For example, the image scene graph model can identify each object in the image modality data sample and analyze the spatial relationships and interactive behaviors between objects to determine which image regions represent combined concepts (for example, in an image of a person riding a bicycle, the overlapping part of the person and the bicycle is the image region representing the combined concept), while the image regions corresponding to "person" and "bicycle" can be determined as the image regions representing object concepts.

[0069] The data processing model training method provided in the embodiments of this specification can efficiently and accurately determine the first type of text data and the second type of text data in the text data sample by inputting text data samples into a text scene graph model and using the text scene graph determined by the text scene graph model.

[0070] In one or more embodiments of this specification, the attribution scores of text data are calculated in various ways. Based on a preset attribution calculation method, a first attribution score for a first type of text data and a second attribution score for a second type of text data are calculated respectively. Specific implementation methods are described below:

[0071] The calculation of the first attribution score for the first type of text data and the second attribution score for the second type of text data includes:

[0072] According to a preset attribution calculation method, the first attribution score of the first type of text data and the second attribution score of the second type of text data are calculated.

[0073] The preset attribution calculation method includes at least one of the following: gradient-based attribution calculation method, attention score-based attribution calculation method, and perturbation-based attribution calculation method.

[0074] Specifically, the attribution score is calculated using a gradient-based attribution method. This involves calculating the similarity loss function between the text data sample and the target modality data sample by applying the data sample to the input data processing model. Through backpropagation, the gradient of the similarity loss function with respect to each layer of the data processing model is calculated. These gradients reflect the contribution of the input text data sample to the similarity loss function. The attribution score is determined based on this contribution. Since the text data sample is processed in the form of tokens (words obtained from word segmentation), the attribution score corresponding to each token can be determined. If the token represents the first type of text data, the first attribution score is determined; if the token represents the second type of text data, the second attribution score is determined.

[0075] In practical applications, attribution scores can be calculated using the Grad-CAM (Gradient-weighted Class Activation Mapping) attribution method. Grad-CAM is a heatmap technique for visualizing deep learning models (especially convolutional neural networks) to help understand the key aspects that the model focuses on when making decisions. Its principle is a gradient-based attribution method. Grad-CAM does not require any modifications to the model or additional training data.

[0076] The attribution score is calculated using an attention score-based attribution method. Specifically, based on the Transform model as the data processing model, the attention score can be determined using the attention mechanism of the Transform model. Therefore, the attribution score can be determined based on the attention score, for example, the attention score can be directly determined as the attribution score.

[0077] The attribution score is calculated using a perturbation-based attribution method. Specifically, by perturbing or occluding the text data samples, the changes in the model output before and after the perturbation or occlusion are compared. The importance of the changed part in the input data is determined based on the degree of output change, and this importance is the attribution score. In other words, the perturbation-based attribution method infers the importance of the changed part in the input data by observing the changes in the model output when the input data changes.

[0078] The data processing model training method provided in the embodiments of this specification can calculate attribution scores using various preset attribution calculation methods. This allows for convenient and flexible use of these preset attribution calculation methods, improving the efficiency of calculating attribution scores. Based on these multiple preset attribution calculation methods, the combinatorial understanding ability of the data processing model can be effectively improved.

[0079] In one or more embodiments of this specification, the first type of text data includes a plurality of first type lexical text data, and the second type of text data includes a plurality of second type lexical text data; a first attribution score of the first type of text data is determined by calculating a first initial attribution score for each first type lexical text data, and a second attribution score of the second type of text data is determined by calculating a second initial attribution score for each second type lexical text data. Specific implementation methods are as follows:

[0080] The calculation of the first attribution score for the first type of text data and the second attribution score for the second type of text data includes:

[0081] Calculate and obtain the first initial attribution score for each of the multiple first-type word element text data;

[0082] Based on the multiple initial attribution scores of the multiple first type of word text data, determine the first attribution score of the first type of text data;

[0083] Calculate and obtain the second initial attribution score for each of the multiple second-type word element text data;

[0084] The second attribution score of the second type of text data is determined based on the multiple second initial attribution scores of the multiple second type of word text data.

[0085] Specifically, word text data (i.e., word units) is the basic unit for data processing in the data processing model. Therefore, the initial attribution score of each word text data in the text data sample is calculated using the above-mentioned preset attribution calculation method. The classification results of each word text data in the text data sample are also obtained by using the text scene graph model. Therefore, based on the classification results of each word text data and the initial attribution score of each word text data, the first initial attribution score of each first type of word text data and the second initial attribution score of each second type of word text data are determined.

[0086] By obtaining the first initial attribution score of each type of word text data, and averaging the first initial attribution scores of multiple type-1 word text data, the first attribution score of the first type of text data is obtained. Similarly, by obtaining the second initial attribution score of each type-2 word text data, and averaging the second initial attribution scores of multiple type-2 word text data, the second attribution score of the second type of text data is obtained. In practical applications, multiple first initial attribution scores and multiple second initial attribution scores can also be weighted separately to obtain the first attribution score and the second attribution score, which is not limited here.

[0087] In practice, the attribution score of each word in the text data sample can be obtained, and the attribution scores of words representing combined concepts in the text data sample can be averaged to obtain the combined concept attribution score of the text data sample; similarly, the attribution scores of words representing object concepts in the text data sample can be averaged to obtain the object concept attribution score of the text data sample.

[0088] The data processing model training method provided in the embodiments of this specification accurately obtains the first attribution score of the first type of text data and the second attribution score of the second type of text data in the text data sample by calculating the initial attribution score of each word text data and classifying it according to the type of word text data.

[0089] Step 206: Based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample, train a data processing model, wherein the data processing model is used to match the text data and the target modality data.

[0090] In practical applications, training a data processing model based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample includes:

[0091] Using the first attribution score and the second attribution score, a constraint loss function is determined, and using the text data sample and the target modality data sample, a similarity loss function is determined;

[0092] The data processing model is trained based on the constraint loss function and the similarity loss function.

[0093] Among them, the constraint loss function is used to constrain the first attribution score of the text data sample to be greater than the second attribution score; the similarity loss function is used to narrow the distance between the text data sample and the corresponding target modality data sample.

[0094] In one or more embodiments of this specification, the number of data sample pairs is multiple, including multiple text data samples and corresponding multiple target modality data samples; when training the data processing model in batches, multiple data sample pairs will be input into the data processing model in one batch.

[0095] The step of determining the similarity loss function using the text data samples and the target modality data samples includes:

[0096] The plurality of text data samples are sequentially determined as the current text data sample, and the first loss function is determined using the current text data sample and the plurality of target modality data samples;

[0097] The plurality of target modality data samples are sequentially determined as the current target modality data sample, and the second loss function is determined using the plurality of text data samples and the current target modality data sample;

[0098] The similarity loss function is determined based on the first loss function and the second loss function.

[0099] Here, the current text data sample can be understood as the text data sample currently being processed, and the current target modality data sample can be understood as the target modality data sample currently being processed; the first loss function is used to shorten the distance between the current text data sample and the corresponding target modality data sample, and to widen the distance between the current text data sample and the non-matching target modality data sample; the second loss function is used to shorten the distance between the current target modality data sample and the corresponding text data sample, and to widen the distance between the current target modality data sample and the non-matching text data sample.

[0100] Specifically, for each text data sample among multiple text data samples, the similarity between each text data sample and multiple target modality data samples is calculated, and a first loss function is defined using the similarity. In this way, when the data processing model is applied, the corresponding target modality data can be determined for a given text data.

[0101] Similarly, for each target modality data sample among multiple target modality data samples, the similarity between each target modality data sample and multiple text data samples is calculated, and a second loss function is defined using the similarity. In this way, when the data processing model is applied, the corresponding text data can be determined for a given target modality data.

[0102] For the first loss function and the second loss function, the final similarity loss function can be obtained by averaging.

[0103] Specifically, determining the first loss function using the current text data sample and the multiple target modality data samples includes:

[0104] The target modality data sample corresponding to the current text data sample is determined as the matching target modality data sample, and the target modality data samples other than the matching target modality data sample among the multiple target modality data samples are determined as other target modality data samples;

[0105] Calculate the first target similarity between the current text data sample and the matching target modality data sample, and calculate the second target similarity between the current text data sample and at least one of the other target modality data samples respectively;

[0106] The first loss function is determined based on the first target similarity and the second target similarity.

[0107] In practical applications, the target modality data sample corresponding to the current text data sample is marked as the "matching target modality data sample," and the target modality data samples other than the matching target modality data sample among multiple target modality data samples are identified as other target modality data samples. These other target modality data samples do not have a matching relationship with the current text data sample. The first target similarity between the current text data sample and the matching target modality data sample is calculated. Specifically, the calculation method can be to calculate the cosine similarity between the text features of the current text data sample and the image features of the matching target modality data sample. Similarly, the second target similarity between the current text data sample and each other target modality data sample is calculated separately.

[0108] Based on the first target similarity and the second target similarity, a first loss function can be designed to guide the training of the model. The purpose of this first loss function is usually to maximize the similarity between the current text data sample and the matching target modality data sample (i.e., the first target similarity), while minimizing the similarity between the current text data sample and other target modality data samples (i.e., the second target similarity).

[0109] Specifically, determining the second loss function using the multiple text data samples and the current target modality data sample includes:

[0110] The text data sample corresponding to the current target modality data sample is determined as the matching text data sample, and the text data samples other than the matching text data sample among the multiple text data samples are determined as other text data samples;

[0111] Calculate the third target similarity between the current target modality data sample and the matching text data sample, and calculate the fourth target similarity between the current target modality data sample and at least one of the other text data samples respectively;

[0112] The second loss function is determined based on the third target similarity and the fourth target similarity.

[0113] The purpose of the second loss function is usually to maximize the similarity between the current target modality data sample and the matching text data sample (i.e., the third target similarity), while minimizing the similarity between the current target modality data sample and other target text data samples (i.e., the fourth target similarity).

[0114] The specific implementation method is similar to the above embodiments, and will not be repeated here.

[0115] The data processing model training method provided in the embodiments of this specification, through the calculation of a first loss function and a second loss function, enables the data processing model to determine the matching target modality data or text data regardless of whether the received input is text data or target modality data, thereby improving the applicability of the data processing model.

[0116] In one or more embodiments of this specification, a similarity loss function is used to match text data samples with target modality data samples, and a constraint loss function is used to increase the importance of the first type of text data in the text data samples during the matching process. Specific implementation methods are described below:

[0117] The step of training a data processing model based on the constraint loss function and the similarity loss function includes:

[0118] Based on the constraint loss function and the similarity loss function, the model parameters of the initial data processing model are updated to obtain the data processing model.

[0119] Specifically, by combining the constraint loss function and the similarity loss function, the target loss function is determined. The gradient of the target loss function with respect to the model parameters is calculated using the backpropagation algorithm. By updating the model parameters of the initial data processing model, the target loss function is minimized, and the updated data processing model is obtained.

[0120] The data processing model training method provided in this specification uses a similarity loss function to match text data samples with target modal data samples, and uses a constraint loss function to increase the importance of the first type of text data in the text data samples during the matching process, thereby improving the accuracy of the data processing model output results.

[0121] In one or more embodiments of this specification, the initial data processing model includes a text encoder; when the constraint loss function is determined by the attribution score on the text side, the network parameters of the text encoder can be updated based on the constraint loss function, thereby optimizing the text encoder. Specific implementation methods are as follows:

[0122] The data processing model training method further includes:

[0123] Based on the constraint loss function, update the network parameters of the text encoder in the initial data processing model to obtain the trained text encoder.

[0124] Specifically, the constraint loss function is determined by the first attribution score of words representing combined concepts in the text data samples and the second attribution score of words representing object concepts in the text data samples. Therefore, when the model parameters of the initial data processing model are adjusted based on the constraint loss function, the network parameters of the text encoder in the initial data processing model will be adjusted to achieve optimization for the text encoder.

[0125] The data processing model training method provided in the embodiments of this specification optimizes the text encoder based on a determined constraint loss function. On this basis, the data processing model training method in the embodiments of this specification is not limited to the image-text multimodal framework, and also improves the feature retrieval capability for pure text. It can be applied to the field of text retrieval and improves the generalizability of the data processing model.

[0126] In one or more embodiments of this specification, after the data processing model is obtained through training, the method further includes:

[0127] Identify the target text data and input the target text data into the data processing model;

[0128] Using the data processing model, target modal data corresponding to the target text data is obtained.

[0129] Alternatively, the target modal data can be determined and input into the data processing model;

[0130] Using the data processing model, target text data corresponding to the target modality data is obtained.

[0131] Specifically, when the target text data is input into the trained data processing model, the target modality data corresponding to the target text data is obtained. For example, when the target modality data is image modality data, the data processing model is a large visual-language multimodal model, and the large visual-language multimodal model is used to match the image modality data corresponding to the target text data.

[0132] Alternatively, when the target modality data is input into a trained data processing model, target text data corresponding to the target modality data can be obtained. Similarly, when the target modality data is image modality data, the data processing model is a large visual-language multimodal model, which is used to match the target text data corresponding to the image modality data.

[0133] The data processing model training method provided in the embodiments of this specification, when the data processing model is trained, can input data of multiple modalities to match target modal data with target text data. It can be applied to various fields such as image and text retrieval and text retrieval, improving the applicability of the data processing model. At the same time, when the data processing model is obtained based on constraint loss function and similarity loss function, it improves the combinatorial understanding ability of the data processing model, thereby improving the accuracy of the output results of the data processing model.

[0134] The data processing model training method provided in this specification enhances the combinatorial understanding ability of the data processing model by explicitly increasing the model's attribution score for combinatorial concepts and decreasing the model's attribution score for object concepts during training. This data processing model training method can be seamlessly applied to the visual language multimodal pre-training framework based on contrastive learning, effectively improving the combinatorial relationship understanding ability of the large visual language multimodal model and increasing the accuracy of the output results of the large visual language multimodal model.

[0135] See Figure 3 , Figure 3 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0136] Step 302: Determine the target text data and input the target text data into the data processing model, wherein the data processing model is trained using the data processing model training method described above.

[0137] Step 304: Using the data processing model, obtain the target modal data corresponding to the target text data.

[0138] The data processing method further includes:

[0139] Determine the target modal data and input the target modal data into the data processing model;

[0140] Using the data processing model, target text data corresponding to the target modality data is obtained.

[0141] The target modal data includes text modal data, visual modal data, and audio modal data; for specific implementation methods, please refer to the above embodiments, which will not be repeated here.

[0142] In one or more embodiments of this specification, before determining the target text data and inputting the target text data into the data processing model, the method further includes:

[0143] Receive target text data sent by the client, wherein the target text data is determined based on the interactive operation on the user interface of the client;

[0144] After obtaining the target modality data corresponding to the target text data using the data processing model, the process further includes:

[0145] The target modal data is returned to the client so that it can be displayed on the user interface of the client.

[0146] Specifically, users can send target text data to the server through interactive operations such as input and clicks on the client's user interface. When the server receives the target text data sent by the client, it uses data processing methods to obtain the target modal data corresponding to the target text data and returns the target modal data to the client so that it can be displayed to the user on the client's user interface.

[0147] In practical applications, users can send target modal data to the server through interactive operations such as input and clicks on the client's user interface. When the server receives the target modal data sent by the client, it uses data processing methods to obtain the target text data corresponding to the target modal data and returns the target text data to the client so that it can be displayed to the user on the client's user interface.

[0148] The data processing method provided in the embodiments of this specification can achieve convenient and efficient multimodal data processing through interaction with the client, and the interactive design allows users to interact with the system intuitively through the interface, which helps to improve the user experience.

[0149] The data processing method provided in the embodiments of this specification, when a data processing model is trained, can input data of multiple modalities to match target modal data with target text data. It can be applied to various fields such as image and text retrieval and text retrieval, improving the applicability of the data processing model. At the same time, when the data processing model is obtained based on constraint loss function and similarity loss function, it improves the combinatorial understanding ability of the data processing model, thereby improving the accuracy of the output results of the data processing model.

[0150] See Figure 4 , Figure 4 The flowchart of a data processing model training method provided in one embodiment of this specification is shown.

[0151] This specification describes the application of a data processing model to the image and text domain, providing a detailed explanation of the training method for the data processing model. In this case, the data processing model is a multimodal data processing model. Specifically, existing multimodal data processing models, such as the CLIP model (Image-Text Multimodal Large Model), lack the ability to combine and understand the relationships between objects and the binding relationships between attributes and objects in images / text. The combined understanding ability of the image-text multimodal large model is of great value for the accurate identification of open image-text search and image-text dialogue items. This lack of combined understanding ability will seriously affect the accuracy of the output results of the multimodal data processing model. For example, when searching for "a person carrying a red backpack", the multimodal data processing model will return an image of "wearing a red shirt".

[0152] In practical applications, by analyzing the multimodal data processing model using the preset attribution calculation method in the above embodiments, it can be determined that the reason for the insufficient combinatorial understanding ability of the multimodal data processing model is that the multimodal data processing model pays insufficient attention to combinatorial concepts in the text, that is, the multimodal data processing model has a problem of insufficient attribution of combinatorial concepts.

[0153] Analysis using a pre-defined attribution calculation method revealed that when performing image-text matching, the multimodal data processing model paid significantly less attention (attribution) to nouns representing objects in the text than to words representing combined concepts (such as relational and attribute words). This indicates that the multimodal data processing model ignores key combined concepts in the text when performing image-text matching, focusing too much on the object concept level. This phenomenon is known as insufficient attribution of combined concepts.

[0154] Existing solutions may involve explicitly mitigating the insufficient combinatorial relationship understanding ability of multimodal data processing models by introducing new network parameters or modules, or by carefully constructing difficult negative samples. These methods introduce additional data collection or training costs. By collecting data (e.g., constructing difficult negative samples) and adding parameters (e.g., adding two new reconstruction modules based on cross-attention to the original model), the model's capabilities are improved like a "black box." However, these methods do not accurately pinpoint the interpretability reasons for the model's combinatorial understanding ability, nor do they address the underlying problem of insufficient attribution of combinatorial concepts. Moreover, these methods require prior knowledge of the target relationship category before constructing difficult negative samples, making them susceptible to the influence of the long-tail distribution of relationship categories in the training data.

[0155] This specification uses image modal data samples as examples to illustrate the data processing model training method. This multimodal data processing model is used for image-text matching.

[0156] Each text data sample in the training dataset containing multiple data sample pairs is parsed using a text scene graph model (such as SceneGraphParser) to obtain words representing combined concepts (i.e., the first type of lexical text data in the above embodiment) and words representing object concepts (i.e., the second type of lexical text data in the above embodiment) in each text data sample.

[0157] In practical applications, based on attribution tracking, a preset attribution calculation method (such as gradient-based, Grad-CAM-based, attention score-based, or perturbation-based attribution calculation method) is used to calculate the attribution score of each word in the text data sample when the multimodal data processing model performs image-text matching. This means that the importance of each word in the text data sample to the final output of the model is obtained.

[0158] The attribution scores of all words representing combined concepts in the text data sample are then averaged to obtain the combined concept attribution score of the text data sample (i.e., the first attribution score in the above embodiment); similarly, the attribution scores of words representing object concepts in the text data sample can be averaged to obtain the object concept attribution score of the text data sample (i.e., the second attribution score in the above embodiment).

[0159] That is, the attribution score of the object concept is Where n represents the number of data sample pairs in the training batch (i.e., n data sample pairs). This represents the attribution score of the first data sample for the lexical units representing object concepts in the Chinese text data sample; the others are similar. A obj This represents the attribution score of the lexical units representing object concepts in multiple text data samples in the current training batch.

[0160] That is, the attribution score of the combined concept is Where n represents the number of data sample pairs in the training batch (i.e., n data sample pairs). This represents the attribution score of the lexical units representing the combined concepts of the first data sample relative to the Chinese text data sample; the others are similar. A comp This represents the attribution score of the lexical units representing the combined concepts of multiple text data samples in the current training batch.

[0161] A MarginalRanking loss function (i.e., the constraint loss function in the above embodiment) is used to constrain the attribution score of the combined concept of each text data sample to be greater than the attribution score of the object concept.

[0162] That is, the constraint function is The goal is to constrain the attribution score of a combined concept to be greater than the attribution score of an object concept by a threshold ∈ , where max represents a larger value. For example Figure 4 In this study, the constrained loss function was used to achieve composite concept attribution enhancement. Specifically, the attribution scores of words representing object concepts, such as "person" and "horse", were reduced from 0.23 and 0.25 to 0.16 and 0.19, respectively, while the attribution score of words representing composite concepts, such as "ride", was increased from 0.08 to 0.15.

[0163] Calculate the image-text comparison learning loss (i.e., the similarity loss function in the above embodiment) within a batch, add the constraint loss function and the image-text comparison learning loss in a certain proportion to obtain the total target loss function, and backpropagate the target loss function to update the gradient of the model.

[0164] Specifically, the method for calculating the image-text contrast learning loss is as follows: the text encoder of the multimodal data processing model encodes the text data samples in the data sample pair into text features, and the image encoder of the multimodal data processing model encodes the image data samples in the data sample pair into image features. Since the text features and image features are encoded features in the same vector space, the image-text contrast learning loss can be determined by calculating the similarity between the text features and the image features.

[0165] In practical applications, the image-to-text contrastive learning loss function (i.e., the second loss function) can be expressed as:

[0166]

[0167] The function is to bring the features of the current image and its corresponding text closer together, and to push the features of the current image away from those of other unpaired texts. S(I,T) represents the similarity between image I and text T, B represents the current batch, and T... i This represents the i-th text in the current batch, and exp() represents the exponential function.

[0168] The text-to-image contrastive learning loss function (i.e., the first loss function) can be expressed as:

[0169]

[0170] The function is to bring the features of the current text and its corresponding image closer together, and to push the features of the current text away from those of other unpaired images. S(I,T) represents the similarity between image I and text T, and B represents the current batch. j This represents the j-th image in the current batch, and exp() represents the exponential function.

[0171] The learning loss for image-text comparison is L. ITC =(L i2t +L t2iThe average of the image-to-text contrastive learning loss function and the text-to-image contrastive learning loss function is used to form the final image-text contrastive learning loss (i.e., the similarity loss function is determined by the first loss function and the second loss function in the above embodiment).

[0172] The target loss function is λ can be set according to the actual situation. The total target loss function is determined by the constraint loss function and the similarity loss function.

[0173] In practical applications, based on the above methods, it can also be combined with difficult negative samples to further enhance the combinatorial understanding ability of the multimodal data processing model. For example, the data sample pairs are output as the initial data processing model. Based on the prediction results of the initial data processing model, difficult negative samples are determined from the data sample pairs (for example, data sample pairs whose prediction results do not meet expectations are determined as difficult negative samples). Thus, the difficult negative samples are used to train the initial data processing model and obtain the data processing model.

[0174] The data processing model training method provided in this specification analyzes the multimodal data processing model using various attribution calculation methods. It reveals a problem of insufficient attribution of combinatorial concepts in the multimodal data processing model. Without carefully constructing difficult negative samples or introducing new network parameters or modules, the method explicitly improves the multimodal data processing model's attribution of combinatorial concepts and reduces its attribution of object concepts during training. This enhances the model's combinatorial understanding ability, saving computational costs and effectively improving its ability to understand combinatorial relationships. This multimodal data processing model is applicable to various retrieval scenarios and has significant value in improving the understanding of specific keyword features in plain text retrieval projects, thus improving the generalizability of the multimodal data processing model.

[0175] See Figure 5 , Figure 5 A schematic diagram of the structure of a cloud training platform provided in one embodiment of this specification is shown.

[0176] This specification also provides a cloud training platform, including a request interface 502 and a response unit 504.

[0177] The request interface 502 is used to receive a task generation request sent by a terminal device, wherein the task generation request includes request information.

[0178] The response unit 504 is used to determine the data sample pair and the initial data processing model according to the request information, and to train and obtain the data processing model using the above-mentioned data processing model training method.

[0179] Specifically, to save on training resource costs for terminal devices, model training tasks can be sent to a cloud training platform, where the model training task can be completed and a data processing model can be trained.

[0180] In practical applications, the cloud training platform can receive task generation requests sent by terminal devices. These task generation requests are requests to generate model training tasks to obtain data processing models. The request information in the task generation request includes data sample pairs and an initial data processing model. Based on the data sample pairs and the initial data processing model, the response unit can train and obtain the data processing model using the data processing model training method.

[0181] Furthermore, once the data processing model is obtained through training of the response unit, the interface information of the data processing model can be returned to the terminal device, so that the terminal device can call the data processing model through the interface information of the data processing model, without having to deploy the data processing model locally, thus saving the storage space of the terminal device.

[0182] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing model training device. Figure 6 A schematic diagram of a data processing model training apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0183] The sample determination module 602 is configured to determine data sample pairs, wherein the data sample pairs include text data samples and corresponding target modality data samples;

[0184] The score calculation module 604 is configured to determine a first type of text data and a second type of text data from the text data sample, and calculate a first attribution score for the first type of text data and a second attribution score for the second type of text data.

[0185] The model training module 606 is configured to train a data processing model based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample, wherein the data processing model is used to match the text data and the target modality data.

[0186] Optionally, the fraction calculation module 604 is further configured to:

[0187] The text data sample is input into a text scene graph model to obtain a text scene graph corresponding to the text data sample. The text scene graph model is a machine learning model. The text scene graph includes nodes and edges. The nodes are used to determine the first type of text data in the text data sample, and the edges are used to determine the second type of text data in the text data sample.

[0188] Based on the text scene diagram, the first type of text data and the second type of text data are determined.

[0189] Optionally, the fraction calculation module 604 is further configured to:

[0190] Calculate and obtain the first initial attribution score for each of the multiple first-type word element text data;

[0191] Based on the multiple first initial attribution scores of the multiple first type of word text data, determine the first attribution score of the first type of text data;

[0192] Calculate and obtain the second initial attribution score for each of the multiple second-type word element text data;

[0193] The second attribution score of the second type of text data is determined based on the multiple second initial attribution scores of the multiple second type of lexical text data.

[0194] Optionally, the fraction calculation module 604 is further configured to:

[0195] According to a preset attribution calculation method, a first attribution score for the first type of text data and a second attribution score for the second type of text data are calculated. The preset attribution calculation method includes at least one of the following: a gradient-based attribution calculation method, an attention score-based attribution calculation method, and a perturbation-based attribution calculation method.

[0196] Optionally, the model training module 606 is further configured to:

[0197] Using the first attribution score and the second attribution score, a constraint loss function is determined, and using the text data sample and the target modality data sample, a similarity loss function is determined;

[0198] The data processing model is trained based on the constraint loss function and the similarity loss function.

[0199] Optionally, the model training module 606 is further configured to:

[0200] The plurality of text data samples are sequentially determined as the current text data sample. Using the current text data sample and the plurality of target modality data samples, a first loss function is determined, wherein the first loss function is used to narrow the distance between the current text data sample and the corresponding target modality data sample.

[0201] The plurality of target modal data samples are sequentially determined as the current target modal data sample. Using the plurality of text data samples and the current target modal data sample, a second loss function is determined, wherein the first loss function is used to narrow the distance between the current target modal data sample and the corresponding text data sample.

[0202] The similarity loss function is determined based on the first loss function and the second loss function.

[0203] Optionally, the model training module 606 is further configured to:

[0204] The target modality data sample corresponding to the current text data sample is determined as the matching target modality data sample, and the target modality data samples other than the matching target modality data sample among the multiple target modality data samples are determined as other target modality data samples;

[0205] Calculate the first target similarity between the current text data sample and the matching target modality data sample, and calculate the second target similarity between the current text data sample and at least one of the other target modality data samples respectively;

[0206] The first loss function is determined based on the first target similarity and the second target similarity.

[0207] Optionally, the model training module 606 is further configured to:

[0208] The text data sample corresponding to the current target modality data sample is determined as the matching text data sample, and the text data samples other than the matching text data sample among the multiple text data samples are determined as other text data samples;

[0209] Calculate the third target similarity between the current target modality data sample and the matching text data sample, and calculate the fourth target similarity between the current target modality data sample and at least one of the other text data samples respectively;

[0210] The second loss function is determined based on the third target similarity and the fourth target similarity.

[0211] Optionally, the model training module 606 is further configured to:

[0212] Based on the constraint loss function and the similarity loss function, the model parameters of the initial data processing model are updated to obtain the data processing model.

[0213] The device further includes:

[0214] The parameter update module is configured to update the network parameters of the text encoder in the initial data processing model according to the constraint loss function, so as to obtain the trained text encoder.

[0215] The device further includes:

[0216] The first acquisition module is configured to determine target text data and input the target text data into the data processing model; and use the data processing model to obtain target modal data corresponding to the target text data.

[0217] The device further includes:

[0218] The second acquisition module is configured to determine target modal data and input the target modal data into the data processing model; and use the data processing model to obtain target text data corresponding to the target modal data.

[0219] The data processing model training apparatus provided in this specification enhances the combinatorial understanding ability of the data processing model by explicitly increasing the model's attribution score for combinatorial concepts and decreasing the model's attribution score for object concepts during training. This data processing model training method can be seamlessly applied to the visual language multimodal pre-training framework based on contrastive learning, effectively improving the combinatorial relationship understanding ability of the large visual language multimodal model and increasing the accuracy of the output results of the large visual language multimodal model.

[0220] The above is an illustrative scheme of a data processing model training device according to this embodiment. It should be noted that the technical solution of this data processing model training device and the technical solution of the data processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the data processing model training device, please refer to the description of the technical solution of the data processing model training method described above.

[0221] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 7 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0222] The first determining module 702 is configured to determine target text data and input the target text data into a data processing model, wherein the data processing model is obtained by training the data processing model training method described above.

[0223] The first acquisition module 704 is configured to use the data processing model to obtain target modality data corresponding to the target text data.

[0224] The device further includes:

[0225] The second determining module is configured to determine target modal data and input the target modal data into the data processing model.

[0226] The device further includes:

[0227] The second acquisition module is configured to use the data processing model to obtain target text data corresponding to the target modality data.

[0228] The device further includes:

[0229] The receiving module is configured to receive target text data sent by the client, wherein the target text data is determined based on the interactive operations on the user interface of the client.

[0230] The return module is configured to return the target modal data to the client for display on the user interface of the client.

[0231] The data processing apparatus provided in the embodiments of this specification, when a data processing model is trained, can input data of multiple modalities to match target modal data with target text data. It can be applied to various fields such as image and text retrieval and text retrieval, improving the applicability of the data processing model. At the same time, when the data processing model is obtained based on constraint loss function and similarity loss function, it improves the combinatorial understanding ability of the data processing model, thereby improving the accuracy of the output results of the data processing model.

[0232] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0233] Figure 8A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0234] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0235] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0236] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0237] The processor 820 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned data processing model training method and data processing method.

[0238] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing model training method and data processing method embodiments, so the description is relatively simple. Relevant parts can be referred to the descriptions of the data processing model training method and data processing method embodiments.

[0239] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0240] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the data processing model training method and data processing method embodiments; relevant parts can be referred to the descriptions of the data processing model training method and data processing method embodiments.

[0241] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing model training method and data processing method.

[0242] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the data processing model training method and the data processing method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the data processing model training method and the data processing method described above.

[0243] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0244] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0245] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0246] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0247] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing model training method, comprising: determining a data sample pair, wherein the data sample pair comprises a text data sample and a corresponding target modality data sample; determining a first type of text data and a second type of text data from the text data sample, and calculating a first attribution score of the first type of text data and a second attribution score of the second type of text data; training a data processing model according to the first attribution score, the second attribution score, the text data sample, and the target modality data sample, wherein the data processing model is used to match text data and target modality data.

2. The data processing model training method of claim 1, wherein the determining a first type of text data and a second type of text data from the text data sample comprises: inputting the text data sample into a text scene graph model to obtain a text scene graph corresponding to the text data sample, wherein the text scene graph model is a machine learning model, and the text scene graph comprises nodes and edges, the nodes are used to determine the first type of text data in the text data sample, and the edges are used to determine the second type of text data in the text data sample; determining the first type of text data and the second type of text data according to the text scene graph.

3. The data processing model training method of claim 1, wherein the first type of text data comprises a plurality of first type of token text data, and the second type of text data comprises a plurality of second type of token text data; the calculating a first attribution score of the first type of text data and a second attribution score of the second type of text data comprises: calculating a first initial attribution score of each first type of token text data in the plurality of first type of token text data; determining a first attribution score of the first type of text data according to a plurality of the first initial attribution scores of the plurality of first type of token text data; calculating a second initial attribution score of each second type of token text data in the plurality of second type of token text data; determining a second attribution score of the second type of text data according to a plurality of the second initial attribution scores of the plurality of second type of token text data.

4. The data processing model training method of claim 1, wherein the calculating a first attribution score of the first type of text data and a second attribution score of the second type of text data comprises: calculating a first attribution score of the first type of text data and a second attribution score of the second type of text data according to a preset attribution calculation manner, wherein the preset attribution calculation manner comprises at least one of a gradient-based attribution calculation manner, an attention score-based attribution calculation manner, and a perturbation change-based attribution calculation manner.

5. The data processing model training method of claim 1, wherein the training of the data processing model based on the first attribution score, the second attribution score, the text data sample, and the target modality data sample comprises: determining a constraint loss function based on the first attribution score and the second attribution score, and determining a similarity loss function based on the text data sample and the target modality data sample; and training the data processing model based on the constraint loss function and the similarity loss function.

6. The data processing model training method of claim 5, wherein the data sample pair is one of a plurality of data sample pairs, and each of the plurality of data sample pairs comprises a text data sample and a corresponding target modality data sample; and the determining of the similarity loss function based on the text data sample and the target modality data sample comprises: determining each of the plurality of text data samples as a current text data sample in sequence, and determining a first loss function based on the current text data sample and the plurality of target modality data samples, wherein the first loss function is used to reduce a distance between the current text data sample and a corresponding target modality data sample; determining each of the plurality of target modality data samples as a current target modality data sample in sequence, and determining a second loss function based on the plurality of text data samples and the current target modality data sample, wherein the second loss function is used to reduce a distance between the current target modality data sample and a corresponding text data sample; and determining the similarity loss function based on the first loss function and the second loss function.

7. The data processing model training method of claim 6, wherein the determining of the first loss function based on the current text data sample and the plurality of target modality data samples comprises: determining a target modality data sample corresponding to the current text data sample as a matching target modality data sample, and determining each of the plurality of target modality data samples other than the matching target modality data sample as an other target modality data sample; calculating a first target similarity between the current text data sample and the matching target modality data sample, and calculating a second target similarity between the current text data sample and each of the other target modality data samples; and determining the first loss function based on the first target similarity and the second target similarity.

8. The data processing model training method of claim 6, wherein the determining of the second loss function based on the plurality of text data samples and the current target modality data sample comprises: determining a text data sample corresponding to the current target modality data sample as a matching text data sample, and determining each of the plurality of text data samples other than the matching text data sample as an other text data sample; and determining the second loss function based on the matching text data sample and the other text data sample. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a third target similarity between the current target modality data sample and the matching text data sample is calculated, and a fourth target similarity between the current target modality data sample and at least one of the other text data samples is calculated respectively; the second loss function is determined based on the third target similarity and the fourth target similarity.

9. The data processing model training method of claim 5, wherein the data processing model is trained based on the constraint loss function and the similarity loss function, comprising: updating model parameters of an initial data processing model based on the constraint loss function and the similarity loss function to obtain the data processing model.

10. The data processing model training method of claim 9, wherein the initial data processing model comprises a text encoder. The data processing model training method further comprises: updating network parameters of the text encoder in the initial data processing model based on the constraint loss function to obtain a trained text encoder.

11. The data processing model training method of any one of claims 1-10, wherein the target modality data sample comprises a text modality data sample, a visual modality data sample, or an audio modality data sample.

12. A data processing method, comprising: determining target text data and inputting the target text data into a data processing model, wherein the data processing model is trained by the data processing model training method of any one of claims 1-10; obtaining target modality data corresponding to the target text data by using the data processing model.

13. The data processing method of claim 12, further comprising: determining target modality data and inputting the target modality data into the data processing model; obtaining target text data corresponding to the target modality data by using the data processing model.

14. The data processing method of claim 12, wherein before the determining target text data and inputting the target text data into a data processing model, the method further comprises: receiving target text data sent by a client, wherein the target text data is determined according to an interaction operation on a user interaction interface of the client; and after the obtaining target modality data corresponding to the target text data by using the data processing model, the method further comprises: returning the target modality data to the client to display the target modality data on the user interaction interface of the client.

15. The data processing method of claim 12, wherein the target modality data comprises a text modality data, a visual modality data, or an audio modality data.

16. A cloud training platform, comprising a request interface and a response unit; The request interface is configured to receive a task generation request sent by the terminal device, wherein the task generation request comprises request information; the response unit is configured to determine a data sample pair and an initial data processing model based on the request information, and train a data processing model by using the data processing model training method of any one of claims 1-11.

17. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the data processing model training method according to any one of claims 1-11 or the data processing method according to any one of claims 12-15. 18.A computer readable storage medium storing computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the data processing model training method according to any one of claims 1-11 or the data processing method according to any one of claims 12-15. 19.A computer program product comprising computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the data processing model training method according to any one of claims 1-11 or the data processing method according to any one of claims 12-15.

Images