A model processing method, apparatus and device

Abstract

Embodiments of the present specification disclose a model processing method, device and equipment, the method comprising: acquiring a first image text pair matched with each other and a second image text pair with noise; respectively performing encoding processing on image data in the first image text pair and image data in the second image text pair, to obtain corresponding first image representation and second image representation; respectively inputting the first image representation and the second image representation into a target model, to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; acquiring a first prediction result from the prediction result corresponding to the first image representation, and matching first text data corresponding to the first prediction result with each other to obtain a third image text pair; and performing model training on the target model based on a similarity between different data in the third image text pair and a similarity between the prediction result corresponding to the second image representation and the first prediction result.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method, apparatus and device for processing a model. Background Technology

[0002] The core idea of ​​most cross-modal matching methods is to transform different modalities into a common space where the goal is to increase the similarity between matching image-text data and decrease the similarity between non-matching image-text data. However, the actual success of these methods depends on a key assumption: the training data needs to remain consistent across modalities. However, obtaining accurately matched data pairs is difficult. Manual annotation is not only a massive task but also prone to subjective errors and can easily lead to the leakage of private data. An alternative approach is to mine co-occurring image-text pairs from the vast space of the internet; however, this approach introduces noise from mismatched data pairs. Therefore, a technical solution is needed that can better mitigate the negative impact of mismatched image-text pairs on model training. Summary of the Invention

[0003] The purpose of the embodiments in this specification is to provide a technical solution that can better mitigate the negative impact of mismatched image text on model training.

[0004] To achieve the above technical solution, the embodiments in this specification are implemented as follows:

[0005] This specification provides a model processing method, comprising: acquiring a first image-text pair that matches each other and a second image-text pair with noise; encoding the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; inputting the first image representation and the second image representation into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtaining a first prediction result from the prediction result corresponding to the first image representation, whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold; using the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair; and training the target model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain a trained target model.

[0006] This specification provides a model processing method, comprising: acquiring a first image-text pair that matches each other and a second image-text pair with noise; inputting the image data from the first image-text pair and the image data from the second image-text pair into a graph-text encoding sub-model in a multimodal model to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair, wherein the graph-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation; inputting the first image representation and the second image representation into a prediction sub-model in the multimodal model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtaining a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold with the prediction result corresponding to the second image representation; and using the first text data from the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain a third image-text pair. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0007] This specification provides a model processing apparatus, comprising: a data pair acquisition module for acquiring a first image-text pair that matches each other and a second image-text pair with noise; a representation determination module for encoding the image data in the first image-text pair and the image data in the second image-text pair to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; a prediction module for inputting the first image representation and the second image representation into a target model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; a data processing module for acquiring a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and using the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair; and a training module for training the target model based on the similarity between the image data and text data in the third image-text pair and the similarity between the prediction result corresponding to the second image representation and the first prediction result to obtain a trained target model.

[0008] This specification provides a model processing apparatus, comprising: a data pair acquisition module for acquiring a first image-text pair that matches each other and a second image-text pair with noise; a first model processing module for inputting the image data from the first image-text pair and the image data from the second image-text pair into a graph-text encoding sub-model in a multimodal model to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair, wherein the graph-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation; a second model processing module for inputting the first image representation and the second image representation into a prediction sub-model in a multimodal model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; and a data processing module for acquiring a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and using the first text data from the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair. The training module trains the multimodal model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0009] This specification provides a model processing device comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire a first image-text pair that matches each other and a second image-text pair containing noise; encode the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; input the first image representation and the second image representation into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtain a first prediction result from the prediction result corresponding to the first image representation whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold; use the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair; and train the target model based on the similarity between the image data and text data in the third image-text pair and the similarity between the prediction result corresponding to the second image representation and the first prediction result to obtain a trained target model.

[0010] This specification provides a model processing device comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire a first image-text pair that matches each other and a second image-text pair with noise; input the image data from the first image-text pair and the image data from the second image-text pair into a graph-text encoding sub-model in a multimodal model to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair, wherein the graph-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation; input the first image representation and the second image representation into a prediction sub-model in the multimodal model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtain a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation; and use the first text data from the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0011] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring a first image-text pair that matches each other and a second image-text pair with noise; encoding the image data in the first image-text pair and the image data in the second image-text pair to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; inputting the first image representation and the second image representation into a target model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtaining a first prediction result from the prediction result corresponding to the first image representation, whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold; using the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair; and training the target model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain a trained target model.

[0012] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring a first image-text pair that matches each other and a second image-text pair with noise; inputting the image data from the first image-text pair and the image data from the second image-text pair into a graph-text encoding sub-model within a multimodal model to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; the graph-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation; inputting the first image representation and the second image representation into a prediction sub-model within the multimodal model to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtaining a first prediction result from the prediction result corresponding to the first image representation, whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold; and using the first text data from the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain a third image-text pair. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0013] This specification also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: acquiring a first image-text pair that matches each other and a second image-text pair with noise; encoding the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair; inputting the first image representation and the second image representation into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation; obtaining a first prediction result from the prediction result corresponding to the first image representation, whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold; using the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair; and training the target model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain a trained target model.

[0014] This specification also provides a computer program product, including a computer program that, when executed by a processor, performs the following process: acquiring a first image-text pair that matches each other and a second image-text pair with noise. Inputting the image data from the first image-text pair and the image data from the second image-text pair into a graph-text encoding sub-model in a multimodal model, respectively, to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. The graph-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation. Inputting the first image representation and the second image representation into a prediction sub-model in the multimodal model, respectively, to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation. Obtaining a first prediction result from the prediction result corresponding to the first image representation, whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Using the first text data from the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair, a third image-text pair is obtained. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an embodiment of a model processing method described in this specification;

[0017] Figure 2 This is a schematic diagram illustrating the processing procedure of one model in this specification;

[0018] Figure 3 This is an embodiment of another model processing method described in this specification;

[0019] Figure 4 This is a schematic diagram illustrating the processing procedure of another model in this specification;

[0020] Figure 5 This is yet another embodiment of the model processing method described in this specification;

[0021] Figure 6 This is an embodiment of a processing device for a model described in this specification;

[0022] Figure 7 This is another embodiment of the processing device for a different model in this specification;

[0023] Figure 8 This is an embodiment of a processing device for one model in this specification. Detailed Implementation

[0024] This specification provides a method, apparatus, and device for processing a model.

[0025] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0026] This specification provides an embodiment of a pseudo-title mechanism based on pseudo-classification for cross-modal matching. To maintain network security, many information service platforms or systems conduct full-network inspections. During these inspections, these platforms possess a wealth of image-text pairs, including screenshots of potentially risky websites and HTML source code. For full-network inspections, many downstream tasks rely on the model's ability to understand multimodal data, such as image and text data. Therefore, pre-training the model backbone using image and text data is essential. Furthermore, information service platforms or systems may have substantial data accumulation; utilizing this rich image and text data in pre-training can significantly improve data utilization. The core idea of ​​most cross-modal matching methods is to transform different modalities into a common space. The goal in this space is to increase the similarity between matched image-text data while reducing the similarity between unmatched image-text data. However, the actual success of these methods depends on a crucial assumption: the training data needs to remain consistent across modalities. However, obtaining accurately matched data pairs is difficult. Manual annotation is not only a massive task but also prone to subjective errors. An alternative is to mine co-occurring image-text pairs from the vast expanse of the internet; however, this process introduces noise from mismatched data pairs, leading to the problem of noise correspondence. Unlike commonly used noise labels, which refer to incorrect category labels, noise correspondence refers to mismatches between different modalities in paired data. Typically, the collected data may contain a mixture of accurately matched data pairs and noisy data pairs, which reduces the effectiveness of cross-modal matching techniques. Therefore, embodiments of this specification propose a pseudo-classification-based pseudo-title algorithm for cross-modal matching.

[0027] Currently, there are few solutions for handling noisy correspondences. Since the initial introduction of NCR, only a few studies have explored it further. The previously proposed NCL solution can be summarized as adjusting the correspondence labels, which can be reinterpreted as a soft margin in the triplet loss, thus mitigating the negative training impact caused by mismatched image-text pairs. This approach can be called a margin-based approach. While it has shown feasibility and effectiveness, it has limitations: adjusting the margin does not directly provide beneficial supervisory information for falsely reported image-text pairs, but only mitigates their erroneous supervision. Furthermore, despite its efforts to resist noisy data, it is still significantly negatively affected. The learning process of NCR exhibits an oscillating pattern, especially upon initial encounter with noisy data, leading to significant fluctuations in the loss associated with accurately matched data pairs. The NCR solution robustly trains the image-text pair matching model through adaptively corrected soft correspondence label information. Following NCR, its successors have improved upon NCL. For example, BiCro introduces an innovative way to correct noisy correspondence label information using bidirectional cross-modal similarity consistency, leveraging the inherent consistency in paired data (i.e., image-text pairs). On the other hand, DECL utilizes cross-modal evidence learning to estimate the uncertainty introduced by noise, thereby isolating noisy data pairs. A significant feature of this approach is its trade-off strategy for handling mismatched image-text pairs; its design primarily revolves around mitigating the negative impact of mismatched image-text pairs by isolating them or adjusting the smaller margins in the triplet ranking loss. Unlike the aforementioned approaches, this embodiment provides direct supervision signals for images within mismatched image-text pairs, enriching the learning process. Specific processing details can be found in the following embodiments.

[0028] like Figure 1 As shown in the embodiments of this specification, a method for processing a model is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial business or online shopping business, or a backend server for an application, etc. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps:

[0029] In step S102, a first image-text pair that matches each other and a second image-text pair with noise are obtained.

[0030] The first image-text pair can be a data pair consisting of matching image data and text data. The text data can include one or more of the following: descriptive information of the image data content, layout information of specified data within the image data, and specified attribute information of the image data. The specific details can be set according to actual conditions, and this embodiment does not limit this. The image data can be data of any image, such as one or more of the following: image data of a building, an animal, a plant, a person, or a specific scene. The specific details can be set according to actual conditions, and this embodiment does not limit this. The second image-text pair with noise can be a data pair where the correspondence between image data and text data is noisy. That is, the second image-text pair with noise can be a data pair where the image data and text data do not match. This mismatch can be indicated by the semantic similarity between the image data and the text data being less than a preset threshold, signifying a significant difference between the image data and the text data. For example, if text data is used to describe the content of image data, and the semantics of the descriptive information corresponding to the text data are inconsistent with the semantics of the image data, then the image data and text data do not match.

[0031] In implementation, when training a specific model (i.e., the target model), data pairs consisting of image data and text data can be obtained in various ways. For example, some business systems record image data and corresponding text information provided by users during business processing. A certain number of data pairs can be obtained from the image data-text data pairs recorded by the business system. The obtained data pairs may include a first image-text pair that matches each other and a second image-text pair with noise. Alternatively, data pairs consisting of image data and text data can be obtained directly from a specified database. The database may include a first image-text pair that matches each other and a second image-text pair with noise. Alternatively, data pairs consisting of image data and text data can be collected from different users. Alternatively, data pairs consisting of image data and text data can be obtained through two or three of the above methods. The obtained data pairs may include a first image-text pair that matches each other and a second image-text pair with noise, etc. In practical applications, in addition to obtaining data pairs consisting of image data and text data through the above methods, other different methods can also be used to obtain the above data pairs, which can be set according to the actual situation.

[0032] It should be noted that the first image-text pair may include one or more, and correspondingly, the second image-text pair may include one or more. Furthermore, in practical applications, the number of the first image-text pair and the number of the second image-text pair can satisfy a specified relationship, such as the ratio of the number of the first image-text pair to the number of the second image-text pair being 10:1 or 50:1, etc., which can be set according to the actual situation.

[0033] In step S104, the image data in the first image-text pair and the image data in the second image-text pair are encoded to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair.

[0034] The first image representation can be data capable of representing the corresponding image data. The first image representation can be represented in various ways; for example, it can be represented as a vector, a matrix of specified dimensions, or one or more numerical values, etc., depending on the specific circumstances. Similarly, the second image representation can also be data capable of representing the corresponding image data. The second image representation can also be represented in various ways; for example, it can be represented as a vector, a matrix of specified dimensions, or one or more numerical values, etc., depending on the specific circumstances.

[0035] In practice, in order to better process image data and text data, the image representation corresponding to the image data and the text representation corresponding to the text data can be determined separately. Specifically, an encoder can be pre-set according to the actual situation. The encoder can be constructed by a neural network or by a specified feature extraction algorithm, etc. Then, the encoder can be trained or fine-tuned using training samples to obtain the trained encoder. Alternatively, a pre-trained or pre-constructed encoder can be used directly. The specific settings can be determined according to the actual situation, and the embodiments in this specification do not limit this.

[0036] like Figure 2As shown, image data contained in the first image-text pair can be extracted. The image data in the first image-text pair is input into the encoder. The encoder can perform feature extraction on the image data to obtain an image representation that can characterize the image data. The obtained image representation can be the first image representation. Similarly, image data contained in the second image-text pair can be extracted. The image data in the second image-text pair is input into the encoder. The encoder can perform feature extraction on the image data to obtain an image representation that can characterize the image data. The obtained image representation can be the second image representation.

[0037] In step S106, the first image representation and the second image representation are input into the target model respectively to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation.

[0038] The target model can be a model that predicts results from corresponding image data based on the input image representation. The target model can be constructed using machine learning network models such as neural networks, or using classification or regression algorithms. In practical applications, the target model can also be a pre-defined large language model, depending on the specific circumstances. The target model can analyze the input image representation to determine the semantic information of the image data corresponding to that representation, thereby obtaining text data corresponding to the determined semantic information, and using the obtained text data and other related data as the prediction result.

[0039] In implementation, handling mismatched data pairs is crucial in NCL. However, many approaches often neglect the protection of learning from mutually matched data pairs. As mentioned earlier, introducing mismatched data pairs (i.e., noisy image-text pairs) into model training can severely impact the learning process from mutually matched data pairs. To enhance the robustness of training on mutually matched data pairs, this embodiment proposes an auxiliary training task to strengthen learning from such data. A key insight provided is that in image-text pairs, the text data (or title) corresponding to the image data can be considered a classification label, where a predefined hyperparameter determines the number of categories. Therefore, training on image-text pairs can be viewed as a multi-class classification task. For example, titles (or text data) in the dataset can be divided into two main categories: descriptions of natural landscapes and descriptions of biological behaviors. The goal is to train the model to classify image data of natural landscapes and image data containing organisms into their respective categories. To achieve this goal, we set up a pseudo-classifier (i.e., the target model) and use the text data from mutually matched data pairs to generate pseudo-labels for training the pseudo-classifier. The role of pseudo-labels is to encourage the model to make confident predictions. From the perspective of supervised learning, pseudo-classification also helps the model capture the similarity relationships between samples. It enhances the model's learning of mutually matching data pairs and improves the model's resistance to noise interference.

[0040] Specifically, a target model can be constructed using machine learning network models, classification algorithms, or regression algorithms. Then, training samples can be used to train or fine-tune the target model to obtain the trained target model. Alternatively, a pre-trained or pre-built encoder can be used directly. The specific settings can be determined according to the actual situation, and the embodiments in this specification do not limit this.

[0041] like Figure 2 As shown, the first image representation can be input into the target model, and the target model can analyze the first image representation to determine the semantic information of the image corresponding to the first image features. Based on the determined semantic information, the prediction result (such as text data, classification category, or numerical value) corresponding to the first image representation can be determined. Similarly, the second image representation can be input into the target model, and the target model can analyze the second image representation to determine the semantic information of the image corresponding to the second image representation. Based on the determined semantic information, the prediction result (such as text data, classification category, or numerical value) corresponding to the second image representation can be determined.

[0042] In step S108, a first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained from the prediction result corresponding to the first image representation. The first text data in the first image-text pair corresponding to the first prediction result is used as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0043] The preset threshold can be set according to the actual situation. For example, the preset threshold can be 80% or 90%.

[0044] In implementation, a similarity algorithm can be pre-defined, such as cosine similarity algorithm or Euclidean distance algorithm. Figure 2 As shown, a similarity algorithm can be used to calculate the similarity between the prediction result corresponding to each first image representation and the prediction result corresponding to each second image representation. The calculated similarity value can be compared with a preset threshold. Based on the comparison result, a first prediction result with a similarity greater than the preset threshold to the prediction result corresponding to the second image representation can be obtained from the prediction results corresponding to the first image representation. The text data in the first image-text pair corresponding to the first prediction result can be obtained; this obtained text data can be considered the first text data. The first text data and the image data in the second image-text pair can be considered a matching data pair, which can be considered the third image-text pair. The text data in the second image-text pair corresponding to the aforementioned image data can be discarded or left unprocessed. Through the above method, matching text data (or pseudo-titles) can be set for the image data in the noisy second image-text pair.

[0045] In step S110, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0046] In implementation, a loss function or objective function can be pre-set according to the actual situation. For example, a corresponding loss function or objective function can be pre-set based on similarity. Alternatively, one or two types of data can be selected from image data and text data, and a corresponding loss function or objective function can be pre-set based on similarity and the selected data. The specific settings can be determined according to the actual situation, and this specification does not limit this. The similarity between the image data and the text data in the third image-text pair can be calculated, and the similarity between the prediction result corresponding to the second image representation and the first prediction result can be calculated. The calculated similarity can be substituted into the loss function or objective function (other relevant data can also be substituted into the loss function or objective function). The corresponding loss information can be obtained through calculation. The model parameters of the target model can be adjusted based on this loss information. If the preset convergence condition is not met, the corresponding loss information can be calculated in the above manner, and the model parameters of the target model can be adjusted until the preset convergence condition is met. This is used to train the target model, ultimately obtaining the trained target model.

[0047] This specification provides a model processing method that involves acquiring a first image-text pair that matches each other and a second image-text pair containing noise. The image data in the first image-text pair and the image data in the second image-text pair are encoded to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model to obtain prediction results corresponding to the first image representation and prediction results corresponding to the second image representation. A first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained from the prediction result corresponding to the first image representation. The first image-text pair corresponding to the first prediction result is then processed. The first text data in the first image-text pair is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model. In this way, the most accurate text data (or pseudo-titles) are assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction with effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the first image-text pair that matches each other can be more stable.

[0048] In practical applications, such as Figure 3As shown, before step S110, the following steps S112 and S114 can also be performed.

[0049] In step S112, the text data in the first image-text pair is encoded to obtain the first text representation corresponding to the text data in the first image-text pair.

[0050] The first text representation can be data that can represent the corresponding text data. The first text representation can be represented in a variety of different ways. For example, the first text representation can be represented in the form of a vector, or it can be represented by a matrix of a specified dimension, or it can be represented by one or more numerical values, etc. The specific representation can be set according to the actual situation.

[0051] In practice, in order to better process text data, the text representation corresponding to the text data can be determined. Specifically, as mentioned above, an encoder can be pre-set according to the actual situation. The encoder can be constructed by a neural network or by a specified feature extraction algorithm, etc. Then, the encoder can be trained or fine-tuned using training samples to obtain the trained encoder. Alternatively, a pre-trained or pre-constructed encoder can be used directly. The specific settings can be determined according to the actual situation, and the embodiments in this specification do not limit this.

[0052] like Figure 4 As shown, the text data contained in the first image text pair can be extracted. The text data in the first image text pair is input into the encoder. The encoder can perform feature extraction on the text data to obtain a text representation that can characterize the text data. The obtained text representation can be the first text representation.

[0053] In step S114, the first text representation is input into the target model to obtain the prediction result corresponding to the first text representation.

[0054] The target model can analyze the input text representation, determine the semantic information of the text data corresponding to the text representation, and then obtain the corresponding prediction results.

[0055] In implementation, as described above, such as Figure 4 As shown, the first text representation can be input into the constructed target model (the construction method (or training method of the target model) can be found in the aforementioned related content, and will not be repeated here). The target model analyzes the first text representation to determine the semantic information of the image corresponding to the first text representation. Based on the determined semantic information, the prediction result corresponding to the first text representation can be determined.

[0056] Based on the processing of steps S112 and S114 above, step S110 may further include: using the prediction result corresponding to the first text representation as the label information corresponding to the image data in the first image-text pair; determining the first loss information based on the label information and the prediction result corresponding to the first image representation; determining the second loss information based on the similarity between the image data and the text data in the third image-text pair and the similarity between the prediction result corresponding to the second image representation and the first prediction result; and training the target model based on the first loss information and the second loss information to obtain the trained target model.

[0057] In implementation, such as Figure 4 As shown, a loss function can be pre-set according to the actual situation. In this embodiment, multiple different loss functions can be set. A corresponding loss function can be set for the prediction result corresponding to the first image representation. This loss function can be set in various ways, such as the mean squared error loss function, the Huber loss function, etc. The specific setting can be based on the actual situation, and this embodiment does not limit this. The prediction result corresponding to the first text representation can be used as the label information corresponding to the image data in the first image-text pair. Then, the label information and the prediction result corresponding to the first image representation can be substituted into the loss function set above to obtain the first loss information.

[0058] According to the actual situation, another loss function can be preset. This loss function can be set by the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result. The similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result can be substituted into the above-set loss function to obtain the second loss information.

[0059] The total loss information can be obtained by adding the first and second loss information, or by weighting the first and second loss information, or by performing a specific calculation based on the first and second loss information, with the result serving as the total loss information. Other methods can also be used to calculate the total loss information. The model parameters of the target model can be adjusted based on the total loss information. If the preset convergence condition is not met, the corresponding total loss information can be calculated using the above methods, and the model parameters of the target model can be adjusted until the preset convergence condition is met. This process is used to train the target model, ultimately obtaining the trained target model.

[0060] In practical applications, the above-mentioned model training based on the first loss information and the second loss information to obtain the trained target model can be processed in various ways. In addition to the above-mentioned processing, it can also be processed in a variety of different ways, which can include the following: training the target model based on the first loss information, the second loss information, and the similarity between the prediction result corresponding to the second image representation and the first prediction result to obtain the trained target model.

[0061] In implementation, the total loss information can be determined based on the first and second loss information, combined with the similarity between the prediction result corresponding to the second image representation and the first prediction result. The model parameters of the target model can be adjusted based on the total loss information. If the preset convergence condition is not met, the corresponding total loss information can be calculated in the above manner, and the model parameters of the target model can be adjusted until the preset convergence condition is met. This is used to train the target model, ultimately obtaining the trained target model. Since it cannot be guaranteed that the found text data accurately reflects the semantic information of the image data in the second image-text pair, the interval can also be dynamically adjusted to ensure that the model benefits from a more accurate correspondence during training. Specifically, the margin in the Triplet Ranking loss function can be adaptively adjusted based on the similarity between the image data corresponding to the selected text data and the image data in the second image-text pair.

[0062] In practical applications, the first loss information is determined by the cross-entropy loss function based on the aforementioned label information and the prediction results corresponding to the first image representation. Specifically, the prediction results corresponding to the first text representation can be used as the label information corresponding to the image data in the first image-text pair. The standard cross-entropy loss function is used to supervise the training of the target model on the prediction results corresponding to the image data, thereby enhancing the training effect of matching image data and text data.

[0063] In addition, such as Figure 4 As shown, the second loss information is determined by the Triplet Ranking loss function based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result. Constructing the third image-text pair aims to provide more accurate supervision signals for model training.

[0064] The similarity between the prediction result corresponding to the second image representation and the first prediction result can be used as the result of SoftMargin.

[0065] In addition, the image data contained in the first image-text pair and the second image-text pair includes screenshots of a preset website, and the text data contained in the first image-text pair and the second image-text pair includes one or more of the following: layout information of the corresponding image data, description information of the image data content, image data content, and code information corresponding to the image data content.

[0066] In practical applications, the specific processing method of the above step S104 can be varied. The following is an optional processing method, which may include the following: inputting the image data in the first image-text pair and the image data in the second image-text pair into the image encoder to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image encoder is used to encode the image data or text data to obtain the corresponding representation.

[0067] Furthermore, the specific processing methods for encoding the text data in the first image-text pair to obtain the first text representation corresponding to the text data in the first image-text pair can be varied. The following provides an optional processing method, which may include the following: inputting the text data in the first image-text pair into the image encoder to obtain the first text representation corresponding to the text data in the first image-text pair.

[0068] The image encoder can be an encoder built based on the CLIP model, or an encoder built based on a preset big oracle model, etc. The specific settings can be made according to the actual situation, and the embodiments in this specification do not limit this.

[0069] This specification provides a model processing method that involves acquiring a first image-text pair that matches each other and a second image-text pair containing noise. The image data in the first image-text pair and the image data in the second image-text pair are encoded to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model to obtain prediction results corresponding to the first image representation and prediction results corresponding to the second image representation. A first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained from the prediction result corresponding to the first image representation. The first image-text pair corresponding to the first prediction result is then processed. The first text data in the first image-text pair is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model. In this way, the most accurate text data (or pseudo-titles) are assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction with effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the first image-text pair that matches each other can be more stable.

[0070] like Figure 5 As shown in the embodiments of this specification, a method for processing a model is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial business or online shopping business, or a backend server for an application, etc. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps:

[0071] In step S502, a first image-text pair that matches each other and a second image-text pair with noise are obtained.

[0072] The specific processing method of step S502 above can be found in the aforementioned related content, and will not be repeated here.

[0073] It should be noted that the above embodiment has only one target model for predicting image and text data, while this embodiment is a multimodal model. This multimodal model not only includes a prediction sub-model that has the same function or role as the target model for predicting image and text data, but may also include an image and text encoding sub-model for encoding image data and text data. That is, the multimodal model includes an image and text encoding sub-model and a prediction sub-model, as detailed below.

[0074] In step S504, the image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text coding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text coding sub-model is used to encode the image data or text data to obtain the corresponding representation.

[0075] In implementation, the aforementioned image-text encoding sub-model has the same function as the encoder or image-text encoder described above. The construction and training methods of the image-text encoding sub-model can refer to the construction and training methods of the aforementioned encoder or image-text encoder. That is, the image-text encoding sub-model can be pre-defined according to actual conditions. This sub-model can be constructed through a neural network or a specified feature extraction algorithm, etc. Then, training samples can be used to train or fine-tune the image-text encoding sub-model to obtain the trained model. Alternatively, a pre-trained or pre-constructed image-text encoding sub-model can be used directly. The specific settings can be determined according to actual conditions, and this specification does not limit this aspect. The image-text encoding sub-model can be used to encode the image data in the first image-text pair and the image data in the second image-text pair to obtain the corresponding image representation.

[0076] In step S506, the first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation.

[0077] In implementation, the aforementioned predictive sub-model has the same function or role as the target model described above. The construction and training methods of the predictive sub-model can refer to those of the target model. That is, the predictive sub-model can be a model that predicts results from corresponding image data based on input image representations. The predictive sub-model can be constructed using machine learning network models such as neural networks, or using classification or regression algorithms. In practical applications, the predictive sub-model can also be a pre-defined large language model, etc., depending on the specific circumstances. The predictive sub-model can analyze the input image representation, determine the semantic information of the image data corresponding to that image representation, and then obtain text data corresponding to the determined semantic information. The obtained text data and other related data can be used as the prediction result. The aforementioned predictive sub-model can be used to predict the results of the first and second image representations to obtain the corresponding prediction results.

[0078] In step S508, a first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained from the prediction result corresponding to the first image representation. The first text data in the first image-text pair corresponding to the first prediction result is used as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0079] The specific processing method of step S508 above can be found in the aforementioned related content, and will not be repeated here.

[0080] In step S510, the multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0081] The specific processing method of step S510 can be found in the aforementioned related content, and will not be repeated here.

[0082] In practical applications, the prediction sub-model can also be trained using two types of loss information, as detailed in steps A2 and A4 below.

[0083] In step A2, the text data in the first image-text pair is input into the image-text coding sub-model for encoding processing to obtain the first text representation corresponding to the text data in the first image-text pair.

[0084] In step A4, the first text representation is input into the prediction sub-model to obtain the prediction result corresponding to the first text representation.

[0085] The specific processing methods for steps A2 and A4 above can be found in the aforementioned related content, and will not be repeated here.

[0086] Based on the processing of steps A2 and A4 above, the specific processing of step S510 above may include: taking the prediction result corresponding to the first text representation as the label information corresponding to the image data in the first image-text pair, determining the first loss information based on the label information and the prediction result corresponding to the first image representation, determining the second loss information based on the similarity between the image data and the text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, and training the multimodal model based on the first loss information and the second loss information to obtain the trained multimodal model.

[0087] The specific processing methods described above can be found in the aforementioned related content, and will not be repeated here.

[0088] In practical applications, the specific processing methods for training the target model based on the first loss information and the second loss information to obtain the trained target model can be varied. The following provides another optional processing method, which can include: training the multimodal model based on the first loss information, the second loss information, and the similarity between the prediction result corresponding to the second image representation and the first prediction result to obtain the trained multimodal model.

[0089] The specific processing methods described above can be found in the aforementioned related content, and will not be repeated here.

[0090] The first loss information is determined by the cross-entropy loss function based on the prediction results corresponding to the aforementioned label information and the first image representation.

[0091] The second loss information is determined by the Triplet Ranking loss function based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result.

[0092] In practical applications, the image data contained in the first image-text pair and the second image-text pair includes screenshots of a preset website, and the text data contained in the first image-text pair and the second image-text pair includes one or more of the following: layout information of the corresponding image data, description information of the image data content, image data content, and code information corresponding to the image data content.

[0093] This specification provides a model processing method. By acquiring a first image-text pair that matches each other and a second image-text pair containing noise, the image data from the first and second image-text pairs are input into a graph-text encoding sub-model within a multimodal model. This yields a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. The graph-text encoding sub-model encodes the image data or text data to obtain the corresponding representation. Then, the first and second image representations are input into a prediction sub-model within the multimodal model to obtain prediction results corresponding to the first and second image representations. From the prediction results corresponding to the first image representation, a prediction result with a similarity greater than a preset threshold is obtained between the prediction results corresponding to the first image representation and the prediction results corresponding to the second image representation. The first prediction result is used as the first text data in the first image-text pair corresponding to the first prediction result, and then used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the multimodal model can be trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model. In this way, the most accurate text data (or pseudo-titles) can be assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction through effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the matching first image-text pair can be more stable.

[0094] The above describes the model processing method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a model processing device, such as... Figure 6 As shown.

[0095] The processing apparatus of the model includes: a data pair acquisition module 601, a representation determination module 602, a prediction module 603, a data processing module 604, and a training module 605, wherein:

[0096] Data acquisition module 601 acquires a first image-text pair that matches each other and a second image-text pair with noise;

[0097] The characterization determination module 602 encodes the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain the first image characterization corresponding to the image data in the first image-text pair and the second image characterization corresponding to the image data in the second image-text pair.

[0098] The prediction module 603 inputs the first image representation and the second image representation into the target model respectively to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation.

[0099] The data processing module 604 obtains a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and uses the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair.

[0100] The training module 605 trains the target model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0101] In the embodiments described in this specification, the device further includes:

[0102] The text encoding module encodes the text data in the first image text pair to obtain the first text representation corresponding to the text data in the first image text pair.

[0103] The text prediction module inputs the first text representation into the target model to obtain the prediction result corresponding to the first text representation;

[0104] The training module 605 uses the prediction result corresponding to the first text representation as the label information corresponding to the image data in the first image-text pair, determines the first loss information based on the label information and the prediction result corresponding to the first image representation, determines the second loss information based on the similarity between the image data and the text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, and trains the target model based on the first loss information and the second loss information to obtain the trained target model.

[0105] In this embodiment of the specification, the training module 605 trains the target model based on the first loss information and the second loss information, as well as the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0106] In this embodiment of the specification, the first loss information is the loss information determined by the cross-entropy loss function based on the prediction results corresponding to the label information and the first image representation.

[0107] In this embodiment of the specification, the second loss information is determined by the Triplet Ranking loss function based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result.

[0108] In the embodiments of this specification, the image data contained in the first image-text pair and the second image-text pair includes screenshots of a preset website, and the text data contained in the first image-text pair and the second image-text pair includes one or more of the following: layout information of the corresponding image data, description information of the content of the image data, content of the image data, and code information corresponding to the content of the image data.

[0109] In this embodiment of the specification, the representation determination module 605 inputs the image data from the first image-text pair and the image data from the second image-text pair into the image encoder to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. The image encoder is used to encode the image data or text data to obtain the corresponding representation; and / or,

[0110] The text encoding module inputs the text data in the first image-text pair into the image-text encoder to obtain the first text representation corresponding to the text data in the first image-text pair.

[0111] This specification provides a model processing apparatus that acquires a first image-text pair and a noisy second image-text pair, encodes the image data in the first image-text pair and the image data in the second image-text pair respectively, and obtains a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation. A first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained from the prediction result corresponding to the first image representation. The first image-text pair corresponding to the first prediction result is then processed. The first text data in the first image-text pair is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model. In this way, the most accurate text data (or pseudo-titles) are assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction with effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the first image-text pair that matches each other can be more stable.

[0112] Based on the same idea, embodiments of this specification also provide a model processing device, such as... Figure 7 As shown.

[0113] The processing device for this model includes: a data pair acquisition module 701, a first model processing module 702, a second model processing module 703, a data processing module 704, and a training module 705, wherein:

[0114] The data pair acquisition module 701 acquires a first image-text pair that matches each other and a second image-text pair with noise.

[0115] The first model processing module 702 inputs the image data in the first image-text pair and the image data in the second image-text pair into the image-text encoding sub-model in the multimodal model, respectively, to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation.

[0116] The second model processing module 703 inputs the first image representation and the second image representation into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation.

[0117] The data processing module 704 obtains a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and uses the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair.

[0118] Training module 705 trains the multimodal model based on the similarity between image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0119] In the embodiments described in this specification, the device further includes:

[0120] The text encoding module encodes the text data in the first image text pair to obtain the first text representation corresponding to the text data in the first image text pair.

[0121] The text prediction module inputs the first text representation into the target model to obtain the prediction result corresponding to the first text representation;

[0122] The training module 705 uses the prediction result corresponding to the first text representation as the label information corresponding to the image data in the first image-text pair, determines the first loss information based on the label information and the prediction result corresponding to the first image representation, determines the second loss information based on the similarity between the image data and the text data in the third image-text pair and the similarity between the prediction result corresponding to the second image representation and the first prediction result, and trains the target model based on the first loss information and the second loss information to obtain the trained target model.

[0123] In this embodiment of the specification, the training module 705 trains the target model based on the first loss information and the second loss information, as well as the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0124] In this embodiment of the specification, the first loss information is the loss information determined by the cross-entropy loss function based on the prediction results corresponding to the label information and the first image representation.

[0125] In this embodiment of the specification, the second loss information is determined by the Triplet Ranking loss function based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result.

[0126] In the embodiments of this specification, the image data contained in the first image-text pair and the second image-text pair includes screenshots of a preset website, and the text data contained in the first image-text pair and the second image-text pair includes one or more of the following: layout information of the corresponding image data, description information of the content of the image data, content of the image data, and code information corresponding to the content of the image data.

[0127] This specification provides a model processing apparatus that acquires a first image-text pair and a second image-text pair containing noise. The image data from the first and second image-text pairs are input into a graph-text encoding sub-model within a multimodal model to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. The graph-text encoding sub-model encodes the image data or text data to obtain the corresponding representation. Then, the first and second image representations are input into a prediction sub-model within the multimodal model to obtain prediction results corresponding to the first and second image representations. A prediction result with a similarity greater than a preset threshold between the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation is obtained. The first prediction result is used as the first text data in the first image-text pair corresponding to the first prediction result, and then used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the multimodal model can be trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model. In this way, the most accurate text data (or pseudo-titles) can be assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction through effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the matching first image-text pair can be more stable.

[0128] The above describes the model processing apparatus provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a model processing device, such as... Figure 8 As shown.

[0129] The processing device for the model can be a terminal device or a server, as described in the above embodiments.

[0130] The processing device of the model can vary considerably due to differences in configuration or performance, and may include one or more processors 801 and memory 802. Memory 802 may store one or more application programs or data. Memory 802 may be temporary or persistent storage. The application programs stored in memory 802 may include one or more modules (not shown in the figures), each module including a series of computer-executable instructions for the model's processing device. Furthermore, processor 801 may be configured to communicate with memory 802 and execute the series of computer-executable instructions in memory 802 on the model's processing device. The model's processing device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, and one or more keyboards 806.

[0131] Specifically, in this embodiment, the processing device for the model includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the processing device of the model, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0132] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0133] The image data in the first image-text pair and the image data in the second image-text pair are encoded respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair.

[0134] The first image representation and the second image representation are respectively input into the target model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0135] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0136] The target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0137] Furthermore, specifically in this embodiment, the processing device for the model includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the processing device of the model, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0138] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0139] The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text encoding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation.

[0140] The first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0141] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0142] The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for processing the model are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0144] This specification provides a model processing device that acquires a first image-text pair and a noisy second image-text pair, encodes the image data in the first image-text pair and the image data in the second image-text pair respectively, and obtains a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation. From the prediction result corresponding to the first image representation, a first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained, and the first image-text pair corresponding to the first prediction result is... The first text data in the first image-text pair is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model. In this way, the most accurate text data (or pseudo-titles) are assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction with effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the first image-text pair that matches each other can be more stable.

[0145] Furthermore, based on the above Figures 1 to 5 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instruction information. In one specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process:

[0146] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0147] The image data in the first image-text pair and the image data in the second image-text pair are encoded respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair.

[0148] The first image representation and the second image representation are respectively input into the target model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0149] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0150] The target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0151] In another specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc., and the computer-executable instruction information stored in the storage medium can achieve the following process when executed by the processor:

[0152] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0153] The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text encoding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation.

[0154] The first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0155] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0156] The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0157] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0158] This specification provides a storage medium that acquires a first image-text pair and a noisy second image-text pair, encodes the image data in the first image-text pair and the image data in the second image-text pair respectively, to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model respectively to obtain prediction results corresponding to the first image representation and prediction results corresponding to the second image representation. From the prediction results corresponding to the first image representation, a first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained. The first image-text pair corresponding to the first prediction result is then... The first text data is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result. This yields the trained target model. In this way, the most accurate text data (or pseudo-titles) can be assigned to the image data in the noisy image-text pair, allowing the model to optimize in a better direction through effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the target model's learning of the matched first image-text pair can be more stable.

[0159] Furthermore, based on the above Figures 1 to 5 The method shown in this specification, along with one or more embodiments, also provides a computer program product including a computer program that, when executed by a processor, performs the following process:

[0160] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0161] The image data in the first image-text pair and the image data in the second image-text pair are encoded respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair.

[0162] The first image representation and the second image representation are respectively input into the target model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0163] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0164] The target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

[0165] Furthermore, in another specific embodiment, the computer program product includes a computer program that, when executed by a processor, performs the following process:

[0166] Obtain a first image-text pair that matches each other and a second image-text pair with noise;

[0167] The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text encoding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation.

[0168] The first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation;

[0169] From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair.

[0170] The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

[0171] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0172] This specification provides a computer program product that acquires a first image-text pair and a noisy second image-text pair, encodes the image data in the first image-text pair and the image data in the second image-text pair respectively, and obtains a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair. Then, the first image representation and the second image representation are input into a target model respectively to obtain a prediction result corresponding to the first image representation and a prediction result corresponding to the second image representation. From the prediction result corresponding to the first image representation, a first prediction result with a similarity greater than a preset threshold to the prediction result corresponding to the second image representation is obtained. The first image-text pair corresponding to the first prediction result is then used to generate the first image-text pair. The first text data in the first image-text pair is used as the text data to match the image data in the second image-text pair to obtain the third image-text pair. Finally, the target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model. In this way, the most accurate text data (or pseudo-titles) are assigned to the image data in the noisy image-text pair, so that the model can be optimized in a better direction with effective supervision information. Moreover, by directly assigning the corresponding text data to the image data in the noisy image-text pair, the learning of the target model on the first image-text pair that matches each other can be more stable.

[0173] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0174] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0175] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0176] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0177] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0178] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0183] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0184] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0188] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0189] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for processing a model, the method comprising: Obtain a first image-text pair that matches each other and a second image-text pair with noise; The image data in the first image-text pair and the image data in the second image-text pair are encoded respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The first image representation and the second image representation are respectively input into the target model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation; From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair. The target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

2. The method according to claim 1, further comprising: The text data in the first image-text pair is encoded to obtain the first text representation corresponding to the text data in the first image-text pair. The first text representation is input into the target model to obtain the prediction result corresponding to the first text representation; The step of training the target model based on the similarity between image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model, includes: The prediction result corresponding to the first text representation is used as the label information corresponding to the image data in the first image-text pair. Based on the label information and the prediction result corresponding to the first image representation, a first loss information is determined. Based on the similarity between the image data and the text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, a second loss information is determined. Based on the first loss information and the second loss information, the target model is trained to obtain the trained target model.

3. The method according to claim 2, wherein training the target model based on the first loss information and the second loss information to obtain the trained target model includes: The target model is trained based on the first loss information and the second loss information, as well as the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

4. The method according to claim 2, wherein the first loss information is the loss information determined by the cross-entropy loss function based on the prediction results corresponding to the label information and the first image representation.

5. The method according to claim 3, wherein the second loss information is a loss information determined by the Triplet Ranking loss function based on the similarity between the image data and the text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result.

6. The method according to claim 2, wherein the image data contained in the first image-text pair and the second image-text pair includes screenshots of a preset website, and the text data contained in the first image-text pair and the second image-text pair includes one or more of the following: layout information of the corresponding image data, description information of the content of the image data, content of the image data, and code information corresponding to the content of the image data.

7. The method according to claim 6, wherein encoding the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain a first image representation corresponding to the image data in the first image-text pair and a second image representation corresponding to the image data in the second image-text pair comprises: The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image encoder to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image encoder is used to encode the image data or text data to obtain the corresponding representation. And / or, The step of encoding the text data in the first image-text pair to obtain the first text representation corresponding to the text data in the first image-text pair includes: The text data in the first image-text pair is input into the image encoder to obtain the first text representation corresponding to the text data in the first image-text pair.

8. A method for processing a model, the method comprising: Obtain a first image-text pair that matches each other and a second image-text pair with noise; The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text encoding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation. The first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation; From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

9. A model processing apparatus, the apparatus comprising: The data pair acquisition module acquires a first image-text pair that matches each other and a second image-text pair with noise. The representation determination module encodes the image data in the first image-text pair and the image data in the second image-text pair respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The prediction module inputs the first image representation and the second image representation into the target model respectively to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation. The data processing module obtains a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and uses the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair. The training module trains the target model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

10. A model processing apparatus, the apparatus comprising: The data pair acquisition module acquires a first image-text pair that matches each other and a second image-text pair with noise. The first model processing module inputs the image data from the first image-text pair and the image data from the second image-text pair into the image-text encoding sub-model in the multimodal model, respectively, to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation. The second model processing module inputs the first image representation and the second image representation into the prediction sub-model in the multimodal model, respectively, to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation. The data processing module obtains a first prediction result from the prediction result corresponding to the first image representation, which has a similarity greater than a preset threshold to the prediction result corresponding to the second image representation, and uses the first text data in the first image-text pair corresponding to the first prediction result as text data that matches the image data in the second image-text pair to obtain a third image-text pair. The training module trains the multimodal model based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

11. A model processing apparatus, the model processing apparatus comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Obtain a first image-text pair that matches each other and a second image-text pair with noise; The image data in the first image-text pair and the image data in the second image-text pair are encoded respectively to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The first image representation and the second image representation are respectively input into the target model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation; From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair. The target model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained target model.

12. A model processing apparatus, the model processing apparatus comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Obtain a first image-text pair that matches each other and a second image-text pair with noise; The image data in the first image-text pair and the image data in the second image-text pair are respectively input into the image-text encoding sub-model in the multimodal model to obtain the first image representation corresponding to the image data in the first image-text pair and the second image representation corresponding to the image data in the second image-text pair. The image-text encoding sub-model is used to encode the image data or text data to obtain the corresponding representation. The first image representation and the second image representation are respectively input into the prediction sub-model in the multimodal model to obtain the prediction result corresponding to the first image representation and the prediction result corresponding to the second image representation; From the prediction results corresponding to the first image representation, obtain the first prediction result whose similarity to the prediction result corresponding to the second image representation is greater than a preset threshold. Use the first text data in the first image-text pair corresponding to the first prediction result as the text data that matches the image data in the second image-text pair to obtain the third image-text pair. The multimodal model is trained based on the similarity between the image data and text data in the third image-text pair, and the similarity between the prediction result corresponding to the second image representation and the first prediction result, to obtain the trained multimodal model.

Images