Feature Generation Method

Abstract

The embodiments of this specification provide a feature generation method, which includes: determining a text-image sample pair, where the text-image sample pair includes an image sample and a text sample corresponding to the image sample; determining an initial image feature of the image sample and an initial text feature of the text sample; inputting the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first text-image feature pair; inputting the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second text-image feature pair. It realizes generating multiple training labels according to one text-image sample pair, ensuring the diversity of the training data of the text-image data processing model.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and particularly to a feature generation method. Background Art

[0002] As a multi-modal model, a graphic and text data processing model can be used to process various different types of data, such as text, images, etc. A modality can be understood as different data representation methods. For example, text and images are different modalities. A multi-modal model can convert the input of one modality into the output of another modality and can be applied to downstream tasks such as image classification, image detection and segmentation, graphic and text retrieval, and image generation.

[0003] However, although the training samples (i.e., image samples and text samples) of current multi-modal models are numerous, the data is messy and has no organizational structure. Most of the data describes the same semantic content, and it is difficult to obtain a large number of effective samples required for model training to cover different semantic contents, resulting in insufficient diversity. When training a multi-modal model, the training efficiency is poor, and the performance of the trained multi-modal model is also poor. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide a feature generation method. One or more embodiments of this specification also relate to a feature generation device, a training method for a graphic and text data processing model, a training device for a graphic and text data processing model, a graphic and text data processing method, a graphic and text data processing device, a text processing method, a text processing device, a VR / AR device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to the first aspect of the embodiments of this specification, a feature generation method is provided, including:

[0006] Determine a graphic and text sample pair, where the graphic and text sample pair includes an image sample and a text sample corresponding to the image sample;

[0007] Determine the initial image feature of the image sample and the initial text feature of the text sample;

[0008] Input the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first graphic and text feature pair;

[0009] Input the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second graphic and text feature pair.

[0010] According to a second aspect of the embodiments of the present specification, a feature generation device is provided, including:

[0011] A first determination module configured to determine a graphic-text sample pair, where the graphic-text sample pair includes an image sample and a text sample corresponding to the image sample;

[0012] A second determination module configured to determine an initial image feature of the image sample and an initial text feature of the text sample;

[0013] A first input module configured to input the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first graphic-text feature pair;

[0014] A second input module configured to input the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second graphic-text feature pair.

[0015] According to a third aspect of the embodiments of the present specification, a training method for a graphic-text data processing model is provided, including:

[0016] Determine a graphic-text sample pair, where the graphic-text sample pair includes an image sample and a text sample corresponding to the image sample;

[0017] Determine an initial image feature of the image sample and an initial text feature of the text sample;

[0018] Input the image sample and the text sample into a graphic-text data processing model to obtain a predicted image feature and a predicted text feature;

[0019] Input the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first graphic-text feature pair;

[0020] Input the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second graphic-text feature pair;

[0021] Train the graphic-text data processing model according to the predicted image feature, the at least two target image features, the predicted text feature, and the at least two target text features until a graphic-text data processing model that meets the training stop condition is obtained.

[0022] According to a fourth aspect of the embodiments of the present specification, there is provided a training device for a graphic and text data processing model, including:

[0023] A first determination module, configured to determine a graphic and text sample pair, where the graphic and text sample pair includes an image sample and a text sample corresponding to the image sample;

[0024] A second determination module, configured to determine an initial image feature of the image sample and an initial text feature of the text sample;

[0025] A third determination module, configured to input the image sample and the text sample into the graphic and text data processing model to obtain a predicted image feature and a predicted text feature;

[0026] A first input module, configured to input the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first graphic and text feature pair;

[0027] A second input module, configured to input the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second graphic and text feature pair;

[0028] A training module, configured to train the graphic and text data processing model according to the predicted image feature, the at least two target image features, the predicted text feature, and the at least two target text features until a graphic and text data processing model that meets the training stop condition is obtained.

[0029] According to a fifth aspect of the embodiments of the present specification, there is provided a graphic and text data processing method, including:

[0030] Determine the graphic and text data to be processed;

[0031] Input the graphic and text data to be processed into the graphic and text data processing model to obtain the graphic and text features corresponding to the graphic and text data to be processed, where the graphic and text data processing model is the graphic and text data processing model that meets the training stop condition described in the training method of the graphic and text data processing model provided in the embodiments of the present specification;

[0032] Determine the graphic and text data processing result corresponding to the graphic and text data to be processed according to the graphic and text features.

[0033] According to a sixth aspect of the embodiments of the present specification, there is provided a graphic and text data processing device, including:

[0034] A first determination module, configured to determine the graphic and text data to be processed;

[0035] An input module, configured to input the text and image data to be processed into a text and image data processing model, and obtain text and image features corresponding to the text and image data to be processed, where the text and image data processing model is the text and image data processing model that meets the training stop condition in the training method of the text and image data processing model provided in the embodiments of the present specification;

[0036] A second determination module, configured to determine a text and image data processing result corresponding to the text and image data to be processed according to the text and image features.

[0037] According to a seventh aspect of the embodiments of the present specification, there is provided a text processing method, which is applied to a cloud-side device and includes:

[0038] Receiving a text processing request sent by a terminal-side device, where the text processing request carries text to be processed;

[0039] Inputting the text to be processed into a text and image data processing model, and obtaining text features corresponding to the text to be processed, where the text and image data processing model is the text and image data processing model that meets the training stop condition in the training method of the text and image data processing model provided in the embodiments of the present specification;

[0040] According to the text features, determining image features corresponding to the text features, and determining an image corresponding to the image features;

[0041] Sending the image to the terminal-side device.

[0042] According to an eighth aspect of the embodiments of the present specification, there is provided a text processing device, which is applied to a cloud-side device and includes:

[0043] A receiving module, configured to receive a text processing request sent by a terminal-side device, where the text processing request carries text to be processed;

[0044] An input module, configured to input the text to be processed into a text and image data processing model, and obtain text features corresponding to the text to be processed, where the text and image data processing model is the text and image data processing model that meets the training stop condition in the training method of the text and image data processing model provided in the embodiments of the present specification;

[0045] A determination module, configured to determine image features corresponding to the text features according to the text features, and determine an image corresponding to the image features;

[0046] A sending module, configured to send the image to the terminal-side device.

[0047] According to a ninth aspect of the embodiments of the present specification, there is provided a computing device, including:

[0048] A memory and a processor;

[0049] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0050] According to the tenth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the above method are implemented.

[0051] According to the eleventh aspect of the embodiments of the present specification, a computer program is provided. When the computer program is executed on a computer, the computer is made to execute the steps of the above method.

[0052] According to the twelfth aspect of the embodiments of the present specification, a VR / AR device is provided, including:

[0053] A memory and a processor;

[0054] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method for processing graphic and text data are implemented.

[0055] An embodiment of the present specification provides a feature generation method, which determines a graphic and text sample pair, where the graphic and text sample pair includes an image sample and a text sample corresponding to the image sample; determines an initial image feature of the image sample and an initial text feature of the text sample; inputs the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on a first graphic and text feature pair; inputs the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on a second graphic and text feature pair.

[0056] The above method extracts initial image features from the image samples in the image-text sample pair, generates at least two target text features based on the initial image features and at least two random first features, extracts initial text features from the text samples in the image-text sample pair, and generates at least two target image features based on the initial text features and at least two random second features. When training the image-text data processing model subsequently, the generated at least two target text features and at least two target image features are used as training labels, realizing the generation of multiple training labels from one image-text sample pair, ensuring the diversity of the training data of the image-text data processing model, expanding the semantic content outside the image-text sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of covering different semantic contents with a large number of effective samples during the model training process, and thus improving the model training efficiency and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 FIG. is a schematic diagram of an application scenario of a feature generation method provided by an embodiment of the present specification;

[0058] Figure 2 FIG. is a flowchart of a feature generation method provided by an embodiment of the present specification;

[0059] Figure 3 FIG. is a flowchart of a processing process of a feature generation method provided by an embodiment of the present specification;

[0060] Figure 4 FIG. is a schematic structural diagram of a feature generation device provided by an embodiment of the present specification;

[0061] Figure 5 FIG. is a flowchart of a training method of an image-text data processing model provided by an embodiment of the present specification;

[0062] Figure 6 FIG. is a schematic structural diagram of a training device of an image-text data processing model provided by an embodiment of the present specification;

[0063] Figure 7 FIG. is a flowchart of an image-text data processing method provided by an embodiment of the present specification;

[0064] Figure 8 FIG. is a schematic diagram of an application scenario of an image-text data processing method provided by an embodiment of the present specification;

[0065] Figure 9 FIG. is a schematic diagram of an application scenario of an image-text data processing method provided by an embodiment of the present specification;

[0066] Figure 10It is a schematic structural diagram of a graphic and text data processing device provided by an embodiment of this specification;

[0067] Figure 11 It is a flowchart of a text processing method provided by an embodiment of this specification;

[0068] Figure 12 It is a schematic diagram of the processing result of a text processing method provided by an embodiment of this specification;

[0069] Figure 13 It is a schematic structural diagram of a text processing device provided by an embodiment of this specification;

[0070] Figure 14 It is a structural block diagram of a computing device provided by an embodiment of this specification. Specific embodiments

[0071] In the following description, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the spirit of this specification. Therefore, this specification is not limited by the specific embodiments disclosed below.

[0072] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0073] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0074] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0075] First, the noun terms involved in one or more embodiments of this specification are explained.

[0076] VR: Virtual Reality, virtual reality technology.

[0077] AR: Augmented Reality, augmented reality technology.

[0078] In this specification, a feature generation method is provided. This specification also involves a feature generation device, a training method for a graphic and text data processing model, a training device for a graphic and text data processing model, a graphic and text data processing method, a graphic and text data processing device, a text processing method, a text processing device, a VR / AR device, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0079] See Figure 1 , Figure 1 which shows a schematic diagram of an application scenario of a feature generation method provided according to an embodiment of this specification.

[0080] Figure 1 It includes a cloud-side device 102. Herein, the cloud-side device 102 can be understood as a cloud server. Of course, in another implementable solution, the cloud-side device 102 can also be replaced with a physical server.

[0081] In specific implementation, the cloud-side device 102 can determine text-image sample pairs. Moreover, feature extraction is respectively performed on the image sample and the text sample in the text-image sample pair to obtain an initial image feature and an initial text feature. Through random sampling, at least two random first features and at least two random second features are obtained. The initial image feature and the at least two random first features are input into a first feature generator deployed on the cloud-side device 102 to obtain at least two target text features. The initial text feature and the at least two random second features are input into a second feature generator deployed on the cloud-side device 102 to obtain at least two target image features. This generation process can be continuously executed to obtain multiple text-image sample pairs, as well as at least two target text features and at least two image features corresponding to each text-image sample pair. According to the text-image sample pair, the at least two target text features, and the at least two target image features, a training data set can be constructed. This facilitates subsequent model training using these training data, achieving the expansion and enrichment of the original training data.

[0082] See Figure 2 , Figure 2 FIG. shows a flowchart of a feature generation method provided according to an embodiment of the present specification, which specifically includes the following steps.

[0083] Step 202: Determine a text-image sample pair, where the text-image sample pair includes an image sample and a text sample corresponding to the image sample.

[0084] Specifically, the feature generation method provided in the embodiments of the present specification can be applied to a server.

[0085] Among them, the text-image sample pair can be understood as any text-image sample pair obtained from a training data set, and the training data set can include multiple text-image sample pairs. The text-image sample pair can be used as training data for a text-image data processing model. The text-image data processing model can be used to process an image or text to obtain text information corresponding to the image or image information corresponding to the text.

[0086] For example, the image sample can be an image showing "a dog running on the grass", then the text sample corresponding to the image sample can be the text description information "a dog running on the grass" of the image sample, or the category information "animal" of the image sample. Specifically, the image sample and the text sample can be determined according to the requirements of model training. For example, in order to enable the text-image data processing model to learn the classification ability of the input image, the text sample can be a category information sample of the image sample. In order to enable the text-image data processing model to learn the description ability of the input image, the text sample can be a text description information sample of the image sample.

[0087] Specifically, the training data set can be determined according to the type of processing task of the image-text data processing model. For example, if the image-text data processing model needs to classify images showing animals, then a training data set in the animal field can be selected. The training data set in the animal field includes images of animals and corresponding category information. When the image-text data processing model needs to process other tasks, the corresponding training data set can be determined according to the actual task requirements, so as to determine the image-text sample pairs from the training data set.

[0088] It can be understood that, as a multi-modal model, the image-text data processing model can process multi-modal data. That is to say, the image-text data processing model can process both the input images and the input text. In order to enable the image-text data processing model to have the ability to process images, when training the image-text data processing model, the image samples in the image-text sample pairs can be used as training samples, and the text samples in the image-text sample pairs can be used as training labels to train the image-text data processing model. In order to enable the image-text data processing model to have the ability to process text, when training the image-text data processing model, the text samples in the image-text sample pairs can be used as training samples, and the image samples in the image-text sample pairs can be used as training labels to train the image-text data processing model.

[0089] In addition, the image-text data processing model can also process other types of data, such as video data and / or audio data. Correspondingly, the training data of the image-text data processing model also includes video samples and / or audio samples, etc.

[0090] Step 204: Determine the initial image features of the image sample and the initial text features of the text sample.

[0091] Specifically, after determining the image-text sample pairs, the initial image features of the image samples in the image-text sample pairs can be determined, and the initial text features of the text samples in the image-text sample pairs can be determined.

[0092] Among them, the initial image features can be understood as the initial image feature vectors of the image samples, and the initial text features can be understood as the initial image feature vectors of the text samples. It can be understood that the features described in the embodiments of this specification, such as initial text features, initial image features, target text features, target image features, random first features, random second features, etc., can all be understood as feature vectors.

[0093] In specific implementation, a feature encoder can be used to extract the initial image features of the image samples and the initial text features of the text samples. An image feature encoder can be used to extract the initial image features of the image samples, and a text feature encoder can be used to extract the initial image features of the text samples.

[0094] In practical applications, the graphic and text data processing model may include an image feature encoder and a text feature encoder. An image sample and a text sample can be input into the graphic and text data processing model to obtain the initial image features output by the image feature encoder in the graphic and text data processing model, and the initial text features output by the text feature encoder in the graphic and text data processing model.

[0095] Step 206: Input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on first graphic and text features.

[0096] Specifically, after determining the initial image features, the initial image features and at least two random first features can be input into the first feature generator to obtain at least two target text features.

[0097] Among them, the random first feature can be understood as a feature vector obtained through random sampling. The first feature generator can be understood as a feature generator used to generate associated text features according to the input image features. In practical applications, the first feature generator can be a pre-trained diffusion model.

[0098] In specific implementation, each random first feature in the initial image features and at least two random first features can be input into the first feature generator to obtain at least two target text features. It can be understood that the vector values of each random first feature can be different.

[0099] For example, 3 random first features X1, X2, and X3 are obtained through random sampling. The initial image features and the random first feature X1 are input into the first feature generator to obtain the target text feature Y1; the initial image features and the random first feature X2 are input into the first feature generator to obtain the target text feature Y2; the initial image features and the random first feature X3 are input into the first feature generator to obtain the target text feature Y3. Based on this, 3 target text features Y1, Y2, and Y3 can be obtained according to the initial image features and the 3 random first features X1, X2, and X3.

[0100] In practical applications, in order to enable the graphic and text data processing model to have the ability to process text, so that after the text is input into the graphic and text data processing model, the corresponding image information of the text can be determined according to the text features output by the graphic and text data processing model, usually the input text sample can be used as the training sample, and the image sample corresponding to the text sample can be used as the training label to train the graphic and text data processing model. Such a training method results in a single training data for the graphic and text data processing model and cannot enable the graphic and text data processing model to learn more abundant knowledge. Based on this, the image sample can be further processed, and at least two target text features can be generated according to the initial image features of the image sample to enrich the training data of the graphic and text data processing model. Specifically, the initial image features and at least two random first features can be input into the first feature generator to generate at least two target text features.

[0101] In order to enable the first feature generator to generate target text features according to the input initial image features, it is necessary to pre-train the first feature generator. The specific implementation method is as follows:

[0102] Before inputting the initial image features and at least two random first features into the first feature generator, it further includes:

[0103] Determine the first graphic and text feature pair and the random first feature sample, where the first graphic and text feature pair includes the first image feature and the first text feature corresponding to the first image feature;

[0104] Use the first image feature and the random first feature sample as the training sample, and use the first text feature as the training label to train the first feature generator until a first feature generator that meets the training stop condition is obtained.

[0105] Among them, the first image feature can be obtained from the image sample in the graphic and text sample pair, and the first text feature can be obtained from the text sample in the graphic and text sample pair. The graphic and text sample pair here and the graphic and text sample pair determined in step 202 can be the same or different. The random first feature sample can be a first feature vector sample obtained by random sampling.

[0106] Specifically, when training the first feature generator, it can be based on the diffusion model, that is, training is carried out on the basis of the diffusion model until a first feature generator that meets the training stop condition is obtained. Correspondingly, at least two target text features output by the first feature generator are diffusion features, realizing the diffusion and enrichment of the initial image features, and thus enriching the training data.

[0107] The training stop condition can be understood as that the model loss value is less than the preset threshold, and / or the number of training rounds reaches the preset number of rounds.

[0108] In a specific implementation provided in this specification, taking the example of stopping the training of the first feature generator when the model loss value is less than a preset threshold, where the preset threshold is 0.3, when the calculated model loss value is less than 0.3, it is determined that the training of the first feature generator is completed.

[0109] In another specific implementation provided in this specification, taking the example of using a preset number of training rounds to stop the training of the first feature generator, where the preset number of training rounds is 10 rounds, when the number of training rounds of the sample data reaches 10 rounds, it is determined that the training of the first feature generator is completed.

[0110] In yet another specific implementation provided in this specification, two training stop conditions, namely the preset loss value threshold and the preset number of training rounds, are set, and both the model loss value and the number of training rounds are monitored. When either one or both of the model loss value and the number of training rounds meet the training stop conditions, it is determined that the training of the first feature generator is completed. That is, when the model loss value is less than the preset loss value threshold or the number of training rounds reaches the preset number of training rounds, or when the model loss value is less than the preset loss value threshold and the number of training rounds reaches the preset number of training rounds, it is determined that the training of the first feature generator is completed.

[0111] In summary, by training the first feature generator, it is convenient to subsequently generate at least two target text features based on the initial image features, thereby realizing the enrichment and diffusion of the training data.

[0112] During specific implementation, in the first feature generator, the initial image features can be used to adjust the random first features, and the specific implementation method is as follows:

[0113] The step of inputting the initial image features and at least two random first features into the first feature generator to obtain at least two target text features includes:

[0114] Taking the initial image features as adjustment conditions and at least two random first features as objects to be adjusted, and inputting them into the first feature generator;

[0115] In the first feature generator, adjusting the at least two random first features according to the initial image features to obtain at least two target text features.

[0116] Specifically, the first feature generator can gradually denoise the random first features input as objects to be adjusted according to the initial image features input as adjustment conditions, and output the target text features. It can be understood that due to the different numerical values of the random first features, different target text features will be output after gradually denoising the random first features.

[0117] Continuing with the above example, taking the initial image features as adjustment conditions and the random first feature X1 as the object to be adjusted, input them into the first feature generator. In this first feature generator, according to the initial image features, gradually denoise the random first feature X1 to obtain the target text feature Y1. Similarly, obtain the target text features Y2 and Y3 output by the first feature generator.

[0118] In summary, by using the first feature generator and inputting at least two different random first features, at least two target text features are obtained, realizing the diffusion and enrichment of the initial image features, and providing more training data for model training.

[0119] During specific implementation, when using the at least two obtained target text features for model training, the at least two target text features can be used as training labels for model training. The specific implementation method is as follows:

[0120] After obtaining the at least two target text features, it further includes:

[0121] Input the text sample into the text-image data processing model to obtain the predicted text feature;

[0122] According to the predicted text feature and the at least two target text features, train the text-image data processing model until a text-image data processing model that meets the training stop condition is obtained.

[0123] Specifically, the text-image data processing model includes a text feature encoder and a text feature mapping layer. Input the text sample into the text feature encoder in the text-image data processing model to obtain the initial text feature output by the text feature encoder, and input the initial text feature into the text feature mapping layer to obtain the predicted text feature output by the text feature mapping layer.

[0124] In an embodiment of this specification, in order to make the semantic content of the predicted text feature output by the image processing model more abundant, in the text feature mapping layer, linear mapping processing can be performed on the initial text feature to obtain the predicted text feature.

[0125] It can be understood that when training the text-image data processing model, text-image sample pairs can be determined. Input the text sample in the text-image sample pair into the text-image data processing model to obtain the predicted text feature. According to the predicted text feature and at least two target text features, train the text-image data processing model, and continue to execute the step of determining text-image sample pairs, and so on, until a text-image data processing model that meets the training stop condition is obtained.

[0126] In summary, by using at least two obtained target text features for model training, more effective training data is augmented outside the original training data, enabling the model training data to cover more semantic content, and thus ensuring the richness of the representational semantics that the model can learn.

[0127] In specific implementation, the second model loss value can be calculated based on the predicted text feature and each target text feature, thereby training the graphic and text data processing model. The specific implementation method is as follows:

[0128] Training the graphic and text data processing model according to the predicted text feature and the at least two target text features until a graphic and text data processing model that meets the training stop condition is obtained, includes:

[0129] Calculating at least two second model loss values according to each target text feature among the predicted text feature and the at least two target text features;

[0130] Determining a second target model loss value according to the at least two second model loss values;

[0131] Training the graphic and text data processing model according to the second target model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

[0132] Specifically, at least two second model loss values can be calculated according to each target text feature among the predicted text feature and the at least two target text features, and the at least two second model loss values can be processed. For example, the at least two second model loss values can be added together, or the at least two second model loss values can be weighted and summed to obtain the second target model loss value. According to the second target model loss value, the graphic and text data processing model is trained until a graphic and text data processing model that meets the training stop condition is obtained.

[0133] In the embodiments of this specification, when training the model according to the predicted text feature and the at least two target text features, the training objective is to narrow the distance between the predicted text feature and each target text feature. Narrowing the distance can be understood as maximizing the similarity between the predicted text feature and each target text feature, thereby reducing the distance between the predicted text feature and each target text feature. The distance between the predicted text feature and the target text feature can be obtained by subtracting the predicted text feature vector from the target text feature vector.

[0134] Moreover, when calculating the second model loss value, the predicted text feature and the target text feature can be input into the target loss function to obtain the second model loss value output by the target loss function.

[0135] In practical applications, an appropriate target loss function can be selected according to the actual application scenario, and the predicted text features and the target text features are input into the target loss function for calculation to obtain the second model loss value. The target loss function can be, for example, a cross-entropy loss function, or an absolute value loss function, a squared loss function, a 0-1 loss function, etc. The embodiments of this specification do not limit the selection of the target loss function.

[0136] Continuing with the above example, the text sample can be input into the text-image data processing model to obtain the predicted text features. According to the predicted text features and the target text features Y1, the second model loss value a1 is calculated; according to the predicted text features and the target text features Y2, the second model loss value a2 is calculated; according to the predicted text features and the target text features Y3, the second model loss value a3 is calculated. According to the second model loss values a1, a2, and a3, the second target model loss value a is calculated. According to the second target model loss value a, the text-image data processing model is trained until a text-image data processing model that meets the training stop condition is obtained.

[0137] In summary, by training the text-image data processing model using the second target model loss value calculated from at least two obtained target text features, the predicted text features can learn the representations of at least two target text features, thereby learning the representations of the initial image features, further achieving the training goal of narrowing the distance between the predicted text features and the image features, and thus improving the processing performance of the text-image data processing model.

[0138] Step 208: Input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on second text-image feature pairs.

[0139] Specifically, after obtaining the initial text features, the initial text features and at least two random second features can be input into the second feature generator to obtain at least two target image features output by the second feature generator.

[0140] Among them, the random second features can be understood as feature vectors obtained through random sampling. The random second features and the random first features can be the same or different. The second feature generator can be understood as a feature generator for generating associated image features according to the input text features. In practical applications, the second feature generator can be a pre-trained diffusion model. The second text-image feature pairs can be the same as or different from the first text-image feature pairs. It can be understood that the vector values of each random second feature can be different.

[0141] For example, three random second features P1, P2, and P3 are obtained through random sampling. The initial text feature and the random second feature P1 are input into the second feature generator to obtain the target image feature Q1; the initial text feature and the random second feature P2 are input into the second feature generator to obtain the target image feature Q2; the initial text feature and the random second feature P3 are input into the second feature generator to obtain the target image feature Q3. Based on this, three target image features Q1, Q2, and Q3 can be obtained according to the initial text feature and the three random second features P1, P2, and P3.

[0142] In one embodiment of this specification, the first feature generator and the second feature generator are diffusion models.

[0143] In practical applications, in order to enable the text-image data processing model to have the ability to process images, so that after an image is input into the text-image data processing model, the text information corresponding to the image can be determined according to the image features output by the text-image data processing model. Usually, the input image samples can be used as training samples, and the text samples corresponding to the image samples can be used as training labels to train the text-image data processing model. Such a training method results in a single training data for the text-image data processing model and cannot enable the text-image data processing model to learn more abundant knowledge. Based on this, the text samples can be further processed, and at least two target image features can be generated according to the initial text features of the text samples to enrich the training data of the text-image data processing model. Specifically, the initial text feature and at least two random second features can be input into the second feature generator to generate at least two target image features.

[0144] In order to enable the second feature generator to generate target image features according to the input initial text features, it is necessary to pre-train the second feature generator. The specific implementation method is as follows:

[0145] Before inputting the initial text feature and at least two random second features into the second feature generator, it further includes:

[0146] Determine the second text-image feature pairs and random second feature samples, where the second text-image feature pairs include second image features and the second text features corresponding to the second image features;

[0147] Use the second text features and the random second feature samples as training samples, and use the second image features as training labels to train the second feature generator until a second feature generator that meets the training stop condition is obtained.

[0148] Among them, the second image feature can be obtained from the image sample in the image-text sample pair, and the second text feature can be obtained from the text sample in the image-text sample pair. The image-text sample pair here and the image-text sample pair determined in step 202 can be the same or different. The random second feature sample can be a second feature vector sample obtained by random sampling.

[0149] Specifically, when training the second feature generator, it can be based on a diffusion model, that is, training is carried out on the basis of the diffusion model until a second feature generator that meets the training stop condition is obtained. Correspondingly, at least two target image features output by the second feature generator are diffusion features, which realizes the diffusion and enrichment of the initial text features, and thus enriches the training data.

[0150] In summary, by training the second feature generator, it is convenient to subsequently generate at least two target image features according to the initial text features, thereby realizing the enrichment and diffusion of the training data.

[0151] In specific implementation, in the second feature generator, the initial text feature can be used to adjust the random second feature. The specific implementation method is as follows:

[0152] The step of inputting the initial text feature and at least two random second features into the second feature generator to obtain at least two target image features includes:

[0153] Taking the initial text feature as the adjustment condition and at least two random second features as the objects to be adjusted, and inputting them into the second feature generator;

[0154] In the second feature generator, according to the initial text feature, the at least two random second features are adjusted to obtain at least two target image features.

[0155] Specifically, the second feature generator can gradually denoise the random second feature input as the object to be adjusted according to the initial text feature input as the adjustment condition, and output the target image feature. It can be understood that due to the different numerical values of the random second feature, the target image features output by gradually denoising the random second feature are also different.

[0156] Continuing with the above example, taking the initial text feature as the adjustment condition and the random second feature P1 as the object to be adjusted and inputting them into the second feature generator. In this second feature generator, according to the initial text feature, the random second feature P1 is gradually denoised to obtain the target image feature Q1. Similarly, the target image features Q2 and Q3 output by the second feature generator are obtained.

[0157] In summary, by using the second feature generator and inputting at least two different random second features, at least two target image features are obtained, realizing the diffusion and enrichment of the initial text features, and providing more training data for model training.

[0158] In specific implementation, when using the obtained at least two target image features for model training, the at least two target image features can be used as training labels for model training. The specific implementation method is as follows:

[0159] After obtaining the at least two target image features, it further includes:

[0160] Input the image sample into the text-image data processing model to obtain predicted image features;

[0161] According to the predicted image features and the at least two target image features, train the text-image data processing model until a text-image data processing model that meets the training stop condition is obtained.

[0162] Specifically, the text-image data processing model includes an image feature encoder and an image feature mapping layer. Input the image sample into the image feature encoder in the text-image data processing model to obtain the initial image features output by the image feature encoder, and input the initial image features into the image feature mapping layer to obtain the predicted image features output by the image feature mapping layer.

[0163] Similar to the foregoing, in the image feature mapping layer, linear mapping processing can be performed on the initial image features to obtain predicted image features.

[0164] It can be understood that when training the text-image data processing model, text-image sample pairs can be determined. Input the image samples in the text-image sample pairs into the text-image data processing model to obtain predicted image features. According to the predicted image features and at least two target image features, train the text-image data processing model, and continue to execute the step of determining text-image sample pairs, and so on, until a text-image data processing model that meets the training stop condition is obtained.

[0165] In summary, by using the obtained at least two target image features for model training, more effective training data is expanded outside the original training data, enabling the model training data to cover more semantic content, and thus ensuring the richness of the semantic representations that the model can learn.

[0166] In specific implementation, the first model loss value can be calculated according to the predicted image features and each target image feature, so as to train the text-image data processing model. The specific implementation method is as follows:

[0167] Training the text and image data processing model according to the predicted image features and the at least two target image features until a text and image data processing model that meets the training stop condition is obtained, includes:

[0168] Calculating at least two first model loss values according to each target image feature among the predicted image features and the at least two target image features;

[0169] Determining a first target model loss value according to the at least two first model loss values;

[0170] Training the text and image data processing model according to the first target model loss value until a text and image data processing model that meets the training stop condition is obtained.

[0171] Specifically, at least two first model loss values can be calculated according to each target image feature among the predicted image features and the at least two target image features, and the at least two first model loss values can be processed. For example, the at least two first model loss values can be added together, or the at least two first model loss values can be weighted and summed to obtain the first target model loss value. The text and image data processing model is trained according to the first target model loss value until a text and image data processing model that meets the training stop condition is obtained.

[0172] In the embodiments of this specification, when training the model according to the predicted image features and the at least two target image features, the training objective is to narrow the distance between the predicted image features and each target image feature. Narrowing the distance can be understood as maximizing the similarity between the predicted image features and each target image feature, so as to make the distance between the predicted image features and each target image feature smaller. The distance between the predicted image feature and the target image feature can be obtained by subtracting the predicted image feature vector from the target image feature vector.

[0173] Moreover, when calculating the first model loss value, the predicted image feature and the target image feature can be input into the target loss function to obtain the first model loss value output by the target loss function.

[0174] In practical applications, a suitable target loss function can be selected according to the actual application scenario, and the predicted image feature and the target image feature are input into the target loss function for calculation to obtain the first model loss value. The target loss function can be, for example, the cross-entropy loss function, or the absolute value loss function, the square loss function, the 0-1 loss function, etc. The embodiments of this specification do not limit the selection of the target loss function.

[0175] Continuing with the above example, the image sample can be input into the graphic and text data processing model to obtain the predicted image features. According to the predicted image features and the target image features Q1, the first model loss value b1 is calculated; according to the predicted image features and the target image features Q2, the first model loss value b2 is calculated; according to the predicted image features and the target image features Q3, the first model loss value b3 is calculated. According to the first model loss values b1, b2, and b3, the first target model loss value b is calculated. According to the first target model loss value b, the graphic and text data processing model is trained until a graphic and text data processing model that meets the training stop condition is obtained.

[0176] In summary, by training the graphic and text data processing model with the first target model loss value calculated using at least two obtained target image features, the predicted image features can learn the representations of at least two target image features, thereby learning the representations of the initial text features, further achieving the training goal of narrowing the distance between the predicted image features and the text features, and thus improving the processing performance of the graphic and text data processing model.

[0177] In specific implementation, the model can also be trained based on the predicted image features and the predicted text features, with the goal of narrowing the distance between the predicted image features and the predicted text features. The specific implementation method is as follows:

[0178] According to the predicted image features and the predicted text features, the graphic and text data processing model is trained until a graphic and text data processing model that meets the training stop condition is obtained.

[0179] Specifically, the model loss value can be calculated based on the predicted image features and the predicted text features, and the graphic and text data processing model is trained according to the model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

[0180] It can be understood that when training the graphic and text data processing model, a graphic and text sample pair can be determined. The image sample and the text sample in the graphic and text sample pair are input into the graphic and text data processing model to obtain the predicted image features and the predicted text features. According to the predicted image features and the predicted text features, the graphic and text data processing model is trained, and the step of determining the graphic and text sample pair is continued, and so on, until a graphic and text data processing model that meets the training stop condition is obtained.

[0181] In summary, by training the graphic and text data processing model through the above method, the features learned between the two different modalities of images and texts can be aligned, enhancing the representation ability of the graphic and text data processing model.

[0182] In summary, the above method extracts initial image features from the image samples in the image-text sample pair, generates at least two target text features based on the initial image features and at least two random first features, extracts initial text features from the text samples in the image-text sample pair, and generates at least two target image features based on the initial text features and at least two random second features. This facilitates subsequent training of the image-text data processing model by using the generated at least two target text features and at least two target image features as training labels, achieving the generation of multiple training labels from one image-text sample pair, ensuring the diversity of the training data of the image-text data processing model, expanding the semantic content outside the image-text sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of a large number of effective samples covering different semantic contents during the model training process, and thus improving the model training efficiency and model performance.

[0183] The following combines the attached Figure 3 drawing, taking the application of the feature generation method provided in this specification in model training as an example, to further illustrate the feature generation method. Among them, Figure 3 FIG. shows a processing flow chart of a feature generation method provided by an embodiment of this specification, which specifically includes the following steps.

[0184] Step 302: Determine an image-text sample pair, where the image-text sample pair includes an image sample and the text sample corresponding to the image sample.

[0185] Taking the training of an image-text data processing model for processing image content description tasks as an example, the determined image sample can be an image showing a dog running on the grass, and the determined text sample can be the text description information "A dog is running on the grass" for the image sample.

[0186] Step 304: Input the text sample into the text feature encoder in the image-text data processing model to obtain initial text features.

[0187] Step 306: Input the initial text features into the text feature mapping layer in the image-text data processing model to obtain predicted text features.

[0188] Specifically, in order to improve the richness of the semantic content of the predicted text features, in the text feature mapping layer, linear mapping processing can be performed on the initial text features to obtain the predicted text feature t.

[0189] Step 308: Input the image sample into the image feature encoder in the image-text data processing model to obtain initial image features.

[0190] Step 310: Input the initial image features into the image feature mapping layer in the image-text data processing model to obtain predicted image features.

[0191] Accordingly, in order to improve the richness of the semantic content of the predicted image features, in the image feature mapping layer, linear mapping processing can be performed on the initial image features to obtain the predicted image features v.

[0192] Step 312: Input the initial image features as adjustment conditions and at least two random first features as objects to be adjusted into the first feature generator to obtain at least two target text features.

[0193] Among them, the first feature generator, as an image-to-text feature generator, can be used to generate text features according to the input image features, and it can be trained based on the diffusion model.

[0194] Specifically, the initial image features can be used as adjustment conditions, and k randomly sampled random first features can be used as objects to be adjusted (i.e., seeds), and input into the image-to-text feature generator to obtain k target text features t1’, t2’... tk’.

[0195] Step 314: Input the initial text features as adjustment conditions and at least two random second features as objects to be adjusted into the second feature generator to obtain at least two target image features.

[0196] Among them, the second feature generator, as a text-to-image feature generator, can be used to generate image features according to the input text features, and it can be trained based on the diffusion model.

[0197] Specifically, the initial text features can be used as adjustment conditions, and k randomly sampled random second features can be used as objects to be adjusted (i.e., seeds), and input into the text-to-image feature generator to obtain k target image features v1’, v2’... vk’.

[0198] Step 316: Train the text-image data processing model according to the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a text-image data processing model that meets the training stop condition is obtained.

[0199] Specifically, during training, the distance between the predicted text feature t and the predicted image feature v can be used as the training objective, and the text-image data processing model can be trained according to the predicted text feature t and the predicted image feature v.

[0200] It is also possible to use as the training objective the reduction of the distance between the predicted image feature v and the k target image features v1', v2',..., vk'. Based on the predicted image feature v and the target image feature v1', based on the predicted image feature v and the target image feature v2',..., based on the predicted image feature v and the target image feature vk', calculate k first model loss values, sum up the k first model loss values to obtain the first target model loss value, and train the graphic and text data processing model according to the first target model loss value.

[0201] It is also possible to use as the training objective the reduction of the distance between the predicted text feature t and the k target text features t1', t2',..., tk'. Based on the predicted text feature t and the target text feature t1', based on the predicted text feature t and the target text feature t2',..., based on the predicted text feature t and the target text feature tk', calculate k second model loss values, sum up the k second model loss values to obtain the second target model loss value, and train the graphic and text data processing model according to the second target model loss value. Until a graphic and text data processing model that meets the training stop condition is obtained.

[0202] In summary, the above method extracts the initial image features from the image samples in the graphic and text sample pairs, generates at least two target text features based on the initial image features and at least two random first features, extracts the initial text features from the text samples in the graphic and text sample pairs, and generates at least two target image features based on the initial text features and at least two random second features. When training the graphic and text data processing model subsequently, the generated at least two target text features and at least two target image features are used as training labels, realizing the generation of multiple training labels from one graphic and text sample pair, ensuring the diversity of the training data of the graphic and text data processing model, expanding the semantic content outside the graphic and text sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of a large number of effective samples covering different semantic contents during the model training process, and thus improving the model training efficiency and model performance.

[0203] Corresponding to the above method embodiment, this specification also provides an embodiment of a feature generation device. Figure 4 The following shows a schematic structural diagram of a feature generation device provided by an embodiment of this specification. As Figure 4 shown, the device includes:

[0204] A first determination module 402, configured to determine a graphic and text sample pair, where the graphic and text sample pair includes an image sample and the text sample corresponding to the image sample;

[0205] A second determination module 404, configured to determine the initial image feature of the image sample and the initial text feature of the text sample;

[0206] The first input module 406 is configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on first image-text feature pairs;

[0207] The second input module 408 is configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on second image-text feature pairs.

[0208] In an optional embodiment, the first input module 406 is further configured to:

[0209] Input the initial image features as adjustment conditions and at least two random first features as objects to be adjusted into the first feature generator;

[0210] In the first feature generator, adjust the at least two random first features according to the initial image features to obtain at least two target text features.

[0211] In an optional embodiment, the second input module 408 is further configured to:

[0212] Input the initial text features as adjustment conditions and at least two random second features as objects to be adjusted into the second feature generator;

[0213] In the second feature generator, adjust the at least two random second features according to the initial text features to obtain at least two target image features.

[0214] In an optional embodiment, the apparatus further includes a training module, which is configured to:

[0215] Input the image samples into the image-text data processing model to obtain predicted image features;

[0216] Train the image-text data processing model according to the predicted image features and the at least two target image features until an image-text data processing model that meets the training stop condition is obtained.

[0217] In an optional embodiment, the training module is further configured to:

[0218] Calculate at least two first model loss values according to each of the predicted image features and the at least two target image features;

[0219] Determine a first target model loss value according to the at least two first model loss values;

[0220] Train the graphic and text data processing model according to the first target model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

[0221] In an optional embodiment, the training module is further configured to:

[0222] Input the text sample into the graphic and text data processing model to obtain predicted text features;

[0223] Train the graphic and text data processing model according to the predicted text features and the at least two target text features until a graphic and text data processing model that meets the training stop condition is obtained.

[0224] In an optional embodiment, the training module is further configured to:

[0225] Calculate at least two second model loss values according to each target text feature among the predicted text features and the at least two target text features;

[0226] Determine a second target model loss value according to the at least two second model loss values;

[0227] Train the graphic and text data processing model according to the second target model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

[0228] In an optional embodiment, the training module is further configured to:

[0229] Train the graphic and text data processing model according to the predicted image features and the predicted text features until a graphic and text data processing model that meets the training stop condition is obtained.

[0230] In an optional embodiment, the training module is further configured to:

[0231] Determine a first graphic and text feature pair and a random first feature sample, where the first graphic and text feature pair includes a first image feature and a first text feature corresponding to the first image feature;

[0232] Use the first image feature and the random first feature sample as training samples, and use the first text feature as a training label to train the first feature generator until a first feature generator that meets the training stop condition is obtained.

[0233] In an optional embodiment, the training module is further configured to:

[0234] Determine a second image-text feature pair and a random second feature sample, where the second image-text feature pair includes a second image feature and a second text feature corresponding to the second image feature;

[0235] Use the second text feature and the random second feature sample as training samples, and use the second image feature as a training label to train the second feature generator until a second feature generator that meets the training stop condition is obtained.

[0236] In an optional embodiment, the first feature generator and the second feature generator are diffusion models.

[0237] In summary, the above device extracts an initial image feature from the image sample in the image-text sample pair, generates at least two target text features based on the initial image feature and at least two random first features, and extracts an initial text feature from the text sample in the image-text sample pair, and generates at least two target image features based on the initial text feature and at least two random second features. When facilitating the subsequent training of the image-text data processing model, using the generated at least two target text features and at least two target image features as training labels realizes generating multiple training labels from one image-text sample pair, ensuring the diversity of the training data of the image-text data processing model, expanding the semantic content outside the image-text sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of a large number of effective samples covering different semantic contents during the model training process, and thus improving the model training efficiency and model performance.

[0238] The above is a schematic solution of a feature generation device in this embodiment. It should be noted that the technical solution of this feature generation device and the technical solution of the above feature generation method belong to the same concept. For the details not described in detail in the technical solution of the feature generation device, reference can be made to the description of the technical solution of the above feature generation method.

[0239] Corresponding to the above method embodiment, this specification also provides an embodiment of a training method for an image-text data processing model. See Figure 5 , Figure 5 shows a flowchart of a training method for an image-text data processing model provided according to an embodiment of this specification, which specifically includes the following steps.

[0240] Step 502: Determine an image-text sample pair, where the image-text sample pair includes an image sample and a text sample corresponding to the image sample;

[0241] Step 504: Determine the initial image feature of the image sample and the initial text feature of the text sample;

[0242] Step 506: Input the image sample and the text sample into the graphic-text data processing model to obtain predicted image features and predicted text features;

[0243] Step 508: Input the initial image features and at least two random first features into the first feature generator to obtain at least two target text features, where the first feature generator is trained based on the first graphic-text feature pairs;

[0244] Step 510: Input the initial text features and at least two random second features into the second feature generator to obtain at least two target image features, where the second feature generator is trained based on the second graphic-text feature pairs;

[0245] Step 512: Train the graphic-text data processing model according to the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a graphic-text data processing model that meets the training stop condition is obtained.

[0246] Specifically, when training the graphic-text data processing model according to the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features, the distance between the predicted text features and the predicted image features can be minimized as the training objective, and the graphic-text data processing model can be trained according to the predicted text features and the predicted image features.

[0247] Alternatively, the distance between the predicted image features and the at least two target image features can be minimized as the training objective. Calculate at least two first model loss values according to the predicted image features and each target image feature, sum the at least two first model loss values to obtain a first target model loss value, and train the graphic-text data processing model according to the first target model loss value.

[0248] Alternatively, the distance between the predicted text features and the at least two target text features can be minimized as the training objective. Calculate at least two second model loss values according to the predicted text features and each target text feature, sum the at least two second model loss values to obtain a second target model loss value, and train the graphic-text data processing model according to the second target model loss value. Until a graphic-text data processing model that meets the training stop condition is obtained.

[0249] Specifically, the above three training processes can be executed separately or simultaneously. The specific training process is similar to the foregoing and will not be repeated here.

[0250] In summary, the above method extracts initial image features from the image samples in the image-text sample pairs, generates at least two target text features based on the initial image features and at least two random first features, extracts initial text features from the text samples in the image-text sample pairs, and generates at least two target image features based on the initial text features and at least two random second features. When facilitating the subsequent training of the image-text data processing model, the generated at least two target text features and at least two target image features are used as training labels, realizing the generation of multiple training labels from one image-text sample pair, ensuring the diversity of the training data of the image-text data processing model, expanding the semantic content outside the image-text sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of a large number of effective samples covering different semantic contents during the model training process, and thus improving the model training efficiency and model performance.

[0251] The above is a schematic solution of a training method for an image-text data processing model in this embodiment. It should be noted that the technical solution of the training method for the image-text data processing model belongs to the same concept as the technical solution of the above feature generation method. For the details not described in the technical solution of the training method for the image-text data processing model, reference can be made to the description of the technical solution of the above feature generation method.

[0252] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for an image-text data processing model. Figure 6 It shows a schematic structural diagram of a training device for an image-text data processing model provided in an embodiment of this specification. As Figure 6 shown, the device includes:

[0253] A first determination module 602, configured to determine an image-text sample pair, where the image-text sample pair includes an image sample and a text sample corresponding to the image sample;

[0254] A second determination module 604, configured to determine the initial image features of the image sample and the initial text features of the text sample;

[0255] A third determination module, configured to input the image sample and the text sample into the image-text data processing model to obtain predicted image features and predicted text features;

[0256] A first input module 608, configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is trained based on first image-text feature pairs;

[0257] A second input module 610, configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is trained based on second text-image features;

[0258] A training module 612, configured to train the text-image data processing model according to the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a text-image data processing model that meets the training stop condition is obtained.

[0259] In summary, the above device extracts initial image features from the image samples in the text-image sample pair, generates at least two target text features according to the initial image features and at least two random first features, extracts initial text features from the text samples in the text-image sample pair, and generates at least two target image features according to the initial text features and at least two random second features. When training the text-image data processing model subsequently, the generated at least two target text features and at least two target image features are used as training labels, realizing generating multiple training labels from one text-image sample pair, ensuring the diversity of the training data of the text-image data processing model, expanding the semantic content outside the text-image sample pairs (i.e., the original training data) that can be learned during model training, meeting the requirement of a large number of effective samples covering different semantic contents during the model training process, and thus improving the model training efficiency and model performance.

[0260] Corresponding to the above method embodiment, this specification also provides a text-image data processing method embodiment. Refer to Figure 7 , Figure 7 which shows a flowchart of a text-image data processing method provided according to an embodiment of this specification, specifically including the following steps.

[0261] Step 702: Determine the text-image data to be processed;

[0262] Step 704: Input the text-image data to be processed into the text-image data processing model to obtain the text-image features corresponding to the text-image data to be processed, where the text-image data processing model is the text-image data processing model that meets the training stop condition described in the training method of the text-image data processing model provided in the embodiments of this specification;

[0263] Step 706: Determine the text-image data processing result corresponding to the text-image data to be processed according to the text-image features.

[0264] Among them, the text-image data to be processed may include the image to be processed and the text to be processed.

[0265] Specifically, when the graphic data to be processed is an image to be processed, the image to be processed can be input into a graphic data processing model to obtain image features corresponding to the image to be processed, and based on the image features, the text features associated with the image features are determined, and based on the text features, the text information corresponding to the image to be processed is determined. The text information is the result of the graphic data processing. In specific implementation, when determining the text features associated with the image features based on the image features, the text features with the greatest similarity to the image features are determined in the candidate text feature set as the text features associated with the image features. The candidate text feature set can be determined based on the specific processing task. For example, when the processing task is animal image classification, the candidate text feature set can be a set of pre-configured text features of animal categories.

[0266] Taking animal image description as an example, the image to be processed may be an image showing a cat drinking water. The image to be processed is input into the graphic data processing model to obtain the image features corresponding to the image to be processed. Based on the image features, the text features associated with the image features are determined from the candidate text feature set in the field of animal image description. Based on the text features, the text information corresponding to the image to be processed is determined to be "cat drinking water".

[0267] When the image data to be processed is text to be processed, the text to be processed can be input into the image data processing model to obtain text features corresponding to the text to be processed. Based on the text features, the image features associated with the text features are determined. Based on the image features, the image information corresponding to the text to be processed is determined. This image information is the image data processing result. Similar to the above, when determining the image features associated with the text features based on the text features, the image feature with the greatest similarity to the text features can be determined from the set of candidate image features as the image feature associated with the text features.

[0268] Based on this, the embodiment of this specification also provides an embodiment of an image data processing method, which specifically includes the following steps.

[0269] Determine the image to be processed;

[0270] Inputting the image to be processed into a graphic data processing model to obtain image features corresponding to the image to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies the training stop condition as described in the training method for the graphic data processing model provided in the embodiments of this specification;

[0271] Determining text features associated with the image features based on the image features;

[0272] The text information corresponding to the image to be processed is determined according to the text feature.

[0273] Moreover, the embodiments of this specification also provide an embodiment of a text data processing method, which specifically includes the following steps.

[0274] Determine the text to be processed;

[0275] Input the text to be processed into the graphic and text data processing model to obtain the text features corresponding to the text to be processed, where the graphic and text data processing model is the graphic and text data processing model that meets the training stop condition described in the training method of the graphic and text data processing model provided by the embodiments of this specification;

[0276] Determine the image features associated with the text features according to the text features;

[0277] Determine the image information corresponding to the text to be processed according to the image features.

[0278] The following combines the attached Figure 8 , taking the application of the graphic and text data processing method provided in this specification in image classification as an example, to further illustrate the image data processing method. Among them, Figure 8 shows a schematic diagram of an application scenario of a graphic and text data processing method provided by an embodiment of this specification, which specifically includes the following steps.

[0279] Step 802: The edge device receives the image to be processed input by the user in the image processing request upload box, and receives the upload instruction of the user for this image processing request, and sends the image processing request to the cloud device.

[0280] Among them, the edge device displays the image processing request upload box to the user. The image processing request carries the image to be processed. The image to be processed is an image with an image classification requirement.

[0281] Specifically, the user clicks the control "OK" on the display interface of the edge device, and the edge device determines the upload instruction of the user for the image processing request based on the click instruction of the user.

[0282] Step 804: The cloud device receives this image processing request and determines the image to be processed.

[0283] Step 806: The cloud device inputs the image to be processed into the graphic and text data processing model to obtain the image features of the image to be processed.

[0284] Among them, the graphic and text data processing model can be deployed on the cloud device. The graphic and text data processing model here is the graphic and text data processing model obtained after the aforementioned training.

[0285] Step 808: The cloud device determines the text features associated with the image features according to the image features, and determines the text information corresponding to the image to be processed according to the text features, and sends the text information to the edge device.

[0286] Specifically, the cloud-side device can determine the text feature with the highest similarity to the image feature from the candidate text feature set in the field of image classification, use this text feature as the text feature associated with the image feature, and determine the text information corresponding to the image to be processed based on this text feature. This text information is the category information of the image to be processed.

[0287] Step 810: The edge-side device displays the text information to the user in the output result display box.

[0288] In addition, the edge-side device can receive the graphic and text data processing model sent by the cloud-side device, and use the graphic and text data processing model to process the image to be processed on the edge-side device.

[0289] The following combines the attached Figure 9 , taking the application of the graphic and text data processing method provided in this specification in graphic and text retrieval as an example, to further illustrate the image data processing method. Among them, Figure 9 shows a schematic diagram of an application scenario of a graphic and text data processing method provided in an embodiment of this specification, specifically including the following steps.

[0290] Step 902: The edge-side device receives the text to be processed input by the user in the text processing request upload box, and receives the upload instruction of the user for this text processing request, and sends the text processing request to the cloud-side device.

[0291] Among them, the edge-side device displays the text processing request upload box to the user. The image processing request carries the text to be processed. The text to be processed is the text with the text retrieval requirement. The text to be processed can be input by the user in the text processing request upload box, or can be selected by the user in the text processing request upload box.

[0292] For example, the text to be processed can be "The clown is holding a red balloon".

[0293] Specifically, the user clicks the control "OK" on the display interface of the edge-side device, and the edge-side device determines the upload instruction of the user for the text processing request based on the click instruction of the user.

[0294] Step 904: The cloud-side device receives this text processing request and determines the text to be processed.

[0295] Step 906: The cloud-side device inputs the text to be processed into the graphic and text data processing model to obtain the text feature of the text to be processed.

[0296] Among them, the graphic and text data processing model can be deployed on the cloud-side device. The graphic and text data processing model here is the graphic and text data processing model obtained after the aforementioned training.

[0297] Step 908: The cloud-side device determines the image features associated with the text features according to the text features, determines the image information corresponding to the text to be processed according to the image features, and sends the image information to the end-side device.

[0298] Specifically, the cloud-side device can determine the image feature with the highest similarity to the text feature from the candidate image feature set in the field of text-image retrieval, use this image feature as the image feature associated with the text feature, and determine the image information corresponding to the text to be processed according to this image feature. This image information is the image described by the text to be processed.

[0299] Continuing with the above example, for the text to be processed "The clown is holding a red balloon", the image information can be an image showing the clown holding a red balloon.

[0300] Step 910: The end-side device renders the image information and displays the image information to the user in the output result display box.

[0301] In addition, the end-side device can receive the text-image data processing model sent by the cloud-side device and use the text-image data processing model to process the text to be processed on the end-side device.

[0302] Corresponding to the above method embodiments, this specification also provides embodiments of a text-image data processing device. Figure 10 The structure diagram of a text-image data processing device provided by an embodiment of this specification is shown. As Figure 10 shown, the device includes:

[0303] The first determination module 1002 is configured to determine the text-image data to be processed;

[0304] The input module 1004 is configured to input the text-image data to be processed into the text-image data processing model to obtain the text-image features corresponding to the text-image data to be processed. The text-image data processing model is the text-image data processing model that meets the training stop condition described in the training method of the text-image data processing model provided by the embodiments of this specification;

[0305] The second determination module 1006 is configured to determine the text-image data processing result corresponding to the text-image data to be processed according to the text-image features.

[0306] Specifically, the text-image data processing model in the above device, as a trained multi-modal representation model, can process text-image data, can be used for multiple different downstream tasks such as image classification, object detection, semantic segmentation, text-image retrieval, and image generation, and has good processing performance.

[0307] Corresponding to the above method embodiments, this specification also provides an embodiment of a text processing method. SeeFigure 11 , Figure 11 shows a flowchart of a text processing method provided according to an embodiment of this specification, which specifically includes the following steps.

[0308] Step 1102: Receive a text processing request sent by a device on the receiving end, where the text processing request carries the text to be processed;

[0309] Step 1104: Input the text to be processed into a graphic and text data processing model to obtain the text features corresponding to the text to be processed, where the graphic and text data processing model is the graphic and text data processing model that meets the training stop condition described in the training method of the graphic and text data processing model provided in the embodiments of this specification;

[0310] Step 1106: Determine the image features corresponding to the text features according to the text features, and determine the image corresponding to the image features;

[0311] Step 1108: Send the image to the device on the end side.

[0312] Specifically, the graphic and text data processing model obtained through model training can be used as a pre-model for graphic and text data processing to process images or texts. Taking the processing of texts as an example for illustration. Specifically, the device on the cloud side can receive a text processing request sent by a user through the device on the end side, determine the text to be processed carried in the text processing request, input the text to be processed into the trained graphic and text data processing model, and obtain the text features corresponding to the text to be processed. According to the text features, determine the image features corresponding to the text features, and determine the image corresponding to the image features. It can be understood that this image is the image corresponding to the text to be processed. And send this image to the device on the end side, and the device on the end side renders the image to display it to the user.

[0313] During specific implementation, when determining the image features corresponding to the text features according to the text features and determining the image corresponding to the image features, it can be implemented by using a trained feature processing model, that is, inputting the text features into the feature processing model and obtaining the image output by the feature processing model.

[0314] In another embodiment of this specification, the candidate image feature set corresponding to the text processing task may also be determined according to the text processing task. When determining the image feature corresponding to the text feature according to the text feature, the similarity between the text feature and each candidate image feature in the candidate image feature set may be calculated, and the candidate image feature with the highest similarity is determined as the image feature corresponding to the text feature. For example, if the text processing task is text processing in the animal field, that is, determining the animal image corresponding to the text according to the text describing animals, then the candidate image feature set corresponding to this text processing task includes candidate image features of a large number of animal images.

[0315] It can be understood that since the text and image data processing model is trained according to the predicted text feature and at least two target text features, and the at least two target text features are obtained based on the initial image feature, the text feature output by the trained text and image data processing model is more similar to the image feature, providing a basis for calculating the similarity between the text feature and the candidate image feature.

[0316] For example, Figure 12 shows a schematic diagram of the processing result of a text processing method provided according to an embodiment of this specification. The text to be processed is "A fine horse is running on the grassland", and the final obtained image is as Figure 12 shown.

[0317] In summary, the above method can realize generating a corresponding image based on the text and is applicable to text processing tasks in different downstream fields, having applicability.

[0318] Corresponding to the above method embodiment, this specification also provides an embodiment of a text processing device. Figure 13 shows a schematic structural diagram of a text processing device provided according to an embodiment of this specification. As Figure 13 shown, the device includes:

[0319] A receiving module 1302, configured to receive a text processing request sent by a terminal device, where the text processing request carries the text to be processed;

[0320] An input module 1304, configured to input the text to be processed into a text and image data processing model to obtain the text feature corresponding to the text to be processed, where the text and image data processing model is the text and image data processing model that meets the training stop condition described in the training method of the text and image data processing model provided in the embodiment of this specification;

[0321] A determining module 1306, configured to determine the image feature corresponding to the text feature according to the text feature, and determine the image corresponding to the image feature;

[0322] A sending module 1308, configured to send the image to the terminal device.

[0323] In summary, the above device can generate a corresponding image based on text and is applicable to text processing tasks in different downstream fields, having applicability.

[0324] The above is a schematic solution of a text processing device according to this embodiment. It should be noted that the technical solution of this text processing device and the technical solution of the above text processing method belong to the same concept. For the details not described in detail in the technical solution of the text processing device, reference can be made to the description of the technical solution of the above text processing method.

[0325] Figure 14 A structural block diagram of a computing device 1400 according to an embodiment of this specification is shown. The components of the computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 through a bus 1430, and a database 1450 is used to store data.

[0326] The computing device 1400 further includes an access device 1440, which enables the computing device 1400 to communicate via one or more networks 1460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1440 may include one or more of any type of wired or wireless network interfaces (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0327] In an embodiment of the present application, the above components of the computing device 1400 and Figure 14Other components not shown may also be connected to each other, for example, via a bus. It should be understood that Figure 14 The block diagram of the computing device shown is for illustrative purposes only and is not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed.

[0328] The computing device 1400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.) or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC, Personal Computer). The computing device 1400 can also be a mobile or stationary server.

[0329] Among them, the processor 1420 is used to execute the following computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0330] The above is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above method belong to the same concept. For the details not described in the technical solution of the computing device, reference can be made to the description of the technical solution of the above method.

[0331] An embodiment of this specification also provides a VR / AR device, including:

[0332] A memory and a processor;

[0333] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above graphic and text data processing method are implemented.

[0334] Specifically, when a user wears the VR / AR device, the VR / AR device can use the above graphic and text data processing method to process the input text, obtain the image corresponding to the text, and display the image to the user to ensure the user experience.

[0335] An embodiment of this specification also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0336] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above method belong to the same concept. For the details not described in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above method.

[0337] An embodiment of this specification also provides a computer program. When the computer program is executed on a computer, the computer is made to execute the steps of the above method.

[0338] The above is a schematic solution of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above method belong to the same concept. For the details not described in the technical solution of the computer program, reference can be made to the description of the technical solution of the above method.

[0339] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0340] The computer instructions include computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0341] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for the embodiments of this specification.

[0342] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0343] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The optional embodiments do not elaborate on all details and do not limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of the present specification. The present specification selects and specifically describes these embodiments to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is only limited by the claims and their full scope and equivalents.

Claims

1. A feature generation method, comprising: Determining a pair of image - text samples, where the pair of image - text samples includes an image sample and a text sample corresponding to the image sample; Determining an initial image feature of the image sample and an initial text feature of the text sample; Inputting the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is a diffusion model pre - trained based on a first pair of image - text features, and the first feature generator is used to generate the at least two target text features associated with the initial image feature; Inputting the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is a diffusion model pre - trained based on a second pair of image - text features, and the second feature generator is used to generate the at least two target image features associated with the initial text feature; Using the at least two target text features and the at least two target image features as training labels to train a model for processing image - text data.

2. The method according to claim 1, wherein the step of inputting the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features includes: Inputting the initial image feature as an adjustment condition and at least two random first features as objects to be adjusted into the first feature generator, where the first feature generator is a diffusion model; In the first feature generator, adjusting the at least two random first features according to the initial image feature to obtain at least two target text features.

3. The method according to claim 1, wherein the step of inputting the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features includes: Inputting the initial text feature as an adjustment condition and at least two random second features as objects to be adjusted into the second feature generator, where the second feature generator is a diffusion model; In the second feature generator, adjusting the at least two random second features according to the initial text feature to obtain at least two target image features.

4. The method according to claim 1, after obtaining the at least two target image features, further comprising: Inputting the image sample into the model for processing image - text data to obtain a predicted image feature; Training the model for processing image - text data according to the predicted image feature and the at least two target image features until a model for processing image - text data that meets the training stop condition is obtained.

5. The method according to claim 4, wherein the step of training the model for processing image - text data according to the predicted image feature and the at least two target image features until a model for processing image - text data that meets the training stop condition is obtained includes: ​ Determine a first target model loss value according to the at least two first model loss values; Train the graphic and text data processing model according to the first target model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

6. The method according to claim 4, after obtaining the at least two target text features, further comprising: Input the text sample into the graphic and text data processing model to obtain predicted text features; Train the graphic and text data processing model according to the predicted text features and the at least two target text features until a graphic and text data processing model that meets the training stop condition is obtained.

7. The method according to claim 6, wherein the training the graphic and text data processing model according to the predicted text features and the at least two target text features until a graphic and text data processing model that meets the training stop condition comprises: Calculate at least two second model loss values according to each target text feature among the predicted text features and the at least two target text features; Determine a second target model loss value according to the at least two second model loss values; Train the graphic and text data processing model according to the second target model loss value until a graphic and text data processing model that meets the training stop condition is obtained.

8. The method according to claim 1, before inputting the initial image features and at least two random first features into the first feature generator, further comprising: Determine a first graphic and text feature pair and a random first feature sample, wherein the first graphic and text feature pair includes a first image feature and a first text feature corresponding to the first image feature; Use the first image feature and the random first feature sample as training samples, and use the first text feature as a training label to train the first feature generator until a first feature generator that meets the training stop condition is obtained.

9. The method according to claim 1, before inputting the initial text features and at least two random second features into the second feature generator, further comprising: Determine a second graphic and text feature pair and a random second feature sample, wherein the second graphic and text feature pair includes a second image feature and a second text feature corresponding to the second image feature; Use the second text feature and the random second feature sample as training samples, and use the second image feature as a training label to train the second feature generator until a second feature generator that meets the training stop condition is obtained.

10. A training method for a graphic and text data processing model, comprising: Determine a graphic and text sample pair, wherein the graphic and text sample pair includes an image sample and a text sample corresponding to the image sample; Determine an initial image feature of the image sample and an initial text feature of the text sample; Input the image sample and the text sample into the graphic and text data processing model to obtain predicted image features and predicted text features; Input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, where the first feature generator is a diffusion model pre-trained based on first text-image feature pairs, and the first feature generator is used to generate the at least two target text features associated with the initial image features; Input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, where the second feature generator is a diffusion model pre-trained based on second text-image feature pairs, and the second feature generator is used to generate the at least two target image features associated with the initial text features; Use the at least two target text features and the at least two target image features as training labels, and train the text-image data processing model according to the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a text-image data processing model that meets the training stop condition is obtained.

11. A text-image data processing method, comprising: Determine the text-image data to be processed; Input the text-image data to be processed into a text-image data processing model to obtain the text-image features corresponding to the text-image data to be processed, where the text-image data processing model is the text-image data processing model that meets the training stop condition described in claim 10; Determine the text-image data processing result corresponding to the text-image data to be processed according to the text-image features.

12. A text processing method, applied to a cloud-side device, comprising: Receive a text processing request sent by a terminal-side device, where the text processing request carries the text to be processed; Input the text to be processed into a text-image data processing model to obtain the text features corresponding to the text to be processed, where the text-image data processing model is the text-image data processing model that meets the training stop condition described in claim 10; Determine the image features corresponding to the text features according to the text features, and determine the image corresponding to the image features; Send the image to the terminal-side device.

13. A computing device, comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium that stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

15. A computer program product, characterized in that, Including computer instructions, and when the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

Images