Feature generation method
By extracting and generating target features, expanding the training data set of the graphical and text data processing model, the problem of insufficient data diversity in the training of existing multimodal models is solved, and the training efficiency and performance of the model are improved.
Patent Information
- Application Number
- PCT/SG2024/050204
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-14
AI Technical Summary
During the training process, existing multimodal models are difficult to obtain effective samples covering different semantic contents due to the messy data and lack of organizational structure, resulting in poor training efficiency and performance.
By determining the graphic sample pair, extracting the initial features and using the feature generator to generate target features, expanding the training data set, ensuring that the model can learn more semantic content.
The training efficiency and performance of multimodal models are improved, ensuring that the model can cover more semantic content and generate diverse training data.
Smart Images

Figure SG2024050204_14112024_PF_FP_ABST
Abstract
Description
[0001] Cross-Reference to Feature Generation Method This disclosure claims priority to Chinese patent application No. 202310526214.E, filed with the China Patent Office on May 10, 2023, entitled "Feature Generation Method," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to feature generation methods. Background: Graphical data processing models, as multimodal models, can be used to process various types of data, such as text and images. Modalities can be understood as different ways of representing data; for example, text and images are different modalities. Multimodal models can convert input from one modality into output from another modality and can be applied to downstream tasks such as image classification, image detection and segmentation, graphic and text retrieval, and image generation. However, while current multimodal model training samples (i.e., image and text samples) are numerous, the data is disorganized and unstructured. Most of the data describes the same semantic content, and it is difficult to obtain the large number of valid samples covering different semantic content required for model training. This lack of diversity leads to poor training efficiency and poor performance of the trained multimodal models. In view of this, embodiments of the present disclosure provide a feature generation method. One or more embodiments of the present disclosure also relate to a feature generation device, a training method for a graphic and text data processing model, a training device for a graphic and text data processing model, a graphic and text data processing method, a graphic and text data processing device, a text processing method, a text processing device, a VR / AR device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, a feature generation method is provided, comprising: determining an image-text sample pair, wherein the image-text sample pair comprises an image sample and a text sample corresponding to the image sample; determining initial image features of the image sample and initial text features of the text sample; inputting the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; and inputting the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair.According to a second aspect of an embodiment of the present disclosure, a feature generation device is provided, comprising: a first determination module, configured to determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; a second determination module, configured to determine initial image features of the image sample and initial text features of the text sample; a first input module, configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; and a second input module, configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair. According to a third aspect of an embodiment of the present disclosure, a method for training a graphic-text data processing model is provided, comprising: determining a graphic-text sample pair, wherein the graphic-text sample pair includes an image sample and a text sample corresponding to the image sample; determining initial image features of the image sample and initial text features of the text sample; inputting the image sample and the text sample into the graphic-text data processing model to obtain predicted image features and predicted text features; inputting the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first graphic-text feature pair; inputting the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second graphic-text feature pair; and training the graphic-text data processing model based on the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a graphic-text data processing model that meets a training stop condition is obtained.According to a fourth aspect of an embodiment of the present disclosure, a training device for a graph-text data processing model is provided, comprising: a first determination module, configured to determine a graph-text sample pair, wherein the graph-text sample pair includes an image sample and a text sample corresponding to the image sample; a second determination module, configured to determine initial image features of the image sample and initial text features of the text sample; a third determination module, configured to input the image sample and the text sample into the graph-text data processing model to obtain predicted image features and predicted text features: a first input module, configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first graph-text feature pair; a second input module, configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second graph-text feature pair; a training module, configured to train a model based on the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features. The graphic data processing model is trained until a graphic data processing model that satisfies a training stop condition is obtained. According to a fifth aspect of an embodiment of the present disclosure, a graphic data processing method is provided, comprising: determining graphic data to be processed; inputting the graphic data to be processed into the graphic data processing model to obtain graphic features corresponding to the graphic data to be processed, wherein the graphic data processing model is the graphic data processing model that satisfies the training stop condition as described in the method for training a graphic data processing model provided in an embodiment of the present disclosure; and determining a graphic data processing result corresponding to the graphic data to be processed based on the graphic features. According to a sixth aspect of an embodiment of the present disclosure, a graphic data processing device is provided, comprising: a first determination module, configured to determine graphic data to be processed; an input module, configured to input the graphic data to be processed into a graphic data processing model to obtain graphic features corresponding to the graphic data to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies a training stop condition as described in the training method for a graphic data processing model provided in an embodiment of the present disclosure; and a second determination module, configured to determine a graphic data processing result corresponding to the graphic data to be processed based on the graphic features.According to a seventh aspect of an embodiment of the present disclosure, a text processing method is provided, which is applied to a cloud-side device, and includes: receiving a text processing request sent by an end-side device, wherein the text processing request carries text to be processed; inputting the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed, wherein the graphic data processing model is a graphic data processing model that meets the training stop condition as described in the training method for the graphic data processing model provided in an embodiment of the present disclosure; determining, based on the text features, image features corresponding to the text features, and determining an image corresponding to the image features; and sending the image to the end-side device. According to an eighth aspect of the embodiments of the present disclosure, a text processing apparatus is provided, applied to a cloud-side device, comprising: a receiving module configured to receive a text processing request sent by a client-side device, wherein the text processing request carries text to be processed; an input module configured to input the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies a training stop condition as described in the training method for a graphic data processing model provided in the embodiments of the present disclosure; a determination module configured to determine, based on the text features, image features corresponding to the text features and an image corresponding to the image features; and a sending module configured to send the image to the client-side device. According to a ninth aspect of the embodiments of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory being configured to store computer-executable instructions, and the processor being configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the above-described method. According to a tenth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by a processor, the instructions implement the steps of the aforementioned method. According to an eleventh aspect of the embodiments of the present disclosure, a computer program is provided. When executed on a computer, the computer program causes the computer to perform the steps of the aforementioned method. According to a twelfth aspect of the embodiments of the present disclosure, a VR / AR device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When executed by the processor, the computer-executable instructions implement the steps of the aforementioned method for processing graphic and text data.An embodiment of the present disclosure provides a feature generation method, which includes determining an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; determining initial image features of the image sample and initial text features of the text sample; inputting the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; and inputting the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair. The above method extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features, and extracts initial text features from text samples in the image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of an image-text data processing model. This achieves the generation of multiple training labels based on one image-text sample pair, ensures the diversity of training data for the image-text data processing model, expands the semantic content that can be learned beyond the image-text sample pair (i.e., the original training data) during model training, meets the requirement for a large number of effective samples covering different semantic content during model training, and thereby improves model training efficiency and model performance.BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a schematic diagram of an application scenario of a feature generation method provided by an embodiment of the present disclosure; Figure 2 is a flow chart of a feature generation method provided by an embodiment of the present disclosure; Figure 3 is a flow chart of a processing process of a feature generation method provided by an embodiment of the present disclosure; Figure 4 is a schematic diagram of the structure of a feature generation device provided by an embodiment of the present disclosure; Figure 5 is a flow chart of a training method for a graphic data processing model provided by an embodiment of the present disclosure; Figure 6 is a schematic diagram of the structure of a training device for a graphic data processing model provided by an embodiment of the present disclosure; Figure 7 is a flow chart of a graphic data processing method provided by an embodiment of the present disclosure; Figure 8 is a schematic diagram of an application scenario of a graphic data processing method provided by an embodiment of the present disclosure; Figure 9 is a schematic diagram of an application scenario of a graphic data processing method provided by an embodiment of the present disclosure; Figure 10 is a schematic diagram of the structure of a graphic data processing device provided by an embodiment of the present disclosure; Figure 11 is a flow chart of a text processing method provided by an embodiment of the present disclosure; Figure 12 is a schematic diagram of the processing results of a text processing method provided by an embodiment of the present disclosure; Figure 13 is a schematic diagram of the structure of a text processing device provided by an embodiment of the present disclosure; Figure 14 is a structural block diagram of a computing device provided by an embodiment of the present disclosure. The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below. The terminology used in one or more embodiments of the present disclosure is for the purpose of describing specific embodiments and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that while the terms "first," "second," and so on may be used in one or more embodiments of the present disclosure to describe various information, such information should not be limited to these terms. These terms are used to distinguish information of the same type from one another. For example, "first" could be referred to as "second," and similarly, "second" could be referred to as "first" without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the word "if" as used herein may be interpreted as "when" or "when" or "in response to determining."Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in one or more embodiments of this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or deny such data. First, the terms used in one or more embodiments of this disclosure are explained.
[0002] VR: Virtual Reality, virtual reality technology.
[0003] AR: Augmented Reality. This disclosure provides a feature generation method and also relates to a feature generation device, a training method for a graphic and text data processing model, a training device for a graphic and text data processing model, a graphic and text data processing method, a graphic and text data processing device, a text processing method, a text processing device, a VR / AR device, a computing device, and a computer-readable storage medium. Each of these features is described in detail in the following embodiments. Referring to FIG1 , FIG1 illustrates an application scenario of a feature generation method according to one embodiment of this disclosure. FIG1 includes a cloud-side device 102, which can be understood as a cloud server. Alternatively, in another implementation, cloud-side device 102 can be replaced by a physical server. In a specific implementation, cloud-side device 102 can determine a graphic and text sample pair. Furthermore, feature extraction is performed on the image sample and text sample in the graphic and text sample pair to obtain initial image features and initial text features. At least two random first features and at least two random second features are obtained through random sampling. The initial image features and the at least two random first features are input into a first feature generator deployed on the cloud-side device 102 to obtain at least two target text features. The initial text features and the at least two random second features are input into a second feature generator deployed on the cloud-side device 102 to obtain at least two target image features. This generation process can be continued to obtain multiple image-text sample pairs, and at least two target text features and at least two image features corresponding to each image-text sample pair. A training dataset can be constructed based on these image-text sample pairs, at least two target text features, and at least two target image features. This facilitates subsequent model training using this training data, thereby expanding and enriching the original training data. Referring to Figure 2, a flow chart of a feature generation method provided according to an embodiment of the present disclosure is shown, specifically including the following steps: Step 202: Determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample. Specifically, the feature generation method provided in this embodiment of the present disclosure can be applied to a server. The image-text sample pair can be understood as any image-text sample pair obtained from a training dataset, which may include multiple image-text sample pairs. The image-text sample pair can serve as training data for an image-text data processing model. The image-text data processing model can be configured to process images or text to obtain text information corresponding to the image or image information corresponding to the text.For example, an image sample might be an image depicting a "dog running on the grass." The text sample corresponding to this image sample could be the text description of the image sample, "dog running on the grass," or the category information of the image sample, "animal." Specifically, the image and text samples can be determined based on model training requirements. For example, to enable a graphic data processing model to learn the ability to classify input images, the text sample could be a sample of the image sample's category information. To enable a graphic data processing model to learn the ability to describe input images, the text sample could be a sample of the image sample's text description information. Specifically, the training dataset can be determined based on the type of processing task for the graphic data processing model. For example, if the graphic data processing model needs to classify images depicting animals, a training dataset in the animal domain can be selected, which includes images of animals and their corresponding category information. If the graphic data processing model needs to handle other tasks, the corresponding training dataset can be determined based on the actual task requirements, thereby determining the graphic and text sample pairs from the training dataset. It is understood that, as a multimodal model, the image-text data processing model can process multimodal data. That is, the image-text data processing model can process both image and text inputs. To enable the image-text data processing model to process images, the image samples in the image-text sample pairs can be used as training samples, and the text samples in the image-text sample pairs can be used as training labels for training. To enable the image-text data processing model to process text, the text samples in the image-text sample pairs can be used as training samples, and the image samples in the image-text sample pairs can be used as training labels for training. Furthermore, the image-text data processing model can also process other types of data, such as video data and / or audio data. Accordingly, the training data for the image-text data processing model also includes video samples and / or audio samples. Step 204: Determine initial image features of the image sample and initial text features of the text sample. Specifically, after determining the image-text sample pair, the initial image features of the image sample in the image-text sample pair and the initial text features of the text sample in the image-text sample pair can be determined. The initial image features can be understood as the initial image feature vector of the image sample, and the initial text features can be understood as the initial image feature vector of the text sample.It is understandable that the features described in the embodiments of the present disclosure, such as initial text features, initial image features, target text features, target image features, random first features, random second features, etc., can all be understood as feature vectors. In specific implementations, a feature encoder can be used to extract initial image features of image samples and initial text features of text samples. An image feature encoder can be used to extract initial image features of image samples, while a text feature encoder can be used to extract initial image features of text samples. In practical applications, a graphic data processing model may include an image feature encoder and a text feature encoder. Image samples and text samples can be input into the graphic data processing model to obtain initial image features output by the image feature encoder and initial text features output by the text feature encoder in the graphic data processing model. Step 206: Input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first graphic feature pair. Specifically, after determining the initial image features, the initial image features and at least two random first features can be input into a first feature generator to obtain at least two target text features. The random first features can be understood as feature vectors obtained through random sampling. The first feature generator can be understood as a feature generator for generating associated text features based on the input image features. In practical applications, the first feature generator can be a pre-trained diffusion model. In specific implementations, the initial image features and each of the at least two random first features can be input into the first feature generator to obtain at least two target text features. It is understood that the vector values of each random first feature can be different. For example, three random first features X1, X2, and X3 are obtained through random sampling. The initial image features and the random first feature X1 are input into the first feature generator to obtain target text feature Y1; the initial image features and the random first feature X2 are input into the first feature generator to obtain target text feature Y2; and the initial image features and the random first feature X3 are input into the first feature generator to obtain target text feature Y3. Based on this, three target text features Y1, Y2 and Y3 can be obtained according to the initial image features and the three random first features X1, X2 and X3.In practical applications, to enable a graphic data processing model to process text, and to ensure that, after inputting text into the graphic data processing model, the model can determine the image information corresponding to the text based on the text features output by the graphic data processing model, the model is typically trained using input text samples as training samples and the image samples corresponding to the text samples as training labels. However, this training method results in a single training data set for the graphic data processing model, preventing it from learning richer knowledge. Therefore, the image samples can be further processed to generate at least two target text features based on the initial image features of the image samples, thereby enriching the training data for the graphic data processing model. Specifically, the initial image features and at least two random first features can be input into a first feature generator to generate the at least two target text features. In order to enable the first feature generator to generate target text features based on the input initial image features, the first feature generator needs to be pre-trained. This is specifically implemented as follows: Before inputting the initial image features and at least two random first features into the first feature generator, the process further includes: determining a first image-text feature pair and a random first feature sample, wherein the first image-text feature pair includes a first image feature and a first text feature corresponding to the first image feature; using the first image feature and the random first feature sample as training samples, and the first text feature as a training label, training the first feature generator until a first feature generator that meets the training stop condition is obtained. The first image feature can be obtained based on the image sample in the image-text sample pair, and the first text feature can be obtained based on the text sample in the image-text sample pair. The image-text sample pair here can be the same as or different from the image-text sample pair determined in step 202. The random first feature sample can be a first feature vector sample obtained through random sampling. Specifically, the training of the first feature generator can be based on a diffusion model, that is, training is performed based on the diffusion model until a first feature generator that meets the training stop condition is obtained. Accordingly, the at least two target text features output by the first feature generator are diffusion features, which achieve diffusion and enrichment of the initial image features, thereby enriching the training data. The training termination condition can be understood as the model loss value being less than a preset threshold and / or the number of training rounds reaching a preset number. In a specific embodiment provided in the present disclosure, terminating the training of the first feature generator when the model loss value is less than a preset threshold is used as an example. The preset threshold is 0.3. When the calculated model loss value is less than 0.3, the training of the first feature generator is considered complete.In another specific embodiment provided by the present disclosure, training of the first feature generator is terminated based on a preset number of training rounds, for example, where the preset number of training rounds is 10. When the sample data reaches 10 training rounds, the first feature generator is considered trained. In yet another specific embodiment provided by the present disclosure, two training termination conditions are set: a preset loss value threshold and a preset number of training rounds. The model loss value and the number of training rounds are monitored simultaneously. Training of the first feature generator is considered completed when either or both the model loss value and the number of training rounds meet the training termination conditions. Specifically, training of the first feature generator is considered completed when the model loss value is less than the preset loss value threshold, the number of training rounds reaches the preset number, or the model loss value is less than the preset loss value threshold, and the number of training rounds reaches the preset number. In summary, training the first feature generator facilitates the subsequent generation of at least two target text features based on initial image features, thereby enriching and disseminating training data. In a specific implementation, the first feature generator can use the initial image features to adjust the random first features. The specific implementation method is as follows: Inputting the initial image features and at least two random first features into the first feature generator to obtain at least two target text features includes: inputting the initial image features as adjustment conditions and the at least two random first features as objects to be adjusted into the first feature generator; and, in the first feature generator, adjusting the at least two random first features based on the initial image features to obtain at least two target text features. Specifically, the first feature generator can gradually denoise the random first features input as the objects to be adjusted based on the initial image features input as the adjustment conditions, and output the target text features. It will be appreciated that, due to different values of the random first features, different target text features will be outputted through gradual denoising of the random first features. Continuing with the above example, the initial image features are used as adjustment conditions, and the random first feature X1 is input into the first feature generator as the object to be adjusted. In this first feature generator, the random first feature X1 is gradually denoised based on the initial image features to obtain the target text feature Y1. Similarly, target text features Y2 and Y3 are output by the first feature generator. In summary, by utilizing the first feature generator and inputting at least two different random first features, at least two target text features are obtained, thereby diversifying and enriching the initial image features and providing more training data for model training.In a specific implementation, when using the at least two obtained target text features for model training, the at least two target text features can be used as training labels for model training. The specific implementation method is as follows: After obtaining the at least two target text features, the method further includes: inputting the text sample into a graphic data processing model to obtain predicted text features; and training the graphic data processing model based on the predicted text features and the at least two target text features until a graphic data processing model that meets a training stop condition is obtained. Specifically, the graphic data processing model includes a text feature encoder and a text feature mapping layer. The text sample is input into the text feature encoder in the graphic data processing model to obtain initial text features output by the text feature encoder. The initial text features are then input into the text feature mapping layer to obtain predicted text features output by the text feature mapping layer. In one embodiment of the present disclosure, to enrich the semantic content of the predicted text features output by the image processing model, the text feature mapping layer can perform linear mapping on the initial text features to obtain the predicted text features. It is understandable that when training a graph-text data processing model, graph-text sample pairs can be determined, and text samples from the graph-text sample pairs can be input into the graph-text data processing model to obtain predicted text features. The graph-text data processing model is then trained based on the predicted text features and at least two target text features. The steps of determining graph-text sample pairs are continued, and so on, until a graph-text data processing model that meets the training termination criteria is obtained. In summary, by utilizing the obtained at least two target text features for model training, more effective training data is expanded beyond the original training data, allowing the model training data to cover more semantic content, thereby ensuring the richness of the semantic representations learned by the model. During specific implementation, a second model loss value can be calculated based on the predicted text feature and each target text feature, thereby training the image-text data processing model. The specific implementation method is as follows: the image-text data processing model is trained based on the predicted text feature and the at least two target text features until a image-text data processing model that meets the training stop condition is obtained, including: calculating at least two second model loss values based on the predicted text feature and each target text feature of the at least two target text features; determining a second target model loss value based on the at least two second model loss values; and training the image-text data processing model based on the second target model loss value until a image-text data processing model that meets the training stop condition is obtained.Specifically, at least two second model loss values can be calculated based on the predicted text features and each of the at least two target text features. The at least two second model loss values can then be processed, for example, by adding the at least two second model loss values or performing a weighted sum of the at least two second model loss values to obtain a second target model loss value. The image-text data processing model is trained based on the second target model loss values until a image-text data processing model that meets a training stop condition is obtained. In the disclosed embodiments, when training the model based on the predicted text features and the at least two target text features, the training objective is to reduce the distance between the predicted text features and each target text feature. Reducing the distance can be understood as maximizing the similarity between the predicted text features and each target text feature, thereby reducing the distance between the predicted text features and each target text feature. The distance between the predicted text features and the target text features can be obtained by subtracting the predicted text feature vector from the target text feature vector. Furthermore, when calculating the second model loss value, the predicted text features and the target text features can be input into a target loss function to obtain the second model loss value output by the target loss function. In practical applications, an appropriate target loss function can be selected based on the actual application scenario. The predicted text features and target text features can be input into the target loss function for calculation to obtain a second model loss value. The target loss function can be, for example, a cross entropy loss function, an absolute value loss function, a square loss function, a 0-1 loss function, etc. The present disclosure does not limit the choice of target loss function. Continuing with the above example, a text sample can be input into the image-text data processing model to obtain predicted text features. Based on the predicted text features and the target text features Y1, a second model loss value a1 is calculated; based on the predicted text features and the target text features Y2, a second model loss value a2 is calculated; and based on the predicted text features and the target text features Y3, a second model loss value a3 is calculated. Based on the second model loss values a1, a2, and a3, a second target model loss value a is calculated. The image-text data processing model is trained based on the second target model loss value a until a training stop condition is obtained. In summary, by using the second target model loss value calculated using the at least two target text features obtained to train the image-text data processing model, the predicted text features can learn the representations of at least two target text features, and thus learn the representations of the initial image features, further achieving the training goal of narrowing the gap between the predicted text features and the image features, thereby improving the processing performance of the image-text data processing model.Step 208: Input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair. Specifically, after obtaining the initial text features, the initial text features and at least two random second features can be input into the second feature generator to obtain at least two target image features as output by the second feature generator. The random second features can be understood as feature vectors obtained through random sampling. The random second features and random first features can be the same or different. The second feature generator can be understood as a feature generator for generating associated image features based on the input text features. In practical applications, the second feature generator can be a pre-trained diffusion model. The second image-text feature pair can be the same or different from the first image-text feature pair. It is understood that the vector values of each random second feature can be different. For example, after random sampling, three random second features P1, P2, and P3 are obtained. The initial text features and the random second feature P1 are input into the second feature generator to obtain the target image feature Q1. The initial text features and the random second feature P2 are input into the second feature generator to obtain the target image feature Q2. The initial text features and the random second feature P3 are input into the second feature generator to obtain the target image feature Q3. Based on this, the three target image features Q1, Q2, and Q3 can be obtained based on the initial text features and the three random second features P1, P2, and P3. In one embodiment of the present disclosure, the first feature generator and the second feature generator are diffusion models. In practical applications, to enable a graphic data processing model to process images, such that after an image is input into the model, the model can determine the text information corresponding to the image based on the image features output by the model, the model is typically trained using input image samples as training samples and the text samples corresponding to the image samples as training labels. However, this training approach results in a single training data set for the graphic data processing model, preventing the model from learning richer knowledge. Therefore, further processing can be performed on the text samples to generate at least two target image features based on the initial text features of the text samples, thereby enriching the training data for the graphic data processing model. Specifically, the initial text features and at least two random second features can be input into a second feature generator to generate at least two target image features.In order to enable the second feature generator to generate target image features based on the input initial text features, the second feature generator needs to be pre-trained. This is specifically implemented as follows: Before inputting the initial text features and at least two random second features into the second feature generator, the process further includes: determining a second image-text feature pair and a random second feature sample, wherein the second image-text feature pair includes a second image feature and a second text feature corresponding to the second image feature; using the second text feature and the random second feature sample as training samples and the second image feature as training labels, the second feature generator is trained until a second feature generator that meets the training stop condition is obtained. The second image feature can be obtained based on the image sample in the image-text sample pair, and the second text feature can be obtained based on the text sample in the image-text sample pair. The image-text sample pair here can be the same as or different from the image-text sample pair determined in step 202. The random second feature sample can be a second feature vector sample obtained through random sampling. Specifically, the second feature generator can be trained based on a diffusion model, that is, training is performed on the basis of the diffusion model until a second feature generator that meets the training stop condition is obtained. Accordingly, the at least two target image features output by the second feature generator are diffusion features, which achieve diffusion and enrichment of the initial text features, thereby enriching the training data. In summary, training the second feature generator facilitates the subsequent generation of at least two target image features based on the initial text features, thereby enriching and diversifying the training data. In a specific implementation, the second feature generator can use the initial text features to adjust the random second features. This is specifically implemented as follows: Inputting the initial text features and at least two random second features into the second feature generator to obtain at least two target image features includes: inputting the initial text features as adjustment conditions and the at least two random second features as objects to be adjusted into the second feature generator; and, in the second feature generator, adjusting the at least two random second features based on the initial text features to obtain at least two target image features. Specifically, the second feature generator can gradually denoise the random second features input as objects to be adjusted based on the initial text features input as adjustment conditions, thereby outputting the target image features. It can be understood that, due to the different values of the random second features, the output target image features are also different when the random second features are gradually denoised.Continuing with the above example, the initial text features are used as adjustment conditions, and the random second feature P1 is input into the second feature generator as the object to be adjusted. Within this second feature generator, the random second feature P1 is gradually denoised based on the initial text features to obtain the target image feature Q1. Similarly, target image features Q2 and Q3 are output by the second feature generator. In summary, by utilizing the second feature generator and inputting at least two different random second features, at least two target image features are obtained, thereby diversifying and enriching the initial text features and providing more training data for model training. In specific implementations, when using the at least two obtained target image features for model training, the at least two target image features can be used as training labels for model training. The specific implementation method is as follows: After obtaining the at least two target image features, the method further includes: inputting the image sample into a graphic data processing model to obtain predicted image features; and training the graphic data processing model based on the predicted image features and the at least two target image features until a graphic data processing model that meets the training stop condition is obtained. Specifically, the image-text data processing model includes an image feature encoder and an image feature mapping layer. Image samples are input into the image feature encoder in the image-text data processing model to obtain initial image features output by the image feature encoder. The initial image features are then input into the image feature mapping layer to obtain predicted image features output by the image feature mapping layer. Similar to the above, the image feature mapping layer can perform feature mapping on the initial image features to obtain predicted image features. It is understood that when training the image-text data processing model, image-text sample pairs can be determined. Image samples from these pairs are input into the image-text data processing model to obtain predicted image features. The image-text data processing model is then trained based on the predicted image features and at least two target image features. The step of determining image-text sample pairs is continued, and so on, until a image-text data processing model that meets the training termination criteria is obtained. In summary, by utilizing the obtained at least two target image features for model training, more effective training data is added beyond the original training data, enabling the model training data to cover more semantic content, thereby ensuring the richness of the semantic representations learned by the model.In a specific implementation, a first model loss value can be calculated based on the predicted image features and each target image feature, thereby training the image-text data processing model. The specific implementation method is as follows: Training the image-text data processing model based on the predicted image features and the at least two target image features until a image-text data processing model that meets a training stop condition is obtained includes: calculating at least two first model loss values based on the predicted image features and each of the at least two target image features; determining a first target model loss value based on the at least two first model loss values; and training the image-text data processing model based on the first target model loss values until a image-text data processing model that meets the training stop condition is obtained. Specifically, at least two first model loss values can be calculated based on the predicted image features and each of the at least two target image features, and the at least two first model loss values can be processed, for example, by adding the at least two first model loss values or performing a weighted sum of the at least two first model loss values to obtain the first target model loss value. The image-text data processing model is trained based on the first target model loss value until a model that meets the training stop condition is obtained. In the disclosed embodiment, when training the model based on the predicted image features and at least two target image features, the training objective is to reduce the distance between the predicted image features and each target image feature. Reducing the distance can be understood as maximizing the similarity between the predicted image features and each target image feature, thereby reducing the distance between the predicted image features and each target image feature. The distance between the predicted image features and the target image features can be obtained by subtracting the predicted image feature vector from the target image feature vector. Furthermore, when calculating the first model loss value, the predicted image features and the target image features can be input into a target loss function to obtain the first model loss value output by the target loss function. In practical applications, an appropriate target loss function can be selected based on the actual application scenario, and the predicted image features and the target image features can be input into the target loss function for calculation to obtain the first model loss value. The target loss function can be, for example, a cross entropy loss function, an absolute value loss function, a square loss function, a 0-1 loss function, etc. The presently disclosed embodiments do not limit the choice of target loss function. Continuing with the above example, an image sample can be input into the image and text data processing model to obtain predicted image features. Based on the predicted image features and the target image features Q1, a first model loss value bl is calculated; based on the predicted image features and the target image features Q2, a first model loss value b2 is calculated; and based on the predicted image features and the target image features Q3, a first model loss value b3 is calculated.A first target model loss value b is calculated based on the first model loss values b1, b2, and b3. The image-text data processing model is trained based on the first target model loss value b until a model that satisfies the training stop condition is obtained. In summary, by using the first target model loss value calculated based on at least two obtained target image features to train the image-text data processing model, the predicted image features can learn the representations of at least two target image features, thereby learning the representations of the initial text features. This further achieves the training goal of narrowing the gap between the predicted image features and the text features, thereby improving the processing performance of the image-text data processing model. In specific implementations, model training can also be performed based on the predicted image features and the predicted text features, with narrowing the gap between the predicted image features and the predicted text features as the training goal. This is specifically implemented as follows: The image-text data processing model is trained based on the predicted image features and the predicted text features until a model that satisfies the training stop condition is obtained. Specifically, a model loss value can be calculated based on the predicted image features and predicted text features, and the image-text data processing model can be trained based on the model loss value until a model that meets the training stopping criteria is obtained. It will be appreciated that when training the image-text data processing model, image-text sample pairs can be determined, and the image and text samples in the image-text sample pairs can be input into the image-text data processing model to obtain predicted image features and predicted text features. The image-text data processing model can then be trained based on the predicted image features and predicted text features. The steps of determining image-text sample pairs are then continued, and so on, until a model that meets the training stopping criteria is obtained. In summary, by training the image-text data processing model using the above method, learned features from two different modalities, image and text, can be aligned, thereby enhancing the representational capabilities of the image-text data processing model. In summary, the above method extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features, and extracts initial text features from text samples in an image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of an image-text data processing model. This achieves the generation of multiple training labels based on one image-text sample pair, ensures the diversity of training data for the image-text data processing model, expands the semantics that can be learned beyond the image-text sample pair (i.e., the original training data) during model training, meets the requirement of a large number of effective samples covering different semantic contents during model training, and thereby improves model training efficiency and model performance.The following, with reference to FIG3 , further illustrates the feature generation method provided by the present disclosure, using its application in model training as an example. FIG3 shows a flowchart of the feature generation method provided by one embodiment of the present disclosure, specifically including the following steps: Step 302: Determine an image-text sample pair, where the image-text sample pair includes an image sample and a text sample corresponding to the image sample. Taking the image-text data processing model trained to process image content description as an example, the determined image sample may be an image of a dog running on grass, and the determined text sample may be the text description "dog running on grass" for the image sample. Step 304: Input the text sample into a text feature encoder within the image-text data processing model to obtain initial text features. Step 306: Input the initial text features into a text feature mapping layer within the image-text data processing model to obtain predicted text features. Specifically, to enhance the semantic content richness of the predicted text features, the text feature mapping layer may perform linear mapping on the initial text features to obtain predicted text features t. Step 308: Input the image sample into the image feature encoder in the image-text data processing model to obtain initial image features. Step 310: Input the initial image features into the image feature mapping layer in the image-text data processing model to obtain predicted image features. Accordingly, to enhance the semantic content richness of the predicted image features, the initial image features can be linearly mapped in the image feature mapping layer to obtain predicted image features v. Step 312: Input the initial image features as adjustment conditions and at least two random first features as objects to be adjusted into a first feature generator to obtain at least two target text features. The first feature generator, acting as an image-to-text feature generator, can be used to generate text features based on the input image features. It can be trained based on a diffusion model. Specifically, the initial image features can be used as adjustment conditions and k randomly sampled random first features can be used as objects to be adjusted (i.e., seeds). These can be input into the image-to-text feature generator to obtain k target text features t1′, t2′, and tk′. Step 314: Input the initial text features as adjustment conditions and at least two random second features as objects to be adjusted into a second feature generator to obtain at least two target image features. The second feature generator, acting as a text-to-image feature generator, can be used to generate image features based on the input text features and can be trained based on a diffusion model.Specifically, the initial text features can be used as adjustment conditions, and k randomly sampled second features can be used as objects to be adjusted (i.e., seeds). The text can be input into an image feature generator to obtain k target image features v1′, v2′, and vk′. Step 316: The image-text data processing model is trained based on the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a model that meets the training stop condition is obtained. Specifically, during training, the image-text data processing model can be trained based on the predicted text features t and the predicted image features v, with the goal of narrowing the distance between the predicted text features t and the predicted image features v. Alternatively, a training goal may be to narrow the distance between the predicted image feature v and k target image features vl', v2' > vk'. Based on the predicted image feature v and the target image feature vl', based on the predicted image feature v and the target image feature v2', ..., based on the predicted image feature v and the target image feature vk', k first model loss values are calculated, the k first model loss values are added together to obtain a first target model loss value, and the graphic data processing model is trained based on the first target model loss value. Alternatively, a training goal may be to narrow the distance between the predicted text feature t and the k target text features t1', t2' > tk'. Based on the predicted text feature t and the target text feature t1', and based on the predicted text feature t and the target text feature t2' > based on the predicted text feature t and the target text feature tk', k second model loss values are calculated. These k second model loss values are added together to obtain a second target model loss value. The image-text data processing model is then trained based on the second target model loss value until a image-text data processing model that meets the training stop condition is obtained.In summary, the above method extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features. It also extracts initial text features from text samples in an image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of an image-text data processing model. This enables the generation of multiple training labels from a single image-text sample pair, ensures the diversity of training data for the image-text data processing model, expands the semantic content that can be learned during model training beyond the image-text sample pair (i.e., the original training data), meets the requirement for a large number of valid samples covering diverse semantic content during model training, and thereby improves model training efficiency and performance. Corresponding to the above method embodiments, the present disclosure also provides an embodiment of a feature generation device. FIG4 shows a schematic structural diagram of a feature generation device provided in one embodiment of the present disclosure. As shown in Figure 4, the device includes: a first determination module 402, configured to determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; a second determination module 404, configured to determine the initial image features of the image sample and the initial text features of the text sample; a first input module 406, configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; a second input module 408, configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair. In an optional embodiment, the first input module 406 is further configured to: input the initial image features as adjustment conditions and at least two random first features as objects to be adjusted into a first feature generator; and the first feature generator adjusts the at least two random first features based on the initial image features to obtain at least two target text features. In an optional embodiment, the second input module 408 is further configured to: input the initial text features as adjustment conditions and at least two random second features as objects to be adjusted into a second feature generator; and the second feature generator adjusts the at least two random second features based on the initial text features to obtain at least two target image features.In an optional embodiment, the apparatus further includes a training module configured to: input the image sample into the image-text data processing model to obtain predicted image features; and train the image-text data processing model based on the predicted image features and the at least two target image features until a image-text data processing model that meets a training stop condition is obtained. In an optional embodiment, the training module is further configured to: calculate at least two first model loss values based on the predicted image features and each of the at least two target image features; determine a first target model loss value based on the at least two first model loss values; and train the image-text data processing model based on the first target model loss values until a image-text data processing model that meets a training stop condition is obtained. In an optional embodiment, the training module is further configured to: input the text sample into the image-text data processing model to obtain predicted text features; and train the image-text data processing model based on the predicted text features and the at least two target text features until a image-text data processing model that meets a training stop condition is obtained. In an optional embodiment, the training module is further configured to: calculate at least two second model loss values based on the predicted text features and each of the at least two target text features; determine a second target model loss value based on the at least two second model loss values; and train the image-text data processing model based on the second target model loss values until a image-text data processing model that meets a training stop condition is obtained. In an optional embodiment, the training module is further configured to: train the image-text data processing model based on the predicted image features and the predicted text features until a image-text data processing model that meets a training stop condition is obtained. In an optional embodiment, the training module is further configured to: determine a first image-text feature pair and a random first feature sample, wherein the first image-text feature pair includes a first image feature and a first text feature corresponding to the first image feature; train the first feature generator using the first image feature and the random first feature sample as training samples and the first text feature as training labels until a first feature generator that meets the training stop condition is obtained.In an optional embodiment, the training module is further configured to: determine a second image-text feature pair and a random second feature sample, wherein the second image-text feature pair includes a second image feature and a second text feature corresponding to the second image feature; and train the second feature generator using the second text feature and the random second feature sample as training samples and the second image feature as training labels until a second feature generator that meets a training stop condition is obtained. In an optional embodiment, the first feature generator and the second feature generator are diffusion models. In summary, the above-described device extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features. It also extracts initial text features from text samples in the image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of the image-text data processing model. This enables the generation of multiple training labels from a single image-text sample pair, ensures the diversity of training data for the image-text data processing model, expands the semantic content that can be learned during model training beyond the image-text sample pair (i.e., the original training data), meets the requirement for a large number of valid samples covering diverse semantic content during model training, and thereby improves model training efficiency and performance. The above is a schematic diagram of a feature generation device according to this embodiment. It should be noted that the technical solution of this feature generation device and the technical solution of the aforementioned feature generation method share the same concept. For details not described in detail in the technical solution of the feature generation device, reference can be made to the description of the technical solution of the aforementioned feature generation method. Corresponding to the above method embodiments, the present disclosure also provides an embodiment of a training method for a graphic data processing model. Referring to FIG5 , FIG5 shows a flowchart of a training method for a graphic data processing model provided according to an embodiment of the present disclosure, specifically including the following steps.Step 502: Determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; Step 504: Determine initial image features of the image sample and initial text features of the text sample; Step 506: Input the image sample and the text sample into an image-text data processing model to obtain predicted image features and predicted text features; Step 508: Input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; Step 510: Input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair; Step 512: Train the image-text data processing model based on the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features until a image-text data processing model that meets a training stop condition is obtained. In a specific implementation, when training the image-text data processing model based on the predicted image features, the at least two target image features, the predicted text features, and the at least two target text features, the image-text data processing model may be trained based on the predicted text features and the predicted image features, with the training objective being to reduce the distance between the predicted image features and the at least two target image features. Alternatively, at least two first model loss values may be calculated based on the predicted image features and each target image feature, the at least two first model loss values may be added together to obtain a first target model loss value, and the image-text data processing model may be trained based on the first target model loss values. Alternatively, at least two second model loss values may be calculated based on the predicted text features and each target text feature, the at least two second model loss values may be added together to obtain a second target model loss value, and the image-text data processing model may be trained based on the second target model loss values, until a image-text data processing model that meets a training stop condition is obtained. Specifically, the above three training processes can be performed separately or simultaneously. The specific training process is similar to the above and will not be repeated here.In summary, the above method extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features. It also extracts initial text features from text samples in the image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of the image-text data processing model. This enables the generation of multiple training labels from a single image-text sample pair, ensures the diversity of the training data for the image-text data processing model, expands the semantic content that can be learned during model training beyond the image-text sample pair (i.e., the original training data), meets the requirement for a large number of valid samples covering diverse semantic content during model training, and thereby improves model training efficiency and performance. The above is an exemplary scheme of a training method for an image-text data processing model according to this embodiment. It should be noted that the technical solution for the training method of this graphic data processing model shares the same concept as the technical solution for the aforementioned feature generation method. For details not described in detail in the technical solution for the training method of this graphic data processing model, reference can be made to the description of the technical solution for the aforementioned feature generation method. Corresponding to the aforementioned method embodiments, the present disclosure also provides embodiments of a training device for a graphic data processing model. Figure 6 shows a schematic structural diagram of a training device for a graphic data processing model provided in one embodiment of the present disclosure.As shown in FIG6 , the apparatus includes: a first determining module 602 configured to determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; a second determining module 604 configured to determine initial image features of the image sample and initial text features of the text sample; a third determining module 606 configured to input the image sample and the text sample into an image-text data processing model to obtain predicted image features and predicted text features; a first input module 608 configured to input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; a second input module 610 configured to input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair; a training module 612 configured to generate a prediction image feature based on the predicted image features, the at least two target image features, The image-text data processing model is trained using the predicted text features and the at least two target text features until a training stop condition is obtained. In summary, the above-mentioned device extracts initial image features from image samples in an image-text sample pair and generates at least two target text features based on the initial image features and at least two random first features, and extracts initial text features from text samples in the image-text sample pair and generates at least two target image features based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of the image-text data processing model. This achieves the generation of multiple training labels based on a single image-text sample pair, ensures the diversity of the training data for the image-text data processing model, expands the semantic content that can be learned during model training beyond the image-text sample pair (i.e., the original training data), meets the requirement for a large number of effective samples covering diverse semantic content during model training, and thereby improves model training efficiency and performance. Corresponding to the above method embodiment, the present disclosure further provides an embodiment of a method for processing graphic data. Referring to FIG. 7 , FIG. 7 shows a flow chart of a method for processing graphic data provided according to an embodiment of the present disclosure, which specifically includes the following steps.Step 702: Determine the image and text data to be processed. Step 704: Input the image and text data to be processed into an image and text data processing model to obtain image and text features corresponding to the image and text data to be processed. The image and text data processing model is a model that satisfies the training stop condition described in the image and text data processing model training method provided in the embodiments of the present disclosure. Step 706: Determine the image and text data processing result corresponding to the image and text data to be processed based on the image and text features. The image and text data to be processed may include an image to be processed and text to be processed. Specifically, if the image and text data to be processed is an image to be processed, the image and text data processing model may be input to obtain image features corresponding to the image to be processed. Based on the image features, text features associated with the image features are determined. Based on the text features, text information corresponding to the image to be processed is determined. This text information is the image and text data processing result. In a specific implementation, when determining the text feature associated with the image feature based on the image feature, the text feature with the greatest similarity to the image feature is determined from a set of candidate text features and used as the text feature associated with the image feature. The candidate text feature set can be determined based on the specific processing task. For example, if the processing task is animal image classification, the candidate text feature set can be a set of pre-configured text features for animal categories. Taking animal image description as an example, the image to be processed can be an image of a cat drinking water. The image to be processed is input into a graphic data processing model to obtain image features corresponding to the image to be processed. Based on the image features, the text features associated with the image features are determined from a set of candidate text features in the field of animal image description. Based on the text features, the text information corresponding to the image to be processed is determined to be "cat drinking water." If the graphic data to be processed is text to be processed, the text to be processed can be input into the graphic data processing model to obtain text features corresponding to the text to be processed. Based on the text features, the image features associated with the text features are determined. Based on the image features, the image information corresponding to the text to be processed is determined. This image information is the result of image and text data processing. Similar to the above, when determining the image feature associated with a text feature based on the text feature, the image feature with the greatest similarity to the text feature can be selected from the candidate image feature set and used as the image feature associated with the text feature. Based on this, the present disclosure also provides an embodiment of an image data processing method, specifically comprising the following steps.Determine an image to be processed; input the image to be processed into a graphic data processing model to obtain image features corresponding to the image to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies the training stop condition described in the graphic data processing model training method provided in the embodiment of the present disclosure; determine text features associated with the image features based on the image features; and determine text information corresponding to the image to be processed based on the text features. Furthermore, the embodiment of the present disclosure also provides an embodiment of a text data processing method, specifically comprising the following steps: Determine text to be processed; input the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies the training stop condition described in the graphic data processing model training method provided in the embodiment of the present disclosure; determine image features associated with the text features based on the text features; and determine image information corresponding to the text to be processed based on the image features. The image data processing method provided in the present disclosure will be further described below, using the application of the graphic data processing method provided in the present disclosure in image classification as an example, with reference to FIG8 . FIG8 illustrates an application scenario of a method for processing image and text data provided by an embodiment of the present disclosure, specifically including the following steps. Step 802: The client-side device receives the image to be processed entered by the user in the image processing request upload box, receives the user's upload instruction for the image processing request, and sends the image processing request to the cloud-side device. The client-side device displays the image processing request upload box to the user. The image processing request carries the image to be processed. The image to be processed is an image that requires image classification. Specifically, the user clicks the "OK" control on the client-side device's display interface, and the client-side device determines the user's upload instruction for the image processing request based on the user's click instruction. Step 804: The cloud-side device receives the image processing request and determines the image to be processed. Step 806: The cloud-side device inputs the image to be processed into a graphic and text data processing model to obtain image features of the image to be processed. The graphic and text data processing model can be deployed on the cloud-side device. The graphic and text data processing model here is the one obtained after the aforementioned training. Step 808: The cloud-side device determines, based on the image feature, a text feature associated with the image feature, and determines, based on the text feature, text information corresponding to the image to be processed, and sends the text information to the terminal-side device.Specifically, the cloud-side device can determine the text feature with the greatest similarity to the image feature from a set of candidate text features in the image classification field, use this text feature as a text feature associated with the image feature, and, based on this text feature, determine text information corresponding to the image to be processed. This text information is the category information of the image to be processed. Step 810: The client-side device displays the text information to the user in the output result display box. Furthermore, the client-side device can receive a graphic data processing model sent by the cloud-side device and use this model to process the image to be processed on the client-side device. The image data processing method provided by the present disclosure will be further described below, using the application of the graphic data processing method in graphic data retrieval as an example, in conjunction with FIG9 . FIG9 illustrates a schematic diagram of an application scenario of the graphic data processing method provided by one embodiment of the present disclosure, which specifically includes the following steps: Step 902: The client-side device receives the text to be processed entered by the user in the text processing request upload box, receives an upload instruction from the user for the text processing request, and sends the text processing request to the cloud-side device. The client device displays a text processing request upload box to the user. The image processing request carries the text to be processed. This text to be processed is text that requires text retrieval. The text to be processed can be entered by the user in the text processing request upload box or selected by the user in the text processing request upload box. For example, the text to be processed can be "A clown holding a red balloon." Specifically, the user clicks the "OK" control on the display interface of the client-side device. The client-side device determines the user's upload instruction for the text processing request based on the user's click instruction. Step 904: The cloud-side device receives the text processing request and determines the text to be processed. Step 906: The cloud-side device inputs the text to be processed into the image-text data processing model to obtain text features of the text to be processed. The image-text data processing model can be deployed on the cloud-side device. The image-text data processing model here is the image-text data processing model obtained after the aforementioned training. Step 908: The cloud-side device determines, based on the text features, the image features associated with the text features, and determines, based on the image features, the image information corresponding to the text to be processed, and sends the image information to the client-side device. Specifically, the cloud-side device can determine, from a set of candidate image features in the field of image-text retrieval, the image feature with the greatest similarity to the text feature, use the image feature as the image feature associated with the text feature, and determine, based on the image features, the image information corresponding to the text to be processed. The image information is the image described by the text to be processed. Continuing with the above example, for the text to be processed being "a clown holding a red balloon", the image information may be an image showing a clown holding a red balloon.Step 910: The next-side device renders the image information and displays it to the user in the output result display box. Furthermore, the Luo-side device can receive the image and text data processing model sent by the cloud-side device and use the image and text data processing model to process the text to be processed on the client-side device. Corresponding to the above-mentioned method embodiments, the present disclosure also provides an embodiment of an image and text data processing apparatus. Figure 10 shows a schematic structural diagram of an image and text data processing apparatus provided by one embodiment of the present disclosure. As shown in Figure 10, the apparatus includes: a first determination module 1002, configured to determine image and text data to be processed; an input module 1004, configured to input the image and text data to be processed into the image and text data processing model to obtain image and text features corresponding to the image and text data to be processed. The image and text data processing model is a image and text data processing model that satisfies the training stop condition described in the image and text data processing model training method provided in the embodiments of the present disclosure; and a second determination module 1006, configured to determine an image and text data processing result corresponding to the image and text data to be processed based on the image and text features. Specifically, the image-text data processing model in the above-described device, as a trained multimodal representation model, can process image-text data and can be used for a variety of downstream tasks such as image classification, object detection, semantic segmentation, image-text retrieval, and image generation, while exhibiting good processing performance. Corresponding to the above-described method embodiment, the present disclosure also provides an embodiment of a text processing method. See FIG11 , which shows a flowchart of a text processing method provided according to one embodiment of the present disclosure, specifically including the following steps. Step 1102: Receive a text processing request sent by a client-side device, wherein the text processing request carries the text to be processed. Step 1104: Input the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed. The graphic data processing model is a graphic data processing model that satisfies the training stop conditions described in the graphic data processing model training method provided in the embodiments of the present disclosure. Step 1106: Determine, based on the text features, image features corresponding to the text features and the image corresponding to the image features. Step 1108: Send the image to the client-side device. Specifically, the graphic data processing model obtained through model training can be used as a pre-model for graphic data processing to process images or text. This example uses text processing as an example. Specifically, the cloud-side device can receive a text processing request sent by a user via a client-side device, determine the text to be processed carried in the text processing request, input the text to be processed into the trained graphic data processing model, and obtain text features corresponding to the text to be processed.Based on the text features, the image features corresponding to the text features are determined, and the image corresponding to the image features is determined. It can be understood that the image corresponds to the text to be processed. The image is then sent to the client-side device, where it is rendered and displayed to the user. In specific implementations, the determination of the image features corresponding to the text features and the image corresponding to the image features based on the text features can be implemented using a trained feature processing model. Specifically, the text features are input into the feature processing model, and the image output by the feature processing model is obtained. In another embodiment of the present disclosure, a set of candidate image features corresponding to the text processing task can be determined based on the text processing task. When determining the image features corresponding to the text features based on the text features, the similarity between the text features and each candidate image feature in the set of candidate image features can be calculated. The candidate image feature with the highest similarity can be determined as the image feature corresponding to the text feature. For example, if the text processing task involves text processing in the animal field, that is, based on text describing animals, the corresponding animal image is determined, then the candidate image feature set corresponding to the text processing task will include a large number of candidate image features for animal images. It can be understood that, since the training of the image-text data processing model is based on predicted text features and at least two target text features, and the at least two target text features are derived based on initial image features, the text features output by the trained image-text data processing model are more similar to image features, providing a basis for calculating the similarity between the text features and the candidate image features. For example, Figure 12 shows a schematic diagram of the processing results of a text processing method according to an embodiment of the present disclosure. The text to be processed is "A horse galloping on the grassland," and the resulting image is shown in Figure 12. In summary, the above method can generate corresponding images based on text and is applicable to text processing tasks in various downstream fields, demonstrating its applicability. Corresponding to the above method embodiments, the present disclosure also provides an embodiment of a text processing device. Figure 13 shows a schematic diagram of the structure of a text processing device according to an embodiment of the present disclosure.As shown in Figure 13, the apparatus includes: a receiving module 1302 configured to receive a text processing request sent by an end-side device, wherein the text processing request carries the text to be processed; an input module 1304 configured to input the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed, wherein the graphic data processing model is a graphic data processing model that satisfies the training stop conditions described in the training method for a graphic data processing model provided in the embodiments of the present disclosure; a determination module 1306 configured to determine, based on the text features, image features corresponding to the text features and determine an image corresponding to the image features; and a sending module 1308 configured to send the image to the end-side device. In summary, the apparatus described above can generate corresponding images based on text and is applicable to text processing tasks in various downstream fields, thus having applicability. The above is a schematic scheme of a text processing apparatus according to this embodiment. It should be noted that the technical solution of the text processing device and the technical solution of the aforementioned text processing method share the same concept. For details not described in detail in the technical solution of the text processing device, please refer to the description of the technical solution of the aforementioned text processing method. Figure 14 shows a block diagram of a computing device 1400 provided according to one embodiment of the present disclosure. The components of computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. Processor 1420 and memory 1410 are connected via a bus 1430, and a data storage device 1450 is configured to store data. Computing device 1400 also includes an access device 1440 that enables computing device 1400 to communicate via one or more networks 1460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.Access device 1440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like. In one embodiment of the present application, the aforementioned components of computing device 1400 and other components not shown in FIG. 14 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG. 14 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed. Computing device 1400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1400 can also be a mobile or stationary server. Processor 1420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method. The above is a schematic diagram of a computing device in this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described method. An embodiment of the present disclosure further provides a VR / AR device, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-mentioned graphic data processing method are implemented.Specifically, when a user wears a VR / AR device, the VR / AR device can utilize the aforementioned graphic data processing method to process input text, obtain an image corresponding to the text, and display the image to the user, ensuring a superior user experience. One embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions implement the steps of the aforementioned method. The above is an exemplary embodiment of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the aforementioned method are based on the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned method. One embodiment of the present disclosure also provides a computer program. When executed on a computer, the computer executes the steps of the aforementioned method. The above is an exemplary embodiment of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned method are based on the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the aforementioned method. The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electrical carrier signals or telecommunication signals.It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely intended to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present disclosure. The present disclosure selects and describes these embodiments in detail to better explain the principles and practical applications of the embodiments of the present disclosure, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is subject to the full scope and equivalents of the claims. Industrial Applicability: The feature generation method provided in the embodiments of the present disclosure is applied to a cloud device. Initial image features are extracted from image samples in an image-text sample pair, and at least two target text features are generated based on the initial image features and at least two random first features. Initial text features are also extracted from text samples in the image-text sample pair, and at least two target image features are generated based on the initial text features and at least two random second features. This facilitates the use of the generated at least two target text features and at least two target image features as training labels during subsequent training of an image-text data processing model. This enables the generation of multiple training labels based on a single image-text sample pair, ensures the diversity of training data for the image-text data processing model, and expands the semantic content that can be learned beyond the image-text sample pair (i.e., the original training data) during model training. This meets the requirement for a large number of effective samples covering different semantic content during model training, thereby improving model training efficiency and performance.
Claims
Claims 1. A feature generation method, comprising: Determine an image-text sample pair, wherein the image-text sample pair includes an image sample and a text sample corresponding to the image sample; determine initial image features of the image sample and initial text features of the text sample; input the initial image features and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first image-text feature pair; input the initial text features and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second image-text feature pair.
2. The method according to claim 1, wherein the step of inputting the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features comprises: The initial image feature is used as an adjustment condition and at least two random first features are used as objects to be adjusted, and are input into a first feature generator, wherein the first feature generator is a diffusion model; in the first feature generator, the at least two random first features are adjusted according to the initial image feature to obtain at least two target text features.
3. The method according to claim 1, wherein the step of inputting the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features comprises: The initial text feature is used as an adjustment condition and at least two random second features are used as objects to be adjusted, and are input into a second feature generator, wherein the second feature generator is a diffusion model; in the second feature generator, the at least two random second features are adjusted according to the initial text feature to obtain at least two target image features.
4. The method according to claim 1, after obtaining at least two target image features, further comprising: Inputting the image sample into a graphic data processing model to obtain predicted image features; and training the graphic data processing model according to the predicted image features and the at least two target image features until a graphic data processing model that meets a training stop condition is obtained.
5. The method according to claim 4, wherein the step of training the image-text data processing model according to the predicted image feature and the at least two target image features until a image-text data processing model that satisfies a training stop condition is obtained comprises: Calculating at least two first model loss values according to the predicted image feature and each of the at least two target image features; Determine a first target model loss value according to the at least two first model loss values; train the graphic data processing model according to the first target model loss value until a graphic data processing model that meets the training stop condition is obtained.
6. The method according to claim 4, after obtaining at least two target text features, further comprising: Inputting the text sample into a graphic data processing model to obtain predicted text features; The image-text data processing model is trained according to the predicted text feature and the at least two target text features until a image-text data processing model that meets a training stop condition is obtained.
7. The method according to claim 6, wherein the step of training the image-text data processing model according to the predicted text feature and the at least two target text features until a image-text data processing model that satisfies a training stop condition is obtained comprises: Calculating at least two second model loss values based on the predicted text feature and each of the at least two target text features; Determine a second target model loss value according to the at least two second model loss values; and train the graphic data processing model according to the second target model loss value until a graphic data processing model that meets a training stop condition is obtained. 22 8. The method according to claim 1, before inputting the initial image feature and at least two random first features into the first feature generator, further comprising: Determine a first image-text feature pair and a random first feature sample, wherein the first image-text feature pair includes a first image feature and a first text feature corresponding to the first image feature; use the first image feature and the random first feature sample as training samples, use the first text feature as a training label, and train the first feature generator until a first feature generator that meets a training stop condition is obtained.
9. The method according to claim 1, before inputting the initial text feature and at least two random second features into a second feature generator, further comprising: Determine a second image-text feature pair and a random second feature sample, wherein the second image-text feature pair includes a second image feature and a second text feature corresponding to the second image feature; use the second text feature and the random second feature sample as training samples, use the second image feature as a training label, and train the second feature generator until a second feature generator that meets the training stop condition is obtained.
10. A training method for a graphic data processing model, comprising: Determine a picture-text sample pair, wherein the picture-text sample pair includes an image sample and a text sample corresponding to the image sample; determine an initial image feature of the image sample and an initial text feature of the text sample; input the image sample and the text sample into a picture-text data processing model to obtain a predicted image feature and a predicted text feature; input the initial image feature and at least two random first features into a first feature generator to obtain at least two target text features, wherein the first feature generator is trained based on the first picture-text feature pair; input the initial text feature and at least two random second features into a second feature generator to obtain at least two target image features, wherein the second feature generator is trained based on the second picture-text feature pair; train the picture-text data processing model according to the predicted image feature, the at least two target image features, the predicted text feature and the at least two target text features until a picture-text data processing model that meets a training stop condition is obtained.
11. A method for processing graphic data, comprising: Determine the graphic data to be processed; input the graphic data to be processed into a graphic data processing model to obtain graphic features corresponding to the graphic data to be processed, wherein the graphic data processing model is the graphic data processing model that meets the training stop condition as described in claim 10; determine the graphic data processing result corresponding to the graphic data to be processed according to the graphic features.
12. A text processing method, applied to a cloud-side device, comprising: A text processing request sent by a receiving end-side device, wherein the text processing request carries a text to be processed; inputting the text to be processed into a graphic data processing model to obtain text features corresponding to the text to be processed, wherein the graphic data processing model is the graphic data processing model that satisfies the training stop condition as described in claim 10; determining, based on the text features, image features corresponding to the text features, and determining an image corresponding to the image features; and sending the image to the end-side device.
13. A computing device, comprising: Memory and processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
An antagonistic cross-media retrieval method based on bilingual semantic space
CN109344266A
Retrieval method, electronic equipment and computer readable medium
CN112287159A
Character-image pair generation method and device based on feature decoupling
CN114677569A
Model processing method and device, storage medium and electronic equipment
CN115221537A
Data enhancement method and device for multi-modal data set
CN115546577A
