Data generation task processing method and model training method applying preference data

Abstract

The embodiment of the invention provides a data generation task processing method and a model training method applying preference data, and the data generation task processing method comprises the steps that task data of a data generation task is acquired, and the task data comprises a sample image, a sample question aiming at the sample image and an answer label of the sample question; generating a corresponding noise image based on the sample image, and generating a candidate answer of the sample question based on the noise image; and on the basis of the answer label and the candidate answer of the sample question, constructing preference data of the sample question, and taking the preference data as a generation result of the data generation task. The candidate answers are obtained based on noise images and are different from answer labels generated based on sample images in large probability, negative samples of sample questions can be efficiently sampled, corresponding preference data are constructed, the economic and time cost of preference data construction is greatly reduced, the construction efficiency of the preference data is improved, and the user experience is improved. And a data basis is provided for large-scale alignment training.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular to a method for processing data generation tasks and a method for training a model of application preference data. Background Art

[0002] With the rapid development of computer technology and artificial intelligence technology, large models are being used more and more widely and are now able to perform various complex language tasks, including writing, translation, conversation, question-answering, etc. Multimodal large models can be used to understand and generate images and videos or to answer questions. However, multimodal large models pre-trained with SFT (Supervised Fine-Tuning) still have hallucination problems.

[0003] At present, one way to solve the hallucination problem is to use alignment training, such as the PPO (proximal policy optimization) algorithm based on reinforcement learning or the DPO (direct preference optimization) algorithm based on direct alignment optimization. Regardless of the alignment algorithm used, preference data is necessary sample data, but preference data is very rare in both general and industry datasets.

[0004] In the existing technology, preference data is often constructed manually, and the data construction link requires a lot of manual work, which is very time-consuming and economically expensive, and has low efficiency. Therefore, a lower-cost and more efficient data generation solution is urgently needed. Summary of the invention

[0005] In view of this, an embodiment of this specification provides a data generation task processing method. One or more embodiments of this specification also involve a model training method for application preference data, a data generation task processing device, a model training platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a method for processing a data generation task is provided, including: Acquire task data of a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; generating a corresponding noise image based on the sample image, and generating a candidate answer to the sample question based on the noise image; Based on the answer label and the candidate answers of the sample question, preference data of the sample question is constructed, and the preference data is used as a generation result of the data generation task.

[0007] According to a second aspect of an embodiment of this specification, a model training method for application preference data is provided, including: Receiving a model training request, wherein the model training request includes model attribute information of a machine learning model to be trained; Determining the machine learning model to be trained according to the model attribute information; Acquire a training sample set, and perform model training on the machine learning model to be trained based on the training sample set to obtain a target model, wherein the training sample set includes a plurality of training samples, the training samples include sample images and corresponding preference data, and the preference data is obtained based on the processing method of the above-mentioned data generation task; The target model is output.

[0008] According to a third aspect of an embodiment of this specification, there is provided a data generation task processing device, including: An acquisition module is configured to acquire task data of a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; a generating module configured to generate a corresponding noise image based on the sample image, and to generate a candidate answer to the sample question based on the noise image; A construction module is configured to construct preference data of the sample question based on the answer label and the candidate answer of the sample question, and use the preference data as a generation result of the data generation task.

[0009] According to a fourth aspect of the embodiments of this specification, there is provided a model training platform, including: a request interface unit, a model training unit and a response unit; A request interface unit is configured to receive a model training request, wherein the model training request includes model attribute information of a machine learning model to be trained; A model training unit is configured to determine the machine learning model to be trained according to the model attribute information; obtain a training sample set, and perform model training on the machine learning model to be trained based on the training sample set to obtain a target model, wherein the training sample set includes a plurality of training samples, the training samples include sample images and corresponding preference data, and the preference data is obtained based on the processing method of the above-mentioned data generation task; The response unit is configured to output the target model.

[0010] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; Among them, the memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned data generation task processing method or the model training method of application preference data are implemented.

[0011] According to a sixth aspect of the embodiments of this specification, there is provided an electronic device, including: A memory and a processor, wherein the memory and the processor are connected via a bus; Among them, the memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned data generation task processing method or the model training method of application preference data are implemented.

[0012] According to the seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned data generation task processing method or the model training method for application preference data.

[0013] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned data generation task processing method or the model training method for application preference data.

[0014] An embodiment of the present specification provides a method for processing a data generation task, which obtains task data for a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; generates a corresponding noise image based on the sample image, and generates a candidate answer to the sample question based on the noise image; constructs preference data for the sample question based on the answer label and the candidate answer to the sample question, and uses the preference data as a generation result of the data generation task.

[0015] An embodiment of the present specification realizes generating a corresponding noise image based on a sample image, and then generating a candidate answer to a sample question based on the noise image. The candidate answer is obtained based on the noise image, and is likely to be different from the answer label generated based on the sample image. It is capable of efficiently sampling negative samples of the sample question and constructing corresponding preference data, providing a link for automatically generating preference data, greatly reducing the economic and time costs of constructing preference data, improving the construction efficiency of preference data, and providing a data basis for large-scale alignment training. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an application architecture diagram of a data generation task provided by an embodiment of this specification; Figure 2 is a flow chart of a data generation task processing method provided by an embodiment of this specification; Figure 3 is a schematic diagram of task data provided by an embodiment of this specification; Figure 4 is a processing flow chart of a data generation task processing method provided by an embodiment of this specification; Figure 5 is a schematic diagram of a data generation task processing interface provided by an embodiment of this specification; Figure 6 is a schematic diagram of another data generation task processing interface provided by an embodiment of this specification; Figure 7 is a flow chart of a model training method for application preference data provided by an embodiment of this specification; Figure 8 It is a structural diagram of a data generation task processing device provided by an embodiment of this specification; Fig. 9 It is a structural diagram of a model training platform provided by an embodiment of this specification; Fig.10 is a structural block diagram of a computing device provided by one embodiment of this specification; Fig.11 It is a structural block diagram of an electronic device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0017] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0018] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0019] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0020] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0022] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0023] First, the terms involved in one or more embodiments of this specification are explained.

[0024] Adversarial Attacks: refers to an attack method against machine learning models, especially deep learning models, by adding carefully designed tiny perturbations, such as noise, to the input data, causing the machine learning / deep learning model to produce erroneous outputs. These perturbations are usually imperceptible to the human eye, but can cause serious misjudgments for the model. Common adversarial attack algorithms may include: Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Projected Gradient Descent (PGD), Carlini & Wagner attack (C&W attack), JSMA (Jacobian-based Saliency Map Attack).

[0025] Fast Gradient Sign Method: It is a simple but effective method for generating adversarial samples. It is a gradient-based method that calculates the gradient on the input data based on the model's loss function and adds a small perturbation information along the gradient direction to make the model make mistakes in predicting the input data. It generates adversarial samples by adding perturbation information along the direction where the loss function rises fastest. It is a simple and effective method for generating adversarial samples.

[0026] Basic Iterative Method: It is a technique for generating adversarial samples. It is an extension of FGSM, or iterative FGSM. It gradually increases the amount of perturbation by iterating FGSM multiple times to generate more covert and effective adversarial samples in order to achieve better results. This method can improve the success rate of adversarial attacks while keeping the perturbation information imperceptible.

[0027] Projected Gradient Descent: It is a powerful adversarial attack method that generates adversarial samples by iteratively applying small perturbations. Similar to the Basic Iterative Method (BIM), PGD is also based on gradient information, but it projects the perturbation information back to a specific range after each iteration to ensure that the perturbation information of the adversarial sample does not exceed the preset limit. It not only considers the change in the gradient direction, but also introduces a projection step to ensure that the generated adversarial sample is located in a specific area. This method can produce more powerful and hidden adversarial samples.

[0028] Carlini & Wagner attack: This method defines the generation process of adversarial samples in the form of an optimization problem, aiming to find the minimum perturbation information that can make the model error.

[0029] JSMA: Uses the Jacobian matrix to determine which pixel changes are most likely to change the model's classification decision.

[0030] Alignment Training: A technique used to improve model performance in machine learning and natural language processing, especially in the field of generative models and reinforcement learning. It is a training method that allows large models to produce expected responses. The main goal of alignment training is to make the model output more consistent with human preferences or standards, thereby improving the quality and practicality of the model, such as making large models produce safe responses that meet human preferences. Alignment training requires the use of aligned data. The general form of the data is that given an input prompt, there is a chosen response that meets the expected response and a rejected response that does not meet the expected response. The chosen response indicates the response selected by the human evaluator in the pairwise comparison. This response is considered to be of higher quality or more consistent with human preferences. The rejected response indicates the response that was not selected by the human evaluator in the pairwise comparison. This response is considered to be of lower quality or inconsistent with human preferences. By introducing additional supervisory signals, the behavior of the model is closer to the behavior or preferences of human experts. It is widely used in dialogue systems, machine translation, text summarization, code generation and other fields.

[0031] Proximal Policy Optimization: is a policy gradient algorithm widely used in reinforcement learning, which aims to improve the stability and efficiency of training by trimming the objective function. The core idea is to control the magnitude of policy updates by trimming the objective function, thereby ensuring that each update is small and stable. In large model training, proximal policy optimization is used as an algorithm for RLHF (Reinforcement Learning from Human Feedback, a technology that combines reinforcement learning and human feedback) human preference alignment.

[0032] Direct Preference Optimization: It is a method for training generative models, especially in the field of natural language processing (NLP). DPO optimizes the generative model by using paired comparison data of human preferences, so that the generated text is more in line with human preferences. This method is particularly suitable for tasks that require high-quality text generation, such as dialogue systems, machine translation, text summarization, etc. The core idea is to guide the learning process of the generative model through paired comparison data, so that the generated text is more in line with human preferences. It is regarded as an alternative to the PPO algorithm in large model training, and its advantage is low training cost.

[0033] Multimodal large models: refers to large machine learning models that can process and understand multiple types of data (such as text, images, audio, video, etc.). These models are usually based on deep learning architectures, especially Transformer architectures, and can effectively fuse and interact information between multiple modalities. Multimodal large models are widely used in many fields.

[0034] It should be noted that multimodal large models can be used to understand and generate images and videos or to answer questions. However, multimodal large models pre-trained with SFT still have hallucination problems. One way to solve the hallucination problem is to use alignment training, such as the PPO algorithm based on reinforcement learning or the DPO algorithm based on direct alignment optimization. Regardless of the alignment algorithm used, preference data is necessary, but preference data is very scarce in both general and industry datasets. Generally, building a link for preference data requires a lot of manual work, which is very time-consuming and economically expensive.

[0035] In one implementation, preference data generation uses a method of directly sampling multiple answers and increasing the model's divide-and-conquer judgment, but the multiple answers generated may not necessarily be different, and the efficiency of preference data generation is not high.

[0036] In another implementation, the model samples multiple answers and uses CLIP (Contrastive Language-Image Pre-training, a multimodal pre-training model) to determine the similarity score with the image. This method is also relatively inefficient, and CLIP does not have a good grasp of long texts and details, and cannot be used directly to judge preference data.

[0037] In another implementation, data augmentation can be used to change the image to generate negative samples. However, the data augmentation scheme has no directionality, so the efficiency of preference data generation is not high and it cannot ensure the generation of negative samples.

[0038] The embodiments of this specification provide a method for processing data generation tasks based on an adversarial attack algorithm. The adversarial attack algorithm is used to add noise to a sample image. The attack is directed at the response in the sample image. The result obtained is very likely to be different from the response in the sample image. Therefore, when the noise image is input into a large multimodal model for inference, a rejected response can be obtained with a high probability. The adversarial attack algorithm is directional and can efficiently sample negative samples. It provides a link for automatically generating preference data, greatly reducing the economic and time costs of constructing preference data. The efficiency of preference data generation can be greatly improved, providing a data basis for large-scale alignment training.

[0039] In order to solve the above technical problems, in this specification, a data generation task processing method is provided. This specification also involves a model training method for application preference data, a data generation task processing device, a model training platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0040] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the data generation task processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiments of this specification.

[0041] In an embodiment of the present specification, a system composed of a client device and a server may perform the following steps: the client device executes task data for uploading a data generation task. The server executes task data for obtaining a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; a corresponding noise image is generated based on the sample image, and a candidate answer to the sample question is generated based on the noise image; based on the answer label and the candidate answer to the sample question, preference data for the sample question is constructed to obtain a generation result of the data generation task. Among them, generating a corresponding noise image based on the sample image and generating a candidate answer to the sample question based on the noise image can be achieved with the help of a large model deployed by the server.

[0042] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0043] See also Figure 2 , Figure 2 A flow chart of a data generation task processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0044] Step 202: Obtain task data for a data generation task, wherein the task data includes sample images, sample questions for the sample images, and answer labels for the sample questions.

[0045] The embodiments of this specification are applied to applications, websites or mini-programs that have data generation task processing functions. The data generation task processing function is implemented on the application, website or mini-program. For example, a website that deploys a target generation model can implement the processing functions of data generation tasks such as image understanding generation and answer generation. For example, a third-party application can implement the corresponding data generation task processing function by calling the deployed target generation model through the application programming interface (Application Programming Interface, referred to as API).

[0046] The data generation task is a task to be processed. The data generation task is used to instruct reasoning based on the task data to generate corresponding preference data as the generation result of the data generation task. For example, the data generation task can be a preference data generation task in an image understanding generation scenario, a preference data generation task in a video understanding generation scenario, etc.

[0047] The task data may include sample images, sample questions for the sample images, and answer labels for the sample questions. The sample image refers to any image to be understood, the sample question refers to a question asked for the sample image, and the answer label for the sample question is the correct answer to the sample question after reasoning for the sample image.

[0048] In one implementation, a supervised fine-tuning sample set of a target generation model of a data generation task in a supervised fine-tuning stage can be obtained, and sample images, sample questions for the sample images, and answer labels for the sample questions can be obtained from the supervised fine-tuning sample set. The target generation model is any large model that is supervised fine-tuned based on the supervised fine-tuning sample set.

[0049] In another implementation method, the task data of the data generation task can also be obtained based on the front-end input, that is, the data generation task is generated in response to the sample image, sample question and corresponding answer label input by the front-end, and the sample image, sample question and corresponding answer label can be obtained by parsing the data generation task.

[0050] For example, Figure 3 is a schematic diagram of a task data provided by an embodiment of this specification, such as Figure 3 As shown in , the sample image obtained is Figure 3 As shown, the sample question is "What else is there in the picture besides the cat?" and the answer label of the sample question is "Besides the cat, there are many pillows."

[0051] Step 204: Generate a corresponding noise image based on the sample image, and generate a candidate answer to the sample question based on the noise image.

[0052] It should be noted that the noise image refers to an image obtained by adding noise to the sample image, and the candidate answer to the sample question refers to the answer obtained by inferring the noise image. The candidate answer is obtained by inferring the noise image and is likely to be different from the answer label based on the sample image, and can serve as the basis for subsequent construction of preference data.

[0053] In actual implementation, we can generate a noise image corresponding to the sample image based on the target generation model corresponding to the data generation task, adopt a set adversarial attack algorithm, and then infer the obtained noise image and sample question based on the target generation model to generate the corresponding candidate answer.

[0054] Among them, the target generation model is a multimodal large model that has been trained through supervised fine-tuning, which can realize reasoning on the input images and questions and output the corresponding answers. The adversarial attack algorithm can be set to be any adversarial attack algorithm, such as the fast gradient symbol method, the basic iterative method, the projected gradient descent, the Carlini & Wagner attack, and the JSMA. An embodiment of this specification is described by taking the fast gradient symbol method as an example.

[0055] In an optional implementation of this embodiment, generating a corresponding noise image based on the sample image includes: Adopting the set adversarial attack algorithm, using the target generation model corresponding to the data generation task, construct the perturbation information of the sample image; The perturbation information is applied to the sample image to generate a noisy image.

[0056] In actual implementation, taking the fast gradient sign method (FGSM) as an example of a setting adversarial attack algorithm, the goal of the FGSM algorithm is to obtain a tiny disturbance information that is not enough to affect human judgment for an input sample image, and add it to the sample image as noise to obtain a noisy image after the adversarial attack. The noisy image increases the loss function of the target generation model, that is, the ability of the target generation model to process the noisy image after the adversarial attack decreases. In other words, the goal of the FGSM algorithm is to obtain a noisy image after the adversarial attack, so that the target generation model produces an incorrect answer to the sample question of the noisy image as a candidate answer.

[0057] In the specific implementation, it is assumed that the target generation model is f(x), x is the input sample image and sample question, and the output of f(x) is the predicted answer y obtained by reasoning about the sample image. pred , the sample label of the sample problem is y true , the goal of the FGSM algorithm is to find a disturbance information , added to the original input sample image, so that the modified input X'=x+ , is incorrectly identified by the target generation model as f(x), and outputs the wrong predicted answer y for the sample question pred Among them, the predicted answer y pred The target generation model performs reasoning on the current input sample image and obtains the answer. The predicted answer y pred Used to determine disturbance information , and then generate a noise image, which is convenient for subsequent reasoning on the noise image to obtain candidate answers to construct preference data.

[0058] In the embodiments of the present specification, an adversarial attack algorithm can be set up, and the target generation model corresponding to the data generation task can be used to construct perturbation information of the sample image, and then the perturbation information is applied to the sample image to generate a noise image. The adversarial attack algorithm is used to add noise to the sample image, and the noise image is input into the target generation model for inference. It is possible to obtain candidate answers that are different from the answer label with a high probability, thereby efficiently constructing preference data.

[0059] In an optional implementation of this embodiment, the adversarial attack algorithm is set to be a fast gradient sign method; the adversarial attack algorithm is set, and the target generation model corresponding to the data generation task is used to construct disturbance information of the sample image, including: Define the target loss function of the target generation model and set the perturbation parameters; Calculate the gradient of the target loss function with respect to the sample image; According to the gradient and perturbation parameters, the perturbation information of the sample image is constructed.

[0060] The perturbation parameter is a parameter used to control the size of the perturbation information and is usually set to a very small positive real number.

[0061] In actual implementation, we can first define a target loss function of the target generation model, usually using cross entropy loss, and then set the perturbation parameters, input the sample image and sample question into the target generation model, obtain the predicted answer output by the target generation model, and calculate the loss value through the constructed target loss function according to the answer label of the sample question and the predicted answer, and calculate the gradient of the loss value with respect to the input sample image, take the sign of the gradient, and construct the perturbation information of the sample image according to the sign of the gradient and the perturbation parameters. Among them, the gradient of the sample image refers to the rate of change of the pixel value (or the feature value in the feature map) in the sample image relative to the model parameter. This gradient indicates how the loss function will change if the pixel value on the input sample image is adjusted; the sign of the gradient refers to the gradient vector expressed in symbolic form, rather than a specific numerical result. This representation method emphasizes that the gradient is an abstract mathematical object, which consists of a series of partial derivatives, and each component corresponds to the rate of change of the original function relative to a certain independent variable.

[0062] In specific implementation, the perturbation information of the sample image can be constructed by the following formula (1): (1) in, is the original input sample image; is the generated adversarial sample, i.e., a noisy image; The target loss function for the target generation model; is the gradient of the target loss function with respect to the input sample image; is the symbol used to obtain the gradient; is the disturbance parameter.

[0063] In the embodiments of this specification, the fast gradient sign method (FGSM) is taken as an example to set the adversarial attack algorithm. The target loss function of the target generation model is first defined, and the perturbation parameters are set. Then, based on the defined target loss function, the gradient of the sample image is calculated. Then, according to the gradient and the perturbation parameters, the perturbation information of the sample image is constructed. The core idea is relatively simple. It misleads the target generation model by increasing the minimum possible change. Only one forward propagation and one back propagation are needed to generate an adversarial sample, and the computational efficiency is very high.

[0064] In an optional implementation of this embodiment, defining a target loss function of a target generation model includes: Using the target generation model, obtain the visual embedding features of the sample image, and obtain the first response text embedding features corresponding to the visual embedding features; A target loss function is generated based on the visual embedding features and the first response text embedding features, wherein a constraint target of the target loss function is to minimize the similarity between the visual embedding features and the first response text embedding features.

[0065] It should be noted that when the visual encoder and text encoder in the target generation model are accessible, the FGSM algorithm can be used for adversarial attack training of contrastive learning. The goal of this attack is to modify the input sample image so that the similarity between the generated visual embedding features and the corresponding text embedding features is reduced.

[0066] In actual implementation, the target generation model includes a visual encoder and a text encoder. The sample image x is input into the visual encoder of the target generation model to obtain the visual embedding feature visual_embedding(x) of the sample image. In addition, based on the text encoder of the target generation model, the first response text embedding feature text_embedding(r1) corresponding to the visual embedding feature visual_embedding(x) can be obtained. The first response text embedding feature text_embedding(r1) is the embedding code of the reasoning response to the sample image and the sample question. Based on the visual embedding feature visual_embedding(x) and the first response text embedding feature text_embedding(r1), the corresponding target loss function can be constructed. The constraint target of the target loss function is to minimize the similarity between the visual embedding feature and the first response text embedding feature.

[0067] Specifically, taking the visual encoder as ViT in the CLIP model as an example, the text encoder corresponding to CLIP can also be obtained, and then the first response text embedding feature text_embedding(r1) corresponding to the visual embedding feature visual_embedding(x) is obtained. Then the corresponding goal of using the FGSM algorithm is to reduce the cosine similarity between the visual embedding feature visual_embedding(x) of the sample image and the first response text embedding feature text_embedding(r1).

[0068] The ViT in the CLIP model refers to Vision Transformer (ViT). Vision Transformer is an image processing model based on the Transformer architecture. It divides the image into a series of patches and treats these patches as sequence data and inputs them into the Transformer for processing. ViT is used as an image encoder in CLIP and is responsible for extracting features from the image. Patches refer to a series of smaller, non-overlapping rectangular areas into which the image is divided. Each patch can be regarded as a local part or fragment of the image.

[0069] Continuing with the above example, the sample image is Figure 3 As shown in the figure, the sample question is "What else is there in the picture besides the cat?" and the answer label is "Besides the cat, there are many pillows." The goal of the adversarial attack algorithm FGSM is to reduce the cosine similarity between the visual embedding feature visual_embedding(x) and the response text embedding feature text_embedding(r1) of the sentence "Besides the cat, there are many pillows."

[0070] In the embodiments of this specification, the fast gradient signed method FGSM is used as an adversarial attack algorithm to add noise to the sample image. The attack is implemented against the answer label of the sample image, that is, the adversarial attack is performed against the correct response text of the sample image. The result obtained is very likely to be different from the answer label of the sample image. The adversarial attack algorithm is directional and can efficiently sample negative samples that are different from the answer label, which greatly reduces the economic and time costs of constructing preference data, and the efficiency of preference data generation can be greatly improved.

[0071] In an optional implementation of this embodiment, defining a target loss function of a target generation model includes: Using the target generation model, obtaining a visual embedding feature of the sample image, and obtaining a second response text embedding feature for setting an expected answer, wherein the second response text embedding feature is different from the first response text embedding feature corresponding to the visual embedding feature; A target loss function is generated based on the visual embedding feature and the second response text embedding feature, wherein a constraint target of the target loss function is to maximize the similarity between the visual embedding feature and the second response text embedding feature.

[0072] Among them, the second response text embedding feature is the embedding feature corresponding to the response text with incorrect reasoning for the sample question, and the first response text embedding feature is the embedding feature corresponding to the response text with correct reasoning for the sample question, and the two are different.

[0073] It should be noted that when the visual encoder and text encoder in the target generation model are accessible, the FGSM algorithm can be used for adversarial attack training of contrastive learning. In addition to the above-mentioned attack target of modifying the input sample image so that the similarity between the generated visual embedding features and the corresponding text embedding features is reduced, the wrong response text can also be specified to maximize the similarity between the visual embedding features and the embedding features of the wrong response text.

[0074] In actual implementation, the target generation model includes a visual encoder and a text encoder. The sample image x can be input into the visual encoder of the target generation model to obtain the visual embedding visual_encoder(x) of the sample image. In addition, the specified wrong response text is input into the text encoder of the target generation model to obtain the second response text embedding feature text_embedding(r2) corresponding to the wrong response text. Based on the visual embedding feature visual_embedding(x) and the second response text embedding feature text_embedding(r2), the corresponding target loss function can be constructed. The constraint target of the target loss function is to maximize the similarity between the visual embedding feature and the second response text embedding feature.

[0075] Specifically, taking the visual encoder as ViT in the CLIP model as an example, the text encoder corresponding to CLIP can also be obtained. The sample image is input into ViT in the CLIP model to obtain the visual embedding feature visual_embedding(x); and the specified wrong response text is input into the text encoder in the CLIP model to obtain the corresponding second response text embedding feature text_embedding(r2). Then the corresponding goal of the FGSM algorithm is to increase the cosine similarity between the visual embedding feature visual_embedding(x) of the sample image and the second response text embedding feature text_embedding(r2).

[0076] Continuing with the above example, the sample image is Figure 3 As shown in the figure, the sample question is "What else is there in the picture besides the cat?", and the specified wrong response text is "Besides the cat, there are also dogs and blankets". The goal of the adversarial attack algorithm FGSM is to increase the cosine similarity between the visual embedding feature visual_embedding(x) and the response text embedding feature text_embedding(r2) of the sentence "Besides the cat, there are also dogs and blankets".

[0077] In the embodiments of this specification, the fast gradient signed method FGSM is used as an adversarial attack algorithm to add noise to the sample image. An incorrect response text can be specified, and an attack is performed in the direction of the incorrect response text to obtain a noise image and a directional incorrect image, thereby obtaining an incorrect answer in the direction as a negative sample. The adversarial attack algorithm is directional and can efficiently sample negative samples that are different from the answer label, greatly reducing the economic and time costs of constructing preference data, and the efficiency of generating preference data can be greatly improved.

[0078] In an optional implementation of this embodiment, defining a target loss function of a target generation model includes: Using the target generation model, obtain the visual embedding features of the sample image; A target loss function is generated based on the visual embedding features, where the constraint objective of the target loss function is to minimize the probability that the target generation model outputs the answer label for the sample image and the sample question.

[0079] It should be noted that when the text encoder in the target generative model cannot be directly obtained, another method can be used to design adversarial attacks. In this case, the autoregressive generation characteristics of the target generative model can be used to affect the performance of the model on the question-answering task by modifying the input image. Specifically, the autoregressive cross entropy loss function can be used as the target loss function to reduce the probability that the model generates the correct answer given a question.

[0080] In actual implementation, if the text encoder in the target generation model cannot be accessed, the visual embedding feature visual_embedding(x) of the sample image can be obtained based on the visual encoder in the target generation model. The target loss function is generated based on the visual embedding feature visual_embedding(x) of the sample image to implement the adversarial attack algorithm. The goal of this attack is to minimize the probability that the target generation model outputs the answer label for the sample image and sample question by modifying the input sample image.

[0081] Continuing with the above example, when the text encoder in the target generation model cannot be directly obtained, the target loss function of FGSM is the autoregressive cross entropy loss function of the target generation model. The constrained target of FGSM is to obtain a noisy image based on the input sample image, so that the probability of the target generation model answering the sample question "What else is there in the picture besides the cat?" "Besides the cat, there are many pillows" is reduced, and the probability of generating other answers such as "Besides the cat, there are dogs and blankets" is increased.

[0082] In the embodiments of the present specification, when the text encoder in the target generation model cannot be directly obtained, the autoregressive generation characteristics of the target generation model can be used to minimize the probability that the target generation model outputs answer labels for sample images and sample questions, and an adversarial attack can be performed to obtain the corresponding noise image. The adversarial attack algorithm can be implemented in a variety of different ways and can be applicable to a variety of different practical application scenarios with high flexibility.

[0083] In an optional implementation of this embodiment, generating candidate answers to sample questions based on the noise image includes: The noisy image and sample question are input into the target generation model corresponding to the data generation task, the target response text is obtained as the output of the target generation model, and the target response text is used as the candidate answer to the sample question.

[0084] It should be noted that after obtaining the noise image corresponding to the sample image based on the target generation model and the adversarial attack algorithm, the noise image and the sample question can be input into the target generation model corresponding to the data generation task to obtain the target response text output by the target generation model. The target response text is obtained based on the noise image reasoning and is likely to be an incorrect response. The target response text is used as a candidate answer to the sample question, that is, the candidate answer is likely to be an incorrect answer to the sample question, and the answer label is the correct answer to the sample question. Subsequently, preference data can be constructed based on the candidate answer (incorrect answer) and the answer label (correct answer).

[0085] In the embodiments of this specification, a target generation model is used to obtain incorrect answers that can be used as negative samples, providing a link for automatically generating preference data, greatly reducing the economic and time costs of constructing preference data, improving the efficiency of constructing preference data, and providing a data foundation for large-scale alignment training.

[0086] Step 206: Based on the answer label and candidate answers of the sample question, construct preference data of the sample question, and use the preference data as the generation result of the data generation task.

[0087] It should be noted that the preference data includes positive samples and negative samples of the sample questions. Based on the answer labels and candidate answers of the sample questions, the preference data of the sample questions can be constructed as the generation result of the data generation task.

[0088] In an optional implementation of this embodiment, based on the answer label and candidate answers of the sample question, preference data of the sample question is constructed, and the preference data is used as the generation result of the data generation task, including: Determine whether the answer label and the candidate answer are consistent and obtain a consistency judgment result; According to the consistency judgment results, the preference data of the sample questions are constructed, and the preference data is used as the generation result of the data generation task.

[0089] In actual implementation, the answer label and candidate answer can be input into the similarity judgment model to determine whether the answer label and candidate answer are consistent, obtain the consistency judgment result, and then construct the preference data of the sample question based on the consistency judgment result, and use the preference data as the generation result of the data generation task.

[0090] The similarity judgment model may be any large model that can judge whether two texts are the same.

[0091] Of course, in actual implementation, other methods may be used to determine whether the answer label and the candidate answer are consistent, such as calculating the similarity, and the embodiments of this specification do not limit this.

[0092] In the embodiments of the present specification, after obtaining a candidate answer to a sample question, the candidate answer is not directly used as a negative sample. Instead, the consistency between the answer label and the candidate answer is first verified to avoid using two identical or similar answers as preference data, thereby improving the accuracy of the preference data and ensuring the effect of alignment training based on the preference data.

[0093] Of course, in actual implementation, after obtaining the candidate answers to the sample questions, the answer labels of the sample questions can be directly used as positive samples and the candidate answers can be used as negative samples. The embodiments of this specification do not limit this.

[0094] In an optional implementation of this embodiment, constructing preference data of sample questions according to the consistency judgment result includes: When the consistency judgment result indicates that the answer label and the candidate answer are inconsistent, the answer label is used as a positive sample of the sample question, and the candidate answer is used as a negative sample of the sample question to construct the preference data of the sample question; When the consistency judgment result indicates that the answer label and the candidate answer are consistent, return to continue the steps of generating a corresponding noise image based on the sample image, and generating a candidate answer to the sample question based on the noise image, and reconstruct the preference data of the sample question.

[0095] It should be noted that the consistency judgment result output by the similarity judgment model may include that the answer label and the candidate answer are inconsistent, or that the answer label and the candidate answer are consistent. Alternatively, the consistency judgment result output by the similarity judgment model may also be the similarity between the answer label and the candidate answer. When the similarity is higher than the similarity threshold, it is determined that the answer label and the candidate answer are consistent, and when it is lower than the similarity threshold, it is determined that the answer label and the candidate answer are inconsistent.

[0096] In actual implementation, if the answer label and the candidate answer are inconsistent, it means that two different answers are obtained for the sample question, the answer label is the correct answer, and the candidate answer is the wrong answer. Therefore, the answer label can be used as a positive sample of the sample question, and the candidate answer can be used as a negative sample of the sample question to construct the preference data of the sample question. If the answer label and the candidate answer are consistent, it means that two consistent or relatively similar answers are obtained for the sample question, which cannot be used as positive and negative samples. Therefore, return to continue to execute the steps of generating the corresponding noise image based on the sample image, and generating the candidate answer of the sample question based on the noise image, and reconstruct the preference data of the sample question.

[0097] It should be noted that when the answer label and the candidate answer are inconsistent, the answer label and the candidate label are used as positive and negative samples respectively to construct the preference data. When the answer label and the candidate answer are consistent, they are regenerated. This ensures the efficiency of generating the preference data and the quality of the preference data, thereby ensuring the effect of the subsequent alignment training.

[0098] The embodiments of this specification provide a method for processing a data generation task, which can add noise to a sample image to generate a corresponding noise image, and then infer the noise image to generate a candidate answer to the sample question. The candidate answer is obtained by inferring the noise image, and is likely to be different from the answer label generated based on the sample image. It can efficiently sample negative samples of the sample question and construct corresponding preference data. It provides a link for automatically generating preference data, which greatly reduces the economic and time costs of constructing preference data, improves the construction efficiency of preference data, and provides a data basis for large-scale alignment training.

[0099] The following combination Figure 4 , taking the application of the data generation task processing method provided in this specification in the image understanding generation scenario as an example, the data generation task processing method is further explained. Among them, Figure 4 A flowchart of a data generation task processing method provided by an embodiment of the present specification is shown. Figure 5 A schematic diagram of a data generation task processing interface provided by an embodiment of this specification is shown. Figure 6 A schematic diagram of another data generation task processing interface provided by an embodiment of the present specification is shown.

[0100] like Figure 5 As shown, the user can select "Data Generation Task" in the data generation task processing interface and click the "Create" control. As an example, the data generation task processing interface can also include a task information configuration component. The user clicks the task information configuration component to configure the task information of the data generation task, such as uploading sample images, sample questions for sample images, answer labels for sample questions, and other task data, or can also configure the application field, scenario, etc. of the data generation task, or configure the model attribute information of the target generation model and / or the corresponding supervised fine-tuning sample set identifier (name or acquisition address). Assuming that the user configures the supervised fine-tuning sample set identifier of the target generation model based on the "Task Information Configuration Component", after clicking the "Create" control, the data generation task can be generated.

[0101] like Figure 4 As shown, the data generation task processing platform responds to the data generation task, obtains the supervised fine-tuning sample set based on the supervised fine-tuning sample set identifier, and obtains the sample image, sample question "What else is there in the picture besides the cat?", and answer label "There are many pillows besides the cat" from the supervised fine-tuning sample set, and directly uses the answer label "There are many pillows besides the cat" as a positive sample. Based on the response text (answer label or specified wrong response) of the sample image, an adversarial attack algorithm is used to generate a directional noise image of the response text; the multimodal large model is used to reason about the noise image and the sample question to obtain the candidate answer "Besides the cat, there are two pillows and a dog on the sofa". The answer label and the candidate answer are input into the similarity judgment large model to determine whether the answer label and the candidate answer are consistent. If not, the answer label is used as a positive sample and the candidate answer is used as a negative sample to construct the corresponding preference data; if consistent, the current data is discarded, and the adversarial attack is performed again to generate the candidate answer.

[0102] After constructing the preference data of the sample questions, the preference data can be used as the generation result of the data generation task and the generation result can be displayed to the front end, such as Figure 6 shown.

[0103] The embodiments of this specification provide a method for processing a data generation task, which can add noise to a sample image to generate a corresponding noise image, and then infer the noise image to generate a candidate answer to the sample question. The candidate answer is obtained by inferring the noise image, and is likely to be different from the answer label generated based on the sample image. It can efficiently sample negative samples of the sample question and construct corresponding preference data. It provides a link for automatically generating preference data, which greatly reduces the economic and time costs of constructing preference data, improves the construction efficiency of preference data, and provides a data basis for large-scale alignment training.

[0104] See also Figure 7 , Figure 7 A flowchart of a model training method for application preference data provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0105] Step 702: Receive a model training request, wherein the model training request includes model attribute information of the machine learning model to be trained.

[0106] Step 704: Determine the machine learning model to be trained based on the model attribute information.

[0107] Step 706: Obtain a training sample set, and perform model training on the machine learning model to be trained based on the training sample set to obtain a target model, wherein the training sample set includes multiple training samples, the training samples include sample images and corresponding preference data, and the preference data is obtained based on the processing method of the above-mentioned data generation task.

[0108] Step 708: Output the target model.

[0109] It should be noted that based on the above-mentioned data generation task processing method, a large amount of preference data can be generated. Based on the preference data, any machine learning model to be trained in the scenario can be aligned and trained to obtain the target model.

[0110] In actual implementation, the user can initiate a model training request based on the terminal device, and carry the model attribute information of the machine learning model to be trained in the model training request. According to the model attribute information, the machine learning model to be trained is determined, and the machine learning model to be trained is aligned and trained based on the preference data obtained in the above embodiment, so as to obtain and output the corresponding target model.

[0111] Among them, alignment training is a technology used to improve model performance in machine learning and natural language processing, especially in the field of generative models and reinforcement learning. It is a training method that allows large models to produce expected answers. The main goal of alignment training is to make the output of the model more consistent with the preferences or standards of positive samples, thereby improving the quality and practicality of the model, such as making large models produce safe answers that meet human preferences. Alignment training requires the use of preference data, which includes positive samples (answer labels) and negative samples (candidate answers) corresponding to a question. By introducing additional supervisory signals, the behavior of the model is closer to the positive sample.

[0112] It should be noted that the generated preference data is used to perform alignment training on the machine learning model to be trained. The process of obtaining the target model can adopt any training strategy, such as proximal strategy optimization, direct preference optimization, etc., and the embodiments of this specification do not limit this.

[0113] The embodiments of this specification provide a model training method using preference data. The preference data generated by the above-mentioned data generation task processing method can be used to perform alignment training on the machine learning model to be trained. Due to the efficiency of constructing the preference data, it provides a data basis for large-scale alignment training and ensures the effect of model training.

[0114] Corresponding to the above method embodiment, this specification also provides a data generation task processing device embodiment, Figure 8 FIG. 1 is a schematic diagram showing a structure of a data generation task processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises: An acquisition module 802 is configured to acquire task data of a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; A generating module 804 is configured to generate a corresponding noise image based on the sample image, and generate a candidate answer to the sample question based on the noise image; The construction module 806 is configured to construct preference data of the sample question based on the answer label and candidate answers of the sample question, and use the preference data as the generation result of the data generation task.

[0115] Optionally, the generating module 804 is further configured to: Adopting the set adversarial attack algorithm, using the target generation model corresponding to the data generation task, construct the perturbation information of the sample image; The perturbation information is applied to the sample image to generate a noisy image.

[0116] Optionally, the anti-attack algorithm is set to the fast gradient sign method; the generation module 804 is further configured to: Define the target loss function of the target generation model and set the perturbation parameters; Calculate the gradient of the target loss function with respect to the sample image; According to the gradient and perturbation parameters, the perturbation information of the sample image is constructed.

[0117] Optionally, the generating module 804 is further configured to: Using the target generation model, obtain the visual embedding features of the sample image, and obtain the first response text embedding features corresponding to the visual embedding features; A target loss function is generated based on the visual embedding features and the first response text embedding features, wherein a constraint target of the target loss function is to minimize the similarity between the visual embedding features and the first response text embedding features.

[0118] Optionally, the generating module 804 is further configured to: Using the target generation model, obtaining a visual embedding feature of the sample image, and obtaining a second response text embedding feature for setting an expected answer, wherein the second response text embedding feature is different from the first response text embedding feature corresponding to the visual embedding feature; A target loss function is generated based on the visual embedding feature and the second response text embedding feature, wherein a constraint target of the target loss function is to maximize the similarity between the visual embedding feature and the second response text embedding feature.

[0119] Optionally, the generating module 804 is further configured to: Using the target generation model, obtain the visual embedding features of the sample image; A target loss function is generated based on the visual embedding features, where the constraint objective of the target loss function is to minimize the probability that the target generation model outputs the answer label for the sample image and the sample question.

[0120] Optionally, the generating module 804 is further configured to: The noisy image and sample question are input into the target generation model corresponding to the data generation task, the target response text is obtained as the output of the target generation model, and the target response text is used as the candidate answer to the sample question.

[0121] Optionally, the construction module 806 is further configured to: Determine whether the answer label and the candidate answer are consistent and obtain a consistency judgment result; According to the consistency judgment results, the preference data of the sample questions are constructed, and the preference data is used as the generation result of the data generation task.

[0122] Optionally, the construction module 806 is further configured to: When the consistency judgment result indicates that the answer label and the candidate answer are inconsistent, the answer label is used as a positive sample of the sample question, and the candidate answer is used as a negative sample of the sample question to construct the preference data of the sample question; When the consistency judgment result indicates that the answer label and the candidate answer are consistent, return to continue the steps of generating a corresponding noise image based on the sample image, and generating a candidate answer to the sample question based on the noise image, and reconstruct the preference data of the sample question.

[0123] The embodiments of this specification provide a data generation task processing device, which can add noise to a sample image to generate a corresponding noise image, and then infer the noise image to generate a candidate answer to the sample question. The candidate answer is obtained by inferring the noise image, and is likely to be different from the answer label generated based on the sample image. It can efficiently sample negative samples of the sample question and construct corresponding preference data. It provides a link for automatically generating preference data, greatly reduces the economic and time costs of constructing preference data, improves the construction efficiency of preference data, and provides a data basis for large-scale alignment training.

[0124] The above is a schematic scheme of a data generation task processing device of this embodiment. It should be noted that the technical scheme of the data generation task processing device and the technical scheme of the above-mentioned data generation task processing method belong to the same concept, and the details not described in detail in the technical scheme of the data generation task processing device can be referred to the description of the technical scheme of the above-mentioned data generation task processing method.

[0125] Corresponding to the above method embodiment, this specification also provides a model training platform embodiment, Fig. 9 FIG. 1 shows a schematic diagram of a model training platform provided by an embodiment of the present specification. Fig. 9 As shown, the model training platform includes: a request interface unit 902, a model training unit 904 and a response unit 906; The request interface unit 902 is configured to receive a model training request, wherein the model training request includes model attribute information of a machine learning model to be trained; The model training unit 904 is configured to determine the machine learning model to be trained according to the model attribute information; obtain a training sample set, and perform model training on the machine learning model to be trained based on the training sample set to obtain a target model, wherein the training sample set includes a plurality of training samples, the training samples include sample images and corresponding preference data, and the preference data is obtained based on the processing method of the above-mentioned data generation task; The response unit 906 is configured to output the target model.

[0126] Optionally, the model training platform further includes a data receiving unit; The data receiving unit is configured to receive a training sample set and send the training sample set to the model training unit.

[0127] Optionally, the model training platform further includes a model library, wherein the model library stores a plurality of initial task models; The model training unit is also used to determine the machine learning model to be trained from the model library according to the model attribute information.

[0128] In the embodiments of this specification, the model training platform adapts to user demand acquisition to perform model training, realizes personalized model training services, provides users with an efficient, flexible and easy-to-use model training service platform, and improves user experience.

[0129] Fig.10 A structural block diagram of a computing device provided by an embodiment of the present specification is shown.

[0130] The computing device 1000 includes: Memory 1010 and processor 1020; The memory 1010 is used to store computer programs / instructions, and the processor 1020 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1020, the steps of the above-mentioned data generation task processing method or the model training method of application preference data are implemented.

[0131] In one or more embodiments of the present specification, the computing device 1000 may be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer), an all-in-one model machine, a mobile phone, a tablet computer or other portable intelligent terminal, etc., and the computing device may be pre-installed with the models in the above embodiments of the present application.

[0132] Specifically, the computing device 1000 can preset multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multimodal task processing, etc., so as to provide a variety of model selections. In different product forms, the computing device 1000 can support one or more model usage methods, including but not limited to model training, model calling, model fine-tuning, model deployment, model reasoning and application, etc. In some product forms, the computing device 1000 also supports model management, including but not limited to multi-type model management (supporting the management of multiple types of models such as discriminants and generative models), model version control (supporting the control of different model versions), model evaluation (based on model evaluation tools, evaluating the performance and effect of models), etc. In other product forms, the computing device 1000 can also create applications based on models, provide API calling capabilities, and can call models to the created applications through API interfaces, while providing application management tools to achieve management and monitoring of applications.

[0133] Furthermore, the computing device 1000 may also include data management (supporting the creation and management of model tuning data sets), a training center (providing rich training resources to help users learn artificial intelligence technology), and basic management and control capabilities (providing enterprise-level basic management and control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive, integrated artificial intelligence development, training, deployment and application device is provided.

[0134] Fig.11 A structural block diagram of an electronic device provided by an embodiment of the present specification is shown.

[0135] The memory 1110 and the processor 1120 are connected via a bus 1130; The memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1120, the steps of the above-mentioned data generation task processing method or the model training method of application preference data are implemented.

[0136] Specifically, the components of the electronic device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 and the memory 1110 may be connected via a bus 1130.

[0137] The electronic device 1100 may also include an access device 1140, which enables the electronic device 1100 to communicate with a database 1150 storing data via one or more networks 1160. Examples of these networks 1160 include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0138] In one embodiment of the present specification, the above components of the electronic device 1100 and Fig.11 Other components not shown in the figure may also be connected to each other, for example, via bus 1130. It should be understood that Fig.11 The electronic device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0139] The electronic device 1100 may be any type of stationary or mobile electronic device, including a mobile computer or mobile electronic device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable electronic device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary electronic device such as a desktop computer or a personal computer (PC). The electronic device 1100 may also be a mobile or stationary server.

[0140] The above is a schematic scheme of an electronic device of this embodiment. It should be noted that the technical scheme of the electronic device and the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data belong to the same concept, and the details not described in detail in the technical scheme of the electronic device can be referred to the description of the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data.

[0141] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned data generation task processing method or the model training method for application preference data.

[0142] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data.

[0143] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned data generation task processing method or the model training method for application preference data.

[0144] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned data generation task processing method or the model training method of application preference data.

[0145] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0146] Computer programs / instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0147] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0148] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data generation task processing method, comprising: Acquire task data of a data generation task, wherein the task data includes a sample image, a sample question for the sample image, and an answer label for the sample question; generating a corresponding noise image based on the sample image, and generating a candidate answer to the sample question based on the noise image; Based on the answer label and the candidate answers of the sample question, preference data of the sample question is constructed, and the preference data is used as a generation result of the data generation task.

2. The data generation task processing method according to claim 1, wherein generating a corresponding noise image based on the sample image comprises: Adopting a set adversarial attack algorithm and utilizing a target generation model corresponding to the data generation task to construct disturbance information of the sample image; The disturbance information is applied to the sample image to generate the noise image.

3. According to the data generation task processing method of claim 2, the set adversarial attack algorithm is a fast gradient sign method; the set adversarial attack algorithm is used to construct the disturbance information of the sample image using the target generation model corresponding to the data generation task, comprising: Defining a target loss function of the target generation model and setting perturbation parameters; Calculating the gradient of the target loss function with respect to the sample image; Constructing disturbance information of the sample image according to the gradient and the disturbance parameter.

4. The data generation task processing method according to claim 3, wherein the target loss function for defining the target generation model comprises: Using the target generation model, obtaining a visual embedding feature of the sample image, and obtaining a first response text embedding feature corresponding to the visual embedding feature; The target loss function is generated based on the visual embedding feature and the first response text embedding feature, wherein a constraint target of the target loss function is to minimize the similarity between the visual embedding feature and the first response text embedding feature.

5. The data generation task processing method according to claim 3, wherein the target loss function for defining the target generation model comprises: Using the target generation model, obtaining a visual embedding feature of the sample image, and obtaining a second response text embedding feature for setting an expected answer, wherein the second response text embedding feature is different from a first response text embedding feature corresponding to the visual embedding feature; The target loss function is generated based on the visual embedding feature and the second response text embedding feature, wherein a constraint target of the target loss function is to maximize the similarity between the visual embedding feature and the second response text embedding feature.

6. The data generation task processing method according to claim 3, wherein the target loss function for defining the target generation model comprises: Using the target generation model, obtaining visual embedding features of the sample image; The target loss function is generated based on the visual embedding feature, wherein the constraint objective of the target loss function is to minimize the probability that the target generation model outputs the answer label for the sample image and the sample question.

7. The data generation task processing method according to any one of claims 1 to 6, wherein generating a candidate answer to the sample question based on the noise image comprises: The noise image and the sample question are input into a target generation model corresponding to the data generation task, a target response text output by the target generation model is obtained, and the target response text is used as a candidate answer to the sample question.

8. The method for processing a data generation task according to any one of claims 1 to 6, wherein the step of constructing preference data for the sample question based on the answer label and the candidate answer for the sample question and using the preference data as a generation result of the data generation task comprises: Determine whether the answer label is consistent with the candidate answer, and obtain a consistency judgment result; According to the consistency judgment result, preference data of the sample question is constructed, and the preference data is used as the generation result of the data generation task.

9. The data generation task processing method according to claim 8, wherein constructing the preference data of the sample question according to the consistency judgment result comprises: When the consistency judgment result indicates that the answer label and the candidate answer are inconsistent, the answer label is used as a positive sample of the sample question, and the candidate answer is used as a negative sample of the sample question to construct preference data of the sample question; When the consistency judgment result indicates that the answer label and the candidate answer are consistent, return to continue executing the steps of generating a corresponding noise image based on the sample image and generating a candidate answer to the sample question based on the noise image, and reconstruct the preference data of the sample question.

10. A model training method for application preference data, comprising: Receiving a model training request, wherein the model training request includes model attribute information of a machine learning model to be trained; Determining the machine learning model to be trained according to the model attribute information; Acquire a training sample set, and perform model training on the machine learning model to be trained based on the training sample set to obtain a target model, wherein the training sample set includes a plurality of training samples, the training samples include sample images and corresponding preference data, and the preference data is obtained based on the processing method for the data generation task described in any one of claims 1 to 9 above; The target model is output.

11. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the processing method for the data generation task described in any one of claims 1 to 9 are implemented.

12. An electronic device comprising: A memory and a processor, wherein the memory and the processor are connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the processing method for the data generation task described in any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the method for processing a data generation task as described in any one of claims 1 to 9.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method for processing a data generation task as claimed in any one of claims 1 to 9.

Images