Training sample generation and model training method and device, equipment and storage medium

Abstract

The invention discloses a training sample generation method and device, a model training method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a training sample generation task; generating training data according to the training sample generation task through a first model; response data corresponding to the training data is generated according to the training data through a second model, and the response data is annotated data of the training data; and generating a training sample data set according to the training data and the response data. By applying the scheme, the data generation efficiency, especially the generation efficiency of the annotation data, can be improved, and the cost is saved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a training sample generation method, a model training method, a device, an electronic device and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, a large number of neural network models have emerged. The training of these models mainly adopts supervised learning, that is, the model is trained through a large number of training samples with labeled data. The quality of sample data plays a vital role in the performance of the training model. In order to obtain high-quality training data, researchers usually use manual labeling to label the data, but this method requires a lot of manpower and time, is costly and inefficient, which is particularly obvious in the training of large models. In addition, errors may be introduced during the manual labeling process, which will also affect the performance of the trained model. Summary of the invention

[0003] To this end, the present invention provides a training sample generation method, a model training method, an apparatus, an electronic device and a storage medium, in an effort to solve or at least alleviate at least one of the above problems.

[0004] According to one aspect of the present invention, a training sample generation method is provided, which is suitable for execution in a computing device, and the method comprises:

[0005] Obtain training sample generation tasks;

[0006] Generate training data according to the training sample generation task by using a first model;

[0007] Generate response data corresponding to the training data according to the training data through a second model, wherein the response data is labeled data of the training data;

[0008] A training sample data set is generated according to the training data and the response data.

[0009] Optionally, the training process of the first model includes the following steps:

[0010] Constructing a first model including a policy network and a judgment network;

[0011] Initializing the parameters of the first model and obtaining a first model training sample generation task;

[0012] Generate prediction data according to the first model training sample generation task through the policy network;

[0013] The prediction data is evaluated by the evaluation network according to the first model training sample generation task to obtain a corresponding prediction data reward value and prediction data generation guidance;

[0014] Adjusting the policy network according to the predicted data reward value;

[0015] The prediction data generation guidance is used as a new training sample generation task, and the above steps are repeated until the first model converges to obtain a trained first model.

[0016] Optionally, while adjusting the policy network according to the predicted data reward value, the judgment network is also adjusted according to the predicted data reward value.

[0017] Optionally, the implementation method of adjusting the policy network according to the predicted data reward value is a gradient strategy method or a Q-Learning method.

[0018] Optionally, both the policy network and the evaluation network are implemented by a large model that has been trained.

[0019] Optionally, the first training data is data with associated information added to the first training sample.

[0020] According to another aspect of the present invention, a model training method is provided, the method comprising:

[0021] Create the initial target model and training sample generation tasks;

[0022] Based on the training sample generation task, generating a training sample data set by using any of the methods described above;

[0023] The initial target model is trained using the training sample data set until the initial target model converges to obtain a trained target model.

[0024] According to another aspect of the present invention, a training sample generating device is provided, the device comprising:

[0025] A task acquisition unit, used to acquire training sample generation tasks;

[0026] A sample data generating unit, configured to generate training data according to the training sample generating task through a first model;

[0027] a labeling data generating unit, configured to generate response data corresponding to the training data according to the training data by using a second model, wherein the response data is labeling data of the training data;

[0028] The training data set generating unit is used to generate a training sample data set according to the training data and the response data.

[0029] According to another aspect of the present invention, a model training device is provided, the device comprising:

[0030] An initial model creation unit, used to create an initial target model and training sample generation tasks;

[0031] A training data generating unit, configured to generate a training sample data set based on the training sample generating task by using any of the methods described above;

[0032] The model training unit is used to train the initial target model according to the training sample data set until the initial target model converges to obtain a trained target model.

[0033] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0034] at least one processor; and

[0035] A memory storing program instructions, wherein the program instructions are configured to be suitable for being executed by the at least one processor, and the program instructions include instructions for executing any of the above methods.

[0036] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes any of the above methods.

[0037] According to the training sample generation method of the present application, training data is obtained through the first model, and corresponding response data is generated according to the training data through the second model, thereby improving data generation efficiency, especially improving the generation efficiency of labeled data, saving costs, and avoiding errors in manual labeling.

[0038] Furthermore, the first model includes a policy network and a critic network. By training the policy network using the actor-critic algorithm in reinforcement learning, the quality of generated training data can be further guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings. The drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0040] Figure 1 is a schematic block diagram of an electronic device according to an embodiment of the present invention;

[0041] Figure 2 is a schematic flow chart of a training sample generation method 200 according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a first model 300 according to an embodiment of the present invention;

[0043] Figure 4 is a flowchart of a first model training process 400 according to an embodiment of the present invention;

[0044] Figure 5 is a schematic block diagram of a training sample generating device 500 according to an embodiment of the present invention;

[0045] Figure 6 is a schematic flow chart of a model training method 600 according to an embodiment of the present invention;

[0046] Figure 7 It is a schematic block diagram of a model training device 700 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0048] The embodiments of the present application provide a training sample generation method, device, electronic device and storage medium, which can be applied to high-quality training data generation. According to the training sample generation method of the present application, training data is obtained through a first model, and corresponding response data is generated according to the training data through a second model, which improves the data generation efficiency, especially improves the generation efficiency of labeled data, saves costs, and avoids errors in manual labeling.

[0049] The training sample generation method provided in the embodiment of the present application can be applied to electronic devices with computing and processing capabilities, and the electronic devices can be mobile phones, tablet computers, desktop computers, laptop computers, servers, vehicle-mounted devices, netbooks, and augmented reality (AR) devices, virtual reality (VR) devices, smart speakers, artificial intelligence robots, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device.

[0050] Figure 1 FIG. 1 is a schematic diagram showing the structure of an electronic device 100 according to an embodiment of the present invention. The electronic device can be used to implement the methods of various embodiments of the present invention. Figure 1 As shown, in a basic configuration 102, the electronic device 100 typically includes a memory 106 and one or more processors 104. A memory bus 108 may be used for communication between the processor 104 and the memory 106.

[0051] Depending on the desired configuration, the processor 104 may be any type of processor, including but not limited to: a microprocessor, a microcontroller, a digital signal processor (DSP), or any combination thereof. The processor 104 may include one or more levels of cache, such as a primary cache 110 and a secondary cache 112, a processor core 114, and registers 116. An example processor core 114 may include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core, or any combination thereof. In some implementations, the memory controller 118 may be a separate component used with the processor 104; in some other implementations, the memory controller 118 may be an internal component of the processor 104.

[0052] Depending on the desired configuration, the memory 106 can be any type of memory, including but not limited to: volatile memory (such as RAM, etc.), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The physical memory in the electronic device generally refers to the volatile memory RAM, and the data in the disk needs to be loaded into the physical memory before it can be read by the processor 104. The memory 106 can store an operating system 120 and one or more applications 122. In some embodiments, the application 122 can be arranged to execute instructions on the operating system by one or more processors 104. The operating system 120 can be, for example, Linux, Windows, Android, Hongmeng, Apple iOS, etc., which includes program instructions for processing basic system services and performing hardware-dependent tasks. The application 122 includes program instructions for implementing various user-desired functions. The application 122 can be, for example, a browser, game software, business software, office software, instant messaging software, software development tools (such as integrated development environment IDE, compiler, etc.), etc., but is not limited thereto. When the application 122 is installed in the electronic device 100, a driver module can be added to the operating system 120 if necessary.

[0053] When the electronic device 100 starts running, the processor 104 reads and executes the program instructions of the operating system 120 from the memory 106. The application 122 runs on the operating system 120 and uses the interface provided by the operating system 120 and the underlying hardware to implement various functions desired by the user. When the user starts the application 122, the processor 104 reads and executes the program instructions of the application 122 from the memory 106.

[0054] The electronic device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, voice input devices, touch input devices, handwriting input devices, etc.) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be configured to facilitate communication with one or more other electronic devices via a network communication link via one or more communication ports 164.

[0055] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. As a non-limiting example, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0056] In the electronic device 100 according to the present invention, the application 122 includes instructions for executing the training sample generation method of the present invention, and the instructions can instruct the processor 104 to execute the training sample generation method of the present invention.

[0057] It is to be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include: Figure 1 More or fewer components are shown, or some components are combined, or some components are separated, or there are different component arrangements. The components shown can be implemented in hardware, software, firmware, or any combination of the three.

[0058] Below, based on the above-mentioned hardware involved in the embodiments of the present application, other embodiments of the present application will be described in detail in conjunction with the accompanying drawings.

[0059] like Figure 2 FIG. 2 is a flow chart of a training sample generation method 200 according to an embodiment of the present invention. The method may include the following steps:

[0060] Step S210: Obtain training sample generation tasks.

[0061] The training sample generation task needs to define at least one of the attributes of the training sample, such as the associated subject, format, and style. The training sample generation task can be implemented by prompt words. In a specific example, the prompt words for the training sample generation task are as follows:

[0062] From now on, your task is to generate training data that meets the following training sample requirements, where the training sample requirements are:

[0063] {

[0064] 1. Generate 1TB of text large language model training samples;

[0065] 2. Generate training data related to everyday drawing and different from the following examples.

[0066] example:

[0067] How is the weather today?

[0068] What do you think of this movie?

[0069] }

[0070] As in the above specific example, the scope and examples of the training data are given through prompt words.

[0071] In another specific example, the pre-trained model needs to be fine-tuned for a specific task. When generating fine-tuning data, a specific task topic can be added to the training sample generation task, and specific task data samples can be given. The training sample generation task can also limit the description of the generated data, such as: the training data should not include enough information, the questioning methods should be diverse, and there should be depth.

[0072] Step S220, generating training data according to the training sample generation task through the first model.

[0073] According to one embodiment of the present invention, the first model is a trained reinforcement learning model, including a policy network (i.e. Figure 3 Actor network in ) and judge network (i.e. Figure 3 The Actor network is used to generate training data according to the training sample generation task. In a specific example, both the Actor network and the Critic network are implemented by a trained large model, which can be a large model trained by the applicant himself or a commercial large model of other manufacturers.

[0074] Obviously, implementing the Actor network and Critic network through a large model that has already been trained can avoid retraining the model and improve the efficiency of data generation.

[0075] The quality of training data is greatly affected by the performance of the model. When selecting training samples to generate a model, a specific example of this embodiment is implemented through a public commercial large model. The public commercial large model is a natural language processing tool driven by artificial intelligence technology. It can generate answers based on the training data and statistical laws in the pre-training stage, and can interact according to the context of the chat, and can really chat and communicate like humans, and even complete tasks such as writing emails, copywriting, codes, and papers. It can be seen that the public commercial large model has powerful text processing and generation functions. Furthermore, when the training sample is a large language model training sample and the training data type is text data, the public commercial large model is selected to implement the policy network and the judgment network. The powerful text data processing capabilities of the public commercial large model can be used to effectively avoid the problem of poor model performance affecting the quality of training data, thereby ensuring the performance of subsequent training models.

[0076] Step S230: Generate response data corresponding to the training data according to the training data using the second model.

[0077] In one example, the second model is implemented by a large model that has been trained. The large model can be a large model trained by the applicant himself, or a commercial large model of other manufacturers. Obviously, by implementing the Actor network and the Critic network through a large model that has been trained, retraining the model can be avoided, which can improve the efficiency of data generation.

[0078] According to a specific example of an embodiment of the present invention, the second model is implemented by a public commercial large model. Based on the powerful language processing capabilities of the public commercial large model, high-quality response data (i.e., labeled data) corresponding to the training data can be generated. Compared with manually labeled data, it can save manpower and time, improve data labeling efficiency, and reduce costs. At the same time, it avoids the problem of introducing errors in the manual labeling process. Furthermore, compared with other models, the public commercial large model has powerful text and language processing capabilities, and can generate high-quality response data (i.e., labeled data). Furthermore, high-quality model training samples can train high-performance target models.

[0079] In a specific example, after the training data is acquired through the first model, the training data is passed to the second model in real time, and the response data corresponding to the training data is acquired through the second model.

[0080] In another specific example, firstly, all the training data are acquired through the first model, and further, the response data of all the training data are acquired through the second model.

[0081] Step S240, generating a training sample data set according to the training data and the response data corresponding to the training data, wherein the response data corresponding to the training data is the labeled data of the training data, that is, a set of data <training data, response data> is a training sample.

[0082] According to the training sample generation method of the present invention, labeled training samples are generated based on a deep network learning model, which saves manpower and improves the efficiency of sample generation and annotation compared to the method of crawling data from the Internet and manual labeling. Furthermore, the training sample generation model is implemented through a public commercial large model. With the help of the powerful language processing capabilities of the public commercial large model, high-quality training data can be obtained, thereby improving the performance of the training target model.

[0083] Figure 3 3 is a schematic diagram of a first model 300 according to an embodiment of the present invention. Figure 3As shown, the first model is implemented through the Actor-Critic model, including an Actor network (i.e., a policy network) 310 and a Critic network (i.e., a judgment network) 320, wherein the Actor network 310 takes the training sample generation task as input, and outputs the prediction data generated according to the training sample generation task, the Critic network 320 takes the training sample generation task as input, and the prediction data output by the Actor network as input, and the output is the evaluation value (i.e., reward value) of the prediction data output by the Actor network according to the training sample generation task, and the prediction data generation guidance, which is to guide the Actor network to perform the training sample generation task in the next round of training, and this cycle is repeated many times until the first model converges. It should be noted that in this application, the Actor network and the policy network are the same description, and the Critic network and the judgment network are the same description.

[0084] Figure 4 A schematic flowchart of a first model training process 400 according to an embodiment of the present invention is shown.

[0085] The following will be combined Figure 3 and Figure 4 The training process 400 of the first model is described, which includes the following steps:

[0086] Step S410, constructing a first model including a policy network and a judgment network, initializing first model parameters and training sample generation tasks.

[0087] The training sample generation task here is a first model training sample generation task, which is usually different from the training sample generation task obtained in step S210.

[0088] According to one embodiment of the present invention, the first model is implemented by an Actor-Critic model (eg Figure 3 ), randomly initialize the parameters of the Actor network and the Critic network, and then initialize the training sample generation task. For example, the training sample generation task is initialized as "create a weather-themed query."

[0089] Step S420, generating prediction data through the strategy network according to the training sample generation task.

[0090] As in the above example, the prediction data generated by the Actor network based on the training sample generation task is "What is the weather like today?"

[0091] Step S430, the prediction data is evaluated by the critic network according to the training sample generation task to obtain the corresponding prediction data reward value. In this step, the prediction data can also be evaluated by the critic network according to the training sample generation task to obtain the corresponding prediction data generation guidance. Among them, the prediction data reward value is the evaluation value of the Critic network on the prediction data generated by the Actor network according to the training sample generation task, and the prediction data generation guidance is the guidance for the Actor network when generating training data next time. For example, the prediction data generation guidance can be a query requiring the Actor network to generate more information, and its format is consistent with the training sample generation task.

[0092] Step S440, adjusting the policy network according to the predicted data reward value. The policy network is adjusted in the direction of obtaining higher quality predicted data.

[0093] In this step, the judgment network can also be adjusted according to the predicted data reward value.

[0094] Exemplarily, the adjustment method of adjusting the policy network or adjusting the judgment network can be implemented by a gradient strategy method or a Q-Learning method.

[0095] According to one embodiment of the present invention, the Actor network parameters are optimized and adjusted according to the reward value through a gradient strategy method, so that the Actor network can generate better prediction data, thereby achieving the goal of maximizing the reward value.

[0096] According to another embodiment of the present invention, the parameters of the Critic network are optimized and adjusted according to the reward value through the Q-Learning method to achieve the goal of maximizing the reward value.

[0097] In other embodiments, the Actor network and the Critic network may be adjusted simultaneously according to the reward value.

[0098] After each round of training is completed, it is determined whether the model has reached the convergence condition. If the convergence condition has not been reached, the process proceeds to step S450.

[0099] In step S450, the prediction data generation guidance is used as a new training sample generation task, and step S420 is re-entered to enter a new round of training process, and steps S420, S430 and S440 are repeated.

[0100] After step S440, if the model reaches the convergence condition, the process proceeds to step S460, the training ends, and a trained first model is obtained.

[0101] After the model is trained, high-quality training data can be generated through the actor network. In a specific example, the trained Actor network generates a question about today's weather, "What's the weather like today? Is it suitable to go out?" Compared with the previous question "What's the weather like today?", it is more informative and not only contains the original question content, but also adds new content.

[0102] According to an embodiment of the present invention, the first model is implemented by a reinforcement learning model. During training, the reinforcement learning model can learn to generate corresponding prediction data according to different prediction data. This ability enables the first model to have strong generalization ability without the need to train for each different generation task. Furthermore, each time the prediction data is generated, a real-time reward value can be obtained, so that the first model has strong real-time decision-making ability.

[0103] Figure 5 A schematic block diagram of a training sample generating apparatus 500 according to an embodiment of the present invention is shown. As shown in the figure, the training sample generating apparatus 500 includes a task acquiring unit 510, a sample data generating unit 520, a labeling data generating unit 530, and a training data set generating unit 540.

[0104] A task acquisition unit 510 is used to acquire a training sample generation task;

[0105] The sample data generating unit 520 is configured to generate training data according to the training sample generating task through the first model 300. According to one embodiment of the present invention, the sample data generating unit 520 is a data processing unit including at least the first model 300.

[0106] The annotated data generating unit 530 is configured to generate response data corresponding to the training data according to the training data by using the second model. According to one embodiment of the present invention, the annotated data generating unit 530 is a data processing unit including at least the second model.

[0107] The training data set generating unit 540 is used to generate a training sample data set according to the training data and the response data corresponding to the training data, wherein the response data corresponding to the training data is the labeled data of the training data.

[0108] Figure 6 A schematic flow chart of a model training method 600 according to an embodiment of the present invention is shown. The method comprises:

[0109] Step S610, creating an initial target model and a training sample generation task. In a specific example, if the target model is a large language model, then an initial large language model and a large language model training sample generation task are created.

[0110] Step S620: Based on the training sample generation task, a training sample data set is generated by the training sample generation method 200. According to one embodiment of the present invention, the following steps are included:

[0111] The training data is generated by the first model according to the target model training sample generation task. According to one embodiment of the present invention, the first model can be obtained by the above-mentioned training through the model training method 400, and the first model is implemented by the Actor-Critic model, including an Actor network and a Critic network, wherein the Actor network and the Critic network are implemented by a public commercial large model. The training sample generation task is used to guide the Actor network to generate training data, and then the corresponding response data is generated according to the training data through the second model. In this embodiment, the second model is implemented by a public commercial large model. Further, a training sample data set is generated according to the training data generated by the first model and the corresponding response data generated by the second model, wherein the response data is the labeled data of the corresponding training data.

[0112] Step S630: train the initial target model using the training sample data set. Input the training data in the training sample data set into the target model, obtain the predicted output through the target model, and adjust the target model according to the loss between the predicted data and the response data corresponding to the training data until the model converges to obtain a trained target model.

[0113] Figure 7 A schematic block diagram of a model training device 700 according to an embodiment of the present invention is shown, which includes an initial model creation unit 710 , a training data generation unit 720 and a model training unit 730 .

[0114] The initial model creation unit 710 is used to create an initial target model and a training sample generation task.

[0115] The training data generating unit 720 is used to generate a training sample data set according to the training sample generating method 200. According to one embodiment of the present invention, the training data generating unit 720 is a data processing unit including at least the training sample generating device 500.

[0116] The model training unit 730 is used to train the initial target model according to the training sample data set until the model converges to obtain a trained target model.

[0117] According to a model training method according to an embodiment of the present invention, label training data is generated with the help of an automated training sample generation method 200, thereby improving the efficiency of training data generation, thereby shortening the model training time and improving the model training efficiency.

[0118] The model training device 700 of the embodiment of the present invention can be composed of Figure 1 The electronic device 100 shown executes program instructions to implement.

[0119] According to an exemplary embodiment of the present invention, an electronic device is provided. The electronic device may include at least one processor and at least one memory. The memory stores program code for implementing the corresponding steps in the training sample generation method or model training method according to the embodiment of the present invention. The processor is used to run the program code stored in the memory to execute the corresponding steps of the training sample generation method or model training method according to the embodiment of the present invention, and to implement the various modules in the training sample generation device or model training device according to the embodiment of the present invention. The electronic device may adopt the aforementioned Figure 1 The electronic devices in the illustrated embodiments and any other suitable electronic devices.

[0120] In one embodiment, when the program code is executed by a processor, the following steps are performed:

[0121] Obtain training sample generation tasks;

[0122] Generate training data according to the training sample generation task by using a first model;

[0123] Generate response data corresponding to the training data according to the training data through a second model, wherein the response data is labeled data of the training data;

[0124] A training sample data set is generated according to the training data and the response data.

[0125] In addition, other steps of the related method are also executed when the program code is executed by the processor.

[0126] In addition, according to another embodiment of the present invention, a computer-readable storage medium is also provided, on which program instructions are stored, and when the program instructions are run by a computer or a processor, they are used to perform the corresponding steps of the training sample generation method or the model training method of the embodiment of the present invention, and are used to implement the corresponding modules in the training sample generation or model training device according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0127] In one embodiment, when the computer program instructions are executed by a computer, they can implement the various functional modules of the training sample generation device or model training device according to the embodiment of the present invention, and / or can execute the training sample generation method or model training method according to the embodiment of the present invention.

[0128] In one embodiment, the computer program instructions, when executed by a computer, perform the following steps: obtaining a training sample generation task;

[0129] Generate training data according to the training sample generation task by using a first model;

[0130] Generate response data corresponding to the training data according to the training data through a second model, wherein the response data is labeled data of the training data;

[0131] A training sample data set is generated according to the training data and the response data.

[0132] In addition, when the computer program instructions are executed by the computer, other steps of each method of the embodiment of the present invention are also executed.

[0133] Each module of the model training device or training sample generating device according to an embodiment of the present invention can be implemented by executing computer program instructions stored in a memory by a processor of an electronic device according to an embodiment of the present invention, or can be implemented when computer instructions stored in a computer-readable storage medium according to an embodiment of the present invention are executed by a computer.

[0134] According to the training sample generation method, model training method and device, electronic device and storage medium of the embodiment of the present invention, the training data is obtained through the first model, and the corresponding response data is generated according to the training data through the second model, which improves the data generation efficiency, especially improves the generation efficiency of the labeled data, saves costs, and avoids errors in manual labeling. Furthermore, the first model includes a policy network and a judgment network, and the quality of the training data generation can be further guaranteed by training the policy network using the actor-critique algorithm in reinforcement learning.

[0135] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0137] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method, apparatus, and device of the present invention, or some aspects or portions of the method, apparatus, and device of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0138] In the case where the program code is executed on a programmable computer, the electronic device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the remote assistance method of the present invention according to the instructions in the program code stored in the memory.

[0139] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules or other data. Communication media generally embody computer-readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium. Any combination of the above is also included in the scope of readable media. In the specification of the present application, algorithms and displays are not inherently related to any specific computer, virtual system or other device. Various general systems can also be used with the examples of the present invention. According to the above description, it is obvious to construct the structure required for such systems. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be used to implement the software and algorithm-related content of the present invention described herein, and the description of the specific language above is only to disclose some specific embodiments of the present invention, but not to limit the use of the specific language.

[0140] In the specification of the application, a lot of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the above-described device and equipment embodiments are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0142] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0143] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0144] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0145] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0146] The components in the various embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that computers, servers, microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the equipment and devices according to embodiments of the present invention. The present invention can also be implemented as a device program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0147] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. Nouns such as "comprise", "include", "have" do not exclude the presence of elements or steps not listed in the claims. The quantifier "one" or "a" preceding an element does not exclude the presence of multiple such elements. The present invention can be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order, and these words may be interpreted as names.

[0148] In addition, in the embodiments of the present application, the expression "and / or" includes any and all combinations of the associated listed words. For example, the expression "A and / or B" may include A, may include B, or may include both A and B. Words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0149] In the description of the embodiments of the present application, when a component is referred to as being "connected" or "accessed" to other components, it should be understood that: the component is not only directly connected to or accessed to other components, but also another component may exist between the component and the other components. On the other hand, when a component is referred to as being "directly connected" or "directly accessed" to other components, it should be understood that there is no component between them. When a component is referred to as being "movably connected" to other components, it means that the positional relationship between the component and the other components is changeable, such as the relationship between a slide rail and a slider, the slider can slide on the slide rail, and the slide rail and the track are movably connected. When a component is referred to as being "fixedly connected" to other components, it means that the positional relationship between the component and the other components after being assembled together is relatively fixed.

[0150] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Although the present application is described according to a limited number of embodiments, it is understood by the technicians in the technical field that other embodiments can be envisioned within the scope of the present application described thereby, thanks to the above description. In addition, it should be noted that the language used in this specification is mainly selected for the purpose of readability and teaching, rather than for explaining or limiting the subject matter of the present application. Therefore, without departing from the scope and spirit of the attached claims, many modifications and changes are obvious to ordinary technicians in the technical field. For the scope of the present application, the disclosure made in the present application is illustrative, not restrictive, and the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A training sample generation method, characterized in that: The method comprises: Obtain training sample generation tasks; Generate training data according to the training sample generation task by using a first model; Generate response data corresponding to the training data according to the training data through a second model, wherein the response data is labeled data of the training data; A training sample data set is generated according to the training data and the response data.

2. The method according to claim 1, characterized in that The training process of the first model includes the following steps: Constructing a first model including a policy network and a judgment network; Initializing the parameters of the first model and the first model training sample generation task; Generate prediction data according to the first model training sample generation task through the policy network; The prediction data is evaluated by the evaluation network according to the first model training sample generation task to obtain a corresponding prediction data reward value and prediction data generation guidance; Adjusting the policy network according to the predicted data reward value; The prediction data generation guidance is used as a new training sample generation task, and the above steps are repeated until the first model converges to obtain a trained first model.

3. The method according to claim 2, characterized in that While adjusting the policy network according to the predicted data reward value, the judgment network is also adjusted according to the predicted data reward value.

4. The method according to claim 2, characterized in that The method for adjusting the policy network according to the predicted data reward value is a gradient policy method or a Q-Learning method; and / or, the policy network and the evaluation network are implemented through a large model that has been trained.

5. The method according to any one of claims 1 to 4, characterized in that: The first training data is data obtained by adding associated information to the first training sample.

6. A model training method, characterized in that: The method comprises: Create the initial target model and training sample generation tasks; Based on the training sample generation task, generating a training sample data set by the method according to any one of claims 1 to 5; The initial target model is trained using the training sample data set until the initial target model converges to obtain a trained target model.

7. A training sample generating device, characterized in that: The device comprises: A task acquisition unit, used to acquire training sample generation tasks; A sample data generating unit, configured to generate training data according to the training sample generating task through a first model; a labeling data generating unit, configured to generate response data corresponding to the training data according to the training data by using a second model, wherein the response data is labeling data of the training data; The training data set generating unit is used to generate a training sample data set according to the training data and the response data.

8. A model training device, characterized in that: The device comprises: An initial model creation unit, used to create an initial target model and training sample generation tasks; A training data generating unit, configured to generate a training sample data set based on the training sample generating task by the method according to any one of claims 1 to 5; The model training unit is used to train the initial target model according to the training sample data set until the initial target model converges to obtain a trained target model.

9. An electronic device, comprising: at least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 6.

10. A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, the computing device executes the method according to any one of claims 1 to 6.

Images