Method, device, apparatus and medium for generating instruction data

Abstract

To provide a method, a device, an electronic apparatus, a storage medium, and a program for generating instruction data for a large model.SOLUTION: A method includes: acquiring a reference instruction based on natural language which can instruct a large model to generate reply data satisfying a plurality of first requests corresponding to a plurality of first request types; acquiring a structured disassembly result for the reference instruction to acquire a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values corresponding to the plurality of first requests; determining a plurality of sample slots and a plurality of sample slot values corresponding to the plurality of sample slots on the basis of the plurality of reference slots, the plurality of reference slot values, and a preset rule; and generating a natural language-based sample instruction capable of instructing the large model to generate reply data satisfying a plurality of second requests corresponding to the plurality of sample slot values on the basis of the plurality of sample slots and the plurality of sample slot values.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to technical fields such as natural language processing, and specifically to a method for generating instruction data for a large model, an apparatus for generating instruction data for a large model, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can imitate some human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning), and includes both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing, while AI software technologies mainly include natural language processing, computer vision, voice recognition, machine learning, big data processing, and knowledge graph technologies.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, it should not be assumed that any approach described in this section is prior art by virtue of its inclusion in this section. Similarly, unless otherwise noted, the subject matter addressed in this section should not be assumed to be an admission of any prior art. Summary of the Invention

[0004] The present disclosure provides a method for generating instruction data for large models, an apparatus for generating instruction data for large models, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method for generating instruction data for a large model executed on a computer is provided, the method including: obtaining a reference instruction based on a natural language, wherein the reference instruction can instruct the large model to generate answer data that satisfies a plurality of first requests corresponding to a plurality of first request types; obtaining a structured decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values ​​and a predetermined rule; and generating a sample instruction based on the plurality of sample slots and the plurality of sample slot values, wherein the sample instruction can instruct the large model to generate answer data that satisfies a plurality of second requests that correspond to the plurality of sample slot values.

[0006] According to another aspect of the present disclosure, there is provided an apparatus for generating instruction data for a large model, the apparatus including: a first acquisition unit configured to acquire a reference instruction based on a natural language, wherein the reference instruction can instruct the large model to generate answer data that satisfies a plurality of first requests corresponding to a plurality of first request types; a second acquisition unit configured to acquire a structured decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; a determination unit configured to determine a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values ​​and a predetermined rule; and a generation unit configured to generate sample instructions based on the natural language based on the plurality of sample slots and the plurality of sample slot values, wherein the sample instruction can instruct the large model to generate answer data that satisfies a plurality of second requests that correspond to the plurality of sample slot values.

[0007] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0009] According to another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the above method.

[0010] According to one or more embodiments of the present disclosure, the present disclosure performs structured decomposition on a reference command based on natural language to obtain a plurality of reference slots corresponding to request types and a plurality of reference slot values ​​corresponding to specific requests of each request type, and then determines a plurality of sample slots and a plurality of sample slot values ​​for generating sample commands based on the plurality of reference slots, the plurality of reference slot values ​​and a predetermined rule, and finally obtains a sample command based on natural language, thereby improving the flexibility of the process of generating command training data, providing more complex, diverse, directional and directed command training data for the generative large model, and effectively improving the command following ability of the large model.

[0011] It should be understood that the material described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief explanation of the drawings]

[0012] The drawings illustratively illustrate examples and constitute a part of the specification, and together with the written description of the specification, serve to explain exemplary embodiments of the examples. The illustrated examples are for purposes of illustration and not limiting the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] FIG. 1 is a schematic diagram of an exemplary system capable of implementing various methods described herein, according to embodiments of the present disclosure. [Figure 2] 1 is a flowchart of a method for generating instruction data for a large model according to an embodiment of the present disclosure. [Figure 3] 10 is a flowchart for determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots, according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method for generating instruction data for a large model according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a structural block diagram of an apparatus for generating instruction data for a large model according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0023] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein, but they should be considered merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0014] In this disclosure, unless otherwise specified, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0015] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, unless the number of elements is specifically limited, the elements may be one or more. Furthermore, as used in this disclosure, the term "and / or" covers any one of the listed items and all possible combinations.

[0016] In related art, to train the command-following ability of a large model, it is generally only possible to rely on manually written command data, but such command data lacks directionality and orientation.

[0017] To solve the above problems, the present disclosure performs structured decomposition on reference commands based on natural language to obtain multiple reference slots corresponding to request types and multiple reference slot values ​​corresponding to specific requests of each request type, and then determines multiple sample slots and multiple sample slot values ​​for generating sample commands based on the multiple reference slots, multiple reference slot values ​​and preset rules, and finally obtains sample commands based on natural language, thereby improving the flexibility of the process of generating command training data and providing more complex, diverse, directional and directed command training data for the generative large model, and effectively improving the command tracking ability of the large model.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0019] 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0020] In an embodiment of the present disclosure, the server 120 may be operable to execute one or more services or software applications for the method of generating instruction data for a large model according to the present disclosure. In one exemplary embodiment, the large model may be deployed on the server.

[0021] In some embodiments, server 120 may further provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0022] In the arrangement shown in Figure 1, server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof, executable by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may utilize the services provided by these assemblies by sequentially utilizing one or more client applications to interact with server 120. It should be understood that a variety of different system configurations are possible and may differ from system 100. As such, Figure 1 is intended to be illustrative of an example system for implementing various methods described herein and is not intended to be limiting.

[0023] A user can generate answer data in the large model using client devices 101, 102, 103, 104, 105, and / or 106. The client devices can provide their users with an interface through which they can interact with the client device. The client devices can also output information to the user via the interface, such as an answer generated by the large model in response to user input. Although only six client devices are shown in FIG. 1 , one skilled in the art will understand that the present disclosure can support any number of client devices.

[0024] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminals, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-based operating systems, Linux or Linux-based operating systems (e.g., Google Chrome OS), etc., and may include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, Android, etc. Portable handheld devices may include cellular phones, smartphones, tablet PCs, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, etc. The client device may run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), and short message service (SMS) applications, and may use a variety of communication protocols.

[0025] Network 110 may be any type of network known to those skilled in the art, which may use any one of several available protocols to support data communications (including, but not limited to, TCP / IP, SNA, IPX, etc.) By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0026] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running a virtual operating system or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.

[0027] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercially available server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, including an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0028] In some embodiments, server 120 may include one or more applications for integrating with analytics data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may further include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0029] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or a smart cloud computing server or smart cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the deficiencies of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0030] System 100 may further include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store audio and video file information. Databases 130 may be located in various locations. For example, a database used by server 120 may be local to server 120 or may be separate from server 120 and in communication with server 120 via a network or dedicated connection. Databases 130 may be of various types. In some embodiments, a database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and to the database in response to commands.

[0031] In some embodiments, one or more of databases 130 may also be used by an application to store the application's data. The databases used by the application may be different types of databases, such as a key-value repository, an object repository, or a general-purpose repository backed by a file system.

[0032] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0033] According to one aspect of the present disclosure, there is provided a method for generating instruction data for a large model executed by a computer. As shown in Fig. 2, the method for generating instruction data for a large model includes: a step S201 of obtaining a reference instruction based on a natural language, where the reference instruction can instruct the large model to generate answer data that satisfy a plurality of first requests corresponding to a plurality of first request types; a step S202 of obtaining a structural decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; a step S203 of determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values, and a predetermined rule; and a step S204 of generating a sample instruction based on the plurality of sample slots and the plurality of sample slot values, where the sample instruction can instruct the large model to generate answer data that satisfy a plurality of second requests corresponding to the plurality of sample slot values.

[0034] Thus, by performing structured decomposition on the reference command based on natural language, a plurality of reference slots corresponding to the request types and a plurality of reference slot values ​​corresponding to the specific requests of each request type are obtained, and then, based on the plurality of reference slots, the plurality of reference slot values ​​and the predetermined rules, a plurality of sample slots and a plurality of sample slot values ​​for generating sample commands are determined, and finally, a sample command based on natural language is obtained, thereby improving the flexibility of the process of generating command training data, and providing more complex, diverse, directional and directed command training data for the generative large model, and effectively improving the command following ability of the large model.

[0035] In this disclosure, the large model is a generative large model based on deep learning. The large model has end-to-end features and can directly generate answer data based on command data input by a user. In other words, the large model itself has generative capabilities. In some embodiments, the large model may be a large-scale language model based on deep learning. Large-scale language models typically have billions or even hundreds of billions of parameters, and they are typically trained on large-scale text data or other modal data. Large-scale language models may be used for various natural language processing tasks, such as text generation, language translation, and question-answering systems.

[0036] The large model may, for example, adopt an N-layer Transformer network structure having an encoder and a decoder, or a Unified Pre-Trained Language Model (UniLM) network structure. It is understood that the large model may also be other neural network models based on the Transformer network structure, and is not limited thereto. The input and output of the large model are both composed of tokens. Each token may correspond to a character, a word, or a special code. The large model may be trained using a pre-training task and then fine-tuned using instruction data and corresponding ground truth answer data to provide the above-mentioned generation function.

[0037] In some embodiments, step S201 may obtain training instructions from a training set or true target instructions from an online user log, and these instructions may be reference instructions. In one exemplary embodiment, an example of a reference instruction may be, "Write a prose piece about alcohol, no longer than 100 characters long, without the word 'alcohol' appearing in the entire sentence." The reference instruction relates to five first requirement types: "Task to be performed by the large model," "Genre of content to be generated," "Topic of content to be generated," "Number of characters in content to be generated," and "Requirements for content to be generated." Each of the five first requirement types has five first requirements: "Task is to create text," "Genre is prose," "Topic is alcohol," "Number of characters is no longer than 100 characters," and "No character 'alcohol' appears in the entire sentence." The reference instruction may instruct the large model to generate answer data that satisfy multiple first requirements corresponding to multiple first requirement types.

[0038] As can be seen from the above example, a "request type" may be understood as a condition that should be met in the process of generating answer data using a large-scale model, such as the genre or subject of the content to be generated. The "request" corresponding to a "request type" is used to explain the specific condition that should be met to generate answer data for that request type. For example, the large-scale model should generate prose for the request type "genre of content to be generated," or should generate content related to alcohol for the request type "subject of content to be generated."

[0039] In some embodiments, in step S202, a structural decomposition result for the defect reference command may be obtained to obtain a plurality of reference slots corresponding to a plurality of first request types and a plurality of reference slot values ​​corresponding to a plurality of first requests. In one exemplary embodiment, a structural decomposition may be performed for the example reference command to obtain five reference slots, namely, "task," "genre," "topic," "number of characters," and "content request," corresponding to the five first request types, and five reference slot values, namely, "text creation," "prose," "alcohol," "<100 characters," and "the character 'alcohol' should not appear," corresponding to the five first requests.

[0040] It should be noted that the first request type and the first request may be understood as abstract concepts about conditions that must be met for the large model to generate answer data, while the reference slot and the reference slot value are concrete, deterministic, and structured representations obtained by decomposing the abstract concepts. In some embodiments, the plurality of reference slots may be selected from a plurality of preset slots, and the plurality of candidate slot values ​​may be selected from a plurality of preset slot values.

[0041] In some embodiments, the structured decomposition result for the reference instruction may be obtained by, for example, processing the reference instruction using a natural language instruction decomposition model. The natural language instruction decomposition model may be a classification model, which selects a plurality of reference slots corresponding to the reference instruction from a plurality of preset slots, and selects a plurality of reference slot values ​​corresponding to the reference instruction from a plurality of preset slot values. The natural language instruction decomposition model may be a generative model (which may be a model different from the large-scale model mentioned above), and directly obtains a plurality of reference slots and a plurality of reference slot values ​​corresponding to the reference instruction through a content generation method.

[0042] Thus, through the above steps, a structured decomposition result for the reference instruction can be obtained, and statistics and analysis can be performed on the reference instruction based on the structured decomposition result, and sample instructions corresponding to which sample slots and sample slot values ​​should be generated can be determined based on the corresponding statistical analysis results. In some embodiments, sample instructions can be generated directly based on the structured decomposition result for the reference instruction.

[0043] In some embodiments, the preset rule may instruct the reference slots to be directly determined as sample slots, and the reference slot values ​​to be directly determined as sample slot values, thereby realizing rewriting of the reference command. Furthermore, by decomposing the reference command into a structure and then rewriting the command based on the decomposition results, it is possible to ensure that each request type and each request in the reference command are reflected in the newly generated sample command, thereby ensuring that the generated sample command is accurate.

[0044] In some embodiments, multiple sample slots and multiple sample slot values ​​may be input into a natural language instruction generation model to obtain sample instructions. The natural language instruction generation model may be a separate large model different from the large model mentioned above (e.g., a language large model similarly based on deep learning). The model may combine multiple sample slot values ​​corresponding to multiple sample slots to generate sample instructions based on natural language. By using a natural language instruction decomposition model to obtain the decomposition result for the reference instruction and using the natural language instruction generation model to generate sample instructions, a fully automatic sample instruction generation process is realized, requiring no human labor during the process, thereby reducing the cost of generating sample instructions and increasing the generation speed.

[0045] According to some embodiments, step S204 of generating sample instructions based on natural language based on the plurality of sample slots and the plurality of sample slot values ​​includes using a natural language instruction generation model to generate a plurality of sample instructions with different expressions but the same meaning based on the plurality of sample slots and the plurality of sample slot values. This method can automatically generate a large number of sample instructions with different expressions but the same meaning, thereby facilitating and accelerating the training of a large model for the corresponding slots and slot values. Furthermore, because these sample instructions have the same meaning and can correspond to the same ground truth answer data, a large amount of sample data (including sample instructions and corresponding ground truth answer data) can be generated quickly.

[0046] In one exemplary embodiment, based on five sample slots, namely, "Task," "Genre," "Topic," "Number of characters," and "Content request," and five sample slot values, namely, "Text creation," "Prose," "Alcohol," "<100 characters," and "Do not include the character 'alcohol'," a sample instruction such as "Write a prose piece, the topic of which is alcohol, the character 'alcohol' should not appear in the sentence, and the number of characters should be within 100 characters" can be generated. In the exemplary embodiment, since the sample slot and the reference slot are the same and the sample slot values ​​and the reference slot values ​​are the same, the plurality of second requests corresponding to the sample instruction and the plurality of first requests corresponding to the reference instruction are the same.

[0047] Return to step S203. According to some embodiments, the preset rule may instruct to randomly determine multiple sample slots for multiple reference slots and randomly determine multiple sample slot values ​​for multiple reference slot values, thereby enabling the random determination method to automatically and quickly generate a large number of sample instructions with different meanings and to generate sample instructions with slot combinations or slot value combinations that have not appeared in the training data or online log data, thereby improving the generalization performance of the large model.

[0048] According to some embodiments, the reference instruction may include a plurality of training instructions obtained from a training set. As shown in FIG. 3, step S203 of determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on a plurality of reference slots, a plurality of reference slot values, and a predetermined rule may include step S301 of collecting statistics on a distribution result of a plurality of conventional slots and a plurality of conventional slot values ​​corresponding to the plurality of training instructions, where the plurality of conventional slots include a plurality of reference slots corresponding to each training instruction in the plurality of training instructions, and the plurality of conventional slot values ​​include a plurality of reference slot values ​​corresponding to each training instruction in the plurality of training instructions; and step S302 of determining a plurality of sample slots and a plurality of sample slot values ​​based on the distribution result and the predetermined rule.

[0049] This allows for structured decomposition of the training commands in the training set, statistical analysis of the distribution of multiple conventional slots and multiple conventional slot values ​​corresponding to multiple training commands, and then determining which sample slots and sample slot values ​​should be included in the sample commands to be generated based on the statistical distribution results, thereby enabling the construction and supplementation of sample command data in a more directional and systematic manner.

[0050] In some embodiments, after decomposing the multiple training instructions, a structured command set can be constructed, which includes all slots (i.e., multiple conventional slots) and all slot values ​​(i.e., multiple conventional slot values) related to the multiple training instructions. The process of randomly determining sample slots / sample slot values ​​described in the above embodiments may also be performed by randomly sampling in the command set and combining new sample instructions. After obtaining the command set, the distribution results of the multiple conventional slots / conventional slot values, for example, the distribution count of each conventional slot / conventional slot value, may be calculated. It can be understood that the distribution result may further include other content, such as the number of co-occurrences described below, which is not limited here.

[0051] In some embodiments, the sample instructions included in a training set are always biased (e.g., have a high distribution of a small number of specific slots / slot values ​​or combinations, but a low distribution of other slots / slot values ​​or combinations), and using such a training set to train a large model will affect the quality of the answer data it generates.

[0052] According to some embodiments, the preset rule may instruct determining the plurality of sample slots based on at least some of the conventional slots with the smallest distribution counts among the plurality of conventional slots, and / or the preset rule may instruct determining the plurality of sample slot values ​​based on at least some of the conventional slot values ​​with the smallest distribution counts among the plurality of conventional slot values, thereby analyzing which slots or slot values ​​are seriously missing in the training instructions based on the distribution results, and determining the seriously missing slots or slot values ​​as the sample slots or sample slot values, thereby realizing directional construction and supplementation of sample instruction data, and further directional construction of training data corresponding to the defects of the large model, thereby overcoming the problem of biased training set, and improving the quality of answer data generated by the large model.

[0053] In some embodiments, the preset rule may instruct to determine some sample slots / sample slot values ​​of a plurality of sample slots by the above method, and to determine other sample slots / sample slot values ​​by another method (e.g., based on co-occurrence relationships or random sampling), or the preset rule may instruct to determine all sample slots / sample slot values ​​by the above method.

[0054] In some embodiments, a predetermined number of conventional slots / conventional slot values ​​with the smallest distribution count among a plurality of conventional slots / conventional slot values ​​may be determined as sample slots / sample slot values, or a conventional slot / conventional slot value with a distribution count that is less than a predetermined value among a plurality of conventional slots / conventional slot values ​​may be determined as sample slots / sample slot values, or conventional slots / conventional slots with a serious lack of training data may be determined as sample slots / sample slot values ​​by other methods, which are not limited here.

[0055] In some embodiments, in step S301, the co-occurrence status of multiple reference slot values / reference slots included in each of the multiple training instructions may be statistically analyzed to obtain a distribution result (e.g., co-occurrence frequency) of multiple conventional slot value combinations / conventional slot combinations. In one exemplary embodiment, a conventional slot combination consisting of two slots, "genre" and "number of characters," has a high co-occurrence frequency, and a conventional slot value combination consisting of two slot values, "essay" and ">800 characters," has a high co-occurrence frequency.

[0056] Generally, the fewer slots / slot values ​​included in a conventional slot combination / conventional slot value combination, the higher the co-occurrence frequency. However, there is little sample data with a large number of slots / slot values. As a result, when a large model processes a user's request type or a command with many requests and complex commands, the quality of the response data it generates is poor.

[0057] According to some embodiments, the preset rule may direct determining a plurality of sample slots based on a plurality of conventional slot combinations having the highest number of co-occurrences, the number of the plurality of sample slots being greater than the number of conventional slots included in any one of the plurality of conventional slot combinations, and / or the preset rule may direct determining a plurality of sample slot values ​​based on a plurality of conventional slot value combinations having the highest number of co-occurrences, the number of the plurality of sample slot values ​​being greater than the number of conventional slot values ​​included in any one of the plurality of conventional slot value combinations.

[0058] In this way, by collecting statistics on the number of co-occurrences of combinations of conventional slot combinations / conventional slot values, and determining multiple sample slots / sample slot values ​​based on the combinations of multiple conventional slot combinations / conventional slot values ​​that co-occur most frequently, it is possible to rationally construct more abundant and more complex sample commands, effectively improving the tracking ability of large models for complex commands.

[0059] In one exemplary embodiment, a first conventional slot combination consisting of two slots, "genre" and "number of characters," has a high co-occurrence frequency, a second conventional slot combination consisting of two slots, "task" and "genre," also has a high co-occurrence frequency, but a third conventional slot combination consisting of three slots, "task," "genre," and "number of characters," has a low co-occurrence frequency. Using the above method, multiple sample slots can be determined based on the first conventional slot combination and the second conventional slot combination. For example, the first conventional slot combination and the second conventional slot combination can be combined to determine the three slots, "task," "genre," and "number of characters," as the sample slot. It should be understood that other methods may also be used to determine multiple sample slots / sample slot values ​​based on the combinations of multiple conventional slot combinations / conventional slot values ​​with the highest co-occurrence frequency, and are not limited thereto.

[0060] In some embodiments, the preset rule may instruct to determine some sample slots / sample slot values ​​of a plurality of sample slots by the above method, and to determine other sample slots / sample slot values ​​by another method (e.g., based on co-occurrence relationships or random sampling), or the preset rule may instruct to determine all sample slots / sample slot values ​​by the above method.

[0061] In some embodiments, a predetermined number of conventional slots / conventional slot values ​​with the smallest distribution count among a plurality of conventional slots / conventional slot values ​​may be determined as sample slots / sample slot values, or a conventional slot / conventional slot value with a distribution count that is less than a predetermined value among a plurality of conventional slots / conventional slot values ​​may be determined as sample slots / sample slot values, or conventional slots / conventional slots with a serious lack of training data may be determined as sample slots / sample slot values ​​by other methods, which are not limited here.

[0062] According to some embodiments, the reference instruction may include a target instruction obtained from an online user log. The plurality of reference slots corresponding to the target instruction may include the target slot, and the plurality of reference slot values ​​corresponding to the target instruction may include the target slot value. The predetermined rule may direct setting the target slot as the sample slot in response to determining that the plurality of conventional slots do not include the target slot and / or that the distribution count of the target slot is less than a first predetermined value or less than the distribution count of at least some other conventional slots, or the predetermined rule may set the target slot value as the sample slot value in response to determining that the plurality of conventional slots do not include the target slot value or that the distribution count of the target slot value is less than a second predetermined value or less than the distribution count of at least some other conventional slot values.

[0063] By analyzing the target commands obtained from the online user logs and determining the sample slots / sample slot values ​​based on the frequency of occurrence of the target slots / target slot values ​​contained therein in the training commands, more directional construction and supplementation of sample command data can be performed, and corresponding training data can be constructed in a directional manner for defects in the large-scale model.Furthermore, by continuously mining the online logs, defects in the large-scale model can be continuously discovered, and corresponding command data can be constructed to supplement capabilities, ultimately promoting the large-scale model to achieve strong command following capabilities.

[0064] In some embodiments, each reference slot / reference slot value among a plurality of reference slots / reference slot values ​​corresponding to a target instruction may be determined as a target slot / target slot value using the above-mentioned preset rule. It can be understood that those skilled in the art may determine the ranges of the above-mentioned first preset value, second preset value, and at least some other conventional slots according to needs, and are not limited herein.

[0065] In some embodiments, the preset rule may instruct to determine some sample slots / sample slot values ​​of a plurality of sample slots by the above method, and to determine other sample slots / sample slot values ​​by another method (e.g., based on co-occurrence relationships or random sampling), or the preset rule may instruct to determine all sample slots / sample slot values ​​by the above method.

[0066] According to some embodiments, the large model may be a large-scale language model. As shown in Figure 4, the method for generating instruction data for the large model further includes step S405 of obtaining ground truth answer data corresponding to sample instructions, and step S406 of training the large model using the sample instructions and the ground truth answer data corresponding to the sample instructions. It can be understood that the operations of steps S401 to S404 in Figure 4 are similar to the operations of steps S201 to S204 in Figure 2, and therefore, description thereof will be omitted here.

[0067] In this way, by obtaining ground truth response data corresponding to the sample command, and using the ground truth response data and the sample command to train the large model, more complex, diverse, directional and directed command training data can be provided, and the command tracking ability of the large model can be effectively improved.

[0068] In some embodiments, in step S405, the ground truth answer data may be obtained by a manual annotation method, a model generation method, or other methods, and the present disclosure does not limit the scope of the present disclosure.

[0069] In some embodiments, in step S406, the sample commands may be input to the large model to obtain predicted answer data output by the large model, and a loss value may be calculated based on the ground truth answer data and the predicted answer data, and the parameters of the large model may be adjusted based on the loss value to train the large model. It should be understood that other methods may also be used to train the large model using the sample commands and the corresponding ground truth answer data, and are not limited thereto.

[0070] According to some embodiments, the step S406 of training the large model using the sample instructions and the ground truth answer data corresponding to the sample instructions may include adding the sample instructions and the ground truth answer data corresponding to the sample instructions to a training set to obtain a target training set, and training the large model using the target training set, thereby obtaining a large model with balanced and comprehensive capabilities by adding the sample instructions constructed based on the distribution results and the corresponding ground truth answer data to the training set and training the large model using the training set.

[0071] In some embodiments, the command set may be updated based on the updated target training set, and the distribution results of the updated command set may be statistically analyzed to generate new sample commands and sample data, thereby continuously improving the command following ability of the large model.

[0072] As described above, in some embodiments, a natural language instruction generation model may be used to generate multiple sample instructions with different expressions but the same meaning for multiple sample slots and multiple sample slot values, and only one ground truth answer data may be obtained for each of these sample instructions, and each sample instruction may be combined with the ground truth answer data and added to the training set.

[0073] According to another aspect of the present disclosure, there is provided an apparatus for generating instruction data for a large model. As shown in FIG. 5 , the apparatus 500 includes: a first acquisition unit 510 configured to acquire a reference instruction based on a natural language, the reference instruction being capable of instructing the large model to generate answer data that satisfy a plurality of first requests corresponding to a plurality of first request types; a second acquisition unit 520 configured to acquire a structured decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; a determination unit 530 configured to determine a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values, and a preset rule; and a generation unit 540 configured to generate sample instructions based on the plurality of sample slots and the plurality of sample slot values, the sample instruction being capable of instructing the large model to generate answer data that satisfy a plurality of second requests corresponding to the plurality of sample slot values.

[0074] As can be understood, the operations of units 510 to 540 in the device 500 are similar to the operations of steps S201 to S204 in FIG. 2, and the description thereof will be omitted here.

[0075] According to some embodiments, the pre-defined rules may direct that reference slots be directly determined as sample slots, and that reference slot values ​​be directly determined as sample slot values.

[0076] According to some embodiments, the pre-defined rules may dictate randomly determining multiple sample slots at multiple reference slots and randomly determining multiple sample slot values ​​at multiple reference slot values.

[0077] According to some embodiments, the reference instructions may include a plurality of training instructions obtained from a training set. The determination unit may include: a statistics subunit configured to calculate a distribution result of a plurality of conventional slots and a plurality of conventional slot values ​​corresponding to the plurality of training instructions, where the plurality of conventional slots include a plurality of reference slots corresponding to each training instruction in the plurality of training instructions and the plurality of conventional slot values ​​include a plurality of reference slot values ​​corresponding to each training instruction in the plurality of training instructions; and a determination subunit configured to determine a plurality of sample slots and a plurality of sample slot values ​​based on the distribution result and a predetermined rule.

[0078] According to some embodiments, the predetermined rule may direct determining the plurality of sample slots based on at least some of the conventional slots having the smallest distribution count among the plurality of conventional slots, and / or the predetermined rule may direct determining the plurality of sample slot values ​​based on at least some of the conventional slot values ​​having the smallest distribution count among the plurality of conventional slot values.

[0079] According to some embodiments, the preset rule may direct determining a plurality of sample slots based on a plurality of conventional slot combinations having the highest number of co-occurrences, the number of the plurality of sample slots being greater than the number of conventional slots included in any one of the plurality of conventional slot combinations, and / or the preset rule may direct determining a plurality of sample slot values ​​based on a plurality of conventional slot value combinations having the highest number of co-occurrences, the number of the plurality of sample slot values ​​being greater than the number of conventional slot values ​​included in any one of the plurality of conventional slot value combinations.

[0080] According to some embodiments, the reference instruction may include a target instruction obtained from an online user log. The plurality of reference slots corresponding to the target instruction includes a target slot, and the plurality of reference slot values ​​corresponding to the target instruction includes a target slot value. The predetermined rule may direct setting the target slot as a sample slot in response to determining that the plurality of conventional slots do not include the target slot or that the distribution count of the target slot is less than a first predetermined value or less than the distribution count of at least some other conventional slots, and / or the predetermined rule may direct setting the target slot value as a sample slot value in response to determining that the plurality of conventional slots do not include the target slot value or that the distribution count of the target slot value is less than a second predetermined value or less than the distribution count of at least some other conventional slot values.

[0081] According to some embodiments, the large model may be a large-scale language model based on deep learning. The apparatus 900 may further include (not shown) a third obtaining unit configured to obtain ground truth answer data corresponding to the sample instructions, and a training unit configured to train the large model using the sample instructions and the ground truth answer data corresponding to the sample instructions.

[0082] According to some embodiments, the training unit may include an update subunit configured to add sample instructions and ground truth answer data corresponding to the sample instructions to a training set to obtain a target training set, and a training subunit configured to train a large model using the target training set.

[0083] According to some embodiments, the generation unit may include a generation subunit configured to generate a plurality of sample instructions with different expressions but the same meaning, based on the plurality of sample slots and the plurality of sample slot values, using a natural language instruction generation model.

[0084] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0085] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.

[0086] Next, referring to FIG. 6 , a structural block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will be described, which is an example of a hardware device applicable to various aspects of the present disclosure. The electronic device represents various forms of digital electronic computers, such as laptop computers, desktop computers, stage computers, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing devices, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0087] 6, the electronic device 600 includes a computing unit 601, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may further store various programs and data necessary for the operation of the electronic device 600. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] The components of the electronic device 600 are connected to an I / O interface 605 and include an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information into the electronic device 600. The input unit 606 can receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackboard, trackball, control lever, microphone, and / or remote control. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 608 may include, but is not limited to, a magnetic disk, an optical disk, etc. The communication unit 609 enables the electronic device 600 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth® device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0089] The computing unit 601 may be any of a variety of general-purpose and / or special-purpose processing assemblies having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs each of the methods and processes described above, such as the method for generating instruction data for a large model. For example, in some embodiments, the method for generating instruction data for a large model may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, some or all of the computer program may be loaded and / or installed into the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, it may perform one or more steps of the method for generating instruction data for a large model described above. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for generating instruction data for a large model in any other suitable manner (eg, firmware).

[0090] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: Implemented in one or more computer programs, which may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and which may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0091] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0092] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection by one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] To provide for user interaction, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including audio input, voice input, and tactile input).

[0094] The systems and techniques described herein may be implemented in a computing system that includes background components (e.g., a data server), a computing system that includes middleware components (e.g., an application server), a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0095] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by running computer programs on the corresponding computers that have the client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server incorporating a blockchain.

[0096] It should be understood that steps may be reordered, added, or deleted using the various types of flows shown above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and this specification is not limited thereto, as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0097] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the appended claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. Furthermore, steps may be performed in a different order than described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In essence, as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. 1. A computer-implemented method for generating instruction data for a large model, the method being carried out by an apparatus for generating instruction data for a large model, the method comprising: obtaining a reference instruction based on natural language, the reference instruction capable of instructing the large model to generate response data satisfying a plurality of first requests corresponding to a plurality of first request types; obtaining a structural decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values, and a predetermined rule; generating sample instructions based on natural language based on the plurality of sample slots and the plurality of sample slot values, wherein the sample instructions can instruct the large model to generate answer data that satisfy a plurality of second requests corresponding to the plurality of sample slot values; the reference instructions include a plurality of training instructions obtained from a training set; Here, determining a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots based on the plurality of reference slots, the plurality of reference slot values, and a predetermined rule includes: Collecting statistics on distribution results of a plurality of conventional slots and a plurality of conventional slot values ​​corresponding to the plurality of training instructions, wherein the plurality of conventional slots includes a plurality of reference slots corresponding to each training instruction in the plurality of training instructions, and the plurality of conventional slot values ​​includes a plurality of reference slot values ​​corresponding to each training instruction in the plurality of training instructions; and determining the plurality of sample slots and the plurality of sample slot values ​​based on the distribution results and the predetermined rule.

2. The predetermined rule indicates determining the plurality of sample slots based on at least some legacy slots with the lowest distribution counts among the plurality of legacy slots; and / or The method of claim 1 , wherein the predetermined rule directs determining the plurality of sample slot values ​​based on at least some of the legacy slot values ​​that are least frequently distributed among the plurality of legacy slot values.

3. the predetermined rule instructs determining the plurality of sample slots based on a plurality of conventional slot combinations with the highest co-occurrence frequency, the number of the plurality of sample slots being greater than the number of conventional slots included in any one of the plurality of conventional slot combinations; and / or 2. The method of claim 1, wherein the predetermined rule instructs determining the plurality of sample slot values ​​based on a combination of a plurality of conventional slot values ​​that co-occur most frequently, and the number of the plurality of sample slot values ​​is greater than the number of conventional slot values ​​included in any one of the plurality of conventional slot value combinations.

4. The reference instruction includes a target instruction obtained from an online user log, the plurality of reference slots corresponding to the target instruction include a target slot, and the plurality of reference slot values ​​corresponding to the target instruction include a target slot value; wherein the predetermined rule indicates setting the target slot as a sample slot in response to determining that the plurality of conventional slots does not include the target slot, or that the distribution count of the target slot is less than a first predetermined value or less than the distribution count of at least some other conventional slots; and / or 2. The method of claim 1, wherein the predetermined rule instructs setting the target slot value as the sample slot value in response to determining that the plurality of conventional slot values ​​does not include the target slot value or that the distribution count of the target slot value is less than a second predetermined value or less than the distribution count of at least some other conventional slot values.

5. generating a natural language based sample instruction based on the plurality of sample slots and the plurality of sample slot values; 2. The method of claim 1, further comprising: utilizing a natural language instruction generation model to generate a plurality of sample instructions that are differently expressed but have the same meaning based on the plurality of sample slots and the plurality of sample slot values.

6. The method of claim 5 , wherein the predetermined rule directs that the reference slots are directly determined as the sample slots, and that the reference slot values ​​are directly determined as the sample slot values.

7. 2. The method of claim 1, wherein the predetermined rule instructs randomly determining the plurality of sample slots among the plurality of reference slots and randomly determining the plurality of sample slot values ​​among the plurality of reference slot values.

8. The large model is a large-scale language model based on deep learning, and the method includes: obtaining ground truth answer data corresponding to the sample instruction; The method of claim 1 , further comprising: training the large model using the sample instructions and ground truth answer data corresponding to the sample instructions.

9. training the large model using the sample instructions and ground truth response data corresponding to the sample instructions; adding the sample instructions and ground truth response data corresponding to the sample instructions to a training set to obtain a target training set; and training the large model using the target training set.

10. An apparatus for generating instruction data for a large model, comprising: a first acquisition unit configured to acquire a reference instruction based on a natural language, the reference instruction being capable of instructing a large model to generate answer data satisfying a plurality of first requests corresponding to a plurality of first request types; a second obtaining unit configured to obtain a structural decomposition result for the reference instruction to obtain a plurality of reference slots corresponding to the plurality of first request types and a plurality of reference slot values ​​corresponding to the plurality of first requests; a determining unit configured to determine a plurality of sample slots and a plurality of sample slot values ​​corresponding to the plurality of sample slots according to the plurality of reference slots, the plurality of reference slot values ​​and a predetermined rule; a generating unit configured to generate sample instructions based on natural language based on the plurality of sample slots and the plurality of sample slot values, the sample instructions being capable of instructing the large model to generate answer data that satisfy a plurality of second requests corresponding to the plurality of sample slot values; the reference instructions include a plurality of training instructions obtained from a training set; wherein the determination unit: a statistical subunit configured to generate statistics on distribution results of a plurality of conventional slots and a plurality of conventional slot values ​​corresponding to the plurality of training instructions, wherein the plurality of conventional slots include a plurality of reference slots corresponding to each training instruction in the plurality of training instructions, and the plurality of conventional slot values ​​include a plurality of reference slot values ​​corresponding to each training instruction in the plurality of training instructions; and a determination subunit configured to determine the plurality of sample slots and the plurality of sample slot values ​​based on the distribution results and the predetermined rule.

11. The predetermined rule indicates determining the plurality of sample slots based on at least some legacy slots with the lowest distribution counts among the plurality of legacy slots; and / or The apparatus of claim 10 , wherein the predetermined rule directs determining the plurality of sample slot values ​​based on at least some of the legacy slot values ​​that are least frequently distributed among the plurality of legacy slot values.

12. the predetermined rule instructs determining the plurality of sample slots based on a plurality of conventional slot combinations with the highest co-occurrence frequency, the number of the plurality of sample slots being greater than the number of conventional slots included in any one of the plurality of conventional slot combinations; and / or 11. The device of claim 10, wherein the predetermined rule instructs determining the plurality of sample slot values ​​based on a combination of a plurality of conventional slot values ​​that co-occur most frequently, and the number of the plurality of sample slot values ​​is greater than the number of conventional slot values ​​included in any one of the combinations of the plurality of conventional slot values.

13. The reference instruction includes a target instruction obtained from an online user log, the plurality of reference slots corresponding to the target instruction include a target slot, and the plurality of reference slot values ​​corresponding to the target instruction include a target slot value; wherein the predetermined rule indicates setting the target slot as a sample slot in response to determining that the plurality of conventional slots does not include the target slot, or that the distribution count of the target slot is less than a first predetermined value or less than the distribution count of at least some other conventional slots; or 11. The apparatus of claim 10, wherein the predetermined rule instructs setting the target slot value as the sample slot value in response to determining that the plurality of conventional slot values ​​does not include the target slot value or that the distribution count of the target slot value is less than a second predetermined value or less than the distribution count of at least some other conventional slot values.

14. The generating unit comprises: The apparatus of claim 10 , further comprising a generation subunit configured to generate a plurality of sample instructions having different expressions but the same meaning, based on the plurality of sample slots and the plurality of sample slot values, using a natural language instruction generation model.

15. The apparatus of claim 14 , wherein the predetermined rule directs directly determining the reference slots as the sample slots and directly determining the reference slot values ​​as the sample slot values.

16. The apparatus of claim 10 , wherein the predetermined rule instructs randomly determining the plurality of sample slots among the plurality of reference slots and randomly determining the plurality of sample slot values ​​among the plurality of reference slot values.

17. The large model is a large-scale language model based on deep learning, and the device: a third acquiring unit configured to acquire ground truth answer data corresponding to the sample instruction; and 14. The apparatus of claim 10, further comprising: a training unit configured to train the large model using the sample instructions and ground truth answer data corresponding to the sample instructions.

18. The training unit comprises: an update subunit configured to add the sample instructions and ground truth answer data corresponding to the sample instructions to a training set to obtain a target training set; and a training subunit configured to train the large model utilizing the target training set.

19. An electronic device, the electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. 10. A non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause the computer to perform the method of any one of claims 1 to 9.

21. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images