Instruction generation method, apparatus, computing system, and computing device

Abstract

The present application relates to the technical field of AI, and provides an instruction generation method, an apparatus, a computing system, and a computing device. In the method, the generative capability of a large language model is utilized; a prompt word is inputted into the large language model to generate an instruction set related to a document fragment; then, the instruction set is evaluated, and when the evaluation result does not satisfy a preset condition, the prompt word is adjusted in time on the basis of the evaluation result, so as to determine a target instruction set related to the document fragment. In this way, a feedback relationship is established between instruction generation and instruction evaluation, enabling timely adjustment of a prompt word on the basis of an instruction evaluation result during the efficient generation of an instruction set, i.e., enabling adjustment of the strategy of a large language model to generate an instruction set, thereby improving the quality of the finally obtained instruction set.

Description

Instruction generation method and device, computing system and computing equipment

[0001] The present application claims priority to the Chinese patent application No. 202411368904.X, filed on September 27, 2024, and entitled "Instruction generation method and device, computing system and computing equipment", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an instruction generation method and device, a computing system and a computing equipment. BACKGROUND

[0003] With the rapid development of large language model (LLM) technology, there is an increasing demand for applying LLM to professional fields. Currently, when applying LLM to professional fields, it is usually necessary to fine-tune LLM based on the instruction set of the field. The instruction set is a data set constructed for a specific task, which contains explicit questions and corresponding expected outputs, and can help LLM learn the specific task of the professional field, so that LLM can accurately generate outputs meeting the requirements according to the given instructions.

[0004] In related technologies, when applying LLM to professional fields, the instruction set is constructed in the following way: a large number of labeled instructions are used to train an instruction generation model, the trained instruction generation model is used to generate instructions for professional fields, and at the same time, an instruction scoring model is trained to score and evaluate the generated instructions, and then instructions meeting the conditions are selected.

[0005] However, the above method needs to rely on a large number of labeled instructions for model training, which is complex and inefficient. SUMMARY

[0006] The present application provides an instruction generation method, device, computing system and computing equipment, which utilizes the generation capability of large language model, and can improve the quality of the instruction set on the basis of efficiently generating the instruction set.

[0007] In a first aspect, the present application provides an instruction generation method applied to a scenario of constructing an instruction set according to a specified field, wherein the present application does not limit the field related to the instruction set, for example, it can be the field of history, literature and art, music, medicine, communication technology, programming technology, etc. The method comprises:

[0008] inputting a first prompt word into a large language model to generate a first instruction set related to a document segment;

[0009] The first instruction set is evaluated to obtain a first evaluation result, and the first evaluation result indicates the quality of each instruction in the first instruction set.

[0010] When the first evaluation result does not satisfy the preset condition, a second prompt word is obtained by adjusting the first prompt word based on the first evaluation result, so as to determine a target instruction set related to the document segment according to the second prompt word.

[0011] In the above method, the generation capability of the large language model is utilized, the prompt word is input into the large language model to generate an instruction set related to the document segment, then the instruction set is evaluated, and when the evaluation result does not satisfy the preset condition, the prompt word is adjusted in a timely manner according to the evaluation result to determine a target instruction set related to the document segment. In this way, a feedback relationship between instruction generation and instruction evaluation is established, and the prompt word can be adjusted in a timely manner according to the instruction evaluation result in the process of efficiently generating the instruction set, that is, the strategy of the large language model for generating instructions is adjusted, so that the quality of the finally obtained instruction set is improved. In addition, the above method can input the prompt word into an existing large language model, so that the large language model quickly generates a corresponding instruction set according to the indication of the prompt word, and the quality of the generated instruction set is ensured by combining the instruction evaluation result, so that a special instruction generation model does not need to be trained by relying on labeled instructions, and convenience and high efficiency are achieved. In contrast, in related technologies, when LLM is applied to a professional field, the way of constructing an instruction set needs to rely on a large number of labeled instructions for model training, which is complex and inefficient.

[0012] In some embodiments, the first prompt word includes the document segment and / or at least one labeled instruction. The first prompt word includes the document segment and / or at least one labeled instruction means that the first prompt word can include the document segment, or the document segment and at least one labeled instruction. The at least one labeled instruction can be a labeled instruction randomly selected from a labeled instruction set, or a labeled instruction selected by a user, which is not limited in the present application. When the first prompt word includes the document segment and at least one labeled instruction, the large language model can have both knowledge of a specified field and examples of instructions when generating instructions, because the first prompt word includes both the document segment of the specified field and the labeled instruction, so that the generated instructions can be applied directly to the specified field, improving the application effect, for example, using the generated instructions to train the large language model, so that the large language model quickly learns the knowledge of the specified field, and the like.

[0013] In some embodiments, the first prompt word further includes a constraint condition, and the constraint condition is used to describe a feature that the instruction needs to have; the second prompt word is obtained by adjusting the first prompt word based on the first evaluation result, including: adjusting at least one of the at least one labeled instruction and the constraint condition based on the first evaluation result to obtain the second prompt word.

[0014] When the first evaluation result does not satisfy the preset condition, it indicates that the overall quality of the instructions in the first instruction set does not meet the requirements (or the number of instructions with qualified quality is small). Since the first evaluation result can indicate the quality of each instruction in the first instruction set, the first prompt can be adjusted based on the first evaluation result, at least one of the at least one labeled instruction and the constraint condition is adjusted to obtain a second prompt. For example, the instructions with qualified quality indicated by the first evaluation result are used as new labeled instructions. In this way, the second prompt can guide the large language model to generate instructions based on the new labeled instructions, thereby increasing the number of instructions with qualified quality. For another example, the constraint condition is adjusted based on the instructions with unqualified quality indicated by the first evaluation result. In this way, the second prompt can guide the large language model to generate instructions based on the adjusted constraint condition, thereby improving the quality of the instructions.

[0015] In some embodiments, based on the first evaluation result, at least one of the at least one labeled instruction and the constraint condition is adjusted to obtain a second prompt, including:

[0016] Based on the first evaluation result, at least one first instruction that meets the evaluation rule is determined from the first instruction set, and the evaluation rule is used to evaluate the characteristics of the instructions to measure the quality of the instructions.

[0017] Based on the at least one first instruction and the constraint condition, a second prompt is obtained.

[0018] In the above manner, when the first evaluation result does not satisfy the preset condition, since the first evaluation result can indicate the quality of each instruction in the first instruction set, the labeled instructions in the first prompt can be adjusted based on the first evaluation result to obtain a second prompt. For example, the first instructions that meet the evaluation rule are used as labeled instructions in the second prompt, or the first instructions are added to the first prompt to obtain the second prompt. In this way, the second prompt can guide the large language model to generate a new instruction set that meets the evaluation rule based on the adjusted labeled instructions, thereby increasing the number of instructions with qualified quality in the instruction set and improving the quality of the instruction set.

[0019] In some embodiments, based on the first evaluation result, at least one of the at least one labeled instruction and the constraint condition is adjusted to obtain a second prompt, including:

[0020] At least one second instruction that does not meet the evaluation rule is determined from the first instruction set;

[0021] Based on the evaluation rule that is not met by the at least one second instruction, the constraint condition is adjusted to obtain an adjusted constraint condition;

[0022] Based on the adjusted constraint condition and the at least one labeled instruction, a second prompt is obtained.

[0023] In this way, when the first evaluation result does not satisfy the preset condition, since the first evaluation result can indicate the quality of each instruction in the first instruction set, the constraint condition in the first prompt word can be adjusted in a timely manner according to the first evaluation result to obtain a second prompt word, for example, a new feature required by the instruction is determined according to the evaluation rule that the second instruction does not meet, and the new feature is added to the constraint condition to obtain the second prompt word, so that the second prompt word can guide the large language model to generate instructions with the adjusted constraint condition, and the instructions that meet the feature requirements are regenerated, thereby improving the quality of the instruction set.

[0024] In some embodiments, the preset condition refers to a qualified rate of the instructions included in the first instruction set being greater than a threshold value, and the qualified rate indicates a proportion of the instructions in the first instruction set that meet the evaluation rule.

[0025] In some embodiments, the first instruction set is evaluated to obtain the first evaluation result, including: evaluating each instruction in the first instruction set by a large language model or an instruction scoring model to obtain the first evaluation result.

[0026] In this way, the quality of each instruction in the first instruction set is quickly evaluated by using the capability of the large language model, and the efficiency of obtaining the first evaluation result is improved on the basis of intuitively reflecting the quality of each instruction. Alternatively, the quality of each instruction in the first instruction set is quantitatively scored by using the instruction scoring model, which can intuitively reflect the quality of the instructions in the first instruction set, and facilitate subsequent screening of the instructions in the first instruction set.

[0027] In some embodiments, the target instruction set is used for at least one of the following: training a large language model; storing to a knowledge base to realize retrieval augmented generation (RAG); and storing to a question and answer database to realize question and answer pair matching. Since the instruction generation method provided in the present application can generate a high-quality instruction set, when the target instruction set is applied to the above scenarios, the accuracy of the answers can be improved. Moreover, the application scenarios of the target instruction set are not limited to the above few scenarios, which are not limited by the present application.

[0028] In a second aspect, the present application provides an instruction generation device, the device comprising at least one functional module, and the at least one functional module is configured to implement the instruction generation method provided in the first aspect or any possible implementation manner of the first aspect.

[0029] In a third aspect, the present application provides a computing system, the system comprising a host and an acceleration card, the host being configured to control the acceleration card, and the acceleration card being configured to run a large language model, and the system being configured to implement the instruction generation method provided in the first aspect or any possible implementation manner of the first aspect.

[0030] In a fourth aspect, the present application provides a computing device, comprising a processor and a memory, wherein the processor is configured to execute at least one piece of program code stored in the memory, so as to enable the computing device to implement the instruction generation method provided in the first aspect or any possible implementation manner of the first aspect.

[0031] In a fifth aspect, the present application provides a computer program product, which, when executed on a computing device, enables the computing device to implement the instruction generation method provided in the first aspect or any possible implementation manner of the first aspect. The computer program product can be a software package, which can be downloaded and executed on the computing device when the foregoing method needs to be implemented.

[0032] In a sixth aspect, the present application provides a computer readable storage medium, which is configured to store at least one piece of program code, and the at least one piece of program code is configured to implement the instruction generation method provided in the first aspect or any possible implementation manner of the first aspect. The storage medium can include, but is not limited to, a volatile memory such as a random access memory, a non-volatile memory such as a flash memory, a hard disk drive (HDD) and a solid state drive (SSD). BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application;

[0034] FIG. 2 is a schematic diagram of a server according to an embodiment of the present application;

[0035] FIG. 3 is a structural schematic diagram of a computing device according to an embodiment of the present application;

[0036] FIG. 4 is a schematic diagram of an instruction generation method according to an embodiment of the present application;

[0037] FIG. 5 is an architectural schematic diagram of a computing system according to an embodiment of the present application;

[0038] FIG. 6 is a flowchart of an instruction generation method according to an embodiment of the present application;

[0039] FIG. 7 is a schematic diagram of an instruction generation method according to an embodiment of the present application;

[0040] FIG. 8 is a schematic diagram of another instruction generation method according to an embodiment of the present application;

[0041] FIG. 9 is a structural schematic diagram of an instruction generation apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] For the purposes of the present application, the technical solutions and advantages will be clearer, the embodiments of the present application will be described in further detail below with reference to the drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the prompt words, document fragments, labeling instructions and the like involved in the present application are obtained under full authorization.

[0043] For the convenience of understanding, the key terms and key concepts involved in the present application will be described first.

[0044] An artificial intelligence (AI) model is a mathematical algorithm model that uses machine learning ideas to solve practical problems. Generally, an AI model includes a large number of parameters and calculation formulas (or calculation rules).

[0045] An acceleration card, also known as an acceleration device, an accelerator or an acceleration chip, is a special hardware device or computer system designed to accelerate the calculation process in AI scenarios. In the embodiments of the present application, the acceleration device is, for example, a graphics processing unit (GPU), a neural network processing unit (XPU), an intelligent processing unit (IPU), a tensor processing unit (TPU), a domain specific architecture (DSA) chip, etc., without being limited thereto.

[0046] A large language model (LLM) is an AI language processing model trained on large-scale text data. LLM is usually trained on a large amount of text data, which comes from a wide range of sources, including Internet web pages, books, news articles, academic papers, social media posts, etc. LLM can understand the questions raised by users and generate accurate answers. Through learning a large amount of knowledge, LLM can answer questions in various fields, including history, science, technology, culture, etc.

[0047] Supervised fine-tuning (SFT), also known as instruction tuning, refers to further fine-tuning a large language model (LLM) that has already been trained by using labeled task-specific data to enable the model to follow instructions. When applying an LLM to a specific field, in order to enable the model to have the ability to understand and respond to human instructions, it is also necessary to fine-tune the model using the instruction set of that field.

[0048] An instruction set is a data set constructed for a specific task, containing explicit questions and corresponding expected outputs, which can help an LLM learn specific tasks in a professional field, enabling the LLM to accurately generate outputs that meet requirements according to given instructions.

[0049] Labeled instructions are instructions obtained by labeling documents by machines or manually, which are usually accurate and reliable. In some scenarios, labeled instructions are also referred to as seed instructions. For example, given a document containing "XXX is a practical programming tool", the labeled instruction obtained by labeling this part of the content is "Question: What is XXX? Answer: XXX is a programming tool".

[0050] A prompt is an input provided to an AI model to guide and stimulate the AI model to generate corresponding responses or content. For example, a prompt is a complete text content (or a sentence), which can be used as an input to an LLM, enabling the LLM to output corresponding reasoning results based on the prompt.

[0051] A prompt template is a template text that includes variables and fixed text content, which can dynamically generate prompts by inputting specific values of variables. For example, in practical applications, by adjusting the specific values of variables in the prompt template, multiple different prompts can be dynamically generated. Illustratively, taking the use of an LLM to generate an instruction set as an example, the prompt template is:

Please generate instructions based on document content {variable}

Please generate instructions based on document content {XXX is a practical programming tool}

[0052] Retrieval augmented generation (RAG) is a method that combines retrieval technology with generative models, aiming to improve the accuracy, relevance and reliability of generated text. For example, when an LLM answers a question about a historical event, the system will retrieve relevant historical records, biographies of historical figures, and other text fragments from a historical literature knowledge base, and then combine these text fragments with the LLM to generate the final answer.

[0053] The application scenarios and implementation environments of the present application are introduced as follows.

[0054] The present application is applied to the scenario of constructing an instruction set according to a specified field, and the present application is not limited to the field involved in the instruction set, for example, it can be a field of history, literature, music, medicine, communication technology, programming technology, etc.

[0055] In the related art, when an LLM is applied to a professional field, the way to construct an instruction set is, for example, to train an instruction generation model using a large number of labeled instructions, to generate instructions in the professional field using the trained instruction generation model, and to train an instruction scoring model to score and evaluate the generated instructions, and then to select instructions that meet the conditions. However, this approach requires a large number of labeled instructions for model training, which is complex and inefficient.

[0056] Based on this, the present application provides an instruction generation method that utilizes the generation capability of a large language model (LLM) and combines the evaluation results of the instruction set during the generation of the instruction set, thereby improving the quality of the instruction set on the basis of efficiently generating the instruction set. The type of LLM is not limited in the present application, and any LLM that can provide instruction generation function is applicable to the present application. For example, the LLM is an open source LLM.

[0057] The implementation environment of the present application is introduced as follows with reference to FIGS. 1-3.

[0058] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application. As shown in FIG. 1, the implementation environment includes a computing system 100, which includes a host 101 and an acceleration card 102, and the host 101 and the acceleration card 102 are in communication connection. In some embodiments, the acceleration card 102 is also referred to as an acceleration device, an accelerator, etc., and the present application is not limited thereto.

[0059] In the embodiments of the present application, the computing system 100 has instruction generation capability, and can determine a prompt word according to a document in a specified field, and input the prompt word into a large language model to obtain an instruction set related to the document. Illustratively, the host 101 is configured to provide an AI service, and the host 101 can control the accelerator card 102 to perform various computing tasks involved in the instruction generation process. For example, in response to an instruction generation request for a document segment, the host 101 determines the prompt word corresponding to the document segment, sends the prompt word to the accelerator card 102, controls the accelerator card 102 to run the large language model to process the prompt word, and generates an instruction set related to the document segment.

[0060] The number of the host 101 can be one or more, which is not limited in the present application. The accelerator card 102 can be a GPU, an XPU, an IPU, a TPU, a DSA chip, etc., which is not limited in the present application. In addition, the number of the accelerator card 102 shown in FIG. 1 is only illustrative, and the number of the accelerator card 102 can be more or less, which is not limited in the present application.

[0061] In the computing system 100, each accelerator card 102 can be connected through a high-speed interconnection link (or a chip bus) for communication, so that different accelerator cards 102 can quickly access the memory of each other through the high-speed interconnection link, and high-efficiency data transmission between different accelerator cards 102 can be realized. For example, the high-speed interconnection link can be a NvLink, a CXL, a UCIe, a HCCS, a CCIX, etc., which is not limited in the present application.

[0062] In addition, in the computing system 100, the host 101 and the accelerator card 102 can be integrated in one server, or can be separately arranged, which is not limited in the present application. Illustratively, taking the case that the host 101 and the accelerator card 102 are integrated in one server as an example, referring to FIG. 2, FIG. 2 is a schematic diagram of a server according to an embodiment of the present application, as shown in FIG. 2, the host 101 and the accelerator card 102 are connected through a PCIe link for communication, and the host 101 and the accelerator card 102 exchange data through the PCIe link. For example, the server is a stand-alone physical server.

[0063] In some embodiments, the computing system 100 can also be deployed in a server cluster composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. Taking the cloud server as an example, the cloud server is also called a cloud platform (i.e., an abbreviation of a cloud computing platform), which is a service based on hardware resources and software resources, providing computing, network, and storage capabilities. Through the network "cloud", huge data computing is processed and analyzed in the remote end and then returned to the user, having characteristics such as large-scale, distribution, virtualization, high availability, scalability, on-demand service, and security. The cloud platform can realize the rapid delivery and release of configurable computing resources with small management cost or low interaction complexity between the user and the service provider.

[0064] In some embodiments, the wireless network or the wired network described above uses standard communication technologies and / or protocols. The network is usually a transmission control protocol / internet protocol (TCP / IP) network and an RDMA network in a data center network, such as an RDMA over converged Ethernet (RoCE) network, an InfiniBand (IB) network, etc., without limitation. In some other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0065] The hardware structure of the host 101 in the computing system 100 described above will be introduced below. The present application provides a computing device that can be configured as the host 101 described above. Referring to FIG. 3, FIG. 3 is a structural schematic diagram of a computing device provided by an embodiment of the present application. As shown in FIG. 3, the computing device 300 includes a memory 301, a processor 302, a communication interface 303, and a bus 304. The memory 301, the processor 302, and the communication interface 303 are communicatively connected to each other through the bus 304.

[0066] The memory 301 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. Illustratively, the memory 301 is used to store at least one program code, and when the program code stored in the memory 301 is executed by the processor 302, the processor 302 is used to execute the following instruction generation method.

[0067] The processor 302 can be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling program execution of the scheme of the present application. The processor 302 can be a single-CPU processor or a multi-CPU processor. The number of processors 302 can be one or more.

[0068] The communication interface 303 uses a transceiving module such as a transceiver to realize communication between the computing device 300 and other devices or communication networks. For example, data can be obtained through the communication interface 303.

[0069] The memory 301 and the processor 302 can be separately arranged or integrated together.

[0070] The bus 304 can include a path for transmitting information between various components (e.g., the memory 301, the processor 302, the communication interface 303) of the computing device 300.

[0071] Based on the above introduction of the application scenario and implementation environment of the present application, the instruction generation method provided by the present application will be introduced as follows.

[0072] For ease of understanding, the principles of the present application will be introduced first with reference to FIG. 4 and FIG. 5.

[0073] FIG. 4 is a schematic diagram of the principles of an instruction generation method according to an embodiment of the present application. As shown in FIG. 4, the instruction generation method provided by the present application involves the following parts: documents of a specified field, a large language model LLM, instruction generation, and instruction evaluation. Illustratively, after the computing system obtains the documents of the specified field, the documents are segmented to obtain multiple document segments (also referred to as domain knowledge), and the annotated instructions (also referred to as seed instructions) obtained based on the documents are obtained. In the process of generating instructions based on the LLM, the prompt words are generated in combination with the domain knowledge and / or the annotated instructions, the prompt words are input into the LLM, the instruction set of the field is generated, and it should be understood that the content of the prompt words can also be understood as an instruction generation strategy, which can guide the LLM to generate the corresponding instruction set according to the prompt words. Then, the instruction set generated by the LLM is evaluated to obtain an evaluation result to indicate the quality of each instruction, and the evaluation result is used to provide feedback for the instruction generation strategy, for example, adjusting the content of the prompt words, so that the LLM regenerates the instruction set of the field according to the adjusted prompt words. For another example, the evaluation result indicates that the quality of the currently generated instruction set meets the requirements, and the currently generated instruction set is taken as the final instruction set (the currently generated instruction set can also be further screened). In this way, a feedback relationship between instruction generation and instruction evaluation is established, which can adjust the instruction generation strategy in time according to the instruction evaluation result during the instruction generation process, and thus improve the quality of the finally generated instruction set.

[0074] Based on the principles shown in FIG. 4, the functions of the computing system provided by the present application will be introduced in combination with the computing system involved in the above-mentioned implementation environment. FIG. 5 is a schematic diagram of the architecture of a computing system according to an embodiment of the present application. As shown in FIG. 5, the computing system is configured to provide an instruction generation function 501 and an instruction evaluation function 502.

[0075] The instruction generation function 501 includes prompt word determination, instruction generation, instruction analysis, and instruction filtering, etc. For example, the computing system determines the prompt words according to the document segments of the specified field and the annotated instructions, inputs the prompt words into the large language model, generates multiple instructions related to the document segments, and obtains the instruction set related to the document segments after analyzing (such as unifying the instruction format, etc.) and filtering (such as screening the instructions containing the same question) the multiple instructions. Illustratively, the instruction generation function 501 is realized by the host and the accelerator card of the computing system in cooperation, for example, the host determines the prompt words and sends the prompt words to the accelerator card, controls the accelerator card to run the large language model, and generates multiple instructions related to the document segments.

[0076] The instruction evaluation function 502 includes instruction evaluation, result feedback, result screening, etc. For example, the computing system evaluates the instruction set generated by the instruction generation function 501 based on the evaluation rules to obtain an evaluation result indicating the quality of each instruction. According to the indication of the evaluation result, feedback is provided for the instruction generation strategy, such as adjusting the content of the prompt word; or further screening the current generated instruction set. Through the above process, the target instruction set related to the document segment is obtained. Illustratively, the instruction evaluation function 502 is implemented by the host and the acceleration card of the computing system in cooperation, for example, the host controls the acceleration card to run the large language model, and evaluates the instruction set to obtain the evaluation result. According to the indication of the evaluation result, the host adjusts the content of the prompt word or performs the instruction set screening step, etc.

[0077] In addition, the function division of the computing system is not limited to the content shown in FIG. 5. In actual application, more functions can be set according to user needs, for example, the computing system is also used to provide storage function, for providing storage space for document segments, annotation instructions, large language models, instruction sets, etc. In addition, the functions provided by the computing system can be allocated to the host and / or the acceleration card according to needs, which is not limited in the present application.

[0078] The process of the instruction generation method performed by the computing system provided by the present application will be introduced below with reference to the embodiment shown in FIG. 6.

[0079] FIG. 6 is a flowchart of an instruction generation method according to an embodiment of the present application. As shown in FIG. 6, taking the method performed by the computing system as an example, the method includes the following steps 601 to 605.

[0080] 601. The computing system obtains a document segment and an annotation instruction.

[0081] In the embodiment of the present application, the document segment and the annotation instruction belong to the same field, wherein the document segment refers to the text obtained by segmenting the document of the specified field, and the annotation instruction refers to the instruction obtained by annotating the document of the specified field. In this step, the present application does not limit the number and source of the document segments obtained by the computing system, and the number and source of the annotation instructions. For example, the document segment can be a document segment obtained by segmenting the document by the computing system after obtaining the document, or a document segment uploaded by the user. For another example, the annotation instruction can be a publicly authorized annotation instruction, or a user-uploaded annotation instruction, etc. Illustratively, when the number of annotation instructions obtained by the computing system is more than one, the plurality of annotation instructions is also referred to as an annotation instruction set. In addition, the present application does not limit the field to which the document segment and the annotation instruction belong, for example, it can be the field of history, literature, music, medicine, communication technology, programming technology, etc.

[0082] In some embodiments, taking the splitting of the obtained document into document segments by the computing system as an example, after obtaining the document, the computing system performs document parsing to convert the document into a format convenient for subsequent processing by the computing system, and then the computing system splits the obtained document into multiple document segments in a sentence segmentation or fixed-length segmentation manner. It should be understood that, since the document formats are various (such as.txt,.csv,.doc,.docx,.xls,.xlsx,.ppt,.pptx, PDF file, web file, and the like), the sources of the document are also various, and may contain noise, impurities, and inconsistent formats, therefore, by performing document parsing, text cleaning and standardization processing, and the like, the document is converted into a unified format, which is convenient for the computing system to continue processing. For example, the text cleaning is to remove special characters, stop words, and redundant spaces; and the standardization processing is to convert the text into a unified case, stem extraction, and part-of-speech tagging.

[0083] 602. The computing system inputs the first prompt word into the large language model to generate a first instruction set related to the document segment.

[0084] In the embodiments of the present application, the first prompt word includes the document segment and / or at least one annotation instruction. After obtaining the document segment and the annotation instruction through the foregoing step 601, for any one document segment, the computing system fills the document segment and / or at least one annotation instruction into a first prompt word template of the large language model to obtain a first prompt word, inputs the first prompt word into the large language model to generate a first instruction set related to the document segment, wherein the first instruction set includes multiple instructions. For example, the host in the computing system sends the first prompt word template to the accelerator card, and sends the document segment and / or at least one annotation instruction to the accelerator card, the accelerator card fills the document segment and / or at least one annotation instruction into the first prompt word template of the large language model to obtain the first prompt word, inputs the first prompt word into the large language model, and runs the large language model to generate the instruction set. In some embodiments, the first prompt word is uploaded by a user, which is not limited in the present application.

[0085] In addition, the first prompt word includes a document segment and / or at least one labeling instruction. The first prompt word can include a document segment, or can include a document segment and at least one labeling instruction. The at least one labeling instruction can be a randomly selected labeling instruction from a labeling instruction set, or can be a labeling instruction selected by a user, which is not limited in the present application. In the case where the first prompt word includes a document segment and at least one labeling instruction, since the first prompt word includes both a document segment of a specified field and a labeling instruction, the large language model can have both knowledge of the specified field and examples of instructions when generating instructions, so as to generate instructions in the specified field that conform to the examples. Based on this, the generated instructions can be directly applied to the specified field, improving the application effect, for example, using the generated instructions to train the large language model, so that the large language model quickly learns the knowledge of the specified field, and the like.

[0086] In some embodiments, the first prompt word further includes a constraint condition for describing a feature that the instruction generated by the large language model needs to have, or in other words, by adding the constraint condition in the first prompt word, the large language model is guided to generate an instruction with a specified feature. The feature that the instruction needs to have includes a subjective feature of the instruction and / or an objective feature of the instruction. The subjective feature refers to a feature that is affected by human factors, and the objective feature refers to a feature that can be quantified and is not easily affected by human factors. Illustratively, the subjective feature of the instruction includes, but is not limited to, at least one of the following: accuracy, used to represent the degree of correspondence between the instruction and the objective fact; logic, used to represent the clarity of the language logic of the instruction; completeness, used to represent the comprehensiveness of the content of the instruction; language quality, used to represent the quality of the textual expression of the instruction; and the like. The objective feature of the instruction includes, but is not limited to, at least one of the following: the number of instruction words, the number of instructions, the sentence structure of the instruction, the language of the instruction, and the like.

[0087] Correspondingly, the constraint condition includes, for example, at least one of the following: the instruction needs to have accuracy, the instruction needs to have logic, the instruction needs to have completeness, the number of instruction words is greater than XX, the number of instructions is greater than YY, the sentence structure of the instruction is ZZ structure, the language of the instruction is WW language, and the like, which is not limited in the present application.

[0088] It should be understood that the content of the first prompt word can also be understood as a kind of instruction generation strategy, which can guide the large language model to generate a corresponding first instruction set according to the first prompt word. The content of the first prompt word is exemplified as follows. Taking the first prompt word template

Please generate instructions based on the following content: document segment {variable 1}; labeling instruction {variable 2}; constraint condition {variable 3}

[0089] For example, the document fragment is "XXX is a practical programming tool", the annotation instruction is "question: what is XXX; answer: XXX is a programming tool", and the constraint condition is "generate 3 instructions; the number of characters in the question of each instruction is not more than 10; the number of characters in the answer of each instruction is greater than 5; the instruction has accuracy". Based on this, the first prompt word is

please generate instructions based on the following content: document fragment {XXX is a practical programming tool}; annotation instruction {question: what is XXX; answer: XXX is a programming tool}; constraint condition {generate 3 instructions; the number of characters in the question of each instruction is not more than 10; the number of characters in the answer of each instruction is greater than 5}

[0090] 603、The computing system evaluates the first instruction set to obtain a first evaluation result, the first evaluation result indicating the quality of each instruction in the first instruction set.

[0091] In the embodiment of the present application, for each instruction in the first instruction set, the computing system evaluates the instruction based on the evaluation rule to obtain the first evaluation result. The evaluation rule can be a default evaluation rule configured by the computing system or an evaluation rule uploaded by the user, which is not limited in the present application. In addition, the number of evaluation rules can be one or more, which is not limited in the present application. It should be understood that the evaluation rule is used to evaluate the characteristics of the instruction to measure the quality of the instruction, that is, for the instruction generated by the large language model, the computing system can evaluate the characteristics currently possessed by the instruction according to the evaluation rule to obtain the quality of the instruction. Illustratively, the characteristics of the instruction evaluated by the evaluation rule include the objective characteristics of the instruction and / or the subjective characteristics of the instruction.

[0092] For example, the instruction feature evaluated by the evaluation rule is the subjective feature-accuracy, and the evaluation rule can be "whether the answer in the instruction contains incorrect or misleading expressions"; the instruction feature evaluated by the evaluation rule is the subjective feature-logic, and the evaluation rule can be "whether the answer in the instruction is self-contradictory or unreasonable"; the instruction feature evaluated by the evaluation rule is the subjective feature-completeness, and the evaluation rule can be "whether the answer in the instruction is a comprehensive response to the question"; the instruction feature evaluated by the evaluation rule is the subjective feature-language quality, and the evaluation rule can be "whether the grammar of the question and answer in the instruction is correct, whether the words are appropriate and accurate, etc." That is, the content of the evaluation rule can be refined according to the subjective feature of the instruction to be evaluated, and the subjective feature of the instruction is measured by specific evaluation indicators, so as to obtain the quality of the evaluated instruction.

[0093] For example, the instruction feature evaluated by the evaluation rule is the objective feature-instruction word count, and the evaluation rule can be "whether the word count of the instruction is greater than XX"; the instruction feature evaluated by the evaluation rule is the objective feature-instruction quantity, and the evaluation rule can be "whether the instruction quantity is greater than YY"; the instruction feature evaluated by the evaluation rule is the objective feature-instruction sentence structure, and the evaluation rule can be "whether the sentence structure of the instruction is ZZ structure"; the instruction feature evaluated by the evaluation rule is the objective feature-instruction language, and the evaluation rule can be "whether the language of the instruction is WW language". That is, the content of the evaluation rule can be set according to the objective feature of the instruction to be evaluated, and the objective feature of the instruction is accurately measured by specific evaluation indicators, so as to obtain the quality of the evaluated instruction.

[0094] It should be noted that the content of the evaluation rule is not limited in the present disclosure, and appropriate evaluation rules can be selected in actual application to improve the comprehensiveness of the evaluation result. In addition, based on the foregoing introduction of the first prompt word, in some scenarios, the first prompt word can include a constraint condition, and the constraint condition is used to describe the features that the instruction generated by the large language model needs to have. The feature evaluated by the evaluation rule can also not include the feature described by the constraint condition, or more than the feature described by the constraint condition. It should be understood that the constraint condition in the first prompt word and the evaluation rule in this step are two types of information with different functions. The constraint condition is to guide the large language model to generate instructions with specified features, and the evaluation rule is to reflect the quality of the instruction by evaluating the features of the instruction. In some embodiments, the feature described by the constraint condition is different from the feature evaluated by the evaluation rule, for example, the constraint condition is used to describe the objective feature of the instruction, and the evaluation rule is used to evaluate the subjective feature of the instruction. In other embodiments, the feature described by the constraint condition and the feature evaluated by the evaluation rule have an overlapping part, for example, the constraint condition is used to describe the objective feature and the subjective feature of the instruction, and the evaluation rule is used to evaluate the subjective feature of the instruction, etc. The present application does not limit this.

[0095] Illustratively, the computing system evaluating the instructions includes: evaluating each instruction in the first instruction set by a large language model or an instruction scoring model to obtain a first evaluation result. The following will introduce these two ways respectively:

[0096] Method one, through the large language model, whether each instruction in the first instruction set conforms to the evaluation rule is evaluated to obtain the first evaluation result. Among them, the computing system fills the first instruction set and the evaluation rule into the second prompt template to obtain the third prompt, inputs the third prompt into the large language model, and evaluates whether each instruction in the first instruction set conforms to the evaluation rule through the large language model to obtain the first evaluation result.

[0097] The content of the third prompt will be illustrated below. Taking the second prompt template

Please evaluate the instructions according to the following evaluation rules: evaluation rule {variable 4}; instruction set {variable 5}

[0098] For example, the evaluation rules include rule A "whether the question of the instruction has a syntax problem", rule B "whether the answer of the instruction is complete and correct", and rule C "whether the answer of the instruction has self-contradiction or unreasonable jump", hereinafter referred to as "rule A, rule B, rule C", and the first instruction set includes instruction 1, instruction 2 and instruction 3. Based on this, the third prompt is

Please evaluate the instructions according to the following evaluation rules: evaluation rule {rule A, rule B, rule C}; instruction set {instruction 1, instruction 2, instruction 3}

[0099] Through the above method, the large language model is used to quickly evaluate each instruction in the first instruction set, and on the basis of intuitively reflecting the quality of each instruction, the efficiency of obtaining the first evaluation result is improved.

[0100] Method two, scoring the quality of each instruction in the first instruction set by an instruction scoring model to obtain the first evaluation result. Among them, the instruction scoring model is an AI model trained based on evaluation rules and sample instructions, which can quantitatively score the quality of instructions. The instruction scoring model can be an instruction scoring model uploaded by a user, or a large language model, and the type and structure of the instruction scoring model are not limited by the present application.

[0101] For example, the first instruction set includes instruction 4, instruction 5, and instruction 6, and the instruction scoring model can score the quality of each instruction according to evaluation rules such as rule D "the number of syntax problems in the question of the instruction", rule E "the accuracy of the answer in the instruction", and rule F "the number of syntax problems in the answer of the instruction", to obtain the quality score (for example, the score interval is 0-10) of each instruction. Correspondingly, the first evaluation result is, for example: instruction 4 is 9 points, instruction 5 is 7.5 points, and instruction 6 is 5 points.

[0102] In the above manner, the quality of each instruction in the first instruction set is quantitatively scored by the instruction scoring model, which can intuitively reflect the quality of the instructions in the first instruction set, facilitating subsequent screening of the instructions in the first instruction set.

[0103] It should be noted that the above several evaluation methods are only for illustration, and in actual application, a suitable evaluation method can be selected according to the needs to obtain the first evaluation result, which is not limited in the present application.

[0104] 604、The computing system adjusts the first prompt word based on the first evaluation result to obtain a second prompt word when the first evaluation result does not satisfy a preset condition.

[0105] In the embodiments of the present application, the preset condition refers to the qualified rate of the instructions included in the first instruction set being greater than a threshold value, wherein the qualified rate indicates the proportion of the instructions in the first instruction set that meet the evaluation rules. The threshold value is a preset threshold value, which can be set according to the needs in actual application, for example, the threshold value is 60%, that is, when the qualified rate of the instructions included in the first instruction set is greater than 60%, it is determined that the first instruction set satisfies the preset condition. Based on the foregoing step 603, the first evaluation result can be obtained by the large language model evaluating whether the quality of each instruction in the first instruction set meets the evaluation rules, or by the instruction scoring model scoring the quality of each instruction in the first instruction set, and accordingly, different types of first evaluation results can correspond to different qualified rate determination methods.

[0106] In some embodiments, the first evaluation result is obtained by the large language model evaluating whether each instruction in the first instruction set meets the evaluation rules, and accordingly, the first evaluation result includes the conclusion of whether each instruction in the first instruction set meets the evaluation rules. Based on this, the computing system can determine the qualified rate of the instructions in the first instruction set according to the conclusion of whether each instruction meets the evaluation rules. For example, the first evaluation result includes: instruction 1 meets rule A and rule B, but does not meet rule C; instruction 2 meets rule A, rule B, and rule C; and instruction 3 meets rule A, but does not meet rule B and rule C. Based on this, the computing system regards the instruction that meets all the evaluation rules as the instruction that meets the evaluation rules (or the instruction with qualified quality), that is, instruction 2 is the instruction that meets the evaluation rules, and the qualified rate is 1 / 3. It should be noted that this is only an example, and the computing system can also regard the instruction that meets part of the evaluation rules as the instruction with qualified quality, which is not limited in the present application.

[0107] In other embodiments, the first evaluation result is obtained by the instruction scoring model scoring the quality of each instruction in the first instruction set, and accordingly, the first evaluation result includes the quality score of each instruction in the first instruction set. Based on this, the computing system can determine the qualified rate of the instructions in the first instruction set according to the quality score of each instruction. For example, the first evaluation result includes: instruction 4 is 9 points, instruction 5 is 7.5 points, and instruction 6 is 5 points. Based on this, the computing system can regard the instruction with a score greater than 7 as the instruction that meets the evaluation rules (or the instruction with qualified quality), that is, instruction 4 and instruction 5 are the instructions that meet the evaluation rules, and the qualified rate is 2 / 3. It should be noted that this is only an example, and the computing system can also set other score thresholds to determine the instruction with qualified quality, which is not limited in the present application.

[0108] In this step, when the first evaluation result indicates that the first instruction set does not meet the preset condition, it means that the overall quality of the instructions in the first instruction set does not meet the demand (or the number of instructions with qualified quality is small). Since the first evaluation result can indicate the quality of each instruction in the first instruction set, the first evaluation result can be used as a reference to adjust at least one of the at least one labeled instruction and the constraint condition in the first prompt to obtain a second prompt. For example, the instruction with qualified quality indicated by the first evaluation result is used as a new labeled instruction, so that the second prompt can guide the large language model to generate instructions using the new labeled instruction as an example, thereby improving the number of instructions with qualified quality. For another example, the instruction with unqualified quality indicated by the first evaluation result is used to adjust the constraint condition, so that the second prompt can guide the large language model to generate instructions using the adjusted constraint condition, thereby improving the quality of the instructions.

[0109] It should be understood that, based on the foregoing introduction, the content of the first prompt word can be understood as an instruction generation strategy that can guide the large language model to generate a corresponding instruction set according to the first prompt word. Accordingly, adjusting the first prompt word to obtain the second prompt word can also be understood as adjusting the instruction generation strategy, thereby guiding the large language model to generate a corresponding instruction set according to the second prompt word.

[0110] Illustratively, since the first prompt word includes at least one of the at least one annotation instruction and the constraint condition, in this step, the computing system can adjust at least one of the at least one annotation instruction and the constraint condition based on the first evaluation result to obtain the second prompt word. The ways of adjusting the at least one annotation instruction and adjusting the constraint condition are introduced as follows:

[0111] Method one, adjusting the at least one annotation instruction in the first prompt word based on the first evaluation result to obtain the second prompt word.

[0112] Among them, the computing system determines at least one first instruction that meets the evaluation rule from the first instruction set based on the first evaluation result; obtains the second prompt word based on the at least one first instruction and the constraint condition. Illustratively, the computing system can replace at least one annotation instruction in the first prompt word with at least one first instruction as annotation instruction; or add at least one first instruction to the annotation instruction set to re-determine at least one annotation instruction from the annotation instruction set; or add at least one first instruction as annotation instruction to the first prompt word to obtain the second prompt word; the application does not limit the way the computing system obtains the second prompt word based on at least one first instruction.

[0113] In some embodiments, the first evaluation result is obtained by the large language model evaluating whether each instruction in the first instruction set meets the evaluation rule, and accordingly, the first evaluation result includes the conclusion of whether each instruction in the first instruction set meets the evaluation rule. Based on this, the computing system can determine at least one first instruction that meets the evaluation rule from the first instruction set according to the conclusion of whether each instruction meets the evaluation rule. For example, the first evaluation result includes: instruction 1 meets rule A and rule B, does not meet rule C; instruction 2 meets rule A, rule B, rule C; instruction 3 meets rule A, does not meet rule B and rule C. Based on this, the computing system takes the instruction that meets all evaluation rules as the first instruction, that is, takes instruction 2 as the first instruction. It should be noted that this is only an example, and the computing system can also take the instruction that meets part of the evaluation rules as the first instruction, which can be set according to the needs in actual application, and the application does not limit this.

[0114] In some embodiments, the first evaluation result is scored by an instruction scoring model, the instruction scoring model scoring the quality of each instruction in the first instruction set, and the first evaluation result includes the quality scores of each instruction in the first instruction set. Based on this, the computing system can determine the pass rate of the instructions in the first instruction set according to the quality scores of each instruction. For example, the first evaluation result includes: instruction 4 scores 9, instruction 5 scores 7.5, and instruction 6 scores 5. Based on this, the computing system can select instructions with a score greater than 7 as the first instructions, or sort the instructions in the first instruction set according to the quality scores and select the top k instructions as the first instructions (k is a positive integer), and the present application does not limit this.

[0115] In this way, when the first evaluation result does not meet the preset condition, since the first evaluation result can indicate the quality of each instruction in the first instruction set, the first evaluation result can be used as a reference to adjust the labeled instructions in the first prompt to obtain a second prompt in a timely manner, for example, selecting the first instructions that meet the evaluation rules as the labeled instructions in the second prompt, or adding the first instructions to the first prompt to obtain the second prompt, so that the second prompt can guide the large language model to regenerate an instruction set that meets the evaluation rules more in the adjusted labeled instructions, improve the number of instructions in the instruction set that meet the quality requirements, and further improve the quality of the instruction set.

[0116] Method two: adjusting the constraint condition in the first prompt based on the first evaluation result to obtain a second prompt.

[0117] In this way, when the first evaluation result does not meet the preset condition, since the first evaluation result can indicate the quality of each instruction in the first instruction set, the first evaluation result can be used as a reference to adjust the labeled instructions in the first prompt to obtain a second prompt in a timely manner, for example, selecting the first instructions that meet the evaluation rules as the labeled instructions in the second prompt, or adding the first instructions to the first prompt to obtain the second prompt, so that the second prompt can guide the large language model to regenerate an instruction set that meets the evaluation rules more in the adjusted labeled instructions, improve the number of instructions in the instruction set that meet the quality requirements, and further improve the quality of the instruction set.

[0118] For example, the constraint condition includes "the instruction needs to be accurate", the second instruction does not conform to the evaluation rule for evaluating this subjective feature of the instruction, the content of the evaluation rule is "whether the answer in the instruction contains an error or misleading expression", and the adjusted constraint condition includes "the instruction needs to be accurate and the answer in the instruction does not contain an error or misleading expression".

[0119] For another example, the constraint condition includes "the number of words in the instruction is greater than XX", the second instruction does not conform to the evaluation rule for evaluating this objective feature of the instruction, the content of the evaluation rule is "whether the number of words in the instruction is greater than XX", and since the number of words in the second instruction is equal to XX, the adjusted constraint condition is "the number of words in the instruction is greater than XX and cannot be equal to XX".

[0120] For another example, the constraint condition includes "the number of words in the instruction is greater than XX", the second instruction does not conform to the evaluation rule for evaluating the accuracy of the instruction, the content of the evaluation rule is "the answer in the instruction does not contain an error or misleading expression", and the adjusted constraint condition includes "the number of words in the instruction is greater than XX; the answer in the instruction does not contain an error or misleading expression".

[0121] It should be noted that this is only an example and does not limit the present application. Based on the foregoing step 603, the feature described by the constraint condition and the feature evaluated by the evaluation rule can be different or can have an overlapping part. In some scenarios, the way of adjusting the constraint condition can be flexibly set according to business needs.

[0122] The following describes the way of adjusting the constraint condition by the computing system, taking different types of first evaluation results as examples.

[0123] In some embodiments, the first evaluation result is obtained by the large language model evaluating whether each instruction in the first instruction set meets the evaluation rule, and accordingly, the first evaluation result includes the conclusion of whether each instruction in the first instruction set meets the evaluation rule. Based on this, the computing system can determine at least one second instruction that does not meet the evaluation rule from the first instruction set according to the conclusion of whether each instruction meets the evaluation rule. For example, the first evaluation result includes: instruction 1 meets rule A and rule B, and does not meet rule C; instruction 2 meets rule A, rule B, and rule C; and instruction 3 meets rule A, and does not meet rule B and rule C. Based on this, the computing system takes the instruction that does not meet all the evaluation rules as the second instruction, that is, takes instruction 1 and instruction 3 as the second instruction, and since instruction 1 and instruction 3 do not meet rule B and rule C, the constraint condition is adjusted based on rule B and rule C. For example, the constraint condition of the first prompt word is “generate 3 instructions; the number of words of the question in each instruction is not more than 10; and the number of words of the answer in each instruction is greater than 5”, rule B is “whether the answer of the instruction is complete and correct”, and rule C is “whether the answer of the instruction is self-contradictory or unreasonable in jumping”, and the adjusted constraint condition is “generate 3 instructions; the number of words of the question in each instruction is not more than 10; the number of words of the answer in each instruction is greater than 5; the answer of the instruction is complete and correct; and the answer of the instruction is not self-contradictory or unreasonable in jumping”.

[0124] In other embodiments, the first evaluation result is obtained by the instruction scoring model scoring the quality of each instruction in the first instruction set, and accordingly, the first evaluation result includes the quality score of each instruction in the first instruction set. Based on this, the computing system can determine at least one second instruction that does not meet the evaluation rule from the first instruction set according to the quality score of each instruction. For example, the first evaluation result includes: instruction 4 is 9 points, instruction 5 is 7.5 points, and instruction 6 is 5 points. Based on this, the computing system can take the instruction with a score less than or equal to 7 as the second instruction, that is, take instruction 6 as the second instruction, and adjust the constraint condition according to whether instruction 6 meets the evaluation rule. It should be noted that the adjustment of the instruction requirement according to the evaluation rule can be set according to business needs, which is not limited in the present application.

[0125] In the above manner, when the first evaluation result does not meet the preset condition, since the first evaluation result can indicate the quality of each instruction in the first instruction set, the constraint condition in the first prompt word can be adjusted in time according to the first evaluation result to obtain the second prompt word, for example, the new characteristics required by the instruction are determined according to the evaluation rule that the second instruction does not meet, and are added to the constraint condition to obtain the second prompt word, so that the second prompt word can guide the large language model to generate the instruction with the adjusted constraint condition, and the instruction set with better quality can be generated again.

[0126] It should be understood that the above-mentioned manner one and manner two can also be used in combination, that is, the computing system adjusts at least one annotation instruction and constraint condition in the first prompt word based on the first evaluation result to obtain a second prompt word. The principle of this process is the same as the aforementioned manner one and manner two, and thus will not be described again. By using manner one and manner two in combination, when the first evaluation result indicates that the first instruction set does not meet the preset condition, the annotation instruction and the constraint condition in the first prompt word are adjusted in a timely manner according to the first evaluation result to obtain the second prompt word, which can enable the large language model to generate a new instruction set that meets the evaluation rules and the feature requirements more in accordance with the adjusted annotation instruction and the adjusted constraint condition, greatly improving the quality of the instruction set.

[0127] 605. The computing system determines the target instruction set related to the document segment according to the second prompt word.

[0128] In the embodiments of the present application, the computing system inputs the second prompt word into the large language model to generate the target instruction set related to the document segment. Since the second prompt word is the prompt word adjusted on the basis of the first prompt word and based on the first evaluation result of the first instruction set, compared with the first prompt word, the second prompt word can better guide the large language model to generate an instruction set with higher quality, and obtain the target instruction set.

[0129] In some embodiments, the computing system inputs the second prompt word into the large language model to generate a second instruction set related to the document segment, evaluates the second instruction set to obtain a second evaluation result, the second evaluation result indicating the quality of each instruction in the second instruction set, and when the second evaluation result indicates that the second instruction set does not meet the preset condition, adjusts the second prompt word based on the second evaluation result to obtain a fourth prompt word, so as to determine the target instruction set related to the document segment according to the fourth prompt word. This process is also that the computing system can regenerate the instruction set related to the document segment according to the second prompt word, and perform the process same as steps 603 to 605 until the instruction set meeting the preset condition is obtained.

[0130] In addition, in steps 604 and 605, the first instruction set is taken as an example to introduce that the first evaluation result indicates that the first instruction set does not meet the preset condition. In some embodiments, when the first evaluation result indicates that the first instruction set meets the preset condition, the computing system filters the target instruction set from the first instruction set based on the first evaluation result. For example, the first evaluation result includes: instruction 1 meets rule A and rule B, and does not meet rule C; instruction 2 meets rule A, rule B, and rule C; instruction 3 meets rule A, and does not meet rule B and rule C, and the like. Based on this, the computing system filters the instructions that meet all evaluation rules to obtain the target instruction set, and the target instruction set includes instruction 2. For another example, the first evaluation result includes: instruction 4 is 9 points, instruction 5 is 7.5 points, instruction 6 is 5 points, and the like. Based on this, the computing system can filter the instructions greater than 7 points to obtain the target instruction set, and the target instruction set includes instruction 4 and instruction 5. It should be noted that this is only an example, and the computing system can also use other ways to filter the target instruction set, which is not limited in the present application. Of course, the computing system can also take the first instruction set as the target instruction set, and the like.

[0131] After obtaining the target instruction set related to the document segment through steps 601 to 605, the target instruction set can be used for at least one of the following:

[0132] (1) Training a large language model. That is, using the target instruction set to train the large language model, so that the large language model learns the knowledge of the field to which the target instruction set belongs, thereby applying the large language model to the field, so that the large language model can accurately generate a required answer according to a question in the field.

[0133] (2) Storing to a knowledge base to realize retrieval augmented generation (RAG). That is, storing the target instruction set to the knowledge base, so that the large language model can retrieve related instructions from the knowledge base when answering a question in the field to which the target instruction set belongs, and combining the retrieved instructions with the large language model to generate a final answer.

[0134] (3) Storing to a question and answer database to realize question and answer pair matching. That is, storing the target instruction set to the question and answer database, and in some question and answer scenarios, the instructions retrieved from the question and answer database can be fed back to the user according to the question input by the user.

[0135] It should be noted that since the instruction generation method provided by the present application can generate a high-quality instruction set, when the target instruction set is applied to the above scenarios, the accuracy of the answer can be improved. Moreover, the application scenarios of the target instruction set are not limited to the above few scenarios, which are not limited in the present application.

[0136] It can be seen that, in the instruction generation method provided in the application, the generation capability of the large language model is utilized, the prompt word is input into the large language model to generate an instruction set related to the document segment, then the instruction set is evaluated, and when the evaluation result indicates that the instruction set does not meet the preset condition, the prompt word is adjusted in a timely manner according to the evaluation result to determine a target instruction set related to the document segment. In this way, a feedback relationship between instruction generation and instruction evaluation is established, which can timely adjust the prompt word according to the instruction evaluation result in the process of efficiently generating the instruction set, that is, adjust the strategy of the large language model to generate instructions, thereby improving the quality of the finally obtained instruction set.

[0137] The method embodiment shown in FIG. 6 is exemplified below with reference to the embodiments shown in FIGS. 7 and 8.

[0138] FIG. 7 is a schematic diagram of an instruction generation method according to an embodiment of the application. As shown in FIG. 7, the instruction generation method includes two parts of instruction generation and instruction evaluation.

[0139] In the instruction generation part, the computing system fills the document segment "XXX is a practical programming tool..." and the labeled instruction "Question: What is XXX? Answer: XXX is a programming tool" into the first prompt word template of the large language model to obtain the first prompt word 701, wherein the first prompt word 701 further includes the constraint condition "YYY". Then, the computing system inputs the first prompt word 701 into the large language model to generate a plurality of instructions related to the document segment. After analyzing (such as unifying the instruction format) and filtering (such as screening instructions containing the same question) the plurality of instructions, a first instruction set related to the document segment is obtained.

[0140] In the instruction evaluation part, the computing system fills the first instruction set and the evaluation rule into the second prompt word template to obtain the third prompt word 702, and inputs the third prompt word 702 into the large language model. Through the large language model, the quality of each instruction in the first instruction set is evaluated whether it meets the evaluation rule (such as rule A, rule B and rule C in the figure) to obtain the first evaluation result 703. Then, the computing system judges whether the first instruction set indicated by the first evaluation result meets the preset condition, that is, the qualified rate r of the instructions included in the first instruction set is greater than the threshold θ (such as 60%). The qualified rate r indicates the proportion of instructions in the first instruction set that meet the evaluation rule. For example, r is calculated by the following formula (1):

[0141] In the above formula (1), I(·) represents an indicator function; n represents the number of instructions in the first instruction set; di represents the quality of the i-th instruction, i is a positive integer. i ∈R represents that the i-th instruction is a qualified instruction, i is a positive integer.

[0142] Further, if the first evaluation result indicates that the first instruction set meets the preset condition, a target instruction set is filtered out from the first instruction set. If the first evaluation result indicates that the first instruction set does not meet the preset condition, a constraint condition in the first prompt word is adjusted according to the first evaluation result to obtain a second prompt word, and the corresponding instruction set is regenerated based on the second prompt word. This process is described with reference to the foregoing steps 604 and 605, and thus will not be described again.

[0143] It can be seen that in the instruction generation method shown in FIG. 7, a feedback relationship between instruction generation and instruction evaluation is established, and the constraint condition in the prompt word can be adjusted in time according to the instruction evaluation result in the process of efficiently generating the instruction set, so as to improve the quality of the finally obtained instruction set. This process can also be understood as an instruction generation process based on self-adaptive iteration of the constraint condition in the prompt word, which can be expressed as formula (2) as follows: n = LLM(p gen , r n-1 ); r n = LLM(p eval , D n ) (2)

[0144] In the foregoing formula (2), D n represents the instruction set generated in the n th iteration; r n represents the evaluation result generated in the n th iteration; p gen represents the prompt word generated based on the first prompt word template, that is, the prompt word used to generate the instruction set; and p eval represents the prompt word generated based on the second prompt word template, that is, the prompt word used to generate the evaluation result.

[0145] FIG. 8 is a schematic diagram of another instruction generation method provided by an embodiment of the present application. As shown in FIG. 8, the instruction generation method includes two parts of instruction generation and instruction evaluation. The instruction generation part is the same as the foregoing embodiment shown in FIG. 7, and thus will not be described again. The instruction evaluation part will be described below.

[0146] In the instruction evaluation part, the computing system scores the quality of each instruction in the first instruction set by using the instruction scoring model to obtain a first evaluation result 801. Then, the computing system determines whether the first instruction set indicated by the first evaluation result meets a preset condition, that is, the qualified rate r of the instructions included in the first instruction set is greater than a threshold value a (for example, 60%, and it should be noted that the threshold value can be the same as or different from that in the embodiment shown in FIG. 7, which is not limited in the present application), where the qualified rate r indicates the proportion of the instructions in the first instruction set that meet the evaluation rule. For example, r is calculated by formula (3) as follows:

[0147] In the above formula (3), I(·) represents an indicator function; n represents the number of instructions in the first instruction set, f(x) represents an instruction scoring model, the input x is an instruction, and the output is a quality score; f(d i ) > σ represents that the i-th instruction is a quality-qualified instruction, i is a positive integer; and σ represents a score threshold (such as 7 points) for instructions to meet the evaluation rules.

[0148] Further, if the first evaluation result indicates that the first instruction set meets the preset condition, a target instruction set is selected from the first instruction set. If the first evaluation result indicates that the first instruction set does not meet the preset condition, the marked instructions in the first prompt word are adjusted according to the first evaluation result, such as selecting the instructions with the top k scores as new marked instructions, replacing the marked instructions in the first prompt word to obtain a second prompt word, and regenerating a corresponding instruction set based on the second prompt word. This process is similar to the aforementioned steps 604 and 605, and thus will not be described again.

[0149] As can be seen, in the instruction generation method shown in FIG. 8, a feedback relationship between instruction generation and instruction evaluation is established, which can timely adjust the marked instructions in the prompt word according to the instruction evaluation result in the process of efficiently generating the instruction set, thereby improving the quality of the finally obtained instruction set. This process can also be understood as an instruction generation process based on adaptive iteration of the marked instructions in the prompt word, which can be expressed as the following formula (4):

[0150] In the above formula (4), D n represents the instruction set generated in the n-th iteration; represents the marked instructions selected in the n-th iteration; p gen represents the prompt word generated based on the first prompt word template, that is, the prompt word used to generate the instruction set; and k represents the number of selected marked instructions.

[0151] Based on the method embodiments shown in FIGS. 4 to 8, the present application further provides an instruction generation device. Illustratively, referring to FIG. 9, FIG. 9 is a structural schematic diagram of an instruction generation device according to an embodiment of the present application. As shown in FIG. 9, the device includes a generation module 901, an evaluation module 902, and an adjustment module 903.

[0152] The generation module 901 is configured to input a first prompt word into a large language model to generate a first instruction set related to a document segment.

[0153] The evaluation module 902 is configured to evaluate the first instruction set to obtain a first evaluation result, the first evaluation result indicating the quality of each instruction in the first instruction set.

[0154] The adjustment module 903 is configured to, when the first evaluation result does not satisfy the preset condition, adjust the first prompt word to obtain a second prompt word based on the first evaluation result, so as to determine the target instruction set related to the document segment according to the second prompt word.

[0155] In some embodiments, the first prompt word comprises the document segment and / or the at least one labeled instruction.

[0156] In some embodiments, the first prompt word further comprises a constraint condition, and the constraint condition is used to describe a feature required by the instruction.

[0157] The adjustment module 903 is configured to adjust at least one of the at least one labeled instruction and the constraint condition based on the first evaluation result to obtain the second prompt word.

[0158] In some embodiments, the adjustment module 903 is configured to:

[0159] determine at least one first instruction that meets an evaluation rule from the first instruction set based on the first evaluation result, and the evaluation rule is used to evaluate a feature of the instruction to measure a quality of the instruction;

[0160] obtain the second prompt word based on the at least one first instruction and the constraint condition.

[0161] In some embodiments, the adjustment module 903 is configured to:

[0162] determine at least one second instruction that does not meet the evaluation rule from the first instruction set;

[0163] adjust the constraint condition based on an evaluation rule that the at least one second instruction does not meet to obtain an adjusted constraint condition;

[0164] obtain the second prompt word based on the adjusted constraint condition and the at least one labeled instruction.

[0165] In some embodiments, the preset condition is that a qualified rate of the instructions included in the first instruction set is greater than a threshold value, and the qualified rate indicates a proportion of the instructions in the first instruction set that meet the evaluation rule.

[0166] In some embodiments, the evaluation module 902 is configured to evaluate each instruction in the first instruction set by using a large language model or an instruction scoring model to obtain the first evaluation result.

[0167] In some embodiments, the target instruction set is used for at least one of the following:

[0168] training of the large language model;

[0169] storage to a knowledge base to implement retrieval augmented generation (RAG);

[0170] storage to a question and answer database to implement question and answer pair matching.

[0171] By means of the above device, the generation capability of the large language model is utilized, the prompt word is input into the large language model to generate an instruction set related to the document segment, then the instruction set is evaluated, when the evaluation result indicates that the instruction set does not meet the preset condition, the prompt word is adjusted in a timely manner according to the evaluation result to determine a target instruction set related to the document segment. In this way, a feedback relationship between instruction generation and instruction evaluation is established, which can timely adjust the prompt word according to the instruction evaluation result in the process of efficiently generating the instruction set, that is, adjust the strategy of the large language model to generate instructions, thereby improving the quality of the finally obtained instruction set.

[0172] Of course, the device can also include other functional units to realize the functions involved by the computing system in the above method embodiments. In actual application, the above functions can be completed by different functional units as needed, that is, the internal structure of the device is divided into different functional units to complete all or part of the functions described above. In some possible implementation manners, each functional unit can be located on the same computing device, or can be located in different computing devices to cooperatively complete all or part of the functions described above. In addition, the instruction generation device and the instruction generation method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0173] The present application also provides a computer readable storage medium, the computer readable storage medium is used to store at least one program code, when the at least one program code is executed by the computing device, the computing device realizes the above-mentioned instruction generation method.

[0174] The present application also provides a computer program product, when the computer program product runs on the computing device, the computing device realizes the above-mentioned instruction generation method.

[0175] In the present application, the terms "first", "second", and the like are used to distinguish between items or similar items having substantially the same function and action, and it should be understood that there is no logical or time sequence between "first", "second", and "nth", and the number and execution order are not limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various described examples, the first prompt word can be called the second prompt word, and similarly, the second prompt word can be called the first prompt word. The first prompt word and the second prompt word can both be prompt words, and in some cases, can be separate and different prompt words.

[0176] The term "at least one" in the present application means one or more, and the term "multiple" in the present application means two or more than two, for example, multiple prompt words means two or more than two prompt words.

[0177] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be encompassed in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0178] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0179] Those of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0180] The above description and the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An instruction generation method, characterized by, The method comprises: inputting a first prompt word into a large language model to generate a first instruction set related to a document segment; evaluating the first instruction set to obtain a first evaluation result, the first evaluation result indicating the quality of each instruction in the first instruction set; when the first evaluation result does not satisfy a preset condition, adjusting the first prompt word based on the first evaluation result to obtain a second prompt word, so as to determine a target instruction set related to the document segment according to the second prompt word.

2. The method of claim 1, wherein, The first prompt word comprises the document segment and / or at least one annotated instruction.

3. The method of claim 2, wherein, The first prompt word further comprises a constraint condition, the constraint condition being used to describe a feature required by the instruction; The first prompt word is adjusted based on the first evaluation result to obtain a second prompt word, comprising: adjusting at least one of the at least one annotated instruction and the constraint condition based on the first evaluation result to obtain the second prompt word.

4. The method of claim 3, wherein, The at least one of the at least one annotated instruction and the constraint condition is adjusted based on the first evaluation result to obtain the second prompt word, comprising: determining at least one first instruction that meets an evaluation rule from the first instruction set based on the first evaluation result, the evaluation rule being used to evaluate the feature of the instruction to measure the quality of the instruction; obtaining the second prompt word based on the at least one first instruction and the constraint condition.

5. The method according to claim 3 or 4, characterized in that, The at least one of the at least one annotated instruction and the constraint condition is adjusted based on the first evaluation result to obtain the second prompt word, comprising: determining at least one second instruction that does not meet the evaluation rule from the first instruction set; adjusting the constraint condition based on the evaluation rule that the at least one second instruction does not meet to obtain an adjusted constraint condition; obtaining the second prompt word based on the adjusted constraint condition and the at least one annotated instruction.

6. The method of claim 1, wherein, The preset condition is that a qualified rate of instructions included in the first instruction set is greater than a threshold value, the qualified rate indicating a proportion of instructions in the first instruction set that meet the evaluation rule.

7. The method of claim 1, wherein, The first instruction set is evaluated to obtain the first evaluation result, comprising: evaluating each instruction in the first instruction set by the large language model or an instruction scoring model to obtain the first evaluation result.

8. A large language model-based instruction generation apparatus, characterized by comprising: The device comprises: a generation module configured to input a first prompt word into a large language model to generate a first instruction set related to a document segment; an evaluation module configured to evaluate the first instruction set to obtain a first evaluation result, the first evaluation result indicating the quality of each instruction in the first instruction set; an adjustment module configured to, when the first evaluation result does not satisfy a preset condition, adjust the first prompt word based on the first evaluation result to obtain a second prompt word, so as to determine a target instruction set related to the document segment according to the second prompt word.

9. The apparatus of claim 8, wherein, The first prompt word comprises the document segment and / or at least one annotated instruction.

10. The apparatus of claim 9, wherein, The first prompt word further comprises a constraint condition, the constraint condition being used to describe a feature required by the instruction; The adjustment module is configured to adjust at least one of the at least one annotation instruction and the constraint condition based on the first evaluation result to obtain the second prompt word.

11. The apparatus of claim 10, wherein, The adjustment module is configured to: determine at least one first instruction that meets an evaluation rule from the first instruction set based on the first evaluation result, the evaluation rule being used to evaluate a feature of the instruction to measure a quality of the instruction; obtain the second prompt word based on the at least one first instruction and the constraint condition.

12. The apparatus of claim 10 or 11, wherein, The adjustment module is configured to: determine at least one second instruction that does not meet the evaluation rule from the first instruction set; adjust the constraint condition based on an evaluation rule that the at least one second instruction does not meet to obtain an adjusted constraint condition; obtain the second prompt word based on the adjusted constraint condition and the at least one annotation instruction.

13. The apparatus of claim 8, wherein, The preset condition refers to a qualified rate of instructions included in the first instruction set being greater than a threshold value, the qualified rate indicating a proportion of instructions in the first instruction set that meet the evaluation rule.

14. The apparatus of claim 8, wherein, The evaluation module is configured to evaluate each instruction in the first instruction set by using the large language model or the instruction scoring model to obtain the first evaluation result.

15. A computing system, comprising: The computing system includes a host and an acceleration card, the host being configured to control the acceleration card, and the acceleration card being configured to run a large language model, and the system is configured to implement the instruction generation method according to any one of the preceding claims 1 to 7.

16. A computing device, comprising: The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the instruction generation method according to any one of the preceding claims 1 to 7.

17. A computer program product, characterised in that, When the computer program product is running on the computing device, the computing device is enabled to implement the instruction generation method according to any one of the preceding claims 1 to 7.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store at least one piece of program code, the at least one piece of program code being used to implement the instruction generation method according to any one of the preceding claims 1 to 7.

Images