Training data generation method, electronic equipment, storage medium and program product

Abstract

The invention provides a training data generation method, electronic equipment, a storage medium and a program product. The training data generation method comprises the steps of obtaining function description information, wherein the function description information at least comprises a function name, a function description, a function input parameter type and a function input parameter description; predicting at least one function input parameter value according to the function input parameter type and the function input parameter description; predicting at least one user instruction according to the function description, the function input parameter description and the at least one function input parameter available value; respectively verifying the at least one function input parameter value and the at least one user instruction to obtain a function input parameter value passing the verification and a user instruction passing the verification; training data are generated at least according to the function name, the function input parameter selectable value passing the verification and the user instruction passing the verification, and the training data are used for training the target large language model to improve the function calling accuracy of the target large language model.

Description

Technical Field

[0001] The present disclosure relates to a training data generation method, an electronic device, a storage medium, and a program product. Background Art

[0002] Large Language Models (LLMs) are rapidly developing in the field of artificial intelligence, with their foundational capabilities continuously improving while demonstrating powerful natural language processing and generation capabilities. While LLMs excel in text understanding and generation, they still have limitations when handling complex numerical calculations and specific business logic.

[0003] Related technologies extend the capabilities of LLMs by using external function calls, enabling them to handle more complex tasks. However, improving the accuracy of LLM function calls requires fine-tuning the LLM using manually constructed training data, which requires significant manpower and time. Summary of the Invention

[0004] The present disclosure provides a training data generation method, an electronic device, a storage medium, and a program product.

[0005] According to one aspect of the present disclosure, a training data generation method is provided, including: obtaining function description information, the function description information including at least a function name, a function function description, a function input parameter type, and a function input parameter description; predicting at least one function input parameter possible value based on the function input parameter type and the function input parameter description; predicting at least one user instruction based on the function function description, the function input parameter description, and at least one function input parameter possible value; verifying at least one function input parameter possible value and at least one user instruction respectively to obtain a function input parameter possible value and a user instruction that pass the verification; and generating training data based on at least the function name, the function input parameter possible value that pass the verification, and the user instruction that pass the verification, wherein the training data is used to train a target large language model to improve the function call accuracy of the target large language model.

[0006] According to one aspect of the technical solution, by obtaining function description information and predicting possible function input parameter values ​​and user instructions, training data is automatically generated, reducing the need for manual intervention, lowering labor and time costs, and improving the efficiency of training data generation. Furthermore, by separately verifying possible function input parameter values ​​and user instructions, the quality and reliability of the generated training data are ensured, helping to improve the target large language model trained on this training data's ability to understand and process function calls, thereby increasing the accuracy of function calls.

[0007] According to at least one embodiment of the present disclosure, a training data generation method predicts at least one possible function input parameter value based on the function input parameter type and the function input parameter description, including: when the function input parameter type is a numeric type or a character type and there is an example function input parameter value in the function input parameter description, using the example function input parameter value as the possible function input parameter value; when the function input parameter type is a numeric type or a character type and there is no example function input parameter value in the function input parameter description, randomly generating at least one possible function input parameter value that conforms to the function input parameter type and the function input parameter description; and when the function input parameter type is an enumeration type, using all function input parameter enumeration values ​​in the function input parameter description as the possible function input parameter values.

[0008] According to the technical solution of this embodiment, different strategies can be adopted to generate possible values ​​for function input parameters based on their types, thereby improving the flexibility and adaptability of the method and enabling flexible processing of different types of function input parameters. Compared to relying solely on function descriptions to predict user instructions, the present disclosure can achieve more accurate predictions of user instructions and fit them to actual application scenarios through precise control of possible values ​​for function input parameters, thereby improving the accuracy of the generated training data.

[0009] According to at least one embodiment of the present disclosure, a training data generation method predicts at least one possible value of a function input parameter based on the function input parameter type and the function input parameter description, including: obtaining a first prompt word, the first prompt word being used to prompt a function input parameter possible value prediction rule; and inputting the first prompt word, the function input parameter type, and the function input parameter description into a first model, and predicting the possible values ​​of the function input parameter based on the function input parameter possible value prediction rule through the first model to obtain at least one possible value of the function input parameter.

[0010] According to the technical solution of this embodiment, the first prompt word instructs the first model to predict possible function input parameter values, thereby enhancing the intelligence of the prediction and improving the accuracy of the prediction results. Furthermore, using the first model to predict possible function input parameter values ​​ensures that the generated possible function input parameter values ​​meet the expected specifications, thereby improving data consistency.

[0011] According to at least one embodiment of the present disclosure, a training data generation method is provided, in which at least one user instruction is predicted based on the function function description, the function input parameter description and at least one possible value of the function input parameter. The method includes: based on the function input parameter description, converting at least one possible value of the function input parameter into at least one first sentence component that conforms to the natural language expression habits, wherein the first sentence component includes phrases and / or words; converting the function function in the function function description into at least one second sentence component that conforms to the natural language expression habits, wherein the second sentence component includes phrases and / or words; and combining each second sentence component in the at least one second sentence component with at least one first sentence component to obtain at least one user instruction.

[0012] According to the technical solution of this embodiment, the function input parameter description and function function description are each converted into sentence components that conform to natural language expression habits. These two sentence components are then combined to generate user instructions. The generated user instructions are more closely aligned with actual application scenarios, improving the readability and practicality of the user instructions. Furthermore, by combining different sentence components to generate user instructions, the diversity of user instructions is enriched, which helps to improve the generalization ability of the target large language model.

[0013] According to the training data generation method of at least one embodiment of the present disclosure, at least one user instruction is predicted based on the function function description, the function input parameter description and at least one possible value of the function input parameter, including: obtaining a second prompt word, the second prompt word is used to prompt the user instruction prediction rule; and inputting the second prompt word, the function function description, the function input parameter description and at least one possible value of the function input parameter into a second model, and performing user instruction prediction based on the user instruction prediction rule through the second model to obtain at least one user instruction.

[0014] According to the technical solution of this embodiment, the second prompt word prompts the second model to predict user commands, improving the intelligent level of user command generation and enhancing the accuracy and rationality of the prediction results. Furthermore, the use of the second model to automatically generate user commands reduces the workload of manually writing user commands and improves generation efficiency.

[0015] According to the training data generation method of at least one embodiment of the present disclosure, at least one of the possible values ​​of the function input parameter and at least one of the user instructions are verified respectively to obtain the possible values ​​of the function input parameter and the user instruction that pass the verification, including: based on the at least one possible value of the function input parameter and the at least one user instruction, verifying whether the at least one user instruction can be correctly inferred by the target large language model and whether the at least one possible value of the function input parameter can be correctly processed by the function corresponding to the function name; and determining the possible values ​​of the function input parameter and the user instruction that pass the verification based on the verification result.

[0016] According to the technical solution of this embodiment, by verifying whether the user instructions can be correctly inferred by the model and whether the function input parameter values ​​can be correctly processed by the function, it is ensured that the user instructions are correctly inferable and the parameters filled in the target large language model are truly callable and the results are consistent with the user instructions, thereby ensuring the validity and accuracy of the generated training data, which is conducive to improving the effect and reliability of subsequent model training.

[0017] According to the training data generation method of at least one embodiment of the present disclosure, verifying whether at least one user instruction can be correctly inferred by the target large language model includes: extracting function input parameters in at least one user instruction through the target large language model to obtain a function input parameter extraction value corresponding to the at least one user instruction; and if the function input parameter extraction value corresponding to the at least one user instruction is respectively the same as the possible function input parameter values ​​used to predict the at least one user instruction, determining that the at least one user instruction can be correctly inferred by the target large language model.

[0018] According to the technical solution of this embodiment, by extracting the function input parameters in the user instruction and comparing the extracted function input parameter values ​​with the possible values ​​of the function input parameters, it is possible to accurately judge whether the user instruction can be correctly inferred by the target large language model, thereby ensuring the correctness and consistency of the user instruction, thereby helping to improve the target large language model's ability to understand and process different user instructions, and enhancing the robustness of the target large language model.

[0019] According to the training data generation method of at least one embodiment of the present disclosure, verifying whether at least one of the possible values ​​of the function input parameter can be correctly processed by the function corresponding to the function name includes: passing the possible value of the function input parameter predicted to be adopted by at least one of the user instructions to the function corresponding to the function name to obtain at least one function return result; and if at least one of the function return results matches at least one of the user instructions respectively, determining that at least one of the possible values ​​of the function input parameter can be correctly processed by the function corresponding to the function name.

[0020] According to the technical solution of this embodiment, by passing the possible values ​​of the function input parameters to the function and checking the function return results, the validity and correctness of the possible values ​​of the function input parameters are ensured, which helps to ensure that the generated training data is consistent with the actual processing logic of the function, thereby improving the credibility of the data.

[0021] According to the training data generation method of at least one embodiment of the present disclosure, the verified function input parameter optional values ​​and the verified user instructions are determined based on the verification results, including: taking at least one of the function input parameter optional values ​​that can be correctly processed by the function as the verified function input parameter optional value; and taking at least one of the user instructions that is predicted based on the verified function input parameter optional values ​​and can be correctly inferred by the target large language model as the verified user instruction.

[0022] According to the technical solution of this embodiment, the optional function input parameters and user instructions that have passed the verification can be accurately screened out based on the verification results, effectively removing invalid or erroneous data, ensuring the quality of the training data finally generated, and helping to improve the training effect of the target large language model.

[0023] According to at least one embodiment of the present disclosure, the training data generation method further includes, before generating training data based on at least the function name, the verified function input parameter optional values, and the verified user instructions, the following: obtaining a plurality of alternative function description information, wherein the alternative function description information includes at least an alternative function name; calculating the similarity between a plurality of the alternative function description information and the function description information respectively; generating training data based on at least the function name, the verified function input parameter optional values, and the verified user instructions, including: generating the training data based on the function name, the verified function input parameter optional values, the verified user instructions, and the alternative function names whose similarities are within a target similarity interval, wherein the alternative function names whose similarities are within the target similarity interval serve as interference items in the training data.

[0024] According to the technical solution of this embodiment, by calculating the similarity between function descriptions and candidate function descriptions, the highly similar candidate function names are introduced as interference items, thereby enriching the content of the training data. Furthermore, the introduction of interference items makes the training data have different learning difficulties, which helps improve the target large language model's ability to distinguish different functions in multi-function call scenarios, thereby enhancing the generalization and robustness of the target large language model.

[0025] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing a computer program; and a processor executing the computer program stored in the memory, so that the processor performs the training data generation method of any embodiment of the present disclosure.

[0026] According to another aspect of the present disclosure, a readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the training data generation method of any embodiment of the present disclosure.

[0027] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the training data generating method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0029] Figure 1 It is a flowchart of a training data generation method according to an embodiment of the present disclosure.

[0030] Figure 2 It is a schematic diagram of the process of predicting the possible values ​​of the input parameters of the function according to one embodiment of the present disclosure.

[0031] Figure 3 It is a schematic diagram of the process of predicting the possible values ​​of the input parameters of the function according to another embodiment of the present disclosure.

[0032] Figure 4 FIG. 4 is a schematic diagram of a process of predicting user instructions according to an embodiment of the present disclosure.

[0033] Figure 5 FIG. 4 is a schematic diagram of a process of predicting user instructions according to another embodiment of the present disclosure.

[0034] Figure 6 The present invention is a process diagram of possible values ​​of verification function input parameters and user instructions according to an embodiment of the present invention.

[0035] Figure 7 4 is a schematic diagram of a process for verifying user instructions according to an embodiment of the present disclosure.

[0036] Figure 8 The present invention is a schematic diagram of a process of determining possible values ​​of a verification function input parameter according to an embodiment of the present invention.

[0037] Figure 9 It is a schematic diagram of a process for determining possible values ​​of function input parameters and user instructions that pass verification according to an embodiment of the present disclosure.

[0038] Figure 10 It is a flowchart of a training data generation method according to another embodiment of the present disclosure.

[0039] Figure 11 4 is a flowchart of a method for generating training data according to another embodiment of the present disclosure.

[0040] Figure 12 It is a schematic structural block diagram of a training data generating device according to one embodiment of the present disclosure.

[0041] Figure 13 is a schematic block diagram of the structure of an electronic device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The present disclosure is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] LLM refers to a deep learning model trained using large amounts of text data. LLM can be used to generate natural language text or understand the meaning of language text. Function call is a feature that allows LLM to call external functions during the text generation process. In related technologies, the capabilities of LLM are expanded through external function calls, allowing LLM to handle more complex tasks. For example, LLM can perform high-precision calculations by calling a specialized numerical calculation library, or LLM can perform specific operations such as addition, deletion, modification, and query by calling the API of an internal business system. However, in order to improve the accuracy of function calls, a large amount of training data needs to be manually constructed. This not only requires a lot of manpower and time, resulting in low efficiency and high cost of training data generation, but also requires the person constructing the training data to have certain professional knowledge, which is also a high requirement for personnel.

[0045] To this end, the present disclosure proposes a training data generation method.

[0046] The training data generation method disclosed herein can be used by an electronic device to automatically generate training data for training a target large language model based on function description information obtained from the device. In this disclosure, electronic devices include, but are not limited to, servers, mobile phones, tablets, laptops, personal computers, wearable devices, and ATMs.

[0047] Figure 1 FIG. 1 shows a schematic diagram of the overall process of a training data generation method M100 according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S150, wherein the method can be executed by electronic devices such as a server, a mobile phone, and a computer.

[0048] S110. Obtain function description information, which at least includes the function name, function function description, function input parameter type, and function input parameter description.

[0049] Function description information is used to describe a specific function that can be called by the target large language model. In addition to the function name, function function description, function input parameter type and function input parameter description, it can also include the function input parameter name or other content, which is not limited here.

[0050] The function description describes the functionality of the function. The function input parameter type describes the type of the function's input parameters. The function input parameter description describes one or more of the following: the Chinese name, definition, and example value of the function input parameter.

[0051] S120. Predict at least one possible value of a function input parameter based on the function input parameter type and the function input parameter description.

[0052] Function input parameter types and descriptions help understand function input parameters. Therefore, you can predict at least one possible function input parameter value based on the function input parameter type and description. A possible function input parameter value is a value that triggers the function to run properly, and can be in numeric, text, or other formats.

[0053] For example, if a function requires only one type of function input parameter, then at least one possible value of the function input parameter can be predicted; if a function requires multiple types of function input parameters, then at least one possible value of each type of function input parameter can be predicted separately.

[0054] S130. Predict at least one user instruction based on the function function description, the function input parameter description, and at least one possible value of the function input parameter.

[0055] A function description can help understand the function's function, and a function parameter description can help understand the parameters. Therefore, at least one user instruction can be predicted based on the function description, the function parameter description, and at least one possible value of the function parameter. A user instruction can be an instruction in the form of text or a statement that may be received during the operation of the target large language model and can trigger the target large language model to call a function.

[0056] Exemplarily, if a function only requires one type of function input parameter, at least one user instruction can be predicted based on the function function description, the function input parameter description, and at least one possible function input parameter value of the function input parameter, and each predicted user instruction may include at least one possible function input parameter value of the function input parameter and / or information that can characterize the possible function input parameter value; if a function requires multiple function input parameters, at least one user instruction can be predicted based on the function function description, the function input parameter description, and at least one possible function input parameter value of the multiple function input parameters, and each predicted user instruction may include at least one possible function input parameter value of each type of the multiple function input parameters and / or information that can characterize the possible function input parameter values.

[0057] S140. Verify at least one possible value of a function input parameter and at least one user instruction respectively, and obtain possible value of the function input parameter and user instruction that pass the verification.

[0058] Although the predicted function input parameter values ​​are those that can trigger the normal operation of the function and the predicted user instructions are those that can trigger the target large language model to call the function, the actual predicted function input parameter values ​​may not trigger the normal operation of the function and / or the actual predicted user instructions may not trigger the target large language model to call the function. Therefore, it is necessary to verify at least one predicted function input parameter value and at least one user instruction, so as to screen out the function input parameter values ​​and user instructions that meet the requirements, and provide high-quality data support for the subsequent generation of accurate training data.

[0059] Exemplarily, at least one possible value for a function input parameter and at least one user instruction can be sent to relevant personnel and prompted to manually verify the at least one possible value for a function input parameter and the at least one user instruction, and obtain the verified possible value for a function input parameter and the user instruction based on the manual verification result; or the electronic device can automatically verify the at least one possible value for a function input parameter and the at least one user instruction to obtain the verified possible value for a function input parameter and the user instruction, which is not limited here.

[0060] S150. Generate training data based on at least the function name, the verified optional function input parameters, and the verified user instructions, wherein the training data is used to train the target large language model to improve the function call accuracy of the target large language model.

[0061] For example, for each verified user instruction, the function name and at least one verified function input parameter value predicted to be used by the user instruction can be used as the annotation content of the user instruction, thereby generating a piece of training data corresponding to the user instruction. If there are multiple verified user instructions, multiple pieces of training data can be generated for each. In order to further improve the accuracy of the training data, after the training data is generated, relevant personnel can also be prompted to manually verify the training data, and the target large language model can be trained based on the manually verified training data.

[0062] The target large language model can be a deep learning model that has been trained using a large amount of text data and can generate natural language text or understand the meaning of language text. In the process of training the target large language model based on the training data, the user instructions in the training data can be used as the input of the target large language model, and the function name and function input parameter values ​​determined by the target large language model for the user instructions are obtained. The loss value of the function name and function input parameter values ​​determined by the target large language model and the function name and function input parameter values ​​corresponding to the user instructions in the training data are calculated, and then the parameters of the target large language model are adjusted according to the loss value until a target large language model that meets the requirements is obtained. Among them, the loss value can be used as the function call accuracy to more accurately guide the training of the target large language model.

[0063] The training data generation method of the disclosed embodiments achieves automated training data generation by acquiring function description information and predicting possible function input parameter values ​​and user instructions. This reduces the need for manual intervention, reduces labor and time costs, and improves the efficiency of training data generation. Furthermore, by separately verifying possible function input parameter values ​​and user instructions, the quality and reliability of the generated training data are ensured, helping to improve the target large language model trained on this training data's ability to understand and process function calls, thereby increasing the accuracy of function calls.

[0064] Regarding step S120, as a possible implementation, it may include the following: Figure 2 Steps S121 to S123 are shown.

[0065] S121. When the function input parameter type is a numeric type or a character type, and there is an example value of the function input parameter in the function input parameter description, the example value of the function input parameter is used as the possible value of the function input parameter.

[0066] Exemplarily, for each type of function input parameter in the function description information, if the type of the function input parameter is a numeric type or a character type, and there are example function input parameter values ​​in the function input parameter description of the function input parameter, then one or more of the example function input parameter values ​​can be used as the possible values ​​of the function input parameter of the function input parameter.

[0067] S122. When the function input parameter type is a numeric type or a character type, and there is no example function input parameter value in the function input parameter description, randomly generate at least one possible function input parameter value that meets the function input parameter type and the function input parameter description.

[0068] For function input parameters of numeric or character types, if there is an example value in the function input parameter description, the example value is directly used as the possible value of the function input parameter. If there is no example value in the function input parameter description, a possible value of the function input parameter is randomly generated, which simplifies the processing flow and improves efficiency.

[0069] For example, for each type of function input parameter in the function description information, if the type of the function input parameter is a numeric type or a character type, and the function input parameter description for that type of function input parameter does not contain an example function input parameter value, then at least one possible function input parameter value that meets the function input parameter type and function input parameter description for that type of function input parameter can be randomly generated based on a relevant algorithm. The relevant algorithm can be set according to actual needs and is not limited here.

[0070] S123. When the function input parameter type is an enumeration type, all function input parameter enumeration values ​​in the function input parameter description are used as possible values ​​of the function input parameters.

[0071] Exemplarily, for each type of function input parameter in the function description information, if the type of the function input parameter is an enumeration type, the function input parameter description of the function input parameter may include the enumeration value of the function input parameter, and then all the function input parameter enumeration values ​​in the function input parameter description may be used as the possible values ​​of the function input parameter of the function input parameter.

[0072] The training data generation method of the above-described embodiment can adopt different strategies to generate possible values ​​for function input parameters based on the type of function input parameter, thereby improving the method's flexibility and adaptability and enabling flexible processing of different types of function input parameters. Compared to relying solely on function descriptions to predict user instructions, the present disclosure can achieve more accurate predictions of user instructions that are more relevant to actual application scenarios through precise control of possible values ​​for function input parameters, thereby improving the accuracy of the generated training data.

[0073] Regarding step S120, as another possible implementation, it may include the following: Figure 3 Step S121' and step S122' are shown.

[0074] S121′, obtaining a first prompt word, where the first prompt word is used to prompt a function input parameter possible value prediction rule.

[0075] The first prompt word can be pre-set according to actual conditions and is not limited here. The function input parameter possible value prediction rule can refer to the contents of step S121 to step S123.

[0076] In one example, the first prompt is "Your task is to add a new enum field for each parameter (i.e., function input parameter) based on the parameter description field (i.e., function input parameter description) in the input method (i.e., function description information). The enum field is an enumeration type, indicating the possible values ​​of this parameter (i.e., the possible values ​​of the function input parameter). The format is: "enum": [element]. If the element type is string, wrap it in double quotes. Please output the enum field strictly according to this format, and do not have any other irrelevant text. Rule 1. The possible values ​​of the parameter are extracted from the description field. When the parameter type (that is, the function input type) is string or integer and there is no clear optional value description in the description, you need to fill in a reasonable value after understanding the meaning of the field. Note that this value should be specific and meaningful. You can forge a specific value yourself, such as 1234, abcd, random characters and numbers, etc. Rule 2. If the description of the parameter description field is of low quality and it is impossible to make a reasonable prediction, please set enum to an empty array []. Rule 3. Your output needs to ensure that it is only a dictionary (dict) like the input, and it must be consistent with the input data format. Do not have any other irrelevant text output."

[0077] S122′, input the first prompt word, the function input parameter type and the function input parameter description into the first model, and predict the function input parameter possible values ​​based on the function input parameter possible value prediction rule by the first model to obtain at least one function input parameter possible value.

[0078] The first model can be a target large language model, or it can be a deep learning model other than the target large language model that has been trained using a large amount of text data and can generate natural language text or understand the meaning of language text.

[0079] For example, if a model specifically designed for predicting possible function input parameter values ​​exists, the first prompt word may not be needed, and only the function input parameter type and function input parameter description may be input into the model, thereby obtaining at least one possible function input parameter value. Alternatively, the first prompt word and function description information may be directly input into the first model, and the first model may predict possible function input parameter values ​​based on the function description information to obtain at least one possible function input parameter value.

[0080] The training data generation method of the above embodiment uses the first prompt word to instruct the first model to predict possible function input parameter values, thereby enhancing the intelligence of the prediction and improving the accuracy of the prediction results. Furthermore, using the first model to predict possible function input parameter values ​​ensures that the generated possible function input parameter values ​​meet expected specifications, thereby improving data consistency.

[0081] Regarding step S130, as a possible implementation, it may include the following: Figure 4 Steps S131 to S133 are shown.

[0082] S131. Based on the function input parameter description, convert at least one possible value of the function input parameter into at least one first sentence component that conforms to the natural language expression habits, wherein the first sentence component includes a phrase and / or a word.

[0083] The function input parameter description can help understand the possible values ​​of the function input parameters, and then facilitate the conversion of the possible values ​​of the function input parameters into the first sentence component that conforms to natural language expression habits.

[0084] For example, when there is a mapping relationship between the possible values ​​of the function input parameters and the first statement component in the function input parameter description, the first statement component corresponding to the possible values ​​of the function input parameters can be directly obtained from the function input parameter description. When there is a mapping relationship between the possible values ​​of the function input parameters and the first statement component in the function input parameter description, the possible values ​​of the function input parameters can be converted into the first statement component based on the understanding of the function input parameters and natural language expression habits, or the possible values ​​of the function input parameters can be directly used as the first statement component.

[0085] S132. Convert the function function in the function function description into at least one second sentence component that conforms to natural language expression habits, wherein the second sentence component includes phrases and / or words.

[0086] For example, the function function may be first extracted from the function function description, and then the function function may be converted into a second sentence component based on natural language expression habits.

[0087] In an example, the function is configuration query, and the components of the multiple second statements obtained by conversion are: what is the setting, the setting, the specific configuration is, and check.

[0088] S133: Combine each second sentence component in the at least one second sentence component with the at least one first sentence component to obtain at least one user instruction.

[0089] When a function includes multiple function input parameters, a combination component that simultaneously includes first statement components corresponding to multiple function input parameters can be determined based on multiple statement components. Each combination component may include a first statement component corresponding to each function input parameter, and then each second statement component in at least one second statement component can be combined with the component to obtain at least one user instruction.

[0090] In the case where a function includes a function input parameter, each second statement component in at least one second statement component can be combined with each first statement component to obtain at least one user instruction.

[0091] The training data generation method of the above embodiment converts function input parameter descriptions and function function descriptions into sentence components that conform to natural language expression conventions, and then combines these two sentence components to generate user instructions. The generated user instructions are more closely aligned with actual application scenarios, improving the readability and practicality of the user instructions. Furthermore, by combining different sentence components to generate user instructions, the diversity of user instructions is enriched, which helps improve the generalization ability of the target large language model.

[0092] Regarding step S130, as another possible implementation, it may include the following: Figure 5 Steps S131' and S132' are shown.

[0093] S131′: Obtain a second prompt word, where the second prompt word is used to prompt the user of the instruction prediction rule.

[0094] The second prompt word can be preset according to actual conditions and is not limited here.

[0095] In one example, the second prompt is "Your task is to output {num} matching prompts (i.e. user instructions) based on the set value of #input parameter# (i.e., optional value of function input parameter) and #function description information#. Please make sure that the prompts you generate are in the form of natural language and should be concise, diverse, and non-repetitive. You cannot directly use the technical terms in the function description information or function input parameter description to generate prompts. Instead, you should generate prompts in the form of daily conversations or easy-to-understand text. Rule 1: Please refer to the function description information and the parameter description of each function input parameter to ask questions and ensure that your questions are clear in meaning. Rule 2: The prompt must be in the format of list[]. Rule 3: Please be sure to generate a Chinese description for the prompt, and each element of the list must be wrapped in double quotes."

[0096] S132′, input the second prompt word, function function description, function input parameter description and at least one possible value of the function input parameter into the second model, and perform user instruction prediction based on the user instruction prediction rule by the second model to obtain at least one user instruction.

[0097] The second model can be the target large language model, or it can be a deep learning model other than the target large language model that has been trained using a large amount of text data and is capable of generating natural language text or understanding the meaning of language text. The first model and the second model can be the same model. The second model can be used to implement steps S131 to S133.

[0098] For example, if a model dedicated to predicting user commands exists, the second prompt word may not be required. Only the function description information and at least one possible function parameter value may be input into the model, and the at least one user command may be obtained through the model. Alternatively, the second prompt word, the function description information, and at least one possible function parameter value may be directly input into a second model, and the second model may predict the user command based on the function description information to obtain the at least one user command.

[0099] The training data generation method described in the above embodiment uses the second prompt word to prompt the second model to predict user commands, thereby improving the intelligent level of user command generation and enhancing the accuracy and rationality of the prediction results. Furthermore, the automatic generation of user commands using the second model reduces the workload of manually writing user commands and improves generation efficiency.

[0100] Regarding step S140, in some embodiments of the present disclosure, it may include the following: Figure 6 Steps S141 and S142 are shown.

[0101] S141. Based on at least one possible function input parameter value and at least one user instruction, verify whether the at least one user instruction can be correctly inferred by the target large language model and whether the at least one possible function input parameter value can be correctly processed by the function corresponding to the function name.

[0102] For example, if a predicted user instruction cannot be correctly inferred by the target large language model due to its own reasons, then the training data generated based on this user instruction cannot effectively train the target large language model. Therefore, it should be verified whether the user instruction can be correctly inferred by the target large language model. If a predicted function input parameter value is not within the valid value range of the function, then the function input parameter value cannot be correctly processed by the function, and the training data generated based on this function input parameter value will cause the target large language model to train in a direction that deviates from the actual application scenario. Therefore, it should be verified whether the function input parameter value can be correctly processed by the function corresponding to the function name.

[0103] S142. Determine, based on the verification result, the function input parameter optional values ​​that have passed the verification and the user instructions that have passed the verification.

[0104] The function input parameter optional values ​​corresponding to the verified user instructions are all the function input parameter optional values ​​that have passed the verification.

[0105] As a possible implementation method, at least one function input parameter may be verified first, and then the user instruction corresponding to the verified function input parameter value may be verified, and then the user instruction that has passed the verification may be determined, and the function input parameter optional value used by the user instruction that has passed the verification may be determined to be the user instruction that has passed the verification.

[0106] As another possible implementation, at least one user instruction may be verified first, and then the possible values ​​of the function input parameters corresponding to the verified user instruction may be verified, and then the possible values ​​of the function input parameters that have passed the verification may be determined, and the user instruction predicted entirely based on the possible values ​​of the function input parameters that have passed the verification may be determined to be the user instruction that has passed the verification.

[0107] As another possible implementation, at least one function input parameter value and at least one user instruction can be verified simultaneously, and then the intersection of the function input parameter value verification results and the user instruction verification results can be taken to determine the function input parameter values ​​and user instructions that have passed the verification.

[0108] The training data generation method of the above embodiment ensures that the user instructions are correct and inferable, and the parameters filled in the target large language model are truly callable and the results conform to the user instructions by verifying whether the user instructions can be correctly inferred by the model and whether the function input parameter values ​​can be correctly processed by the function. This ensures the validity and accuracy of the generated training data, which is conducive to improving the effect and reliability of subsequent model training.

[0109] Regarding checking whether at least one user instruction can be correctly inferred by the target large language model in step S141, in some embodiments of the present disclosure, it may include the following: Figure 7 Steps S1411 and S1412 are shown.

[0110] S1411. Extract function input parameters from at least one user instruction using the target large language model to obtain a function input parameter extraction value corresponding to the at least one user instruction.

[0111] Exemplarily, for each user instruction in at least one user instruction, a function input parameter in the user instruction is extracted using the target large language model to obtain a function input parameter extraction value corresponding to the user instruction.

[0112] S1412. If the function input parameter extraction values ​​corresponding to the at least one user instruction are respectively the same as the function input parameter possible values ​​used to predict the at least one user instruction, it is determined that the at least one user instruction can be correctly inferred by the target large language model.

[0113] Exemplarily, for each user instruction in at least one user instruction, if the function input parameter extraction values ​​corresponding to the user instruction are respectively the same as the possible values ​​of the function input parameters used to predict the user instruction, it can be determined that the user instruction can be correctly inferred by the target large language model; if the function input parameter extraction values ​​corresponding to the user instruction are respectively different from the possible values ​​of the function input parameters used to predict the user instruction, it can be determined that the user instruction cannot be correctly inferred by the target large language model.

[0114] Exemplarily, whenever a user instruction is verified, such as before, after, or at the same time as verifying the possible values ​​of the function input parameters, the verification steps of the user instruction can refer to the description of step S1411 and step S1412. For the sake of brevity, they will not be repeated here.

[0115] In an example, the function name is query_haozan_conf, and the function function is described as querying the configuration details of a given city based on the given city name and configuration type. The function input parameters include city_name and type. The function input parameter type of city_name is string type, and the function input parameter description of city_name is "city name, such as City A". The function input parameter type of type is integer type, and the function input parameter description of type is "configuration type, a limited enumeration set. To query the signing points and fulfillment period, pass 1; to query the write-off and rewriting, pass 2; to query whether the signing is restricted after the price increase after entering the pool, pass 3; to query whether the transaction is free of charge within N hours, pass 4."

[0116] Based on the function input parameter type and description of city_name, we can predict that the value of the function input parameter city_name can be City A. Based on the function input parameter type and description of type, we can predict that the value of the function input parameter type can be 1, 2, 3, or 4. Therefore, we can determine four possible combinations of city_name and type: City A-1, City A-2, City A-3, and City A-4. Each combination of values ​​predicts two user instructions. Among them, the two user instructions predicted for City A-1 are "What are the settings of City A in terms of signing points and fulfillment period" and "What are the settings of City A in terms of querying write-offs and rewriting?" The two user instructions predicted for City A-2 are "Query the configuration related to signing points and fulfillment period in City A" and "What is the specific configuration of write-offs and rewriting in City A"; the two user instructions predicted for City A-3 are "After A's properties are added to the pool, are there any restrictions on signing for price increases?" and "Check whether A has any restrictions on signing for price increases after properties are added to the pool"; the two user instructions predicted for City A-4 are "Check A's N-hour transaction-free rules" and "Will A's properties be free of handling fees within N hours?"

[0117] Furthermore, the target large language model extracts the function input parameters from each user instruction, obtaining the corresponding function input parameter extraction value for each user instruction. Since the function input parameter extraction value for the user instruction "Querying the settings for write-offs and rewriting in City A" is A-2, which is different from the corresponding function input parameter value A-1 for "Querying the settings for write-offs and rewriting in City A," it can be determined that the user instruction "Querying the settings for write-offs and rewriting in City A" cannot be correctly inferred by the target large language model. Since the function input parameter extraction value for the user instruction "Querying the available points and fulfillment period configuration in City A" is A-1, which is different from the corresponding function input parameter value A-2 for "Querying the available points and fulfillment period configuration in City A," it can be determined that the user instruction "Querying the available points and fulfillment period configuration in City A" cannot be correctly inferred by the target large language model. Since the function input parameter extraction values ​​of other user instructions are the same as the corresponding function input parameter values, they can be considered to be correctly inferred by the target large language model.

[0118] It should be noted that the specific numerical values ​​mentioned above are only used as examples to illustrate the implementation of the present disclosure in detail and should not be understood as limiting the present disclosure. In other examples, implementation methods, or embodiments, other numerical values ​​can be selected according to the present disclosure and are not specifically limited here.

[0119] The training data generation method of the above embodiment extracts function input parameters from user instructions and compares the extracted function input parameter values ​​with the possible function input parameter values, thereby accurately judging whether the user instructions can be correctly inferred by the target large language model, ensuring the correctness and consistency of the user instructions, thereby helping to improve the target large language model's ability to understand and process different user instructions, and enhancing the robustness of the target large language model.

[0120] Regarding checking whether the possible values ​​of at least one function input parameter in step S141 can be correctly processed by the function corresponding to the function name, in some embodiments of the present disclosure, it may include the following: Figure 8 Steps S1413 and S1414 are shown.

[0121] S1413. Pass the possible values ​​of the function input parameters used by the predicted at least one user instruction to the function corresponding to the function name to obtain at least one function return result.

[0122] Exemplarily, for each user instruction in at least one user instruction, the possible values ​​of the function input parameters used by the predicted user instruction are passed to the function corresponding to the function name to obtain the function return result associated with the user instruction.

[0123] S1414. If at least one function return result matches at least one user instruction, determine that at least one function input parameter value can be correctly processed by the function corresponding to the function name.

[0124] Exemplarily, for each user instruction in at least one user instruction, if the function return result associated with the user instruction matches the user instruction, it can be determined that the possible values ​​of the function input parameters used by the user instruction can be correctly processed by the function corresponding to the function name; if the function return result associated with the user instruction does not match the user instruction, it can be determined that the possible values ​​of the function input parameters used by the user instruction cannot be correctly processed by the function corresponding to the function name.

[0125] For example, since the target large language model has natural language understanding capabilities, the function return result and user instructions can be input into the target large language model, and the target large language model can determine whether the function return result semantically matches the user instruction. Relevant personnel can also be prompted to make a manual judgment.

[0126] Exemplarily, whenever the possible values ​​of function input parameters are checked, for example, before, after, or at the same time as checking user instructions, the steps for checking the possible values ​​of function input parameters can refer to the description of step S1413 and step S1414. For the sake of brevity, they will not be repeated here.

[0127] The training data generation method of the above embodiment ensures the validity and correctness of the function input parameter values ​​by passing the function input parameter values ​​to the function and checking the function return result, which helps to ensure that the generated training data is consistent with the actual processing logic of the function and improves the credibility of the data.

[0128] Regarding step S142, in some embodiments of the present disclosure, it may include the following: Figure 9 Steps S1421 and S1422 are shown.

[0129] S1421. Take at least one function input parameter value that can be correctly processed by the function as a function input parameter optional value that has passed verification.

[0130] S1422. The user instruction in at least one user instruction that is predicted based on the verified function input parameter optional values ​​and can be correctly inferred by the target large language model is regarded as a verified user instruction.

[0131] The training data generation method of the above embodiment can accurately filter out the function input parameter optional values ​​and user instructions that pass the verification based on the verification results, effectively remove invalid or erroneous data, ensure the quality of the training data finally generated, and help improve the training effect of the target large language model.

[0132] In some embodiments of the present disclosure, before step S150, the following steps may also be included: Figure 10 As shown in step S160 and step S170, correspondingly, step S150 may include step S151.

[0133] S160: Acquire multiple candidate function description information, where the candidate function description information at least includes the candidate function name.

[0134] Exemplarily, the candidate function description information may also include one or more of a candidate function function description, a candidate function input parameter type, and a candidate function input parameter description.

[0135] S170: Calculate the similarity between the plurality of candidate function description information and the function description information.

[0136] The similarity between the plurality of candidate function description information and the function description information may be calculated using a relevant algorithm (eg, BGE algorithm).

[0137] S151. Generate training data based on the function name, the verified optional values ​​of the function input parameters, the verified user instructions and the alternative function names whose similarities are within the target similarity interval, wherein the alternative function names whose similarities are within the target similarity interval serve as interference items in the training data.

[0138] The target similarity interval can be set according to actual conditions. In one example, the target similarity interval includes a high similarity interval, a medium similarity interval, and a low similarity interval, wherein the high similarity interval includes similarities greater than 0.7, the medium similarity interval includes similarities greater than or equal to 0.4 and less than or equal to 0.7, and the low similarity interval includes similarities less than 0.4.

[0139] Exemplarily, for each user instruction that passes the verification, the function name, the alternative function name whose similarity is within the target similarity range, and at least one optional value of the verified function input parameter used to predict the user instruction can be used as the annotation content of the user instruction, and then a training data corresponding to the user instruction is generated, wherein the function name can be used as a positive sample, the alternative function name can be used as a negative sample, and the number of alternative function names can be multiple.

[0140] For each target similarity interval, multiple training data can be generated based on the alternative function names corresponding to the target similarity interval, wherein the alternative function names in the multiple training data are not exactly the same. The number of training data generated for different target similarity intervals can be the same or different. For example, the number of training data corresponding to the high similarity interval, the number of training data corresponding to the medium similarity interval, and the number of training data corresponding to the low similarity interval can be set to 1:1:1, 3:2:1 or other proportional relationships according to actual needs. It can be understood that the more training data corresponding to the high similarity interval, the more difficult it is for the target large language model to identify the expected function during the training process, and thus the generalization ability and robustness of the trained target large language model are stronger.

[0141] The training data generation method described in the above embodiment calculates the similarity between function descriptions and candidate function descriptions and introduces highly similar candidate function names as interference items, thereby enriching the content of the training data. Furthermore, the introduction of interference items imparts varying learning difficulty to the training data, helping to improve the target large language model's ability to distinguish between different functions in multi-function call scenarios, thereby enhancing the target large language model's generalization and robustness.

[0142] Please combine Figure 11 In one example, the training data generation method may include the following steps S201 to S209. The contents related to steps S201 to S209 can refer to the description of the above embodiment. For the sake of brevity, they will not be repeated here.

[0143] In step S201, the target function description information, the alternative function description information, the first prompt word and the second prompt word are obtained. The target function description information includes at least the target function name, function function description, function input parameter type and function input parameter description, and the alternative function description information includes at least the alternative function name.

[0144] In step S202, the first prompt word and function description information are input into the first model, and the first model predicts the possible values ​​of the function input parameter based on the function input parameter possible value prediction rule to obtain at least one possible value of the function input parameter.

[0145] In step S203, the second prompt word, the target function description information and at least one possible value of the function input parameter are input into the second model, and the second model is used to predict the user instruction based on the user instruction prediction rule to obtain at least one user instruction.

[0146] In step S204, for each user instruction in the at least one user instruction, a function input parameter in the user instruction is extracted using the target large language model to obtain a function input parameter extraction value corresponding to the user instruction.

[0147] In step S205, based on the extracted function input parameter values, a check is performed to determine whether the at least one user instruction can be correctly inferred by the target large language model. Specifically, for each user instruction in the at least one user instruction, if the extracted function input parameter values ​​corresponding to the user instruction are respectively the same as the possible function input parameter values ​​used to predict the user instruction, then the user instruction can be determined to be correctly inferred by the target large language model. If the extracted function input parameter values ​​corresponding to the user instruction are respectively different from the possible function input parameter values ​​used to predict the user instruction, then the user instruction can be determined to be incorrectly inferred by the target large language model.

[0148] In step S206, based on the target function, check whether the function input parameter extraction value can be correctly processed by the target function corresponding to the target function name. Specifically, for user instructions that can be correctly inferred by the target large language model, the function input parameter extraction value corresponding to the user instruction (that is, the function input parameter value used by the user instruction is predicted to be acceptable) is passed to the target function corresponding to the target function name to obtain the function return result associated with the user instruction. If the function return result associated with the user instruction matches the user instruction, it can be determined that the function input parameter value used by the user instruction is predicted to be correctly processed by the target function corresponding to the target function name; if the function return result associated with the user instruction does not match the user instruction, it can be determined that the function input parameter value used by the user instruction is predicted to be unable to be correctly processed by the target function corresponding to the target function name.

[0149] In step S207, the function input parameter values ​​that can be correctly processed by the target function are taken as the function input parameter optional values ​​that have passed the verification, and the user instructions that are predicted based on the function input parameter optional values ​​that have passed the verification and can be correctly inferred by the target large language model are taken as the user instructions that have passed the verification.

[0150] In step S208 , the similarities between the plurality of candidate function description information and the target function description information are calculated.

[0151] In step S209, training data is generated based on the target function name, the verified function input parameter optional values, the verified user instructions and the alternative function names whose similarities are within the target similarity interval, wherein the alternative function names whose similarities are within the target similarity interval serve as interference items in the training data.

[0152] Based on any of the above embodiments, the present disclosure also provides a training data generating device.

[0153] Figure 12 It is a schematic block diagram of the structure of a training data generating device according to an embodiment of the present disclosure.

[0154] like Figure 12 As shown, the training data generating device includes: an acquisition module 110, which is used to obtain function description information, where the function description information includes at least a function name, a function function description, a function input parameter type and a function input parameter description; a first prediction module 120, which is used to predict at least one function input parameter possible value based on the function input parameter type and the function input parameter description; a second prediction module 130, which is used to predict at least one user instruction based on the function function description, the function input parameter description and at least one function input parameter possible value; a verification module 140, which is used to verify at least one function input parameter possible value and at least one user instruction respectively to obtain a function input parameter possible value and a user instruction that pass the verification; and a generation module 150, which is used to generate training data based on at least the function name, the function input parameter possible value that pass the verification and the user instruction that pass the verification, wherein the training data is used to train the target large language model to improve the function call accuracy of the target large language model.

[0155] The training data generating device may be in the form of computer software, and each module of the training data generating device may be implemented by a computer software module.

[0156] In some embodiments of the present disclosure, the first prediction module 120 is used to: when the function input parameter type is a numeric type or a character type and there is an example function input parameter value in the function input parameter description, use the example function input parameter value as the function input parameter possible value; when the function input parameter type is a numeric type or a character type and there is no example function input parameter value in the function input parameter description, randomly generate at least one function input parameter possible value that meets the function input parameter type and the function input parameter description; and when the function input parameter type is an enumeration type, use all function input parameter enumeration values ​​in the function input parameter description as the function input parameter possible values.

[0157] In some embodiments of the present disclosure, the first prediction module 120 is used to: obtain a first prompt word, the first prompt word is used to prompt a function input parameter possible value prediction rule; and input the first prompt word, the function input parameter type and the function input parameter description into the first model, and predict the function input parameter possible value based on the function input parameter possible value prediction rule through the first model to obtain at least one function input parameter possible value.

[0158] In some embodiments of the present disclosure, the second prediction module 130 is used to: based on the function input parameter description, convert at least one possible value of the function input parameter into at least one first sentence component that conforms to the natural language expression habits, wherein the first sentence component includes phrases and / or words; convert the function function in the function function description into at least one second sentence component that conforms to the natural language expression habits, wherein the second sentence component includes phrases and / or words; and combine each second sentence component in the at least one second sentence component with at least one first sentence component to obtain at least one user instruction.

[0159] In some embodiments of the present disclosure, the second prediction module 130 is used to: obtain a second prompt word, which is used to prompt a user instruction prediction rule; and input the second prompt word, function function description, function input parameter description and at least one function input parameter possible value into the second model, and perform user instruction prediction based on the user instruction prediction rule through the second model to obtain at least one user instruction.

[0160] In some embodiments of the present disclosure, the verification module 140 is used to: verify, based on at least one function input parameter optional value and at least one user instruction, whether at least one user instruction can be correctly inferred by the target large language model and whether at least one function input parameter optional value can be correctly processed by the function corresponding to the function name; and determine the function input parameter optional value and the user instruction that have passed the verification based on the verification result.

[0161] In some embodiments of the present disclosure, the verification module 140 is used to: extract function input parameters in at least one user instruction through the target large language model to obtain the function input parameter extraction value corresponding to the at least one user instruction; and if the function input parameter extraction value corresponding to the at least one user instruction is respectively the same as the possible function input parameter values ​​used to predict the at least one user instruction, then determine that the at least one user instruction can be correctly inferred by the target large language model.

[0162] In some embodiments of the present disclosure, the verification module 140 is used to: pass the possible values ​​of the function input parameters used by the predicted at least one user instruction to the function corresponding to the function name to obtain at least one function return result; and if the at least one function return result matches the at least one user instruction respectively, determine that the possible values ​​of the at least one function input parameter can be correctly processed by the function corresponding to the function name.

[0163] In some embodiments of the present disclosure, the verification module 140 is used to: take, among at least one possible function input parameter values, a function input parameter value that can be correctly processed by the function as a function input parameter optional value that has passed verification; and take, among at least one user instruction, a user instruction that is predicted based on the function input parameter optional value that has passed verification and can be correctly inferred by the target large language model as a user instruction that has passed verification.

[0164] In some embodiments of the present disclosure, the training data generation apparatus further includes: a second acquisition module for acquiring multiple candidate function description information, wherein the candidate function description information includes at least a candidate function name; and a calculation module for calculating the similarity between the multiple candidate function description information and the function description information. Generation module 150 is configured to generate training data based on the function name, verified optional function input parameters, verified user instructions, and candidate function names whose similarity falls within a target similarity interval, wherein the candidate function names whose similarity falls within the target similarity interval serve as interference items in the training data.

[0165] The execution subject of the training data generation method in the specific embodiment of the present disclosure can be a server, a mobile phone, a computer or other electronic device.

[0166] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, which can execute the training data generation method of any of the above embodiments of the present disclosure.

[0167] Figure 13 1 is a schematic block diagram of the structure of an electronic device 1000 according to an embodiment of the present disclosure.

[0168] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0169] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0170] The present disclosure also provides a readable storage medium having a computer program stored therein, which is used to implement the above-mentioned method when the computer program is executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0171] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the process or function of the present disclosure is performed in whole or in part.

[0172] A computer program or instruction can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be a magnetic medium such as a floppy disk, hard disk, or magnetic tape; an optical medium such as a digital video disk; or a semiconductor medium such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0173] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, electronic devices, and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0177] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are contradictory.

[0178] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A training data generation method, characterized in that: include: Obtain function description information, which includes at least the function name, function function description, function input parameter type, and function input parameter description; Predicting at least one possible value of a function input parameter according to the function input parameter type and the function input parameter description; Predicting at least one user instruction based on the function description, the function input parameter description, and at least one possible value of the function input parameter; Verifying at least one of the possible values ​​of the function input parameter and at least one of the user instructions respectively, and obtaining a possible function input parameter value and a user instruction that pass the verification; as well as Training data is generated based at least on the function name, the verified optional function input parameter values, and the verified user instructions, wherein the training data is used to train a target large language model to improve the function call accuracy of the target large language model.

2. The training data generation method according to claim 1, characterized in that Predicting at least one possible value of a function input parameter based on the function input parameter type and the function input parameter description includes: If the function input parameter type is a numeric type or a character type, and there is a function input parameter example value in the function input parameter description, the function input parameter example value is used as the function input parameter possible value; If the function input parameter type is a numeric type or a character type and there is no example function input parameter value in the function input parameter description, randomly generate at least one possible function input parameter value that meets the function input parameter type and the function input parameter description; and In the case where the function input parameter type is an enumeration type, all function input parameter enumeration values ​​in the function input parameter description are used as the possible values ​​of the function input parameters.

3. The training data generation method according to claim 1, characterized in that Predicting at least one possible value of a function input parameter based on the function input parameter type and the function input parameter description includes: Obtaining a first prompt word, where the first prompt word is used to prompt a function input parameter possible value prediction rule; and The first prompt word, the function input parameter type and the function input parameter description are input into a first model, and the first model is used to predict the function input parameter possible values ​​based on the function input parameter possible value prediction rules to obtain at least one possible value of the function input parameter.

4. The training data generation method according to claim 1, wherein: Predicting at least one user instruction according to the function function description, the function input parameter description, and at least one possible value of the function input parameter includes: Based on the function input parameter description, converting at least one possible value of the function input parameter into at least one first sentence component that conforms to natural language expression habits, wherein the first sentence component includes a phrase and / or a word; Converting the function in the function description into at least one second sentence component that conforms to natural language expression habits, wherein the second sentence component includes phrases and / or words; and Each second sentence component in the at least one second sentence component is combined with at least one first sentence component to obtain at least one user instruction.

5. The training data generation method according to claim 1, wherein: Predicting at least one user instruction according to the function function description, the function input parameter description, and at least one possible value of the function input parameter includes: Obtaining a second prompt word, where the second prompt word is used to prompt the user of the instruction prediction rule; and The second prompt word, the function function description, the function input parameter description and at least one possible value of the function input parameter are input into a second model, and the user instruction prediction is performed based on the user instruction prediction rule by the second model to obtain at least one user instruction.

6. The training data generation method according to claim 1, characterized in that Verifying at least one of the function input parameter values ​​and at least one of the user instructions respectively, and obtaining the function input parameter values ​​and the user instructions that pass the verification, including: Verifying, based on at least one possible function input parameter value and at least one possible user instruction, whether the at least one possible user instruction can be correctly inferred by the target large language model and whether the at least one possible function input parameter value can be correctly processed by the function corresponding to the function name; and The optional values ​​of the function input parameters that pass the verification and the user instructions that pass the verification are determined according to the verification result.

7. The training data generation method according to claim 6, characterized in that: Verifying whether at least one of the user instructions can be correctly inferred by the target large language model includes: Extracting a function input parameter from at least one of the user instructions using the target large language model, and obtaining a function input parameter extraction value corresponding to at least one of the user instructions; and If the function input parameter extraction values ​​corresponding to at least one of the user instructions are respectively the same as the function input parameter possible values ​​used to predict at least one of the user instructions, it is determined that at least one of the user instructions can be correctly inferred by the target large language model.

8. An electronic device, characterized in that: include: a memory storing a computer program; as well as A processor, wherein the processor executes the computer program stored in the memory, so that the processor performs the training data generation method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the training data generation method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the training data generating method according to any one of claims 1 to 7 is implemented.

Images