Data processing methods, apparatus, electronic devices and computer-readable storage media

Abstract

This application discloses a data processing method, apparatus, electronic device, and computer-readable storage medium. In this embodiment, the method involves obtaining description information of an auxiliary object, which assists a first target model in processing a question-answering task; using a second target model, based on the description information, generating target information required when the first target model calls the auxiliary object to process the question-answering task; and using the second target model, predicting the questions that need to be input into the first target model to trigger the generation of the target information. The target information and questions are used to generate training samples for the first target model, thereby improving the efficiency of target information and question generation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data that can handle question-answering tasks.

[0003] To improve the performance of large language models in handling question-answering tasks in a specific domain, question-answering data related to the auxiliary object in that domain will be used to fine-tune the large language model. For example, when the auxiliary object is an article in the medical field, the question-answering data related to the auxiliary object can be information from the article and questions entered by the user. Similarly, when the auxiliary object is a function tool, the question-answering data related to the auxiliary object can be the answers output by the function tool and questions entered by the user.

[0004] However, currently, the question-and-answer data related to auxiliary objects used for fine-tuning large language models are all manually generated by users, which reduces the efficiency of question-and-answer data generation. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium that can improve the efficiency of generating question-and-answer data related to the assisting object.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0007] Obtain the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task;

[0008] Based on the above description information, the second target model generates the target information required when the first target model calls the above auxiliary object to perform question-answering task processing.

[0009] The second target model is used to predict the problem that needs to be input into the first target model to trigger the generation of the target information. The target information and the problem are used to generate training samples for the first target model.

[0010] Secondly, embodiments of this application also provide a data processing apparatus, the apparatus comprising:

[0011] The information acquisition module is used to acquire the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task.

[0012] The first generation module is used to generate target information required when the first target model calls the auxiliary object to perform question-answering task processing, based on the above description information and using the second target model.

[0013] The second generation module is used to predict, through the second target model, the question that needs to be input into the first target model to trigger the generation of the target information, and the target information and the question are used to generate training samples for the first target model.

[0014] Thirdly, embodiments of this application also provide an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute any of the data processing methods provided in embodiments of this application.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the data processing methods provided in embodiments of this application.

[0016] In this embodiment, the description information of the auxiliary object is obtained, and the auxiliary object is used to assist the first target model in processing the question-answering task. Based on the description information, the second target model generates the target information required when the first target model calls the auxiliary object to process the question-answering task. The second target model predicts the questions that need to be input into the first target model to trigger the generation of target information. The target information and questions are used to generate training samples for the first target model, thereby realizing the automatic generation of question-answering data related to the auxiliary object without manual generation, thus improving the generation efficiency of question-answering data related to the auxiliary object. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of one embodiment of the data processing method provided in this application.

[0019] Figure 2 This is a schematic diagram of the model training process provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the confusion matrix provided in an embodiment of this application;

[0021] Figure 4This is a schematic diagram of the model evaluation method provided in the embodiments of this application;

[0022] Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Before providing a detailed explanation of the embodiments of this application, some terms involved in the embodiments of this application will be explained.

[0026] In the description of the embodiments of this application, the terms "first," "second," etc., may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. "A plurality of" refers to two or more.

[0027] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium. Specifically, the data processing method of this application can be executed by an electronic device, which can be a terminal or a server, etc.

[0028] The terminal can be a smartphone, tablet, laptop, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. The terminal can also include a client, which can be a game application client, a browser client with a game program, or an instant messaging client, etc.

[0029] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0030] The following is a detailed description in conjunction with the accompanying drawings. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the drawings.

[0031] In this embodiment, a terminal is used as an example for illustration. This embodiment provides a data processing method, such as... Figure 1 As shown, the specific flow of this data processing method can be as follows:

[0032] 101. Obtain the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task.

[0033] The first target model can be either an untrained neural network model or a trained neural network model. When the second target model is a trained neural network model, it can refer to a Large Language Model (LLM), which can handle natural language tasks, including question answering tasks.

[0034] The auxiliary object is used to assist the primary target model in handling question-answering tasks. It can be understood as improving the ability of the trained target model to handle question-answering tasks in the target domain after training the primary target model with question-answering data related to the auxiliary object. The target domain refers to the domain in which the auxiliary object exists. The target domain can include a general domain and / or a specialized target domain.

[0035] The target professional field refers to a specific knowledge category, which can be set according to the actual situation. For example, the target professional field can be the medical field or the automotive field. This embodiment does not limit it.

[0036] It is understandable that after training the first target model using question-and-answer data related to the auxiliary object, the trained target model can understand and answer questions in the target domain. Therefore, when using the trained target model to handle question-and-answer tasks in the target domain, the trained target model can do so without relying on the auxiliary object. Alternatively, after training the target model using question-and-answer data related to the auxiliary object, the trained target model can understand questions in the target domain and invoke the auxiliary object. Therefore, when using the trained target model to handle question-and-answer tasks in the target domain, the trained target model can invoke the auxiliary object to obtain the answer.

[0037] The type of auxiliary object can be set according to the actual situation. For example, when it is necessary to improve the first target model in handling question-and-answer tasks in the medical field, the auxiliary object can be articles in the medical field. Or, when the first target model needs to handle question-and-answer tasks in a certain professional field by calling function tools, the auxiliary object can be function tools. This application embodiment does not limit this.

[0038] The descriptive information of an auxiliary object can refer to the text that explains or describes the auxiliary object. For example, when the auxiliary object is an article, its descriptive information is the content of the article; when the auxiliary object is a function tool, its descriptive information is the function description of the tool and the parameter description of the input parameters required to call the tool; and when the auxiliary object is a code library, its descriptive information is the code in the code library.

[0039] 102. Based on the description information, the second target model generates the target information required when the first target model calls the auxiliary object to process the question-answering task.

[0040] The second target model can refer to a trained neural network model, such as a Large Language Model (LLM), which can handle natural language tasks, including question-answering tasks. The specific type of LLM can be determined based on the actual situation; for example, the second target model could be Azure-OpenAI-GPT-3.5 or Microsoft Azure-GPT-4.0, but this embodiment does not impose any limitations. When the first target model is also a LLM, the second target model and the first target model can be the same model or different models.

[0041] The target information that the first target model needs to generate when calling the auxiliary object to process the question-and-answer task can refer to the answer generated when the first target model calls the auxiliary object to process the question-and-answer task, or it can refer to the instruction of the first target model to call the auxiliary object, which is used to call the auxiliary object to get the answer.

[0042] For example, when the auxiliary object is an article, the target information can be the answer to the question; when the auxiliary object is a function tool, the target information can be the instruction to call the function tool, or the answer obtained after calling the function tool.

[0043] In this embodiment, the target information required for the first target model to call the auxiliary object to perform question-and-answer task processing is automatically generated based on the description information, eliminating the need for users to manually generate the target information and improving the efficiency of target information generation.

[0044] In some embodiments, when the auxiliary object is an article or code repository, the target information is the answer. The second target model, based on the descriptive information, generates the target information required when the first target model calls the auxiliary object to perform the question-answering task, including:

[0045] The second target model is used to select key content from the descriptive information;

[0046] By combining the key information, we can arrive at the answer.

[0047] In some embodiments, when the auxiliary object is a function tool, the target information can be the instruction to call the function tool.

[0048] In some embodiments, when the auxiliary object is a function tool, the target information required for the first target model to call the auxiliary object to perform question-answering task processing is generated based on the description information using the second target model, including:

[0049] Based on the functional and parameter descriptions, the second target model generates the target parameters required by the first target model to call the function tool.

[0050] By assembling the target parameters and the tool name of the function tool, the instruction to call the function tool is obtained.

[0051] In this context, "function tool" refers to a tool that encapsulates code that performs certain functionalities, and may include at least one such tool. The type of code implementing these functionalities can be set according to actual needs; for example, the code implementing these functionalities may be a function, interface, model, or workflow, etc., and this implementation does not impose any limitations on this. The implemented functionalities can be set according to actual needs; for example, the implemented functionalities may be a weather query function or a product query function, and this embodiment does not impose any limitations on this.

[0052] When the auxiliary object is a function tool, its description information includes the function tool's functional description and the parameter description of the input parameters required to call the function tool. The function tool's functional description refers to information describing the function tool's functionality. For example, if the function tool's function is a weather query function, its functional description could be: "This function tool is used to query the weather in a specific location."

[0053] When calling a function tool, you need to input some parameters. The description of these parameters is called the parameter description. When the function tool performs different functions, the parameters required to call it will also be different, and so will the parameter description. For example, if the function tool's function is to query the weather, the required parameters are location and time, and the parameter description would be: "Required parameters representing location and time." Another example is a function tool that queries salaries, where the required parameter is name, and the parameter description would be: "Required parameters representing name."

[0054] The target parameters required to call a function tool refer to the specific values ​​of the parameters that need to be entered. For example, if the function tool's function is to query the weather, the parameters required to call the function tool are parameters representing the location and time, and the target parameters required to call the function tool are Wuhan and tomorrow.

[0055] After obtaining the target parameters, the terminal can assemble the target parameters and the tool name of the function tool to obtain the instruction to call the function tool.

[0056] For example, if the function tool is for salary lookup, the target parameter is "Zhang San", the tool name is salary_search, and the command to call the function tool can be "salary_search(name='Zhang San')".

[0057] Understandably, when calling a function tool requires multiple input parameters, and each required input parameter corresponds to multiple target parameters, the terminal can first generate each target parameter through the second target model, then combine the target parameters to obtain the combined parameters, and then assemble the combined parameters and the tool name of the function tool to obtain the instruction to call the function tool.

[0058] For example, the parameters required to call a function tool include parameters representing time and parameters representing location. Target parameters generated for the time parameters include tomorrow and the day after tomorrow, and target parameters generated for the location parameters include Wuhan and Shenzhen. Combining tomorrow and Wuhan yields a combined parameter, combining tomorrow and Shenzhen yields a combined parameter, combining the day after tomorrow and Shenzhen yields a combined parameter, and combining the day after tomorrow and Wuhan yields a combined parameter, resulting in a total of four combined parameters. These four combined parameters are then combined with the tool name to obtain four commands for calling the function tool.

[0059] Alternatively, when calling a function tool requires multiple input parameters, and each required input parameter corresponds to multiple target parameters, the terminal can directly generate combined parameters through a second target model. These combined parameters include at least two target parameters. For example, if the input parameters for calling a function tool include parameters representing time and location, the terminal can directly generate the target parameters "Wuhan" and "tomorrow" through the second target model, thus obtaining the combined parameters.

[0060] Optionally, when one of the target parameters affects the generation of other target parameters, the combined parameters can be generated directly through the first target model.

[0061] Optionally, when generating the target parameters required for the function calling tool based on the function description and parameter description using the second target model, a parameter generation prompt template can also be input into the second target model so that the target parameters required for the function calling tool can be generated using the second target model based on the function description, parameter description, and parameter generation prompt template.

[0062] Optionally, if generating target parameters requires samples or generation rules, the samples or generation rules can be stored in the parameter description. In this case, the parameter description includes not only a description of the required input parameters, but also samples or generation rules, so that the second target model can refer to the samples or generation rules and generate target parameters based on the functional description and parameter description.

[0063] In some embodiments, the terminal can generate instructions based on the function information of the function tool. The function information includes the tool name, function description, parameter name, parameter description, parameter type, and parameter requirement of the function tool.

[0064] 103. Using the second target model, predict the problem that needs to be input into the first target model to trigger the generation of target information.

[0065] In this context, the question is equivalent to the content input by the user. When processing question-answering tasks using the first objective model, the user typically needs to manually input the question, and the first objective model then generates an answer based on the question. In this case, the answer can be directly generated by the first objective model, or it can be an answer obtained by the first objective model calling a function tool.

[0066] For example, if a user inputs the question "Why do I have frequent nosebleeds?", the first target model will directly output the answer "It could be nasal dryness or injury to the nasal cavity wall".

[0067] In this embodiment, the answer "It could be nasal dryness or nasal cavity wall injury" is first generated based on the auxiliary object, and then the question "What's wrong with frequent nosebleeds?" is generated based on the answer. "What's wrong with frequent nosebleeds?" is the question that needs to be input into the first target model to trigger the generation of "It could be nasal dryness or nasal cavity wall injury".

[0068] For example, if a user inputs the question "How much is Zhang San's salary?", the first target model outputs the instruction "salary_search(name='Zhang San')" based on "How much is Zhang San's salary?", and the answer obtained by calling the function tool based on this instruction is "Zhang San's salary is XXXX".

[0069] In this embodiment, the instruction "salary_search(name='Zhang San')" is first generated based on the function tool, and then "How much is Zhang San's salary?" is generated based on the instruction. "How much is Zhang San's salary?" is the question that needs to be input into the first target model to trigger the generation of "salary_search(name='Zhang San')".

[0070] In this embodiment, target information is first automatically generated, and then the user-inputted question is simulated based on the target information. This eliminates the need for the user to manually input the question, improving the efficiency of question generation and thus improving the efficiency of question-and-answer data generation. The question-and-answer data includes target information and questions.

[0071] In some embodiments, the question can be a short sentence or a long sentence. For example, when the auxiliary object is a function tool and the function tool is a weather query function, the question can be "Wuhan, tomorrow's weather" (a short sentence), or "I want to ask about the weather in Wuhan tomorrow" (a long sentence).

[0072] In some embodiments, when the auxiliary object is a function tool and the target information is an instruction, before predicting the problem that needs to be input into the first target model to trigger the generation of target information using the second target model, the method further includes:

[0073] Perform quality checks on the instructions to obtain the quality check results;

[0074] Determine the instructions corresponding to the quality inspection results that meet the preset quality conditions.

[0075] The second objective model predicts the questions that need to be input into the first objective model to trigger the generation of objective information, including:

[0076] The second target model predicts the problems that need to be input into the first target model to trigger the generation of target instructions. The target instructions refer to the instructions corresponding to the quality inspection results that meet the preset quality conditions.

[0077] The process of performing quality checks on instructions and obtaining the quality check results can be as follows:

[0078] The quality of the instruction is checked based on at least one of the following: the instruction's structure, the tool name in the instruction, the parameters in the instruction, and the type of the parameters in the instruction, and the quality check result of the instruction is obtained.

[0079] For example, when performing quality checks on instructions based on their structure, we can analyze whether the instruction includes a tool name and parameters. If the instruction includes both, it indicates that the quality check result is the first quality check result. The first quality check result indicates a quality check result that meets preset quality conditions.

[0080] For example, when performing quality checks on an instruction based on the parameters in the instruction, it is possible to analyze whether the parameters in the instruction are mandatory and whether they are required input parameters. If the parameters in the instruction are required input parameters, the quality check result of the instruction is the first quality check result. If the parameters in the instruction are not required input parameters, the quality check result of the instruction is the second quality check result, which is a quality check result that does not meet the preset quality conditions.

[0081] Understandably, when performing quality checks on an instruction based on its composition structure, the tool name within the instruction, the parameters within the instruction, and the types of the parameters within the instruction, the quality check result of the instruction can be determined based on the results of various sub-quality checks.

[0082] For example, based on the composition structure of the instruction, a quality check is performed on the instruction, and the resulting sub-quality check result is the first quality check result. Based on the parameters in the instruction, a quality check is performed on the instruction, and the resulting sub-quality check result is the second quality check result. Based on the first quality check result and the second quality check result, the quality check result of the instruction is determined.

[0083] It is understandable that when the sub-quality inspection result includes the second quality inspection result, the quality inspection result of the instruction can be determined as the second quality inspection result.

[0084] In some embodiments, when the helper object is a function tool, the problem includes the target parameters that need to be input.

[0085] In some embodiments, when the question includes the target parameters that need to be input, the question includes a complete question and an incomplete question. A complete question includes all the target parameters that need to be input when calling the function tool, while an incomplete question includes some of the target parameters that need to be input when calling the function tool.

[0086] For example, the parameters required to call a function tool include a parameter representing the location and a parameter representing the time. A complete question could be "What will the weather be like in Wuhan tomorrow?" In this case, the complete question includes the target parameter "Wuhan" representing the location and the target parameter "tomorrow" representing the time. An incomplete question could be "What will the weather be like tomorrow?" In this case, the incomplete question only includes the target parameter "tomorrow" representing the time and lacks the target parameter "Wuhan" representing the location.

[0087] In this embodiment, the questions include complete questions and incomplete questions. Complete questions include all target parameters required to call the function tool, while incomplete questions include only some target parameters required to call the function tool. This diversifies the generated questions, thereby improving the ability of the trained target model to handle question-answering tasks in the target domain after training the first target model based on the questions and target information.

[0088] In some embodiments, this embodiment further includes:

[0089] Using the second objective model, a supplementary question is generated based on the incomplete problem. The supplementary question is used to prompt the input of the target parameters that are missing in the incomplete problem.

[0090] Using the second target model, supplementary information is generated based on the supplementary question. This supplementary information includes the missing target parameters. The supplementary question and supplementary information are used to generate training samples for the first target model.

[0091] This can involve generating complete and incomplete problems based on instructions. Alternatively, a complete problem can be generated based on instructions, and then an incomplete problem can be generated based on the target parameters.

[0092] After obtaining the incomplete question, a supplementary question is generated based on the incomplete question, and supplementary information is generated based on the supplementary question.

[0093] For example, an incomplete question might be "I want to check my salary for January." Since the incomplete question lacks the target parameter representing the name, the generated supplementary question could be "Are you checking for January of this year? Also, could you provide the name you are checking?" Based on the supplementary question, the generated supplementary information could be something like "It's for January 2023, and the name is Zhang San."

[0094] After receiving the supplementary questions and information, the terminal can generate training samples for the first target model based on the supplementary questions, information, instructions, and questions.

[0095] Specifically, the terminal can generate training samples based on incomplete questions and supplementary questions. In these training samples, incomplete questions serve as data and supplementary questions serve as labels. These training samples are used to train the first target model's ability to ask follow-up questions. Alternatively, the terminal can generate training samples based on incomplete questions, supplementary questions, supplementary information, and instructions. In these training samples, incomplete questions, supplementary questions, and supplementary information serve as data and instructions serve as labels. These training samples are used to train the first target model's ability to call function tools.

[0096] It is understandable that the training samples generated based on incomplete questions and supplementary questions, as well as the training samples generated based on incomplete questions, supplementary questions, supplementary information, instructions, and function tools, belong to the multi-turn dialogue type.

[0097] In this embodiment, target information and questions are used to generate training samples for a first target model. A supplementary question is generated based on an incomplete question using a second target model. This supplementary question prompts for the missing target parameters in the incomplete question. Supplementary information, including the missing target parameters, is then generated based on the supplementary question and the second target model. The supplementary question and supplementary information are used to generate training samples for the first target model. These training samples, when trained on the first target model using the training samples, enable the trained target model to ask follow-up questions. This allows the trained target model to better understand the user's intent and further enhances its ability to call function tools.

[0098] In some embodiments, this embodiment further includes:

[0099] According to the instructions, the function tool is invoked to obtain the parameter values ​​corresponding to the problem;

[0100] Based on the parameter values, the answers to the questions are generated, and the parameter values ​​and answers are used to generate training samples.

[0101] For example, if the question is "What is Zhang San's salary in January 2023?", the command is "salary_search(name='Zhang San')". Based on the command, the function tool is called, and the corresponding parameter values ​​are "{'emp_no':'123456','name':'Zhang San','time':'2023 / 01,'salary':'xxxx'}". The answer to the question is "Hello, we found that Zhang San's salary in January 2023 is xxxx".

[0102] At this point, training samples can be generated based on instructions and questions. In these training samples, questions serve as data and instructions as labels. These training samples are used to train the first target model's ability to call function tools. Training samples can also be generated based on parameter values ​​and answers. In these training samples, parameter values ​​serve as data and answers as labels. These training samples are used to train the first target model's ability to summarize the parameter values ​​returned by the function tools into answers. Training samples can also be generated based on instructions, questions, function toolsets, parameter values, and answers. In these training samples, instructions, questions, and parameter values ​​serve as data, and answers as labels. These training samples are used to train the first target model's ability to summarize the parameter values ​​returned by the function tools into answers.

[0103] It is understandable that the training samples generated based on parameter values ​​and answers, as well as the training samples generated based on instructions, questions, function tools, parameter values, and answers, belong to the multi-turn dialogue type.

[0104] It is understandable that when the question includes an incomplete question, training samples can be generated not only from the incomplete question and supplementary question, or from the incomplete question, supplementary question, supplementary information, and instructions, but also from the incomplete question, supplementary question, supplementary information, instructions, function tools, parameter values, and answers. In this training sample, the incomplete question, supplementary question, supplementary information, instructions, and parameter values ​​serve as data, and the answers serve as labels. This training sample is used to train the first target model's ability to summarize the parameter values ​​returned by the function tools into answers. The type of training sample generated from incomplete questions, supplementary questions, supplementary information, instructions, parameter values, and answers belongs to the multi-turn dialogue type.

[0105] In this embodiment of the application, according to the instruction, the function tool is called to obtain the parameter value corresponding to the question. Based on the parameter value, the answer corresponding to the question is generated. The parameter value and the answer are used to generate training samples. After training the first target model based on the training samples, the ability of the trained target model to summarize the parameter value into the answer can be improved, making it easier for users to understand the answer.

[0106] In some embodiments, the second target model predicts the problem that needs to be input into the first target model to trigger the generation of target information, including:

[0107] Retrieve the set character information;

[0108] The second target model, based on target information and role information, predicts the questions that need to be input into the first target model to trigger the generation of target information.

[0109] Role information refers to information describing a user's identity. Different role information indicates different users. For example, role information could indicate the identity of a human resources person or the identity of a regular employee.

[0110] For example, if the function tool is for querying salaries, and the role information indicates the identity of the personnel, then the question that needs to be input into the first target model to trigger the generation of target information could be "I need to query Zhang San's salary". If the role information indicates the identity of an ordinary employee, then the question that needs to be input into the first target model to trigger the generation of target information could be "I want to know how much Zhang San earns each month".

[0111] Since different users may input different questions for the same instruction, in this embodiment, the second target model predicts the questions that need to be input into the first target model to trigger the generation of target information based on the target information and role information. This makes the generated questions more diverse and further improves the ability of the trained target model to handle question-answering tasks in the target domain.

[0112] In some embodiments, after predicting the problem that needs to be input into the first target model to trigger the generation of target information using the second target model, the method further includes:

[0113] The problem is augmented to obtain the augmented problem.

[0114] In this embodiment, the enhanced problem and target information are used together to generate training samples for the first target model. The type of enhancement processing can be set according to the actual situation. For example, enhancement processing includes at least one of replacement processing, rewriting processing, simplification processing, and error correction processing. This embodiment does not limit the specific type of enhancement processing.

[0115] Replacement processing refers to replacing non-critical content in the question; the replaced content does not affect the generation of instructions. Rewriting processing refers to changing the syntax and question format without altering the key content. Simplification processing refers to reducing the question to a shorter sentence. Error correction processing refers to adding typos to the question.

[0116] In this embodiment of the application, after obtaining the problem, the problem is enhanced to diversify the problems ultimately used to generate training samples.

[0117] In some embodiments, after predicting the problem that needs to be input into the first target model to trigger the generation of target information using the second target model, the method further includes:

[0118] Based on the target information and the problem, generate training samples for the first target model;

[0119] Based on the training samples, the first target model is trained to obtain the trained target model.

[0120] In this embodiment, training samples for a first target model are generated based on the target information and the question. The first target model is then trained based on the training samples to obtain a trained target model, thereby improving the ability of the trained target model to handle question-answering tasks in the target domain.

[0121] Furthermore, since the target information and questions are automatically generated question-and-answer data for the second target model, the efficiency of generating question-and-answer data is improved. Therefore, the efficiency of training the first target model can be improved based on the target information and questions.

[0122] Furthermore, since the target information is generated first through the second target model, and then questions are generated based on the target information, not only can the quality of the questions and the quality of the target information not heavily depend on the capabilities of the second target model, but the questions and target information can also have diversity, structural completeness, and stability, thereby further improving the ability of the trained target model to handle question-answering tasks in the target domain.

[0123] In some embodiments, the auxiliary object is a function tool that generates training samples for the first target model based on the target information and the problem, including:

[0124] The problem, target information, and the function tool set containing the function tools are assembled to obtain the training samples for the first target model.

[0125] The process of assembling the problem, target information, and the set of function tools to obtain the training samples for the first target model can be described as follows:

[0126] Based on some instructions, some questions, and the function toolset where the function tools are located, generate training samples for the first target model;

[0127] The order of the function tools in the function toolset is adjusted to obtain the adjusted function toolset;

[0128] Based on additional instructions, additional questions, and the adjusted function toolkit, training samples for the first target model are generated.

[0129] For example, a function tool set includes function tool c1, function tool c2, and function tool c3. Function tool c1 is the first function tool in the function tool set, function tool c2 is the second function tool in the function tool set, and function tool c3 is the third function tool in the function tool set. After adjusting the order of the function tools in the function tool set, function tool c1 becomes the second function tool in the adjusted function tool set, function tool c3 becomes the first function tool in the adjusted function tool set, and function tool c2 becomes the third function tool in the adjusted function tool set.

[0130] In this embodiment, training samples for the first target model are generated based on some instructions, some questions, and the function toolset containing the function tools. The order of the function tools in the function toolset is adjusted to obtain an adjusted function toolset. Training samples for the first target model are then generated based on other instructions, other questions, and the adjusted function toolset. This ensures that the order of the function tools in the function toolset is different in different training samples, preventing the trained target model from overfitting to the order of the function tools in the function toolset and improving the generalization ability of the trained target model in the target domain.

[0131] In some embodiments, the auxiliary object is a function tool, the target information is instructions, and training samples for the first target model are generated based on the target information and the problem, including:

[0132] A training sample is obtained by assembling a single instruction, the corresponding problem, and a set of function tools.

[0133] The training samples are obtained by assembling the two instructions, the problems and function tools corresponding to the two instructions respectively.

[0134] The two instructions can be any two instructions generated. Optionally, the order of the questions corresponding to the two instructions in the training samples is determined based on the order of the two instructions in the training samples. For example, the instruction to call function tool c1 is instruction d1, and the question that triggers the generation of instruction d1 is q1. The instruction to call function tool c2 is instruction d2, and the question that triggers the generation of instruction d2 is q2. In the training samples generated based on instruction d1, question q1, instruction d2, and question q2, the order of instruction d1 precedes the order of instruction d2, and the order of question q1 precedes the order of question q2.

[0135] It is understandable that assembling the training samples obtained by combining the two instructions and the problem and function toolkits corresponding to the two instructions respectively results in a parallel type of training samples.

[0136] In this application, a training sample is obtained by assembling a single instruction, the corresponding problem, and the function tool set. Similarly, a training sample is obtained by assembling two instructions, the corresponding problems, and the function tool sets. This allows a training sample to contain two instructions, enabling the parallel invocation of two function tools and improving the usability of the target model after training.

[0137] In some embodiments, the training samples include at least two of the following types: single-tool type, multi-tool type, parallel type, and multi-turn dialogue type. The single-tool type refers to a type in which the function tool set includes only one function tool. The multi-tool type refers to a type in which the function tool set includes at least two function tools. The parallel type refers to a type in which the training samples include two instructions and two questions that trigger the generation of the two instructions. The multi-turn dialogue type refers to a type in which the training samples contain multi-turn dialogue information, wherein the multi-turn dialogue information is used to ask follow-up questions based on the parameters required to call the function tool to obtain all the parameters required for the input.

[0138] In the embodiments of this application, the training samples include at least two types, which enhances the diversity of training samples and further improves the ability of the trained target model to handle question-answering tasks in the target domain.

[0139] In some embodiments, the order of training samples of various types in the training set is shuffled, so that the trained target model obtained by training the first target model based on the training samples will not overfit, thereby improving the generalization ability of the trained target model.

[0140] In some embodiments, the first target model can also be trained based on training samples of an irrelevant type, which are training samples generated based on question-and-answer data unrelated to function tool calls.

[0141] In some embodiments, the first target model is trained based on training samples to obtain a trained target model, including:

[0142] The problems and function toolset from the training samples are input into the first target model for processing to obtain the first prediction instruction;

[0143] The loss function value is determined based on the first prediction instruction and the instructions in the training samples;

[0144] If the loss function value meets the preset loss condition, then the first target model is determined as the target model after training;

[0145] If the loss function value does not meet the preset loss condition, the weights of the first target model are updated according to the loss function value, and the process of inputting the problem and function tool set from the training samples into the first target model for processing is returned.

[0146] Understandably, the loss function value determined based on the first prediction instruction and the instructions in the training samples is called the first loss function value. When the training samples include incomplete questions and supplementary questions, the incomplete questions are input into the first target model for processing to obtain the predicted supplementary questions. The second loss function value is determined based on the predicted supplementary questions and the supplementary questions. When the training samples include parameter values ​​and answers, the parameter values ​​are input into the first target model for processing to obtain the predicted answers. The third loss function value is determined based on the answers and the predicted answers. At this point, the loss function value of the first target model can be determined based on the first, second, and third loss function values. If the loss function value meets the preset loss condition, the first target model is determined as the post-trained target model. If the loss function value does not meet the preset loss condition, the weights of the first target model are updated according to the loss function value, and the process returns to the step of inputting the training samples into the first target model for processing.

[0147] In some embodiments, the first target model is trained based on training samples to obtain a trained target model, including:

[0148] Obtain real question and answer data;

[0149] Generate real training samples based on real question-and-answer data;

[0150] The first target model is trained based on the training samples and the real training samples to obtain the trained target model.

[0151] In this embodiment, the target information and questions generated by the second target model can be referred to as simulated question-and-answer data. Real question-and-answer data includes questions manually entered by the user and the target information corresponding to those questions.

[0152] In this embodiment, real question-and-answer data is obtained, real training samples are generated based on the real question-and-answer data, and the first target model is trained based on the training samples and the real training samples to obtain the trained target model. This realizes the training of the first target model based on simulated question-and-answer data and real question-and-answer data, thereby further improving the ability of the trained target model to handle question-and-answer tasks in the target domain.

[0153] In some embodiments, the auxiliary object is an auxiliary object in the target professional field. Based on training samples, the first target model is trained to obtain the trained target model, including:

[0154] Obtain training samples from a general domain;

[0155] The first target model is trained based on training samples and training samples from the general domain to obtain the trained target model.

[0156] In this embodiment of the application, training samples from a general domain are obtained, and a first target model is trained based on the training samples and the training samples from the general domain to obtain a trained target model. This allows the trained target model to not only excel at handling question-answering tasks in the target professional domain, but also maintain its ability to handle question-answering tasks in the general domain.

[0157] In some embodiments, a first target model is trained based on training samples to obtain a trained target model, including:

[0158] Set the adapter for the first target model;

[0159] The training samples are input into the first target model for processing to obtain the loss function value;

[0160] If the loss function value does not meet the preset loss condition, the weights of the adapter are updated based on the loss function value, and the process returns to the step of inputting the training samples into the first target model for processing to obtain the loss function value.

[0161] If the loss function value meets the preset loss condition, the weights of the adapter and the weights of the network layers in the first target model are fused to obtain the trained target model.

[0162] The adapter setting for the first target model can refer to adding a new network layer in a position parallel to at least one network layer in the first target model. The weights of the new network layer are obtained by multiplying two low-rank matrices. The size of the weights of the new network layer is the same as the size of the weights of the at least one network layer. If the size of the weights of the at least one network layer is m*n, splitting the m*n matrix yields two matrices of size m*r and r*n, meaning the sizes of the two matrices of the newly added network layer are m*r and r*n, respectively. If the weights of the at least one network layer are updated directly, the number of parameters that need to be updated is m*n. However, if the weights of the additional network layer are updated, the number of parameters that need to be updated is m*r + r*n, where m*r + r*n is less than m*n.

[0163] Therefore, in this embodiment, an adapter for the first target model is set up, and training samples are input into the first target model for prediction processing to obtain a loss function value. If the loss function value does not meet the preset loss condition, the weights of the adapter are updated based on the loss function value, and the process of inputting training samples into the first target model for processing to obtain a loss function value is returned. If the loss function value meets the preset loss condition, the weights of the adapter after the last update are fused with the weights of at least one network layer in the first target model to obtain the trained target model, thereby reducing the number of parameters that need to be updated and further improving the training efficiency of the first target model.

[0164] For example, when training a first target model based on training samples and real training samples to obtain a trained target model, the training process of the first target model can be as follows: Figure 2 As shown.

[0165] In some embodiments, this embodiment further includes:

[0166] Get the preset number of updates;

[0167] Get the adapter's weight after the preset number of updates;

[0168] The weights of the adapter and the weights of the network layers in the first target model are fused to obtain the trained target model, including:

[0169] The weights after being updated a preset number of times will be merged to obtain the adapter weights;

[0170] The weights of the adapter and the weights of the network layers in the second target model are fused to obtain the trained target model.

[0171] The preset update count, also known as breakpoints, refers to certain numbers of updates made by the adapter. Since the training count of the model can be set before training, the preset update count can be determined based on the training count. For example, the preset update count could be the 1st / 3rd, 2nd / 3rd, and last training count, but this embodiment does not impose any limitations on this.

[0172] For example, if the training iterations are 30, and the adapter weights are updated 30 times when the loss function value meets the preset loss condition (the preset update times are the 10th, 20th, and 30th), the adapter weights after the 10th, 20th, and 30th updates are fused to obtain the adapter weights. Then, the adapter weights are fused with the weights of at least one network layer in the first target model to obtain the trained target model. At this point, the adapter weights fused with the weights of at least one network layer in the first target model are not the last updated adapter weights, but rather the weights fused with the adapter weights after the preset update times.

[0173] It is understandable that the terminal can record the weight of the adapter after a preset number of updates, and then obtain the weight of the adapter after a preset number of updates when the loss function value meets the preset loss condition.

[0174] As training time increases, the ability of the first target model to handle general domain question-answering tasks decreases. In this embodiment, a preset number of updates is obtained, and the weights of the adapter after the preset number of updates are obtained. The weights after the preset number of updates are fused to obtain the adapter weights. The adapter weights are then fused with the weights of the network layers in the first target model to obtain the trained target model. This not only improves the ability of the trained target model to handle question-answering tasks in the target professional domain, but also maintains its ability to handle general domain question-answering tasks, preventing the trained target model from overfitting to the training samples and improving its generalization ability.

[0175] In some embodiments, the auxiliary object is an auxiliary object in the target professional field, and before generating training samples for the first target model based on the target information and the problem, the method further includes:

[0176] Obtain model description information for each candidate large language model;

[0177] Based on the target professional field and model introduction information, the first target model is determined from various candidate large language models.

[0178] The model description information includes at least one of the following: basic information about the model, the characteristics of the model, and the model's logs.

[0179] The terminal can determine the matching degree between each candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, and then determine the candidate large language model with the highest matching degree as the first target model. Alternatively, the terminal can predict the matching degree between the candidate large language models and the target professional field based on the model introduction information and the requirements of the target professional field, recommend each candidate large language model to the user according to the matching degree, and then determine the candidate large language model corresponding to the user's selection operation as the first target model in response to the user's selection operation.

[0180] Specifically, the matching degree between each candidate large language model and the target professional field can be determined based on the characteristics of the model and the requirements of the target professional field. For example, if the target professional field is hotel customer service, the questions in this field are usually in the form of text. In this case, the matching degree of the candidate large language model that is good at handling text-based questions is greater than that of the candidate large language model that is good at handling image-based questions. As another example, if the target professional field is law, the questions in this field are usually longer. In this case, the matching degree of the candidate large language model that is good at handling long questions is greater than that of the candidate large language model that is good at handling short questions.

[0181] In this embodiment of the application, model description information of each candidate large language model is obtained. Based on the target professional field and model description information, a first target model is determined from each candidate large language model. This allows the training of the first target model to be completed more quickly, further improving the efficiency of obtaining the trained target model.

[0182] In some embodiments, before generating training samples for the first target model based on the target information and the problem, the method further includes:

[0183] Display the various candidate large language models;

[0184] In response to the selection operation of each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the first target model.

[0185] In this embodiment, various candidate large language models are displayed. In response to a selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the first target model, allowing the user to select the first target model and improving the user experience.

[0186] In some embodiments, the auxiliary object is an auxiliary object in the target professional field, and obtaining the description information of the auxiliary object includes:

[0187] Display the data generation interface;

[0188] In response to input operations on the data generation interface, obtain the name of the target professional field;

[0189] Using the second target model, based on the name of the target professional field, we obtain descriptive information about auxiliary objects in the target professional field.

[0190] In this embodiment, a data generation interface is displayed. In response to input operations on the data generation interface, the name of the target professional field is obtained. Based on the name of the target professional field, the description information of the auxiliary object of the target professional field is obtained through the second target model. The target information and questions are generated based on the description information, so that the user can select the target professional field according to his or her own needs. This makes the trained target model good at handling the question-and-answer tasks of the professional field required by the user, and realizes the user's customization of the model.

[0191] It is understandable that when a user manually selects the first target model, they can either select to generate target information and questions before selecting the first target model, or select to generate target information and questions at the same time as selecting the first target model. This embodiment does not limit this.

[0192] In some embodiments, users may also provide auxiliary objects in the target professional field to make the trained target model more in line with user needs.

[0193] In some embodiments, after obtaining the trained target model, the method further includes:

[0194] Obtain a test set that includes at least one test sample;

[0195] Based on the evaluation samples, assess the ability of the trained target model to handle question-answering tasks in the target domain.

[0196] The evaluation samples in the evaluation set can be generated manually or based on the target information and the question. The generation process can refer to the process of generating training samples based on the target information and the question, which will not be repeated here in this embodiment.

[0197] When the auxiliary object is a function tool, the evaluation samples include question-and-answer data related to the function tools in the function tool set, as well as the function tool set itself. The question-and-answer data includes questions and instructions. For ease of description, the questions in the training samples are referred to as the first question, the instructions in the training samples as the first instruction, the questions in the evaluation samples as the second question, and the instructions in the evaluation samples as the second instruction. In this case, based on the evaluation samples, the ability of the trained target model to handle question-and-answer tasks in the target domain is evaluated, including:

[0198] The second question and the function tool set are input into the trained target model for processing, generating a second prediction instruction that calls the function tools in the function tool set;

[0199] Based on the function tools invoked by the second prediction instruction and the function tools invoked by the second instruction, evaluate the ability of the trained target model to invoke the function tools in the function toolset set to handle question-answering tasks.

[0200] Specifically, after inputting the second question and the set of function tools into the trained target model, the trained target model can predict which function tool in the function tools will provide the answer to the question, and then generate a second prediction instruction to call that function tool.

[0201] For example, the function toolkit includes function tool c1, function tool c2, and function tool c3. It is predicted that the answer to the question will be obtained through function tool c2, and a second prediction instruction will be generated to call function tool c2.

[0202] The second prediction instruction includes a tool name, and the second instruction also includes a tool name. The function tool called by the second prediction instruction refers to the function tool corresponding to the tool name in the second prediction instruction. The function tool called by the second instruction is equivalent to a label. Based on the label, it is determined whether the function tool called by the second prediction instruction is correct.

[0203] For the same problem, if the function tool called by the second prediction instruction is the same as the function tool called by the second instruction, it indicates that the target training model's call to the function tool after training is accurate. If the function tool called by the second prediction instruction is different from the function tool called by the second instruction, it indicates that the target training model's call to the function tool after training is incorrect, and other function tools in the function tool set are interfering with the call to the function tool.

[0204] For example, the evaluation sample includes a question q1 and an instruction d1. Instruction d1 is an instruction to call function tool c3. The function tool set includes function tool c1, function tool c2, and function tool c3. The question q1 and the function tool set are input into the trained target model for processing to obtain the second prediction instruction d11. If the second prediction instruction d11 is a prediction instruction to call function tool c3, it indicates that the call to function tool c3 is accurate. If the second prediction instruction d11 is a prediction instruction to call function tool c2, it indicates that the call to function tool c3 is incorrect, and function tool c2 interferes with the call to function tool c3.

[0205] After receiving the second prediction instruction, the terminal can count the number of correct calls and the number of incorrect calls for each function tool according to the function tools called by the second prediction instruction and the function tools called by the second instruction. Finally, based on the statistical results of each function tool, the terminal can evaluate the ability of the trained target model to call the function tools in the function tool set to handle question answering tasks.

[0206] In this embodiment, based on the function tools invoked by the second prediction instruction and the function tools invoked by the second instruction, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated. This simplifies the evaluation task of the trained target model's ability to handle question-answering tasks into a function tool classification task, enabling the trained target model to obtain its calling capabilities on each function tool. This refines the granularity of evaluating the trained target model's ability to handle question-answering tasks and improves the accuracy of the evaluation.

[0207] In some embodiments, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated based on the function tools invoked according to the second prediction instruction and the function tools invoked by the second instruction, including:

[0208] Based on the function tools invoked by the second prediction instruction and the function tools invoked by the second instruction, the influence coefficient between the function tools is determined. The influence coefficient is used to indicate the degree of mutual influence between the function tools.

[0209] Based on the impact coefficient, we evaluate the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks.

[0210] In this embodiment, when the function tool called by the second prediction instruction is different from the function tool called by the second instruction, it can be determined that the function tool called by the second prediction instruction is interfering with the function tool called by the second instruction. This allows us to obtain the degree of mutual influence between the various function tools, so that the training samples can be adjusted according to the degree of mutual influence between the various function tools. Based on the training samples, the target model after training can be fine-tuned. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the target model after training, and improve the efficiency of obtaining the final target model.

[0211] In some embodiments, the ability of the trained target model to call function tools in the function tool set to handle question-answering tasks is evaluated based on the function tools invoked according to the second prediction instruction and the function tools invoked by the second instruction, including:

[0212] Based on the function tools called by the second prediction instruction and the function tools called by the instruction, a confusion matrix is ​​constructed. The column data in the confusion matrix indicates the function tools called by the second instruction, and the row data in the confusion matrix indicates the function tools called by the second prediction instruction.

[0213] Based on the confusion matrix, we evaluate the ability of the trained target model to call functions from the function toolkit to handle question-answering tasks.

[0214] The confusion matrix, also known as the error matrix, is used to show the classification results of the target model on the function tool after training.

[0215] For example, a confusion matrix can be like this: Figure 3 As shown, from Figure 3 As can be seen, there are 11 evaluation samples for function tool c1. Among the second prediction instructions generated based on the problems in the 11 evaluation samples, the function tool in 8 of the second prediction instructions is function tool c1, indicating that the 8 second prediction instructions are accurate and the call to function tool c1 is accurate. The function tool in 3 of the second prediction instructions is function tool c3, indicating that the call to function tool c1 was misidentified as a call to function tool c3, indicating that there was an error in the call to function tool c3.

[0216] In this embodiment, a confusion matrix is ​​constructed based on the function tools called by the second prediction instruction and the function tools called by the second instruction. The column data in the confusion matrix indicates the function tools called by the second instruction, and the row data in the confusion matrix indicates the function tools called by the second prediction instruction. Based on the confusion matrix, the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks is evaluated. This allows not only a direct view of the classification results of the trained target model on the function tools, but also a comprehensive evaluation of the trained target model.

[0217] In some embodiments, based on the confusion matrix, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated, including:

[0218] Based on the row data in the confusion matrix, the influence coefficients between the function tools are determined. The influence coefficients are used to indicate the degree of mutual influence between the function tools.

[0219] Based on the impact coefficient, we evaluate the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks.

[0220] The type of influence coefficient can be set according to the actual situation. For example, the influence coefficient can be at least one of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). This application embodiment does not limit this.

[0221] True positives represent the number of evaluation samples that predict a function tool as that function tool. True negatives represent the number of evaluation samples that predict other function tools as other function tools. False positives represent the number of evaluation samples that predict other function tools as that function tool, which can represent the interference that function tool causes to other function tools. False negatives represent the number of evaluation samples that predict a function tool as other function tools, which can represent the interference that other function tools cause to that function tool.

[0222] In this embodiment, the influence coefficients between functional tools are determined based on the row data in the confusion matrix. The influence coefficients are used to indicate the degree of mutual influence between functional tools. Based on the influence coefficients, the ability of the trained target model to call functional tools in the functional tool set to handle question-answering tasks is evaluated. This allows the training samples to be adjusted according to the degree of mutual influence between the various functional tools. Based on the training samples, the trained target model can be further fine-tuned. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the final target model.

[0223] In some embodiments, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated based on an impact coefficient, including:

[0224] Based on the influence coefficient, the call capability indicator parameters for each function tool are determined. The call capability indicator parameters include at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve.

[0225] Based on the call capability indicator parameter for each function tool, the ability of the trained target model to call each function tool is determined.

[0226] For example, based on the influence coefficient of function tool c1, determine the calling capability indicator parameter of function tool c1; based on the calling capability indicator parameter of function tool c1, determine the ability of the target model to call function tool c1 after training; based on the influence coefficient of function tool c2, determine the calling capability indicator parameter of function tool c2; based on the calling capability indicator parameter of function tool c2, determine the ability of the target model to call function tool c2 after training.

[0227] In the embodiments of this application, based on the influence coefficient, the calling capability indicator parameter for each function tool is determined. The calling capability indicator parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve. Based on the calling capability indicator parameter for each function tool, the ability of the trained target model to call the function tool is determined, thereby obtaining the performance capability of the trained target model on each function tool.

[0228] In some embodiments, the ability of the trained target model to invoke functions from a set of functions to handle question-answering tasks is evaluated based on an impact coefficient, including:

[0229] Based on the influence coefficient, determine the overall evaluation coefficient of the target model after training for each function tool;

[0230] Based on the overall evaluation coefficient, the ability of the target model after training to call functions in the function toolkit to handle question-answering tasks is evaluated.

[0231] The process of determining the overall evaluation coefficients of the trained target model for each function tool based on the influence coefficient can be described as follows:

[0232] Based on the impact coefficient, the invocation capability indicator parameters for each function tool are determined. The invocation capability indicator parameters include at least one of precision and recall.

[0233] The call capability indicator parameters are weighted to obtain the overall evaluation coefficient of the target model for each function tool after training; or the call capability indicator parameters are averaged to obtain the overall evaluation coefficient of the target model for each function tool after training.

[0234] For example, when the invocation capability indicator parameter is precision, the overall evaluation coefficient can be weighted precision or average precision; when the invocation capability indicator parameter is recall, the overall evaluation coefficient can be weighted recall.

[0235] The weighted precision rate can be calculated using the following formula:

[0236]

[0237] p_i represents the accuracy of the i-th function tool, Number_i represents the number of evaluation samples associated with the i-th function tool, and Total represents the total number of evaluation samples in the evaluation set.

[0238] Average precision can be calculated using the following formula:

[0239]

[0240] p_i represents the accuracy of the i-th function tool, and N represents the number of function tools in the function tool set.

[0241] In this embodiment, the overall evaluation coefficient of the trained target model for each function tool is determined based on the influence coefficient. Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks is evaluated. This realizes the evaluation of the trained target model based on the overall evaluation coefficient, thereby further improving the accuracy of the evaluation of the trained target model.

[0242] In some embodiments, after receiving the second prediction instruction, the method further includes:

[0243] The correctness of the second prediction instruction is analyzed based on the target dimension, and the analysis results are obtained. The target dimension includes at least one of the following: the structure of the second prediction instruction, the parameters in the second prediction instruction, and the applicable scenarios of the second prediction instruction.

[0244] Based on the function tools invoked by the second prediction instruction and the function tools invoked by the second instruction, evaluate the ability of the trained target model to invoke the function tools in the function toolset set to handle question-answering tasks, including:

[0245] Based on the function tools invoked by the second prediction instruction, the function tools invoked by the second instruction, and the analysis results, evaluate the ability of the trained target model to invoke the function tools in the function toolset to handle question-answering tasks.

[0246] When the target dimension is the structure of the second prediction instruction, the process of analyzing the correctness of the second prediction instruction based on its structure and obtaining the analysis results can be as follows:

[0247] Determine whether the second prediction instruction includes a tool name and parameters;

[0248] If the second prediction instruction includes a tool name and parameters, it indicates that the structure of the second prediction instruction is correct, and the analysis result will be determined as the correct result.

[0249] When the target dimension is a parameter in the second prediction instruction, the process of analyzing the correctness of the second prediction instruction based on its parameters and obtaining the analysis results can be as follows:

[0250] The analysis results are obtained by examining whether the parameters in the second prediction instruction are missing, whether there are unnecessary parameters, whether the parameter types are incorrect, and whether the parameters match.

[0251] When the target dimension is the applicable scenario of the second prediction instruction, the process of analyzing the correctness of the second prediction instruction based on the applicable scenario and obtaining the analysis results can be as follows:

[0252] Determine the applicable scenario for the second prediction instruction, which indicates whether or not a function tool needs to be called.

[0253] Based on the applicable scenarios of the second prediction instruction, the correctness of the second prediction instruction is analyzed, and the analysis results are obtained.

[0254] In the case where the second prediction instruction is applicable in a scenario that requires calling a function tool, it may misidentify that a function tool should not be called as one should be called.

[0255] In this embodiment, the correctness of the second prediction instruction is analyzed based on the target dimension to obtain the analysis results. The target dimension includes at least one of the following: the structure of the second prediction instruction, the parameters in the second prediction instruction, and the applicable scenarios of the second prediction instruction. Based on the function tools called by the second prediction instruction, the function tools called by the instruction, and the analysis results, the ability of the trained target model to call the function tools in the function tool set to handle question-answering tasks is evaluated, thereby realizing the evaluation of the trained target model from multiple dimensions and further improving the accuracy of the evaluation of the trained target model.

[0256] In some embodiments, when the question-answering data includes real question-answering data and simulated question-answering data, the terminal can generate evaluation samples based on the real and simulated question-answering data. Based on the evaluation samples, the number of correct prediction instructions obtained through the trained target model and the number of correct prediction instructions obtained through the second target model are determined. Based on the number of correct prediction instructions, the accuracy of the trained target model and the accuracy of the second target model are obtained. The accuracy of the trained target model indicates its ability to handle question-answering tasks, and the accuracy of the second target model indicates its ability to handle question-answering tasks. This allows for a comparison of the ability of the trained target model and the second target model to handle question-answering tasks. For example, ... Figure 4 As shown.

[0257] As can be seen from the above, in this embodiment of the application, the description information of the auxiliary object is obtained, and the auxiliary object is used to assist the first target model in processing the question-and-answer task; the second target model generates the target information required by the first target model to call the auxiliary object to process the question-and-answer task based on the description information; the second target model predicts the questions that need to be input into the first target model to trigger the generation of target information, thereby realizing the automatic generation of question-and-answer data related to the auxiliary object without manual generation, and improving the generation efficiency of question-and-answer data related to the auxiliary object.

[0258] To better implement the above methods, this application also provides a data processing device, which can be integrated into an electronic device, such as a computer device, which can be a terminal, server, or other device.

[0259] The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0260] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the data processing device specifically integrated into the terminal as an example. This embodiment provides a data processing device, such as... Figure 5 As shown, the data processing apparatus may include:

[0261] The information acquisition module 501 is used to acquire the description information of the auxiliary object, which is used to assist the first target model in processing the question-and-answer task.

[0262] The first generation module 502 is used to generate target information required when the first target model calls the auxiliary object to perform question-answering task processing, based on the description information and using the second target model.

[0263] The second generation module 503 is used to predict, through the second target model, the question that needs to be input into the first target model to trigger the generation of target information, and the target information and the question are used to generate training samples for the first target model.

[0264] In some embodiments, the auxiliary object is a function tool, the description information is a functional description of the function tool and a parameter description of the input parameters required to call the function tool, the target information is the instruction to call the function tool, and the question includes the target parameters that need to be input.

[0265] In some embodiments, the first generation module 502 is specifically used to perform:

[0266] Based on the functional and parameter descriptions, the second target model generates the target parameters required by the first target model to call the function tool.

[0267] By assembling the target parameters and the tool name of the function tool, the instruction to call the function tool is obtained.

[0268] In some embodiments, the problem includes complete problems and incomplete problems, and the second generation module 503 is further configured to perform:

[0269] Using the second objective model, a supplementary question is generated based on the incomplete problem. The supplementary question is used to prompt the input of the target parameters that are missing in the incomplete problem.

[0270] Using the second target model, supplementary information is generated based on the supplementary question. This supplementary information includes the missing target parameters. The supplementary question and supplementary information are used to generate training samples for the first target model.

[0271] In some embodiments, the second generation module 503 is specifically used to perform:

[0272] Retrieve the set character information;

[0273] The second target model, based on target information and role information, predicts the questions that need to be input into the first target model to trigger the generation of target information.

[0274] In some embodiments, the data processing apparatus further includes a training module, which is used to perform:

[0275] Based on the target information and the problem, generate training samples for the first target model;

[0276] Based on the training samples, the first target model is trained to obtain the trained target model.

[0277] In some embodiments, the training module is specifically used to perform:

[0278] Based on some instructions, some questions, and the function toolset where the function tools are located, generate training samples for the first target model;

[0279] The order of the function tools in the function toolset is adjusted to obtain the adjusted function toolset;

[0280] Based on additional instructions, additional questions, and the adjusted function toolkit, training samples for the first target model are generated.

[0281] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0282] Accordingly, embodiments of this application also provide an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Figure 6 As shown, Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 600 includes a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, and a computer program stored on the memory 602 and executable on the processor. The processor 601 and the memory 602 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0283] The processor 601 is the control center of the electronic device 600. It connects various parts of the electronic device 600 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device 600 and processes data, thereby monitoring the electronic device 600 as a whole.

[0284] In this embodiment, the processor 601 in the electronic device 600 loads the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 runs the applications stored in the memory 602 to achieve various functions, such as:

[0285] Obtain the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task;

[0286] Based on the descriptive information, the second target model generates the target information required when the first target model calls the auxiliary object to process the question-answering task.

[0287] The second target model predicts the questions that need to be input into the first target model to trigger the generation of target information. The target information and questions are used to generate training samples for the first target model.

[0288] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the data processing methods above, which will not be repeated here.

[0289] Optional, such as Figure 6 As shown, the electronic device 600 also includes: a touch display screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch display screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607. Those skilled in the art will understand that... Figure 6The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0290] The touch display screen 603 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 603 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 601. It can also receive and execute commands from the processor 601. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 601 to determine the type of touch event. Subsequently, the processor 601 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 603 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 603 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 603 can also be used as part of the input unit 606 to achieve input functions.

[0291] The radio frequency circuit 604 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0292] Audio circuit 605 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 605 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 605, converted back into audio data, and then processed by processor 601 before being transmitted via radio frequency circuit 604 to, for example, another electronic device, or output to memory 602 for further processing. Audio circuit 605 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0293] The input unit 606 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0294] Power supply 607 is used to supply power to various components of electronic device 600. Optionally, power supply 607 can be logically connected to processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 607 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0295] although Figure 6 As not shown in the diagram, the electronic device 600 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0296] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0297] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0298] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute any of the data processing methods provided in embodiments of this application. For example, the computer program can perform the following steps:

[0299] Obtain the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task;

[0300] Based on the descriptive information, the second target model generates the target information required when the first target model calls the auxiliary object to process the question-answering task.

[0301] The second target model predicts the questions that need to be input into the first target model to trigger the generation of target information. The target information and questions are used to generate training samples for the first target model.

[0302] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the data processing methods above, which will not be repeated here.

[0303] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0304] Since the computer program stored in the computer-readable storage medium can execute any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0305] The foregoing has provided a detailed description of a data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the description information of the function tool, which is used to assist the first target model in processing the question-answering task. The description information is a functional description of the function tool and a parameter description of the input parameters required to call the function tool. Based on the description information, the second target model generates the instructions or answers required by the first target model to call the function tool for question-answering task processing. The second target model predicts the questions that need to be input into the first target model to trigger the generation of the instruction or the answer. The instruction and the question are used to generate training samples for the first target model, or the answer and the question are used to generate training samples for the first target model.

2. The data processing method as described in claim 1, characterized in that, The step of generating the instructions required for the first target model to call the function tool for question-answering task processing based on the description information using the second target model includes: Based on the function description and parameter description, the second target model generates the target parameters required for the first target model to call the function tool. The target parameters and the tool name of the function tool are assembled to obtain the instruction to call the function tool.

3. The data processing method as described in claim 1, characterized in that, The problem includes complete problems and incomplete problems, and the method further includes: Based on the incomplete problem, a supplementary problem is generated using the second target model. The supplementary problem is used to prompt for input of the target parameters missing in the incomplete problem. Using the second target model, supplementary information is generated based on the supplementary question. The supplementary information includes the missing target parameters. The supplementary question and the supplementary information are used to generate training samples for the first target model.

4. The data processing method as described in claim 1, characterized in that, The step of predicting the question that needs to be input into the first target model to trigger the generation of the instruction or the answer through the second target model includes: Retrieve the set character information; Based on the instructions and the role information, the second target model predicts the problem that needs to be input into the first target model to trigger the generation of the instructions; or; Based on the answer and the role information, generate a question that triggers the generation of the answer.

5. The data processing method as described in claim 1, characterized in that, After predicting the question that needs to be input into the first target model to trigger the generation of the instruction or the answer using the second target model, the method further includes: Based on the instructions and the question, generate training samples for the first target model; or, based on the answer and the question, generate training samples for the first target model. The first target model is trained based on the training samples to obtain the trained target model.

6. The data processing method as described in claim 5, characterized in that, The step of generating training samples for the first target model based on the instructions and the question includes: Based on some of the instructions, some of the questions, and the function tool set to which the function tool is located, training samples for the first target model are generated; The order of the function tools in the function toolset is adjusted to obtain the adjusted function toolset; Training samples for the first target model are generated based on the instructions, the questions, and the adjusted function toolset described in the additional section.

7. A data processing apparatus, characterized in that, The device includes: The information acquisition module is used to acquire the description information of the function tool, which is used to assist the first target model in processing the question-answering task. The description information is a functional description of the function tool and a parameter description of the input parameters required to call the function tool. The first generation module is used to generate, based on the description information, the instructions required by the first target model when calling the function tool to perform question-answering task processing, or the answers generated by calling the function tool, using the second target model. The second generation module is used to predict, through the second target model, the question that needs to be input into the first target model to trigger the generation of the instruction or the answer, wherein the instruction and the question are used to generate training samples for the first target model, or the answer and the question are used to generate training samples for the first target model.

8. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to execute the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the data processing method according to any one of claims 1 to 6.

Images