Method and device for obtaining training sample, medium, equipment and product

Abstract

The invention relates to a method and device for obtaining a training sample, a medium, equipment and a product, and the method comprises the steps: obtaining first information corresponding to a first question through a first large language model, and displaying the first information; in response to a received first interaction instruction of the user on the first reasoning process information, obtaining second reasoning process information; obtaining first execution result information corresponding to the second code information; returning the first question, the second reasoning process information and the first execution result information to the first big language model, and re-executing the step of obtaining the first information corresponding to the first question through the first big language model to the step of displaying the first information until the first information comprises the first answer information; and according to the first question, the first answer information, the second reasoning process information and the first execution result information, obtaining a training sample for the second large language model. By adopting the scheme, an accurate training sample can be obtained, and the problem reasoning ability of the trained second large language model is enhanced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of large language models, and in particular, to a method, apparatus, medium, device and product for obtaining training samples. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models perform well in processing questions and generating answers, and can understand and analyze user questions and provide corresponding answers. The training of large language models requires a large number of samples, and accurate samples are very important to ensure the accuracy of the trained large language models. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for obtaining a training sample, comprising: Obtaining first information corresponding to a first question through a first large language model, wherein the first information includes first answer information or first reasoning process information, and the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; displaying the first information; In response to receiving a first interactive instruction from a user on the first reasoning process information, acquiring second reasoning process information, where the second reasoning process information includes second reasoning plan information and second code information; Obtaining first execution result information corresponding to the second code information; Returning the first question, the second reasoning process information, and the first execution result information to the first large language model, and re-performing the step of obtaining the first information corresponding to the first question through the first large language model to the step of displaying the first information, until the first information includes the first answer information; According to the first question, the first answer information, the second reasoning process information and the first execution result information, a training sample for a second language model is obtained.

[0005] In a second aspect, the present disclosure provides a device for acquiring a training sample, comprising: A first acquisition module, configured to obtain first information corresponding to a first question through a first large language model, wherein the first information includes first answer information or first reasoning process information, and the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; A display module, used for displaying the first information; A second acquisition module, configured to acquire second reasoning process information in response to receiving a first interaction instruction of a user for the first reasoning process information, wherein the second reasoning process information includes second reasoning plan information and second code information; A third acquisition module, used to acquire first execution result information corresponding to the second code information; a return module, configured to return the first question, the second reasoning process information, and the first execution result information to the first large language model, and trigger the first acquisition module to obtain the first information corresponding to the first question through the first large language model and the display module to display the first information, until the first information includes the first answer information; The fourth acquisition module is used to acquire training samples for a second language model according to the first question, the first answer information, the second reasoning process information and the first execution result information.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method for obtaining training samples provided in the first aspect of the present disclosure.

[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method for obtaining training samples provided in the first aspect of the present disclosure.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for obtaining training samples provided in the first aspect of the present disclosure.

[0009] By adopting the above technical solution, the first information corresponding to the first question is obtained through the first language model, the first information includes the first answer information or the first reasoning process information, the first information is displayed, and in response to receiving the first interactive instruction of the user for the first reasoning process information, the second reasoning process information is obtained, the second reasoning process information includes the second reasoning plan information and the second code information, and the second reasoning process information is the information after the user adjusts the first reasoning process information, which is more accurate than the first reasoning process information. Afterwards, the first execution result information corresponding to the first question, the second reasoning process information and the second code information is returned to the first language model, and the first language model continues to reason until the first information obtained through the first language model includes the first answer information, and the training sample for the second language model is obtained according to the first question, the first answer information, the second reasoning process information and the first execution result information. In this way, the first reasoning process information is obtained through the first language model, and the user can use the first reasoning process information as a reference, and perform operations such as editing on the basis of the first reasoning process information to obtain the second reasoning process information as a training sample, which is more efficient and less time-consuming than the method of directly manually marking sample data. Moreover, since the reasoning process indicated by the second reasoning process information is more accurate and reasonable, the training samples used for the second largest language model can enhance the reasoning ability of the trained second largest language model for problems, making the reasoning process of the trained second largest language model to obtain answers more accurate, thereby enabling the trained second largest language model to obtain more accurate answers based on a reasonable reasoning process.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 The figure is a flowchart of a method for obtaining training samples according to an exemplary embodiment.

[0012] Figure 2 The figure is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0013] Figure 3 It is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0014] Figure 4It is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0015] Figure 5 It is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0016] Figure 6 It is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0017] Figure 7 It is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment.

[0018] Figure 8 It is a block diagram of a device for acquiring training samples according to an exemplary embodiment.

[0019] Fig. 9 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0022] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] Figure 1 is a flow chart of a method for obtaining training samples according to an exemplary embodiment. The method can be applied to an electronic device with processing capability. The electronic device can be provided with a display device, such as a display screen. Figure 1 As shown, the method may include steps 11 to 16.

[0032] In step 11, first information corresponding to the first question is obtained through a first large language model.

[0033] For example, the first large language model may be any large language model. The first question may be any type of question described in natural language, such as a math question, a data analysis question, a knowledge question, etc. For example, the first question is "What is the sales volume of the product whose product name includes "Ice and Snow World"?"

[0034] The first information may include first answer information or first reasoning process information, and the first reasoning process information may include first reasoning plan information and first code information for implementing the first reasoning plan information. Among them, the large language model has reasoning ability, that is, it can divide complex problems into multiple steps for step-by-step reasoning and solving. In the present disclosure, if the first large language model determines that the answer to the first question can be obtained at present, the first information output by the first large language model may include the first answer information corresponding to the first question. If the first large language model determines that the answer to the first question cannot be obtained at present, the first large language model may output the subsequent first reasoning plan information for solving the first question, and the first code information for implementing the first reasoning plan information. Among them, the first reasoning plan information may be a reasoning plan to be executed to obtain the answer to the first question described in natural language, and the first code information may be any form of code, such as Python code.

[0035] In one embodiment, the first large language model may be integrated on an electronic device that executes the method for obtaining training samples provided by the present disclosure. In this embodiment, the first information corresponding to the first question may be directly obtained through the first large language model. In another embodiment, the first large language model may be integrated on a server, and the electronic device may send relevant information of the first question to the server, and the server may input the relevant information into the first large language model, and obtain the first information corresponding to the first question output by the first large language model, and then the server may send the first information to the electronic device, so that the electronic device may obtain the first information corresponding to the first question.

[0036] In step 12, first information is displayed.

[0037] The electronic device may be provided with a display device, such as a display screen, and the first information may be displayed by the display device.

[0038] In step 13, in response to receiving a first interaction instruction of the user to the first reasoning process information, the second reasoning process information is acquired.

[0039] If the first information obtained through the first large language model includes first reasoning process information, the first reasoning process information may be displayed by the display device. Since the first reasoning process information obtained through the first large language model may not be reasonable or accurate, in the present disclosure, the user may edit the first reasoning process information, and in response to receiving the user's first interactive instruction for the first reasoning process information, the second reasoning process information may be obtained, and the second reasoning process information may include second reasoning plan information and second code information.

[0040] In step 14, first execution result information corresponding to the second code information is obtained.

[0041] For example, the second code information may be run through a code execution tool such as a sandbox, and the corresponding first execution result information may be obtained according to the result output by the code execution tool.

[0042] In step 15, the first question, the second reasoning process information and the first execution result information are returned to the first large language model. Then, steps 11 to 12 are re-executed until the first information includes the first answer information.

[0043] If the first information obtained by the first large language model in step 11 includes the first reasoning process information, it can be indicated that the first large language model determines that the answer to the first question cannot be obtained at present, so the first question, the second reasoning process information and the first execution result information can be returned to the first large language model, and the first large language model continues to perform the reasoning operation until the first information includes the first answer information, indicating that the large language model determines that no further reasoning is required and the answer corresponding to the first question can be obtained. Among them, returning the first question, the second reasoning process information and the first execution result information to the first large language model at the same time can provide richer reference information for the reasoning of the first large language model.

[0044] In step 16, a training sample for a second language model is obtained according to the first question, the first answer information, the second reasoning process information and the first execution result information.

[0045] The second largest language model may be any large language model that needs to be trained, for example, a large language model that has not been trained, or a large language model that has been trained and needs to be retrained to improve its performance. In the present disclosure, the first large language model and the second large language model may be the same large language model or different large language models, without limitation.

[0046] By adopting the above technical solution, the first information corresponding to the first question is obtained through the first language model, the first information includes the first answer information or the first reasoning process information, the first information is displayed, and in response to receiving the first interactive instruction of the user for the first reasoning process information, the second reasoning process information is obtained, the second reasoning process information includes the second reasoning plan information and the second code information, and the second reasoning process information is the information after the user adjusts the first reasoning process information, which is more accurate than the first reasoning process information. Afterwards, the first execution result information corresponding to the first question, the second reasoning process information and the second code information is returned to the first language model, and the first language model continues to reason until the first information obtained through the first language model includes the first answer information, and the training sample for the second language model is obtained according to the first question, the first answer information, the second reasoning process information and the first execution result information. In this way, the first reasoning process information is obtained through the first language model, and the user can use the first reasoning process information as a reference, and perform operations such as editing on the basis of the first reasoning process information to obtain the second reasoning process information as a training sample, which is more efficient and less time-consuming than the method of directly manually marking sample data. Moreover, since the reasoning process indicated by the second reasoning process information is more accurate and reasonable, the training samples used for the second largest language model can enhance the reasoning ability of the trained second largest language model for problems, making the reasoning process of the trained second largest language model to obtain answers more accurate, thereby enabling the trained second largest language model to obtain more accurate answers based on a reasonable reasoning process.

[0047] In the present disclosure, a visual front-end page for obtaining training samples may be provided to improve user operation efficiency. Figure 2 FIG. 1 is a schematic diagram showing a page for obtaining training samples according to an exemplary embodiment. Figure 2 As shown, page 101 for obtaining training samples may include area 102, and area 102 may be used to display the first question. For example, the initial question to be labeled is first displayed in area 102. If the user does not edit the initial question to be labeled, the question to be labeled is used as the first question. For example, the initial question to be labeled displayed in area 102 is "What is the sales volume of products including "Ice and Snow World"?" If the user edits it to "What is the sales volume of products whose product names include "Ice and Snow World"?", the question edited by the user is used as the first question.

[0048] like Figure 2As shown, button 103 in area 102 can be used to expand or collapse the display box of the first question. The functions of buttons with the same shape as button 103 in other areas of the present disclosure are similar. Button 104 can be used to delete the display box of the first question, that is, to cancel the label of the first question. Button 105 is a model execution request button. When the user clicks button 105, the electronic device can send the first question to the first language model, and obtain the first information corresponding to the first question through the first language model. The first information may include first reasoning plan information and first code information for implementing the first reasoning plan information. Figure 3 As shown, for example, the first reasoning plan information can be displayed in area 106, and the first code information can be displayed in area 107. The first reasoning plan information is, for example, "first, a table related to this problem needs to be found." The button in area 106 that has the same shape as button 104 is used to delete area 106, that is, to delete the corresponding reasoning plan information, and the button in area 107 that has the same shape as button 104 is used to delete area 107, that is, to delete the corresponding code information. The functions of buttons in other areas of the present disclosure that have the same shape as button 104 are similar.

[0049] For example, the first interaction instruction includes a first confirmation instruction for the first reasoning plan information and / or a second confirmation instruction for the first code information; step 13 may include: In response to receiving the first confirmation instruction, determining the first reasoning plan information as the second reasoning plan information; In response to receiving the second confirmation instruction, the first code information is determined as the second code information.

[0050] Among them, if the first reasoning plan information obtained through the first large language model is relatively accurate, the user does not need to edit the first reasoning plan information. For example, a confirmation button may be set in area 106, and the user clicks the confirmation button, and the electronic device may receive a first confirmation instruction, or if the user's editing operation on the first reasoning plan information is not detected for a certain period of time, it can be considered that the first confirmation instruction is received. In response to receiving the first confirmation instruction, the first reasoning plan information can be determined as the second reasoning plan information. If the first code information obtained through the first large language model is relatively accurate, the user does not need to edit the first code information. For example, a confirmation button may be set in area 107, and the user clicks the confirmation button, and the electronic device may receive a second confirmation instruction, or if the user's editing operation on the first code information is not detected for a certain period of time, it can be considered that the second confirmation instruction is received. In response to receiving the second confirmation instruction, the first code information can be determined as the second code information.

[0051] In this way, the first reasoning process information is obtained through the first large language model, and the user can use the first reasoning process information as a reference. When the first reasoning plan information and the first code information obtained through the first large language model are relatively reasonable and accurate, the second reasoning plan information and the second code information can be obtained without user editing. Compared with the method of directly manually labeling sample data, it is more efficient and less time-consuming.

[0052] For example, the first interaction instruction includes a first editing instruction for the first reasoning plan information and / or a second editing instruction for the first code information; step 13 may include: In response to receiving the first editing instruction, modifying the first reasoning plan information according to the first editing instruction to obtain second reasoning plan information; In response to receiving the second editing instruction, the first code information is modified according to the second editing instruction to obtain the second code information.

[0053] If the first reasoning plan information obtained by the first language model is inaccurate, the user may implement a first editing instruction on the first reasoning plan information. In response to receiving the first editing instruction, the electronic device may modify the first reasoning plan information according to the first editing instruction to obtain the second reasoning plan information, such as Figure 4 As shown, the modified second reasoning plan information is displayed in area 106, which is "first need to find a table related to sales". If the first code information obtained by the first language model is inaccurate, the user can implement a second editing instruction on the first code information. In response to receiving the second editing instruction, the electronic device can modify the first code information according to the second editing instruction to obtain the second code information, such as Figure 4 As shown, area 107 displays the second code information obtained after modification.

[0054] For example, if only the expression of the first reasoning plan information is inappropriate, the user may only edit the first reasoning plan information without editing the first code information. Alternatively, the user may not edit the first reasoning plan information but only edit the first code information.

[0055] Through the above technical solution, the second reasoning process information is the information obtained after the user adjusts the first reasoning process information, which is more accurate than the first reasoning process information. Therefore, the training samples used for the second largest language model can enhance the reasoning ability of the trained second largest language model for problems.

[0056] In the present disclosure, in step 14, obtaining the first execution result information corresponding to the second code information may include: Run the second code information to obtain second execution result information corresponding to the second code information; Display the second execution result information; In response to receiving a fourth editing instruction from the user for the second execution result information, the second execution result information is modified to obtain the first execution result information.

[0057] like Figure 4 As shown, the button 108 in the area 107 may be a code execution request button. When the user clicks the button 108, the electronic device may receive the code execution instruction for the second code information input by the user, and may execute the second code information, for example, by executing the second code information through a sandbox, to obtain the second execution result information corresponding to the second code information, and display the second execution result information, such as Figure 5 As shown, for example, the second execution result information is displayed in area 109. If it is not detected that the user edits the second execution result information, the second execution result information can be used as the first execution result information. If the user edits the second execution result information in area 109, the electronic device can modify the second execution result information in response to receiving the fourth editing instruction of the user on the second execution result information to obtain the first execution result information, so that more accurate first execution result information can be obtained.

[0058] In this disclosure, you can preview any code execution result information, such as Figure 5 As shown, button 111 in area 109 is, for example, a preview request button, and buttons in other areas of the present disclosure that have the same shape as button 111 have similar functions, namely, they are used to preview the code execution results. By way of example, after area 109 displays the second execution result information, if the user clicks button 111, the electronic device may receive a preview request for the second execution result information. By way of example, if area 109 displays the first execution result information obtained after the second execution result information is modified according to the fourth editing instruction, when it is detected that the user clicks button 111, the electronic device may receive a preview request for the first execution result information. That is, button 111 can be used to preview the second execution result information, and can also be used to preview the modified first execution result information.

[0059] Taking the preview of the second execution result information as an example, in one embodiment, the second execution result information obtained by running the second code information may include a structure or link of a chart. In response to receiving a preview request, the electronic device may parse the structure or link and display the parsed chart, and the user may intuitively view the content of the second execution result information through the chart. In another embodiment, the second execution result information may be in a Markdown text format. In response to receiving a preview request, the Markdown text format may be converted into a table format, so that the user can more intuitively view the second execution result information.

[0060] The button 111 in the area 109 is only an example. For example, the area 109 may include multiple preview request buttons, and different preview request buttons correspond to different preview modes, such as chart preview, table preview, bar chart preview, and so on.

[0061] The method for obtaining training samples provided in the present disclosure may also include: The second reasoning process information and the first execution result information corresponding to the first question obtained this time are stored in the memory. In the present disclosure, the second reasoning process information and the first execution result information obtained each time can be stored in the memory. Figure 5 For example, the second reasoning plan information in area 106, the second code information in area 107, and the first execution result information in area 109 obtained this time may be stored in the memory of the electronic device.

[0062] like Figure 5 As shown, the button 110 in area 109 is a model execution request button. If the user clicks the button 110, the electronic device can receive the model execution request and return the first question, the second reasoning process information and the first execution result information obtained this time to the first language model. After that, the steps of obtaining the first information corresponding to the first question through the first language model to the step of displaying the first information are re-executed. If the first language model determines that the first execution result information obtained this time can solve the first question, the first answer information can be output, and the first information obtained by the electronic device may include the first answer information. For example, the first question, the first answer information, the second reasoning process information, and the first execution result information can be used as training samples for the second language model.

[0063] If the first language model determines that the first execution result information obtained this time cannot solve the first problem, the first reasoning plan information for the next step and the first code information for implementing the first reasoning plan information can be output, and the first information obtained by the electronic device may include the first reasoning plan information for the next step and the first code information. Figure 6 As shown, for example, the first reasoning plan information "how much is the sales of each product" is displayed in area 112, and the first code information is displayed in area 113, wherein the implementation method of obtaining the second reasoning process information in response to receiving the first interactive instruction of the user to the first reasoning process information has been described in detail above, for example, if the user's editing operation in area 112 is detected, the first reasoning plan information in area 112 can be modified according to the user's editing instruction to obtain the second reasoning plan information, and if the user's editing operation in area 113 is detected, the first code information in area 113 can be modified according to the user's editing instruction to obtain the second code information.

[0064] like Figure 6 As shown, button 114 in area 113 is a code execution request button. If it is detected that the user clicks button 114, the second code information in area 113 can be executed to obtain the corresponding second execution result information, and the second execution result information is displayed in area 115. If it is detected that the user edits the second execution result information in area 115, the second execution result information can be modified according to the user's editing instruction to obtain the first execution result information. Button 116 in area 115 is a model execution request button. If the user clicks button 116, the electronic device can receive a model execution request.

[0065] In the present disclosure, step 15 may include: returning the first question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time stored in the memory to the first large language model.

[0066] by Figure 6 For example, the second reasoning plan information in area 112, the second code information in area 113, and the first execution result information in area 115 serve as the second reasoning process information and first execution result information corresponding to the first problem obtained this time, and the second reasoning plan information in area 106, the second code information in area 107, and the first execution result information in area 109 serve as the second reasoning process information and first execution result information corresponding to the first problem obtained before this time and stored in the memory.

[0067] Afterwards, the steps of obtaining the first information corresponding to the first question through the first large language model to the step of displaying the first information are re-executed. If the large language model determines that the first execution result information obtained this time can solve the first question, the first answer information can be output, and the first information obtained by the electronic device may include the first answer information.

[0068] In the present disclosure, step 16 may include: In response to receiving a second interaction instruction from the user to the first answer information, obtaining second answer information; According to the first question, the second answer information, the second reasoning process information and the first execution result information, a training sample for a second language model is obtained.

[0069] Illustratively, the second interaction instruction includes a third editing instruction for the first answer information; In which, in response to receiving a second interaction instruction from the user for the first answer information, obtaining the second answer information may include: in response to receiving a third editing instruction, modifying the first answer information according to the third editing instruction to obtain the second answer information.

[0070] like Figure 6 As shown, for example, the first answer information is displayed in area 117, and the user can edit the first answer information in area 117. In response to receiving the third editing instruction, the electronic device modifies the first answer information according to the third editing instruction to obtain the second answer information. If it is not detected that the user edits the first answer information in area 117, or if a confirmation button is provided in area 117, for example, and the user clicks the confirmation button, indicating that the first answer information does not need to be modified, the electronic device can receive the confirmation instruction and can use the first answer information as the second answer information.

[0071] For example, obtaining a training sample for a second language model according to the first question, the second answer information, the second reasoning process information, and the first execution result information may include: A training sample of the second largest language model is formed based on the first question information, the second answer information, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory.

[0072] In this way, since the reasoning process indicated by the second reasoning process information is more accurate, and the thinking chain from the first question to the second answer information forms a training sample of the second largest language model, the reasoning ability of the trained second largest language model for the problem can be enhanced, making the reasoning process of the trained second largest language model to obtain the answer more accurate.

[0073] In one embodiment, the first question may be a related question of the second question, that is, the first question depends on the second question, and the first question may be a further question raised on the basis of the second question. For example, the first question may be “What is the sales volume of products whose product names include “Ice and Snow World””, and the second question may be “What are the products whose product names include “Ice and Snow World””. Since it is necessary to first find out which products have “Ice and Snow World” as the product names, and then calculate the sales volume, the first question is a related question of the second question.

[0074] In this disclosure, for multiple questions with related relationships, users can operate on the same page. Figure 7 is a schematic diagram of a page for obtaining training samples, as shown in FIG. Figure 7 As shown, the first round indicates marking the second question, and the second round indicates marking the first question. It should be noted that the number of second questions associated with the first question can be one or more without limitation. Figure 7 The types and numbers of second questions shown are for illustration only.

[0075] In one embodiment, if there is a second question related to the first question, when the first question is marked, the information returned to the first large language model may include the first question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time stored in the memory.

[0076] In another embodiment, considering that the first question depends on the answer information and reasoning process information corresponding to the second question, preferably, in order to provide a complete thinking chain for the first large language model, returning the first question, the second reasoning process information and the first execution result information to the first large language model may include: The first question, the second question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory, the answer information corresponding to the second question stored in the memory, the reasoning plan information corresponding to the second questions obtained previously, the code information used to implement the reasoning plan information corresponding to the second question, and the execution result information corresponding to the code information used to implement the reasoning plan information corresponding to the second question, are returned to the first large language model.

[0077] like Figure 7 As shown, area 201 is used to display the second question "What are the products whose product names include "Ice and Snow World"", area 202 is used to display the reasoning plan information corresponding to the second question "First, you need to find the table related to the product name", area 203 displays the code information for implementing the reasoning plan information corresponding to the second question, area 204 displays the execution result information of the code information for implementing the reasoning plan information corresponding to the second question, and area 205 is used to display the answer information corresponding to the second question. Since it is necessary to mark the second question first, and then mark the first question after obtaining the answer information of the second question, when marking the first question, the memory has already stored the answer information corresponding to the second question, the reasoning plan information corresponding to the second question obtained previously, the code information for implementing the reasoning plan information corresponding to the second question, and the execution result information corresponding to the code information for implementing the reasoning plan information corresponding to the second question.

[0078] Through the above technical solution, for multiple questions with related relationships, users can operate on the same page, and at the same time return the reasoning process information, execution result information and answer information corresponding to the second question to the first language model, so that the first language model can have richer reasoning basis.

[0079] In addition, after obtaining the second answer information corresponding to the first question, the second answer information can be stored in the memory, so that if there is a third question that needs to be marked, the third question is a related question of the first question, and the second answer information of the first question can be obtained from the memory. In addition, the first question, the second question, the second answer information corresponding to the first question, and all the second reasoning process information and the first execution result information, the answer information corresponding to the second question, and all the reasoning process information and execution result information corresponding to the second question can be used as training samples of the second largest language model.

[0080] The method for obtaining training samples provided in the present disclosure may also include: In response to receiving a user selection instruction for a first code segment, displaying at least one preset code label, the first code segment being a code segment in the second code information; In response to a user's selection instruction for a first preset code tag, a first preset code tag is marked for the first code segment, wherein the first preset code tag is one of the at least one preset code tag.

[0081] Among them, for any section of the second code information, the user can label the code fragment in the second code information. For example, the second code information includes 'p_date'>='2023-11-10'. If a user's selection instruction for the first code fragment '2023-11-10' is detected, such as a word-marking operation, at least one preset code label can be displayed. The preset code labels include, for example, a Time Code label, a Dataframe Code label, etc. In one embodiment, the user clicks on the preset code label Time Code label, and the user's selection instruction for the first preset code label Time Code can be received, and the first preset code label Time Code label can be marked for the first code fragment '2023-11-10'. The user can label multiple code fragments in the second code information, and the code labels marked for different code fragments can be the same or different. The first preset code tag marked for the first code fragment can be displayed in the form of a list below the second code information, for example, and the memory can also store the correspondence between the first code fragment and the first preset code tag. The first preset code tag marked for the first code fragment can also be returned to the first large language model at the same time, so as to facilitate the first large language model to parse the second code information more accurately.

[0082] In one embodiment, for example, a page for obtaining training samples may also have a reference area, which may display reference reasoning process information and reference answers for the first question. The reference reasoning process information and reference answers may be obtained through any large language model. Users may use the content in the reference area as a reference to facilitate comparison. In addition, the area on the page for user operation annotation may initially be the same as the reference area. Figure 3 For example, region 106 may initially display reference reasoning plan information for the first step of the first question, and region 107 may initially display reference code information corresponding to the reference reasoning plan information. If the reference reasoning process information is relatively accurate, the second reasoning process information and the second answer information corresponding to the first question can be quickly obtained without excessive editing and operation by the user, thereby reducing the time consumption for annotation. In addition, if it is detected that the user clicks button 105 in region 102, i.e., the model execution request button, the first reasoning plan information obtained by the first language model may be displayed in region 106, which will cover the previous reference reasoning plan information, and the first code information obtained by the first language model may be displayed in region 107, which will cover the previous reference code information.

[0083] It should be noted that the reasoning process shown in the page for obtaining training samples provided by the present disclosure is only for illustration and does not constitute a limitation to the implementation method.

[0084] Through the above technical solution, training samples for the second largest language model can be obtained. Since the reasoning process indicated by the second reasoning process information is more accurate, the training samples for the second largest language model can enhance the reasoning ability of the trained second largest language model for problems, making the reasoning process of the trained second largest language model for obtaining answers more accurate, thereby enabling the trained second largest language model to obtain more accurate answers based on a reasonable reasoning process.

[0085] Based on the same inventive concept, the present disclosure also provides a device for obtaining training samples. Figure 8 is a block diagram of a device for obtaining training samples according to an exemplary embodiment. Figure 8 As shown, the device 80 may include: A first acquisition module 81 is used to obtain first information corresponding to a first question through a first large language model, where the first information includes first answer information or first reasoning process information, where the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; A display module 82, used for displaying the first information; A second acquisition module 83, configured to acquire second reasoning process information in response to receiving a first interaction instruction of a user for the first reasoning process information, wherein the second reasoning process information includes second reasoning plan information and second code information; A third acquisition module 84 is used to acquire first execution result information corresponding to the second code information; a return module 85, configured to return the first question, the second reasoning process information, and the first execution result information to the first large language model, and trigger the first acquisition module 81 to obtain the first information corresponding to the first question through the first large language model and the display module 82 to display the first information, until the first information includes the first answer information; The fourth acquisition module 86 is used to acquire training samples for a second language model according to the first question, the first answer information, the second reasoning process information and the first execution result information.

[0086] Optionally, the first interaction instruction includes a first editing instruction for the first reasoning plan information and / or a second editing instruction for the first code information; The second acquisition module 83 includes: A first acquisition submodule is used for, in response to receiving the first editing instruction, modifying the first reasoning plan information according to the first editing instruction to obtain the second reasoning plan information; The second acquisition submodule is used to modify the first code information according to the second editing instruction in response to receiving the second editing instruction to obtain the second code information.

[0087] Optionally, the first interaction instruction includes a first confirmation instruction for the first reasoning plan information and / or a second confirmation instruction for the first code information; The second acquisition module 83 includes: a third acquisition submodule, configured to determine the first reasoning plan information as the second reasoning plan information in response to receiving the first confirmation instruction; The fourth acquisition submodule is configured to determine the first code information as the second code information in response to receiving the second confirmation instruction.

[0088] Optionally, the device 80 further includes: A storage module, used for storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The return module 85 is used for: The first question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory are returned to the first large language model.

[0089] Optionally, the fourth acquisition module 86 includes: A fifth acquisition submodule, configured to acquire second answer information in response to receiving a second interaction instruction from a user to the first answer information; The sixth acquisition submodule is used to acquire training samples for the second large language model according to the first question, the second answer information, the second reasoning process information and the first execution result information.

[0090] Optionally, the device 80 further includes: A storage module, used for storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The sixth acquisition submodule is used for: A training sample of the second large language model is formed based on the first question information, the second answer information, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory.

[0091] Optionally, the second interaction instruction includes a third editing instruction for the first answer information; The fifth acquisition submodule is used for: In response to receiving the third editing instruction, the first answer information is modified according to the third editing instruction to obtain the second answer information.

[0092] Optionally, the first question is a related question of the second question; The device 80 further comprises: A storage module, used for storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The return module 85 is used to: The first question, the second question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory, the answer information corresponding to the second question stored in the memory, the reasoning plan information corresponding to the second question obtained in previous times, the code information used to implement the reasoning plan information corresponding to the second question, and the execution result information corresponding to the code information used to implement the reasoning plan information corresponding to the second question, are returned to the first large language model.

[0093] Optionally, the device 80 further includes: A first display module, configured to display at least one preset code label in response to receiving a user's selection instruction for a first code segment, wherein the first code segment is a code segment in the second code information; The marking module is used to mark the first code segment with the first preset code tag in response to a user's selection instruction for the first preset code tag, wherein the first preset code tag is one of the at least one preset code tag.

[0094] Optionally, the third acquisition module 84 includes: A running submodule, used for running the second code information to obtain second execution result information corresponding to the second code information; A display submodule, used to display the second execution result information; The seventh acquisition submodule is used to modify the second execution result information in response to receiving a fourth editing instruction from the user on the second execution result information to obtain the first execution result information.

[0095] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0096] like Fig. 9As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0097] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0098] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0099] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0100] In some embodiments, the client can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0102] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains first information corresponding to a first question through a first large language model, the first information includes first answer information or first reasoning process information, the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; displays the first information; obtains second reasoning process information in response to receiving a first interactive instruction from a user for the first reasoning process information, the second reasoning process information includes second reasoning plan information and second code information; obtains first execution result information corresponding to the second code information; returns the first question, the second reasoning process information and the first execution result information to the first large language model, and re-executes the step of obtaining the first information corresponding to the first question through the first large language model to the step of displaying the first information until the first information includes the first answer information; obtains training samples for the second large language model according to the first question, the first answer information, the second reasoning process information and the first execution result information.

[0103] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring first information".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] According to one or more embodiments of the present disclosure, Example 1 provides a method for obtaining training samples, including: obtaining first information corresponding to a first question through a first large language model, the first information including first answer information or first reasoning process information, the first reasoning process information including first reasoning plan information and first code information for implementing the first reasoning plan information; displaying the first information; in response to receiving a first interactive instruction from a user for the first reasoning process information, obtaining second reasoning process information, the second reasoning process information including second reasoning plan information and second code information; obtaining first execution result information corresponding to the second code information; returning the first question, the second reasoning process information and the first execution result information to the first large language model, and re-executing the step of obtaining the first information corresponding to the first question through the first large language model to the step of displaying the first information until the first information includes the first answer information; obtaining training samples for the second large language model based on the first question, the first answer information, the second reasoning process information and the first execution result information.

[0109] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the first interaction instruction includes a first editing instruction for the first reasoning plan information and / or a second editing instruction for the first code information; in response to receiving the first interaction instruction from the user for the first reasoning process information, obtaining the second reasoning process information includes: in response to receiving the first editing instruction, modifying the first reasoning plan information according to the first editing instruction to obtain the second reasoning plan information; in response to receiving the second editing instruction, modifying the first code information according to the second editing instruction to obtain the second code information.

[0110] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the first interaction instruction includes a first confirmation instruction for the first reasoning plan information and / or a second confirmation instruction for the first code information; in response to receiving the first interaction instruction from the user for the first reasoning process information, obtaining the second reasoning process information includes: in response to receiving the first confirmation instruction, determining the first reasoning plan information as the second reasoning plan information; in response to receiving the second confirmation instruction, determining the first code information as the second code information.

[0111] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, which also includes: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time in a memory; returning the first question, the second reasoning process information and the first execution result information to the first large language model includes: returning the first question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time stored in the memory to the first large language model.

[0112] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein obtaining training samples for a second large language model based on the first question, the first answer information, the second reasoning process information and the first execution result information includes: obtaining the second answer information in response to receiving a second interaction instruction from a user to the first answer information; obtaining training samples for the second large language model based on the first question, the second answer information, the second reasoning process information and the first execution result information.

[0113] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, which also includes: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time in a memory; obtaining training samples for the second large language model based on the first question, the second answer information, the second reasoning process information and the first execution result information, including: forming training samples for the second large language model based on the first question information, the second answer information, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time stored in the memory.

[0114] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 5, wherein the second interaction instruction includes a third editing instruction for the first answer information; the obtaining of the second answer information in response to receiving the second interaction instruction from the user for the first answer information includes: in response to receiving the third editing instruction, modifying the first answer information according to the third editing instruction to obtain the second answer information.

[0115] According to one or more embodiments of the present disclosure, Example 8 provides a method of Example 1, wherein the first question is an associated question of the second question; the method also includes: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time in a memory; the returning the first question, the second reasoning process information and the first execution result information to the first large language model includes: returning the first question, the second question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, the second reasoning process information and the first execution result information corresponding to the first question obtained before this time stored in the memory, the answer information corresponding to the second question stored in the memory, the reasoning plan information corresponding to the second question obtained in previous times, the code information used to implement the reasoning plan information corresponding to the second question, and the execution result information corresponding to the code information used to implement the reasoning plan information corresponding to the second question, to the first large language model.

[0116] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 1, which further includes: in response to receiving a user's selection instruction for a first code fragment, displaying at least one preset code tag, the first code fragment being a code fragment in the second code information; in response to a user's selection instruction for a first preset code tag, marking the first code fragment with the first preset code tag, wherein the first preset code tag is one of the at least one preset code tag.

[0117] According to one or more embodiments of the present disclosure, Example 10 provides the method of Example 1, wherein obtaining the first execution result information corresponding to the second code information includes: running the second code information to obtain the second execution result information corresponding to the second code information; displaying the second execution result information; and in response to receiving a fourth editing instruction from the user for the second execution result information, modifying the second execution result information to obtain the first execution result information.

[0118] According to one or more embodiments of the present disclosure, Example 11 provides a device for obtaining training samples, including: a first acquisition module, used to obtain first information corresponding to a first question through a first large language model, the first information including first answer information or first reasoning process information, the first reasoning process information including first reasoning plan information and first code information for implementing the first reasoning plan information; a display module, used to display the first information; a second acquisition module, used to obtain second reasoning process information in response to receiving a first interactive instruction from a user for the first reasoning process information, the second reasoning process information including second reasoning plan information and second code information; a third acquisition module, used to obtain first execution result information corresponding to the second code information; a return module, used to return the first question, the second reasoning process information and the first execution result information to the first large language model, and trigger the first acquisition module to obtain the first information corresponding to the first question through the first large language model and the display module to display the first information until the first information includes the first answer information; a fourth acquisition module, used to obtain training samples for the second large language model based on the first question, the first answer information, the second reasoning process information and the first execution result information.

[0119] According to one or more embodiments of the present disclosure, Example 12 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in any one of Examples 1 to 10 when executed by a processing device.

[0120] According to one or more embodiments of the present disclosure, Example 13 provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method described in any one of Examples 1 to 10.

[0121] According to one or more embodiments of the present disclosure, Example 14 provides a computer program product, including a computer program, which implements the steps of the method described in any one of Examples 1 to 10 when executed by a processor.

[0122] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0123] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0124] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for obtaining training samples, characterized in that: include: Obtaining first information corresponding to a first question through a first large language model, wherein the first information includes first answer information or first reasoning process information, and the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; displaying the first information; In response to receiving a first interactive instruction from a user on the first reasoning process information, acquiring second reasoning process information, where the second reasoning process information includes second reasoning plan information and second code information; Obtaining first execution result information corresponding to the second code information; Returning the first question, the second reasoning process information, and the first execution result information to the first large language model, and re-performing the step of obtaining the first information corresponding to the first question through the first large language model to the step of displaying the first information, until the first information includes the first answer information; According to the first question, the first answer information, the second reasoning process information and the first execution result information, a training sample for a second language model is obtained.

2. The method according to claim 1, characterized in that The first interaction instruction includes a first editing instruction for the first reasoning plan information and / or a second editing instruction for the first code information; The step of acquiring the second reasoning process information in response to receiving a first interaction instruction from a user for the first reasoning process information includes: In response to receiving the first editing instruction, modifying the first reasoning plan information according to the first editing instruction to obtain the second reasoning plan information; In response to receiving the second editing instruction, the first code information is modified according to the second editing instruction to obtain the second code information.

3. The method according to claim 1, characterized in that The first interaction instruction includes a first confirmation instruction for the first reasoning plan information and / or a second confirmation instruction for the first code information; The step of acquiring the second reasoning process information in response to receiving a first interaction instruction from a user for the first reasoning process information includes: In response to receiving the first confirmation instruction, determining the first reasoning plan information as the second reasoning plan information; In response to receiving the second confirmation instruction, the first code information is determined as the second code information.

4. The method according to claim 1, characterized in that The method further comprises: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The returning the first question, the second reasoning process information, and the first execution result information to the first large language model includes: The first question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory are returned to the first large language model.

5. The method according to claim 1, characterized in that The acquiring, according to the first question, the first answer information, the second reasoning process information and the first execution result information, a training sample for a second language model includes: In response to receiving a second interaction instruction from the user to the first answer information, obtaining second answer information; According to the first question, the second answer information, the second reasoning process information and the first execution result information, a training sample for the second large language model is obtained.

6. The method according to claim 5, characterized in that The method further comprises: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The acquiring, according to the first question, the second answer information, the second reasoning process information and the first execution result information, a training sample for the second language model includes: A training sample of the second large language model is formed based on the first question information, the second answer information, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, and the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory.

7. The method according to claim 5, characterized in that The second interaction instruction includes a third editing instruction for the first answer information; The step of acquiring the second answer information in response to receiving the second interaction instruction of the user to the first answer information includes: In response to receiving the third editing instruction, the first answer information is modified according to the third editing instruction to obtain the second answer information.

8. The method according to claim 1, characterized in that The first question is a related question of the second question; The method further comprises: storing the second reasoning process information and the first execution result information corresponding to the first question obtained this time into a memory; The returning the first question, the second reasoning process information, and the first execution result information to the first large language model includes: The first question, the second question, the second reasoning process information and the first execution result information corresponding to the first question obtained this time, the second reasoning process information and the first execution result information corresponding to the first question obtained before this time and stored in the memory, the answer information corresponding to the second question stored in the memory, the reasoning plan information corresponding to the second question obtained in previous times, the code information used to implement the reasoning plan information corresponding to the second question, and the execution result information corresponding to the code information used to implement the reasoning plan information corresponding to the second question, are returned to the first large language model.

9. The method according to claim 1, characterized in that: The method further comprises: In response to receiving a user selection instruction for a first code segment, displaying at least one preset code label, the first code segment being a code segment in the second code information; In response to a user's selection instruction for a first preset code tag, the first code segment is marked with the first preset code tag, wherein the first preset code tag is one of the at least one preset code tag.

10. The method according to claim 1, characterized in that The obtaining of first execution result information corresponding to the second code information includes: Run the second code information to obtain second execution result information corresponding to the second code information; Displaying the second execution result information; In response to receiving a fourth editing instruction from the user for the second execution result information, the second execution result information is modified to obtain the first execution result information.

11. A device for obtaining training samples, characterized in that: include: A first acquisition module, configured to obtain first information corresponding to a first question through a first large language model, wherein the first information includes first answer information or first reasoning process information, and the first reasoning process information includes first reasoning plan information and first code information for implementing the first reasoning plan information; A display module, used for displaying the first information; A second acquisition module, configured to acquire second reasoning process information in response to receiving a first interaction instruction of a user for the first reasoning process information, wherein the second reasoning process information includes second reasoning plan information and second code information; A third acquisition module, used to acquire first execution result information corresponding to the second code information; a return module, configured to return the first question, the second reasoning process information, and the first execution result information to the first large language model, and trigger the first acquisition module to obtain the first information corresponding to the first question through the first large language model and the display module to display the first information, until the first information includes the first answer information; The fourth acquisition module is used to acquire training samples for a second language model according to the first question, the first answer information, the second reasoning process information and the first execution result information.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method described in any one of claims 1 to 10 are implemented.

13. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images