Large language model training method and device, information interaction method and device and storage medium

Abstract

The invention provides a big language model training method and device, an information interaction method and device and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of big language models, agents, deep learning, man-machine interaction and the like. According to the specific implementation scheme, a second operation process of a first sample text is determined through a first large language model, and a second webpage address obtained by interaction between the first large language model and a sample browser based on the second operation process is obtained, determining a target reward value obtained by interaction between the first large language model and the sample browser according to the second webpage address and a first webpage address corresponding to the first sample text, and performing reinforcement learning training on the first large language model according to the target reward value; therefore, the obtained second large language model can accurately determine the operation process used for obtaining the query target in the target website related to the corresponding text, and the webpage address of the query target can be conveniently and accurately obtained based on the operation process subsequently.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of large language models, agents, deep learning, human-computer interaction, and the like, and in particular to a large language model training method, an information interaction method, an apparatus, and a storage medium. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, large language models have shown great potential in various application scenarios. In particular, in a web navigation task, after obtaining a target text input by a user in the web navigation task, how to train a large language model such that the output based on the large language model can interact with a browser to efficiently locate a web page address corresponding to a query target on a target website involved in the target text has become a key problem to improve human-computer interaction efficiency. SUMMARY

[0003] The present disclosure provides a large language model training method, an information interaction method, an apparatus, and a storage medium.

[0004] According to an aspect of the present disclosure, a large language model training method is provided, which includes: obtaining a first sample text in a web navigation task, a first operation process corresponding to the first sample text, and a first web page address obtained based on the first operation process, wherein the first operation process is used to obtain a web page address of a first query target on a first target website involved in the first sample text, and the first query target is determined based on the first sample text; determining a second operation process of the first sample text based on the first sample text and a first large language model, wherein the second operation process is used to obtain a web page address of a second query target on a second target website involved in the first sample text, and the second query target is determined based on the first sample text; obtaining a second web page address obtained by the first large language model interacting with a sample browser based on the second operation process; determining a target reward value obtained by the first large language model interacting with the sample browser based on the first web page address and the second web page address; and performing reinforcement learning training on the first large language model based on the target reward value to obtain a second large language model.

[0005] According to another aspect of the present disclosure, an information interaction method is provided, the method comprising: obtaining a to-be-processed text under a webpage navigation task; determining, according to a second large language model, a target operation process corresponding to the to-be-processed text, wherein the target operation process is used to obtain a webpage address of a seventh query target on a seventh target website involved by the to-be-processed text, and the seventh query target is determined based on the to-be-processed text, wherein the second large language model is obtained according to training of a large language model disclosed in the present application; based on the target operation process, interacting with the seventh target website on a preset browser to obtain a target webpage address of the seventh query target; and displaying a webpage of the target webpage address on the preset browser.

[0006] According to another aspect of the present disclosure, a training device of a large language model is provided, the device comprising: a first obtaining module configured to obtain a first sample text under a webpage navigation task, a first operation process corresponding to the first sample text, and a first webpage address obtained based on the first operation process, wherein the first operation process is used to obtain a webpage address of a first query target on a first target website involved by the first sample text, and the first query target is determined based on the first sample text; a first determining module configured to determine, according to the first sample text and a first large language model, a second operation process of the first sample text, wherein the second operation process is used to obtain a webpage address of a second query target on a second target website involved by the first sample text, and the second query target is determined based on the first sample text; a second obtaining module configured to obtain a second webpage address obtained by the first large language model interacting with a sample browser based on the second operation process; a second determining module configured to determine, according to the first webpage address and the second webpage address, a target reward value obtained by the first large language model interacting with the sample browser; and a training module configured to perform reinforcement learning training on the first large language model according to the target reward value to obtain a second large language model.

[0007] According to another aspect of the present disclosure, an information interaction device is provided, the device comprising: a third acquisition module configured to acquire a to-be-processed text under a webpage navigation task; a third determination module configured to determine, according to a second large language model, a target operation process corresponding to the to-be-processed text, wherein the target operation process is used to acquire a webpage address of a seventh query target on a seventh target website involved by the to-be-processed text, and the seventh query target is determined based on the to-be-processed text, wherein the second large language model is obtained based on the training method of the large language model disclosed in the present disclosure; an interaction module configured to interact with the seventh target website on a preset browser based on the target operation process to obtain a target webpage address of the seventh query target; and a display module configured to display a webpage of the target webpage address on the preset browser.

[0008] According to another aspect of the present disclosure, an agent is provided, wherein the agent comprises: an input module configured to acquire a to-be-processed text under a webpage navigation task; a processing module configured to determine, according to a second large language model, a target operation process corresponding to the to-be-processed text, wherein the target operation process is used to acquire a webpage address of a seventh query target on a seventh target website involved by the to-be-processed text, and the seventh query target is determined based on the to-be-processed text; interact with the seventh target website on a preset browser based on the target operation process to obtain a target webpage address of the seventh query target, wherein the second large language model is obtained based on the training method of the large language model disclosed in the present disclosure; and an output module configured to output the target webpage address.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training method of the large language model proposed in the present disclosure, or the information interaction method.

[0010] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to enable a computer to perform the training method of the large language model proposed in the present disclosure, or the information interaction method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the training method of the large language model proposed in the present disclosure, or the steps of the information interaction method proposed in the present disclosure.

[0012] It should be understood that the matters described in this section are not intended to identify key or critical elements of the embodiments of the present disclosure or to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram of an intelligent agent according to an embodiment of the present disclosure;

[0021] Figure 8 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0023] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure, wherein it should be noted that the training method of the large language model of the embodiments of the present disclosure can be applied to a training device of the large language model, which can be an electronic device, or can be configured in an electronic device, so that the electronic device can perform the training function of the large language model.

[0024] The electronic device can be any device with computing capability, such as a personal computer (PC), a mobile terminal, a server, and the like. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and the like, which are hardware devices with various operating systems, touch screens, and / or display screens.

[0025] It should be noted that the training device of the large language model is taken as an example in the following embodiments.

[0026] As shown in Figure 1 The training method of the large language model can include the following steps:

[0027] Step 101, obtaining a first sample text under a web navigation task, a first operation process corresponding to the first sample text, and a first web address obtained based on the first operation process, wherein the first operation process is used to obtain a web address of a first query target on a first target website involved in the first sample text, and the first query target is determined based on the first sample text.

[0028] The first query target is a query target on the first website determined based on the first sample text.

[0029] The first operation process is determined based on the first sample text.

[0030] For example, the first sample text is "find the latest variety show on website 1", the first target website involved in the first sample text is website 1, and the first query target on the first target website can be determined as "the latest variety show" according to the first sample text.

[0031] In some embodiments, in order to reduce the cost of obtaining the first sample text, the first sample text can be obtained in the following manner: obtaining a text template and a prompt word template corresponding to the text template under the web navigation task, the text template including a to-be-filled position, the prompt word template including: a plurality of candidate contents that can be filled into the to-be-filled position; selecting a target content from the plurality of candidate contents, and filling the target content into the to-be-filled position in the text template to obtain the first sample text. Thus, based on the text template, the first sample text can be generated, avoiding the tediousness of manually writing the first sample text, and greatly reducing the cost of obtaining the first sample text.

[0032] The to-be-filled position is pre-set according to actual needs, for example, the text template is "search for the latest variety show on {to-be-filled position}", and the plurality of candidate contents corresponding to the to-be-filled position in this example can include website name 1, website name 2, website name 3, and the like.

[0033] For another example, the text template can be: search the latest {to-be-filled position} on website 1, assuming that website 1 is a website that can provide video resources, and correspondingly, the plurality of candidate contents corresponding to the to-be-filled position can include but are not limited to TV series, movies, variety shows, etc.

[0034] For another example, the text template can be: search {to-be-filled position} on website 2, assuming that website 2 is a website that can provide various commodity purchase services, and correspondingly, the plurality of candidate contents corresponding to the to-be-filled position can include but are not limited to commodity name 1, commodity name 2, commodity name 3, and commodity name 4, etc.

[0035] It can be understood that the to-be-filled position in the text template can be one or more, and the number of to-be-filled positions is not specifically limited in this embodiment.

[0036] For example, the to-be-filled position is two, and the two to-be-filled positions can be represented by to-be-filled position 1 and to-be-filled position 2, and correspondingly, the text template can be: search the latest {to-be-filled position 1} on {to-be-filled position 2}, the plurality of candidate contents corresponding to the to-be-filled position 1 can include: website 1, website 2, website 3, and website 4 for providing video resources, and the plurality of candidate contents corresponding to the to-be-filled position 2 can include but are not limited to TV series, movies, variety shows, etc.

[0037] In some embodiments, in order to accurately and quickly obtain the first sample text, and correspondingly, a possible implementation manner of selecting a target content from the plurality of candidate contents and filling the target content into the to-be-filled position in the text template to obtain the first sample text can be: generating a first prompt according to the text template and the prompt word template, wherein the first prompt is used to instruct the sample generation large model to select a target content from the plurality of candidate contents and fill the target content into the to-be-filled position in the text template to obtain the first sample text; inputting the first prompt into the sample generation large model to obtain the first sample text output by the sample generation large model. Thus, the sample generation large model is prompted by the first prompt, so that the sample generation large model can accurately know the background and target of the generated content, and accurately generate the first sample text, thereby reducing the cost of obtaining the first sample text.

[0038] It can be understood that the number of first sample texts in this embodiment can be multiple.

[0039] It should be noted that the sample generation large model can generate a plurality of different first sample texts. For example, the sample generation large model can generate a plurality of different first sample texts at a time, or the sample generation large model generates a single first sample text sequentially.

[0040] In some embodiments, to reduce the cost of obtaining the first operation process, a second prompt word can be generated according to the first sample text, the second prompt word being used to instruct analyzing the first sample text, determining the thought chain of the first sample text, and determining the corresponding operation process based on the thought chain. Correspondingly, the second prompt word can be input into the target large model, and the first operation process can be determined according to the operation process output by the target large model.

[0041] As an example, the operation process output by the target large model can be directly taken as the first operation process. As another example, the operation process output by the target large model can be manually processed, and the operation process after the manual processing can be taken as the first operation process.

[0042] Here, the target large model refers to an existing general-purpose large model.

[0043] Step 102, determining a second operation process of the first sample text according to the first sample text and the first large language model, the second operation process being used to obtain a web page address of a second query target on a second target website involved in the first sample text, the second query target being determined based on the first sample text.

[0044] In some embodiments, the first sample text can be input into the first large language model, and the first large language model can analyze the first sample text, perform logical reasoning on the process of obtaining the second query target from the second target website involved in the first sample text, obtain a thought chain for obtaining the second query target on the second target website, determine a second operation process for obtaining the second query target on the second target website based on the thought chain, and output the second operation process.

[0045] Step 103, obtaining the second web page address obtained by the sample browser interacting with the first large language model based on the second operation process.

[0046] Here, the sample browser can be any one of the browsers in the electronic device, and the embodiment does not make specific limitation on the sample browser.

[0047] It can be understood that the second operation process includes a plurality of operation steps with a sequence of execution, and the plurality of operation steps can be executed in sequence according to the execution sequence, so that the first large language model interacts with the sample browser based on the executed operation step. Correspondingly, for the first executed operation step, the first large language model determines the operation instruction required by the sample browser to execute according to the first operation step, calls the sample browser to execute the operation instruction, and returns the operation result. For the i-th executed operation step, the first large language model obtains the operation result obtained by interacting with the sample browser based on the i-th operation step, and determines the operation instruction required by the sample browser to execute according to the i-th operation step, calls the sample browser to execute the corresponding operation instruction, and receives the operation result returned by the sample browser based on the operation instruction, wherein i is an integer greater than or equal to 1 and less than N, and N represents the number of operation steps included in the second operation process.

[0048] In the operation result in the embodiment, the web source code of the web page obtained by the sample browser based on the operation instruction can be included.

[0049] Step 104, determining the target reward value obtained by the first large language model interacting with the sample browser according to the first web page address and the second web page address.

[0050] In some embodiments, the first web page address and the second web page address can be matched to obtain a matching result, and the target reward value obtained by the first large language model interacting with the sample browser can be determined according to the matching result.

[0051] The matching result is used to indicate whether the first web page address and the second web page address match.

[0052] It can be understood that if the first web page address and the second web page address are the same, it means that the first web page address and the second web page address match, and if the first web page address and the second web page address are not the same, it means that the first web page address and the second web page address do not match.

[0053] In some embodiments, based on the second operation process determined by the first large language model, there can be multiple, and correspondingly, one possible implementation of determining the target reward value obtained by the first large language model interacting with the sample browser according to the first web page address and the second web page address can be: matching the second web page address obtained by each of the plurality of second operation processes with the first web page address respectively to obtain a matching result, and determining the target reward value obtained by the first large language model interacting with the sample browser according to the matching result.

[0054] In some embodiments, a first quantity of the webpage addresses in the plurality of second webpage addresses that match the first webpage address can be determined according to the matching result, and the target reward value obtained by the first large language model and the sample browser interaction can be determined according to the first quantity.

[0055] In some other embodiments, a first quantity of the webpage addresses in the plurality of second webpage addresses that match the first webpage address and a second quantity of the webpage addresses in the plurality of second webpage addresses that do not match the first webpage address can be determined according to the matching result, and the target reward value obtained by the first large language model and the sample browser interaction can be determined according to the first quantity and the second quantity.

[0056] In some other embodiments, the second webpage address obtained by each of the plurality of second operation processes is matched with the first webpage address respectively, and another possible implementation of determining the target reward value obtained by the first large language model and the sample browser interaction according to the matching result is as follows: for each second operation process, the second webpage address obtained based on the second operation process is matched with the first webpage address, and the target score corresponding to the first operation process is determined according to the matching result of the first webpage address and the second webpage address, the average score and the score standard deviation are determined according to the target scores corresponding to the plurality of first operation processes, then the target scores corresponding to the plurality of first operation processes are standardized to obtain the standardized scores corresponding to the plurality of first operation processes, and the standardized scores corresponding to the plurality of first operation processes are averaged to obtain the target reward value obtained by the first large language model and the sample browser interaction.

[0057] In the present embodiment, the target reward value obtained by the first large language model and the sample browser interaction is determined in combination with the matching result of the second webpage address obtained by each of the plurality of second operation processes and the first webpage address, and the first large language model is reinforced in combination with the determined target reward value, which can improve the generalization and stability of the obtained second large language model, while avoiding local optimization and overfitting, so that the second large language model can generate higher quality and more diverse outputs.

[0058] In step 105, the first large language model is reinforced for learning training according to the target reward value to obtain a second large language model.

[0059] In some embodiments, the parameters of the first large language model are optimized and adjusted based on the target reward value, and the adjusted model is iteratively trained until a preset ending condition is met, so as to obtain a second large language model with optimized performance.

[0060] The preset end condition refers to a preset condition for terminating the first large language model training process. For example, the preset end condition can be that the number of training times reaches a preset number of training times, or the target reward value is greater than a preset value, or the change of the target reward value approaches stability, that is, the difference between the target reward values corresponding to adjacent two or more times of training is less than a set value, that is, the target reward value basically no longer changes.

[0061] The training method of the large language model provided in the embodiments of the present disclosure can determine the second operation process of the first sample text through the first large language model, obtain the second web page address obtained by the first large language model based on the second operation process and the sample browser interacting with each other, match the first web page address with the first web page address obtained based on the first operation process of the first sample text, and determine the target reward value obtained by the first large language model and the sample browser interacting with each other according to the matching result. The first large language model is trained through reinforcement learning according to the target reward value, so that the obtained second large language model can analyze the text under the web navigation task, accurately determine the operation process for obtaining the query target in the target website related to the corresponding text, and facilitate subsequent accurate acquisition of the web page address of the query target meeting the user demand based on the operation process.

[0062] In some embodiments, in order to improve the efficiency of obtaining the second large language model, the first large language model in the present embodiment can be a trained large language model.

[0063] It can be understood that the first large language model is a trained large language model, so that the first large language model can provide a good initial state, so that the subsequent reinforcement learning stage can find an effective strategy more quickly, which helps to improve the efficiency of obtaining the second large language model.

[0064] It can be understood that the way of obtaining the first large language model is different in different application scenarios.

[0065] As an example, one possible implementation way of obtaining the first large language model can be: the first sample text can be input into an initial large language model to obtain a predicted operation process of the first sample text output by the initial large language model, and the initial large language model is supervised fine-tuned according to the predicted operation process and the first operation process to obtain the first large language model. The predicted operation process is used to obtain the web page address of the query target on the website related to the first sample text, and the query target is obtained by the initial large language model performing intent analysis on the first sample text.

[0066] The following will be described in combination with Figure 2 Another possible implementation way of obtaining the first large language model is exemplarily described.

[0067] Figure 2 is a schematic diagram according to a second embodiment of the disclosure.

[0068] As Figure 2 illustrated, the method can include:

[0069] Step 201, acquiring a first sample text under a web page navigation task, a first operation process corresponding to the first sample text, and a first web page address obtained based on the first operation process, wherein the first operation process is used to acquire a web page address of a first query target on a first target website involved in the first sample text, and the first query target is determined based on the first sample text.

[0070] It should be noted that the specific description of step 201 can refer to the related description in other embodiments, which will not be repeated here.

[0071] Step 202, inputting the first sample text into the fine-tuned large language model, and sampling the output of the fine-tuned large language model to obtain a first preference data pair of the first sample text, wherein the first preference data pair includes: a third operation process and a fourth operation process.

[0072] Among them, the third operation process is the operation process with the highest Upper Confidence Bound (UCB) score in the operation processes output by the fine-tuned large language model, and the fourth operation process is the operation process with the lowest UCB score in the operation processes output by the fine-tuned large language model.

[0073] Among them, the third operation process and the fourth operation process are both used to acquire a web page address of a third query target on a third target website involved in the first sample text, and the third query target is determined according to the first sample text.

[0074] Among them, it can be understood that the third query target is determined by the fine-tuned large language model according to the first sample text.

[0075] In some embodiments, the UCB score of the third operation process is determined according to the matching result of the third web page address obtained by the third operation process and the first web page address and the probability of the fine-tuned large language model outputting the third operation process; correspondingly, the UCB score of the fourth operation process is determined according to the matching result of the fourth web page address obtained by the fourth operation process and the first web page address and the probability of the fine-tuned large language model outputting the fourth operation process. Therefore, the matching result of the web page address obtained by the corresponding operation process and the first web page address and the probability of the first preference optimization large language model outputting the corresponding operation process are combined to accurately determine the UCB score of the corresponding operation process.

[0076] In some embodiments, when it is known that the third web page address and the first web page address match based on the matching result of the third web page address and the first web page address, the corresponding score can be the first score. Correspondingly, when it is known that the third web page address and the first web page address do not match based on the matching result of the third web page address and the first web page address, the corresponding score can be the second score.

[0077] The first score and the second score may be preset according to actual needs. For example, the first score may be 0, and the second score may be 1 correspondingly.

[0078] In some embodiments, when it is known that the fourth web page address and the first web page address match based on the matching result of the fourth web page address and the first web page address, the corresponding score can be the first score. Correspondingly, when it is known that the fourth web page address and the first web page address do not match based on the matching result of the fourth web page address and the first web page address, the corresponding score can be the second score.

[0079] The first score and the second score may be preset according to actual needs. For example, the first score may be 0, and the second score may be 1 correspondingly.

[0080] In other embodiments, the UCB score of the third operation process can be determined based on the probability of the fine-tuned large language model outputting the third operation process, and correspondingly, the UCB score of the fourth operation process can be determined based on the probability of the fine-tuned large language model outputting the fourth operation process.

[0081] In some embodiments, in the process of sampling multiple operation processes output by the fine-tuned large language model, the probability of the fine-tuned large language model outputting each of the above-mentioned operation processes can be determined based on the Monte-Carlo Tree Search (MCTS) algorithm, and based on the probability of each operation process and the UCB algorithm, the UCB score corresponding to each operation process is determined, and the operation process with the highest UCB score among the multiple operation processes is used as the third operation process, and the operation process with the lowest UVB score among the multiple operation processes is used as the fourth operation process.

[0082] In some embodiments, the fine-tuned large language model may be obtained by performing supervised fine-tuning on the initial large language model based on the first sample text and the corresponding first operation flow.

[0083] In some embodiments, a possible implementation of the supervised fine-tuning of the initial large language model based on the first sample text and the corresponding first operation process can be: inputting the first sample text into the initial large language model to obtain an eighth operation process, where the eighth operation process is used to obtain a web page address of a sixth query target on a sixth target website involved in the first sample text, and the sixth query target is determined according to the first sample text; and performing supervised fine-tuning training on the initial large language model according to the eighth operation process and the first operation process to obtain the fine-tuned large language model. Thus, the fine-tuned large language model is obtained based on the first sample text and the first operation process, so that the fine-tuned large language model can provide an initial state of high quality, facilitating subsequent improvement of preference learning efficiency and helping to improve the efficiency of obtaining the first large language model.

[0084] It can be understood that the sixth query target is determined by the initial large language model based on the first sample text.

[0085] In some embodiments, in order to improve the logical reasoning ability of the obtained fine-tuned large language model, and then improve the logical reasoning ability of the second large language model obtained based on the fine-tuned large language model, correspondingly, the original thinking chain based on which the first operation process corresponding to the first sample text is based on can be improved based on an advanced large model to obtain an improved thinking chain, and the improved first operation process can be obtained based on the improved thinking chain, where it should be noted that the logical reasoning ability of the advanced large model is higher than that of a target large model, and the target large model is a large model used to determine the original thinking chain based on which the first operation process is based on.

[0086] Correspondingly, the initial large language model can be supervised fine-tuned based on the first sample text and the improved first operation process to obtain the fine-tuned large language model. Thus, the logical reasoning ability of the obtained fine-tuned large language model can be improved.

[0087] Step 203: performing at least one round of direct preference optimization training on the fine-tuned large language model based on the first sample text and the first preference data pair to obtain the first large language model.

[0088] In some embodiments, to further improve the stability of the obtained first large language model, a second sample text under a web navigation task can also be obtained correspondingly, where the difficulty level of the second sample text is greater than that of the first sample text. Then, at least one round of direct preference optimization training is performed on the fine-tuned large language model according to the first sample text and the first preference data pair, to obtain a first preference optimized large language model. The second sample text is input into the first preference optimized large language model, and the output of the first preference optimized large language model is sampled to obtain a second preference data pair of the second sample text. The second preference data pair includes a fifth operation process and a sixth operation process. The fifth operation process is the operation process with the highest upper confidence bound (UCB) score in the operation processes output by the first preference optimized large language model, and the sixth operation process is the operation process with the lowest upper confidence bound (UCB) score in the operation processes output by the first preference optimized large language model. Both the fifth operation process and the sixth operation process are used to obtain the web page address of a fourth query target on a fourth target website related to the second sample text, and the fourth query target is determined according to the second sample text. Finally, at least one round of direct preference optimization is performed on the first preference optimized large language model according to the second sample text and the second preference data pair, to obtain the first large language model. Thus, after at least one round of direct preference optimization is performed on the fine-tuned large language model, and the second sample text with a difficulty level greater than that of the first sample text is used, at least one round of direct preference optimization is continued on the preference optimized large language model, which helps to improve the stability of the obtained first large language model.

[0089] In some embodiments, the difficulty level of the second sample text is determined based on the number of operation steps included in the seventh operation process corresponding to the second sample text, and correspondingly, the difficulty level of the first sample text is determined based on the number of operation steps included in the first operation process.

[0090] In some other embodiments, when both the first sample text and the second sample text are obtained based on a text template, the difficulty level of the first sample text can be determined based on the number of to-be-filled positions in the text template relied on for generating the first sample text. Correspondingly, the difficulty level of the second sample text can be determined based on the number of to-be-filled positions in the text template relied on for generating the second sample text.

[0091] In some other embodiments, the difficulty level of the first sample text can be determined based on the number of operation steps included in the first operation process and the number of to-be-filled positions in the text template relied on for generating the first sample text. Correspondingly, the difficulty level of the second sample text can be determined based on the number of operation steps included in the seventh operation process and the number of to-be-filled positions in the text template relied on for generating the second sample text.

[0092] In some embodiments, a fifth webpage address obtained based on the seventh operation process corresponding to the second sample text can also be obtained, wherein the seventh operation process is used to obtain the webpage address of the fifth query target on the fifth target website involved in the second sample text, and the fifth query target is determined based on the second sample text. Correspondingly, the UCB score of the fifth operation process is determined based on the matching result of the sixth webpage address obtained by the fifth operation process with the fifth webpage address and the probability that the first preference optimized large language model outputs the fifth operation process; correspondingly, the UCB score of the sixth operation process is determined based on the matching result of the seventh webpage address obtained by the sixth operation process with the fifth webpage address and the probability that the first preference optimized large language model outputs the sixth operation process. Thus, the UCB score of the corresponding operation process is accurately determined by combining the matching result of the webpage address obtained by the corresponding operation process with the fifth webpage address and the probability that the first preference optimized large language model outputs the corresponding operation process.

[0093] In some embodiments, when it is known that the sixth web page address and the fifth web page address match based on the matching result of the sixth web page address and the fifth web page address, the corresponding score can be a first score. Correspondingly, when it is known that the sixth web page address and the fifth web page address do not match based on the matching result of the sixth web page address and the fifth web page address, the corresponding score can be a second score.

[0094] The first score and the second score may be preset according to actual needs. For example, the first score may be 0, and the second score may be 1 correspondingly.

[0095] In some embodiments, when it is known that the seventh web page address and the fifth web page address match based on the matching result of the seventh web page address and the fifth web page address, the corresponding score can be a first score. Correspondingly, when it is known that the seventh web page address and the fifth web page address do not match based on the matching result of the seventh web page address and the fifth web page address, the corresponding score can be a second score.

[0096] The first score and the second score may be preset according to actual needs. For example, the first score may be 0, and the second score may be 1 correspondingly.

[0097] Step 204: Determine a second operation process for the first sample text based on the first sample text and the first language model. The second operation process is used to obtain the webpage address of a second query target on a second target website involved in the first sample text. The second query target is determined based on the first sample text.

[0098] Step 205 : Obtain a second webpage address obtained by the first language model interacting with the sample browser based on the second operation process.

[0099] In step 206, the target reward value of the first large language model and the sample browser interaction is determined according to the first web page address and the second web page address.

[0100] In step 207, the first large language model is trained by reinforcement learning according to the target reward value, and a second large language model is obtained.

[0101] It should be noted that the specific description of steps 204 to 207 can refer to the related description in other embodiments, which will not be described here.

[0102] In this embodiment, the first preference data pair is automatically constructed by combining the first sample text and the fine-tuned large language model, and the fine-tuned large language model is directly optimized and trained according to the constructed first preference data and the first sample text, and the first large language model is obtained, so that the first large language model can provide a more stable and reasonable initial state, which helps the first large language model to find an effective strategy faster in the reinforcement learning stage, and helps to improve the efficiency of obtaining the second large language model.

[0103] In order to clearly understand the present disclosure, the following will be described in combination with Figure 3 The training method of the large model of this embodiment is exemplarily described.

[0104] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure.

[0105] As Figure 3 indicated, it can include:

[0106] In step 301, a first sample text under a web navigation task, a first operation process corresponding to the first sample text, and a first web page address obtained based on the first operation process are obtained, wherein the first operation process is used to obtain a web page address of a first query target on a first target website involved in the first sample text, and the first query target is determined based on the first sample text.

[0107] In some embodiments, a text template under the web navigation task and a prompt word template corresponding to the text template can be obtained, the text template includes a to-be-filled position, and the prompt word template includes a plurality of candidate contents that can be filled into the to-be-filled position. The first prompt word is generated according to the text template and the prompt word template, wherein the first prompt word is used to instruct the sample generation large model to select a target content from the plurality of candidate contents, and fill the target content into the to-be-filled position in the text template to obtain the first sample text. The first prompt word is input into the sample generation large model, and the first sample text output by the sample generation large model is obtained.

[0108] In some embodiments, a second prompt word is generated according to the first sample text, the second prompt word being used to instruct to analyze the first sample text, determine a thinking chain of the first sample text, and determine a corresponding operation flow based on the thinking chain. Correspondingly, the second prompt word can be input into the target large model, and the first operation flow can be determined according to the operation flow output by the target large model.

[0109] As an example, the operation flow output by the target large model can be directly taken as the first operation flow. As another example, the operation flow output by the target large model can be manually processed, and the operation flow after the manual processing can be taken as the first operation flow.

[0110] Here, the target large model refers to an existing general-purpose large model.

[0111] As an example, the text template can be obtained by analyzing historical input texts under a web navigation task, or can be obtained by other manners, and the embodiment does not limit the manner of obtaining the text template.

[0112] In some other embodiments, the first sample text can also be expanded by a general-purpose large language model.

[0113] In some other embodiments, the target large model can also be used to determine a ninth operation flow of the expanded first sample text, and obtain a first target web address obtained based on the ninth operation flow, where the ninth operation flow is used to obtain a web address of an eighth query target on an eighth target website involved in the expanded first sample text, and the eighth query target is obtained according to the expanded first sample text.

[0114] Step 302: Supervised fine-tuning training is performed on the initial large language model according to the first sample text and the first operation flow, to obtain a fine-tuned large language model.

[0115] For specific description of step 302, reference can be made to the related description in other embodiments, which will not be repeated here.

[0116] Step 303: The first sample text is input into the fine-tuned large language model, and the output of the fine-tuned large language model is sampled to obtain a first preference data pair of the first sample text, where the first preference data pair includes a third operation flow and a fourth operation flow.

[0117] The third operation process is an operation process with the highest upper confidence bound (UCB) score in the operation processes output by the fine-tuned large language model, and the fourth operation process is an operation process with the lowest upper confidence bound (UCB) score in the operation processes output by the fine-tuned large language model. The third operation process and the fourth operation process are both used to obtain a web page address of a third query target on a third target website related to the first sample text, and the third query target is determined according to the first sample text.

[0118] It can be understood that the third query target is determined by the fine-tuned large language model according to the first sample text.

[0119] The UCB score of the third operation process is determined according to a matching result of the third web page address obtained by the third operation process and the first web page address and a probability of the fine-tuned large language model outputting the third operation process. Correspondingly, the UCB score of the fourth operation process is determined according to a matching result of the fourth web page address obtained by the fourth operation process and the first web page address and a probability of the fine-tuned large language model outputting the fourth operation process.

[0120] In step 304, a fine-tuned large language model is trained according to the first sample text and the first preference data pair, to obtain a first preference-optimized large language model.

[0121] In step 305, a second sample text under a web page navigation task is obtained, wherein the difficulty level of the second sample text is higher than that of the first sample text.

[0122] The difficulty levels of the first sample text and the second sample text can be obtained in the manner described in other embodiments, which will not be described here again.

[0123] In step 306, the second sample text is input into the first preference-optimized large language model, and the output of the first preference-optimized large language model is sampled to obtain a second preference data pair of the second sample text, the second preference data pair including a fifth operation process and a sixth operation process.

[0124] The fifth operation process is an operation process with the highest upper confidence bound (UCB) score in the operation processes output by the first preference-optimized large language model.

[0125] The sixth operation process is an operation process with the lowest upper confidence bound (UCB) score in the operation processes output by the first preference-optimized large language model.

[0126] The fifth operation process and the sixth operation process are both used to obtain a web page address of a fourth query target on a fourth target website related to the second sample text.

[0127] The fourth query target is determined according to the second sample text.

[0128] The fourth query target is determined according to the second sample text.

[0129] The UCB score of the fifth operation process is determined according to a matching result of the sixth web page address obtained by the fifth operation process and the fifth web page address and a probability of the first preference optimization large language model outputting the fifth operation process.

[0130] In some embodiments, the fifth web page address obtained by a seventh operation process based on the second sample text can also be obtained, where the seventh operation process is used to obtain a web page address of a fifth query target on a fifth target website involved in the second sample text, and the fifth query target is determined according to the second sample text.

[0131] The UCB score of the fifth operation process is determined according to a matching result of the sixth web page address obtained by the fifth operation process and the fifth web page address and a probability of the first preference optimization large language model outputting the fifth operation process.

[0132] Step 307: At least one round of direct preference optimization is performed on the first preference optimization large language model according to the second sample text and the second preference data pair, to obtain a first large language model.

[0133] Step 308: A second operation process for obtaining a web page address of a second query target on a second target website involved in the first sample text is determined according to the first sample text and the first large language model, where the second query target is determined based on the first sample text.

[0134] Step 309: A second web page address obtained by the first large language model interacting with the sample browser based on the second operation process is obtained.

[0135] Step 310: A target reward value obtained by the first large language model interacting with the sample browser is determined according to the first web page address and the second web page address.

[0136] Step 311: Reinforcement learning training is performed on the first large language model according to the target reward value, to obtain a second large language model.

[0137] In the embodiment, the initial large language model is supervised fine-tuning trained by the first sample text and the corresponding first operation process to obtain a fine-tuned large language model, and a first preference data pair is automatically constructed by the fine-tuned large language model and the first sample text. The fine-tuned large language model is trained based on the first sample text and the first preference data pair for one round of direct preference optimization to obtain a first preference optimized large language model. A second sample text with a difficulty level greater than the first sample text is obtained, and the second sample text is input into the first preference optimized large language model to construct a second preference data pair. The first preference optimized large language model is trained based on the second sample text and the second preference data pair to obtain a first large language model. Thus, the stability of the trained first large language model can be improved. The second operation process of the first sample text is determined by the first sample text and the first large language model. The target reward value obtained by the interaction between the first large language model and the sample browser is determined based on the first web page address and the second web page address. The first large language model is trained based on the target reward value to obtain a second large language model. Thus, the target reward value determined based on the first web page address and the second web page address is used to efficiently implement reinforcement learning of the first large language model, so that the generalization ability of the obtained second large language model is better.

[0138] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. It should be noted that the information interaction method of the embodiments of the present disclosure can be applied to an agent. The information interaction agent refers to a computer program based on a large language model, which has planning ability, memory ability, and tool function using ability, and can autonomously complete a given task. The agent can be configured in an electronic device to enable the electronic device to perform an information interaction function.

[0139] As shown in Figure 4 , the method can include:

[0140] Step 401, obtaining a to-be-processed text under a web navigation task.

[0141] Step 402, determining a target operation process corresponding to the to-be-processed text according to the second large language model, wherein the target operation process is used to obtain a web page address of a seventh query target on a seventh target website involved in the to-be-processed text, and the seventh query target is determined based on the to-be-processed text.

[0142] The second large language model is trained according to the training method of the large language model disclosed in the present application.

[0143] Step 403, interacting with the seventh target website on a preset browser based on the target operation process to obtain a target web page address of the seventh query target.

[0144] It should be noted that the preset browser is any one of the browsers in the electronic device that can communicate with the intelligent agent.

[0145] In some embodiments, the target operation process includes a plurality of operation steps with an execution sequence, and the plurality of operation steps can be executed in sequence according to the execution sequence, so that the first large language model interacts with the sample browser based on the executed operation step. Correspondingly, for the first executed operation step, the first large language model determines the operation instruction required by the sample browser to execute according to the first operation step, calls the sample browser to execute the operation instruction, and returns the operation result. For the ith executed operation step, the first large language model obtains the operation result obtained by interacting with the sample browser based on the ith operation step, determines the operation instruction required by the sample browser to execute according to the ith operation step, calls the sample browser to execute the corresponding operation instruction, and receives the operation result returned by the sample browser based on the operation instruction, where i is an integer greater than or equal to 1 and less than N, and N represents the number of operation steps included in the target operation process.

[0146] For example, the text to be processed is: "Find the latest variety show on website 1", and the obtained target operation process includes 3 operation steps with an execution sequence, the first operation step is: open the target website 1; the second operation step is: find the label or page of the latest variety show; and the third operation step is: analyze the page content to obtain the latest variety show information. Correspondingly, the first operation instruction of opening website 1 can be sent to the preset browser based on the first operation step, where the first operation instruction can include the web address of website 1. Correspondingly, the preset browser opens the web address information of website 1, and returns the web source code of website 1 to the intelligent agent. Correspondingly, the intelligent agent determines the second operation instruction required by the preset browser to execute according to the web source code of website 1 and the second operation step. Correspondingly, the preset browser executes the second operation instruction to obtain the operation result. Assuming that the second operation instruction is: input "latest variety show" in the search box of website 1 and click the search button on website 1, the operation result returned by the preset browser can be the web source code of the search result page. The intelligent agent determines the third operation instruction required by the preset browser to execute according to the operation result and the third operation step, which can be clicking the first search result in the search result page. Correspondingly, the preset browser is called to click the first search result in the search result page. Correspondingly, the web address information corresponding to the first search result is provided to the intelligent agent. The web address information corresponding to the first search result is the web address of the query target on the third target website. Correspondingly, the intelligent agent can also display the web address information.

[0147] Step 404, displaying the webpage of the target webpage address on the preset browser.

[0148] The information interaction method provided by the embodiments of the present disclosure obtains the to-be-processed text under the webpage navigation task, and determines the target operation process corresponding to the to-be-processed text according to the second large language model. Based on the target operation process, the seventh target website is interacted on the preset browser to obtain the target webpage address of the seventh query target, and the webpage of the target webpage address is displayed on the preset browser. In this way, based on the to-be-processed text input by the user under the webpage navigation task, the webpage of the query target involved in the to-be-processed text can be automatically displayed on the preset browser.

[0149] In order to realize the above-mentioned embodiments, the present disclosure further provides a large language model training device.

[0150] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure.

[0151] As Figure 5 shown, the large language model training device 50 can include a first acquisition module 501, a first determination module 502, a second acquisition module 503, a second determination module 504, and a training module 505.

[0152] The first acquisition module 501 is configured to acquire a first sample text under a webpage navigation task, a first operation process corresponding to the first sample text, and a first webpage address obtained based on the first operation process. The first operation process is used to acquire a webpage address of a first query target on a first target website involved in the first sample text, and the first query target is determined based on the first sample text.

[0153] The first determination module 502 is configured to determine a second operation process of the first sample text according to the first sample text and a first large language model. The second operation process is used to acquire a webpage address of a second query target on a second target website involved in the first sample text, and the second query target is determined based on the first sample text.

[0154] The second acquisition module 503 is configured to acquire a second webpage address obtained by the first large language model interacting with a sample browser based on the second operation process.

[0155] The second determination module 504 is configured to determine a target reward value obtained by the first large language model interacting with the sample browser according to the first webpage address and the second webpage address.

[0156] The training module 505 is configured to perform reinforcement learning training on the first large language model according to the target reward value to obtain a second large language model.

[0157] As a possible implementation manner of the embodiment of the present disclosure, the sample text acquisition module is specifically configured to: acquire a text template and a prompt word template corresponding to the text template under a web navigation task, the text template comprising a to-be-filled position, and the prompt word template comprising: a plurality of candidate contents that can be filled into the to-be-filled position; select a target content from the plurality of candidate contents, and fill the target content into the to-be-filled position in the text template to obtain a first sample text.

[0158] The implementation manner of selecting a target content from the plurality of candidate contents and filling the target content into the to-be-filled position in the text template to obtain the first sample text is: generating a first prompt word according to the text template and the prompt word template, wherein the first prompt word is used to instruct the sample generation large model to select a target content from the plurality of candidate contents and fill the target content into the to-be-filled position in the text template to obtain the first sample text; inputting the first prompt word into the sample generation large model to acquire the first sample text output by the sample generation large model.

[0159] As a possible implementation manner of the embodiment of the present disclosure, the second operation process is multiple, and the second determination module 504 is specifically configured to: match the second web addresses obtained by the multiple second operation processes respectively with the first web address to obtain matching results; and determine a target reward value obtained by the first large language model and the sample browser interaction according to the matching results.

[0160] As a possible implementation manner of the embodiment of the present disclosure, the device further comprises: a first processing module configured to: input the first sample text into the fine-tuned large language model, and sample the output of the fine-tuned large language model to obtain a first preference data pair of the first sample text, wherein the first preference data pair comprises: a third operation process and a fourth operation process, the third operation process is an operation process with the highest upper confidence bound (UCB) score in the operation processes output by the fine-tuned large language model, the fourth operation process is an operation process with the lowest upper confidence bound (UCB) score in the operation processes output by the fine-tuned large language model, the third operation process and the fourth operation process are both used to acquire a web address of a third query target on a third target website involved in the first sample text, and the third query target is determined according to the first sample text;

[0161] According to the first sample text and the first preference data pair, the fine-tuned large language model is directly preference-optimized and trained to obtain the first large language model.

[0162] As a possible implementation manner of the embodiment of the present disclosure, the UCB score of the third operation process is determined according to a matching result of the third web page address obtained by the third operation process and the first web page address and a probability of the fine-tuned large language model outputting the third operation process; and the UCB score of the fourth operation process is determined according to a matching result of the fourth web page address obtained by the fourth operation process and the first web page address and a probability of the fine-tuned large language model outputting the fourth operation process.

[0163] As a possible implementation manner of the embodiment of the present disclosure, the device further comprises a second processing module configured to obtain a second sample text under a web navigation task; and the difficulty level of the second sample text is higher than that of the first sample text.

[0164] The first processing module performs direct preference optimization training on the fine-tuned large language model according to the first sample text and the first preference data pair, and the specific manner of obtaining the first large language model is as follows: performing at least one round of direct preference optimization training on the fine-tuned large language model according to the first sample text and the first preference data pair to obtain a first preference-optimized large language model; inputting the second sample text into the first preference-optimized large language model, and sampling the output of the first preference-optimized large language model to obtain a second preference data pair of the second sample text, the second preference data pair comprising: a fifth operation process and a sixth operation process, the fifth operation process being an operation process with the highest upper confidence bound UCB score in the operation processes output by the first preference-optimized large language model, and the sixth operation process being an operation process with the lowest upper confidence bound UCB score in the operation processes output by the first preference-optimized large language model, the fifth operation process and the sixth operation process both being used to obtain a web page address of a fourth query target on a fourth target website involved in the second sample text, the fourth query target being determined according to the second sample text; and performing at least one round of direct preference optimization on the first preference-optimized large language model according to the second sample text and the second preference data pair to obtain the first large language model.

[0165] As a possible implementation manner of the embodiment of the present disclosure, the device further comprises an address obtaining module configured to obtain a fifth web page address obtained based on a seventh operation process corresponding to the second sample text, the seventh operation process being used to obtain a web page address of a fifth query target on a fifth target website involved in the second sample text, the fifth query target being determined according to the second sample text.

[0166] The UCB score of the fifth operation process is determined according to a matching result of a sixth web page address obtained by the fifth operation process and the fifth web page address and a probability of the first preference optimization large language model outputting the fifth operation process; and the UCB score of the sixth operation process is determined according to a matching result of a seventh web page address obtained by the sixth operation process and the fifth web page address and a probability of the first preference optimization large language model outputting the sixth operation process.

[0167] As a possible implementation manner of the embodiment of the present disclosure, the apparatus can further include a third processing module configured to input the first sample text into the initial large language model to obtain an eighth operation process, wherein the eighth operation process is used to obtain a web page address of a sixth query target on a sixth target website involved by the first sample text, and the sixth query target is determined according to the first sample text; and the initial large language model is supervised fine-tuning trained according to the eighth operation process and the first operation process to obtain the fine-tuning large language model.

[0168] It should be noted that the above explanation of the training method of the large language model embodiment is also applicable to the training apparatus of the large language model of this embodiment, which will not be described here.

[0169] The training apparatus of the large language model of the embodiment of the present disclosure determines the second operation process of the first sample text through the first large language model, obtains the second web page address obtained by the first large language model based on the second operation process and the sample browser, matches the first web page address with the first web page address obtained based on the first operation process of the first sample text, determines the target reward value obtained by the first large language model interacting with the sample browser according to the matching result, and performs reinforcement learning training on the first large language model according to the target reward value, so that the obtained second large language model can analyze the text under the web navigation task and accurately determine the operation process for obtaining the query target in the target website involved by the corresponding text, which facilitates subsequent accurate acquisition of the web page address of the query target meeting the user demand based on the operation process.

[0170] In order to realize the above-mentioned embodiments, the present disclosure further provides an information interaction apparatus.

[0171] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure.

[0172] As Figure 6 shown, the information interaction apparatus 60 can include a third acquisition module 601, a third determination module 602, an interaction module 603 and a display module 604, wherein:

[0173] The third acquisition module 601 is configured to acquire the text to be processed under the web navigation task.

[0174] The third determination module 602 is configured to determine, according to the second large language model, a target operation process corresponding to the to-be-processed text, wherein the target operation process is used to obtain a web page address of a seventh query target on a seventh target website involved by the to-be-processed text, and the seventh query target is determined based on the to-be-processed text, and the second large language model is obtained based on the training method of the large language model disclosed in the present application.

[0175] The interaction module 603 is configured to interact with the seventh target website on the preset browser based on the target operation process to obtain the target web page address of the seventh query target.

[0176] The display module 604 is configured to display the web page of the target web page address on the preset browser.

[0177] It should be noted that the above description of the information interaction method is also applicable to the information interaction device, which will not be described here.

[0178] The information interaction device provided by the embodiment of the present disclosure obtains the to-be-processed text under the web navigation task, and determines, according to the second large language model, a target operation process corresponding to the to-be-processed text. The target operation process is used to obtain a target web page address of a seventh query target on a seventh target website involved by the to-be-processed text. The information interaction device interacts with the seventh target website on the preset browser based on the target operation process to obtain the target web page address of the seventh query target, and displays the web page of the target web page address on the preset browser. Thus, based on the to-be-processed text input by the user under the web navigation task, the web page of the query target involved by the to-be-processed text can be automatically displayed on the preset browser.

[0179] Figure 7 FIG. 7 is a schematic diagram of an agent according to an embodiment of the present disclosure.

[0180] As shown in FIG. 7, the agent 700 can include an input module 701, a processing module 702, and an output module 703. Figure 7

[0181] The input module 701 is configured to obtain a to-be-processed text under a web navigation task.

[0182] The processing module 702 is configured to determine, according to a second large language model, a target operation process corresponding to the to-be-processed text, wherein the target operation process is used to obtain a target web page address of a seventh query target on a seventh target website involved by the to-be-processed text, and the seventh query target is determined based on the to-be-processed text. The target operation process is used to interact with the seventh target website on a preset browser based on the target operation process to obtain the target web page address of the seventh query target, and the second large language model is obtained based on the training method of the large language model disclosed in the present application. ​

[0183] The output module 703 is configured to output the target webpage address.

[0184] The agent provided by the embodiment of the present application obtains the to-be-processed text under the webpage navigation task, and determines the target operation process corresponding to the to-be-processed text according to the second large language model. Based on the target operation process, the agent interacts with the seventh target website on the preset browser, obtains the target webpage address of the seventh query target, and displays the webpage of the target webpage address on the preset browser. In this way, based on the to-be-processed text input by the user under the webpage navigation task, the webpage of the query target related to the to-be-processed text can be automatically displayed on the preset browser.

[0185] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out on the premise of obtaining the consent of the user, and all comply with the relevant legal regulations and do not violate public order and good customs.

[0186] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0187] Figure 8 is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0188] As shown in Figure 8 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0189] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0190] The computing unit 801 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the training method of a large language model, or the information interaction method. For example, in some embodiments, the training method of a large language model, or the information interaction method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method of a large language model or the information interaction method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the training method of a large language model, or the information interaction method by any other appropriate means, such as by means of firmware.

[0191] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0192] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0193] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0194] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0195] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0196] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between a client and a server is one of client-server. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0197] It should be understood that the steps shown in the various forms above can be reordered, added to, or deleted from. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved, and the present disclosure is not limited herein.

[0198] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for training a large language model, comprising: Obtaining a first sample text for a webpage navigation task, a first operation process corresponding to the first sample text, and a first webpage address obtained based on the first operation process, wherein the first operation process is used to obtain a webpage address of a first query target on a first target website related to the first sample text, wherein the first query target is determined based on the first sample text; determining, based on the first sample text and the first language model, a second operation process for the first sample text, wherein the second operation process is used to obtain a webpage address of a second query target on a second target website involved in the first sample text, the second query target being determined based on the first sample text; Obtaining a second webpage address obtained by the first language model interacting with the sample browser based on the second operation process; determining a target reward value obtained by interacting the first language model with the sample browser according to the first webpage address and the second webpage address; According to the target reward value, reinforcement learning training is performed on the first language model to obtain a second language model.

2. The method according to claim 1, wherein The first sample text is obtained as follows: Acquire a text template under the webpage navigation task and a prompt word template corresponding to the text template, wherein the text template includes a position to be filled, and the prompt word template includes: a plurality of candidate contents that can be filled into the position to be filled; A target content is selected from the multiple candidate contents, and the target content is filled into the to-be-filled position in the text template to obtain the first sample text.

3. The method according to claim 2, wherein: The step of selecting a target content from the plurality of candidate contents and filling the target content into the to-be-filled position in the text template to obtain the first sample text includes: generating a first prompt word according to the text template and the prompt word template, wherein the first prompt word is used to instruct the sample generation model to select a target content from the multiple candidate contents and fill the target content into the to-be-filled position in the text template to obtain the first sample text; The first prompt word is input into the sample generation model to obtain the first sample text output by the sample generation model.

4. The method according to claim 1, wherein The second operation process includes multiple steps, wherein determining a target reward value obtained by the interaction between the first language model and the sample browser based on the first webpage address and the second webpage address includes: Matching the second webpage addresses obtained by each of the plurality of second operation processes with the first webpage address to obtain matching results; A target reward value obtained by the interaction between the first language model and the sample browser is determined according to the matching result.

5. The method according to claim 1, wherein The method further comprises: Inputting the first sample text into a fine-tuned large language model and sampling the output of the fine-tuned large language model to obtain a first preference data pair for the first sample text, wherein the first preference data pair includes: a third operation process and a fourth operation process, wherein the third operation process is an operation process with the highest upper confidence bound (UCB) score among the operation processes output by the fine-tuned large language model, and the fourth operation process is an operation process with the lowest upper confidence bound (UCB) score among the operation processes output by the fine-tuned large language model, and both the third operation process and the fourth operation process are used to obtain a webpage address of a third query target on a third target website involved in the first sample text, wherein the third query target is determined based on the first sample text; According to the first sample text and the first preference data pair, direct preference optimization training is performed on the fine-tuned large language model to obtain the first large language model.

6. The method according to claim 5, wherein: The UCB score of the third operation process is determined based on the matching result between the third web page address and the first web page address obtained by the third operation process and the probability that the fine-tuned large language model outputs the third operation process; the UCB score of the fourth operation process is determined based on the matching result between the fourth web page address and the first web page address obtained by the fourth operation process and the probability that the fine-tuned large language model outputs the fourth operation process.

7. The method according to claim 5, wherein: The method further comprises: Obtaining a second sample text for a webpage navigation task; wherein the difficulty level of the second sample text is greater than the difficulty level of the first sample text; The step of performing direct preference optimization training on the fine-tuned large language model based on the first sample text and the first preference data pair to obtain the first large language model includes: performing at least one round of direct preference optimization training on the fine-tuned large language model according to the first sample text and the first preference data pair to obtain a first preference-optimized large language model; Inputting the second sample text into the first preference-optimized large language model, and sampling the output of the first preference-optimized large language model to obtain a second preference data pair for the second sample text, the second preference data pair comprising: a fifth operation flow and a sixth operation flow, the fifth operation flow being the operation flow with the highest upper confidence bound (UCB) score among the operation flows output by the first preference-optimized large language model, and the sixth operation flow being the operation flow with the lowest upper confidence bound (UCB) score among the operation flows output by the first preference-optimized large language model, the fifth operation flow and the sixth operation flow both being used to obtain a webpage address of a fourth query target on a fourth target website involved in the second sample text, the fourth query target being determined based on the second sample text; According to the second sample text and the second preference data pair, at least one round of direct preference optimization is performed on the first preference-optimized large language model to obtain the first large language model.

8. The method according to claim 7, wherein: The method further comprises: Obtaining a fifth webpage address obtained based on a seventh operation process corresponding to the second sample text, wherein the seventh operation process is used to obtain a webpage address of a fifth query target on a fifth target website involved in the second sample text, the fifth query target being determined based on the second sample text; Among them, the UCB score of the fifth operation process is determined based on the matching result between the sixth web page address and the fifth web page address obtained by the fifth operation process and the probability that the first preference optimized large language model outputs the fifth operation process; the UCB score of the sixth operation process is determined based on the matching result between the seventh web page address and the fifth web page address obtained by the sixth operation process and the probability that the first preference optimized large language model outputs the sixth operation process.

9. The method according to any one of claims 5 to 8, wherein The method further comprises: Inputting the first sample text into an initial large language model to obtain an eighth operation process, wherein the eighth operation process is used to obtain a webpage address of a sixth query target on a sixth target website involved in the first sample text, the sixth query target being determined based on the first sample text; According to the eighth operation process and the first operation process, supervised fine-tuning training is performed on the initial large language model to obtain the fine-tuned large language model.

10. An information interaction method, comprising: Get the text to be processed under the web page navigation task; determining, based on the second largest language model, a target operation process corresponding to the text to be processed, wherein the target operation process is used to obtain a webpage address of a seventh query target on a seventh target website related to the text to be processed, the seventh query target being determined based on the text to be processed, wherein the second largest language model is trained according to the method of any one of claims 1 to 9; Based on the target operation process, interacting with the seventh target website on a preset browser to obtain a target webpage address of the seventh query target; The web page of the target web page address is displayed on the preset browser.

11. A large language model training device, comprising: A first acquisition module is configured to acquire a first sample text for a webpage navigation task, a first operation process corresponding to the first sample text, and a first webpage address obtained based on the first operation process, wherein the first operation process is configured to acquire a webpage address of a first query target on a first target website associated with the first sample text, wherein the first query target is determined based on the first sample text; a first determining module configured to determine, based on the first sample text and the first language model, a second operation process for the first sample text, wherein the second operation process is configured to obtain a webpage address of a second query target on a second target website referred to by the first sample text, the second query target being determined based on the first sample text; A second acquisition module is configured to acquire a second webpage address obtained by the first language model interacting with the sample browser based on the second operation process; a second determining module, configured to determine a target reward value obtained by the interaction between the first language model and the sample browser according to the first webpage address and the second webpage address; A training module is used to perform reinforcement learning training on the first language model according to the target reward value to obtain a second language model.

12. An information interaction device, comprising: The third acquisition module is used to obtain the text to be processed under the web page navigation task; a third determination module, configured to determine a target operation process corresponding to the text to be processed based on a second large language model, wherein the target operation process is used to obtain a webpage address of a seventh query target on a seventh target website involved in the text to be processed, the seventh query target being determined based on the text to be processed, wherein the second large language model is trained based on the method of any one of claims 1-9; an interaction module, configured to interact with the seventh target website on a preset browser based on the target operation process to obtain a target webpage address of the seventh query target; The display module is used to display the web page of the target web page address on the preset browser.

13. An intelligent agent comprising: Input module, used to obtain the text to be processed under the web page navigation task; a processing module configured to determine, based on a second language model, a target operation process corresponding to the text to be processed, wherein the target operation process is configured to obtain a webpage address of a seventh query target on a seventh target website involved in the text to be processed, the seventh query target being determined based on the text to be processed; and to interact with the seventh target website on a preset browser based on the target operation process to obtain a target webpage address of the seventh query target, wherein the second language model is trained based on the method of any one of claims 1 to 9; The output module is used to output the target web page address.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9, or the method according to claim 10.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 9, or to execute the method according to claim 10.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9, or implements the method according to claim 10.

Images