Intelligent agent training method and device, equipment, medium and product

Abstract

The invention discloses an agent training method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence. The method comprises the steps that for a time step in an expert track, an expansion interface operation corresponding to the time step is generated, the time step in the expert track corresponds to a reference interface operation executed in an interface when a training task is executed, and the expansion interface operation corresponding to the same time step is different from the reference interface operation; obtaining prediction interface operations executed at different time steps when an intelligent agent executes the training task, wherein the intelligent agent is used for executing the interaction task in an interface; for the same time step, determining an operation reward corresponding to the time step by taking the extended interface operation and the reference interface operation corresponding to the time step as targets of predicting the interface operation; the agent is trained based on the operational reward.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an agent training method and device, equipment, medium and product. BACKGROUND

[0002] An agent refers to an entity that can perceive an environment and autonomously perform a specific task by taking actions. Common agents include graphical user interface (GUI) agents that can simulate human operations such as clicking, inputting and reading interface information in a GUI to complete complex tasks.

[0003] In related technologies, when training an agent, an expert trajectory is usually considered as a target of the agent, a reward is calculated according to achievement of the target, and the agent is trained through the reward.

[0004] However, the above training method of giving a reward according to an expert trajectory can cause overfitting of the agent and reduce the generalization ability of the agent. SUMMARY

[0005] Embodiments of the present application provide an agent training method, device, equipment, medium and product. The technical solutions provided by the present application include the following aspects.

[0006] In one aspect, an agent training method is provided by embodiments of the present application, and the method comprises: For a time step in an expert trajectory, an extended interface operation corresponding to the time step is generated, the time step in the expert trajectory corresponds to a reference interface operation performed in an interface when performing a training task, and the extended interface operation corresponding to the same time step is different from the reference interface operation; obtaining predicted interface operations performed by the agent at different time steps when performing the training task, the agent being configured to perform an interactive task in an interface; For the same time step, the extended interface operation and the reference interface operation corresponding to the time step are taken as targets of the predicted interface operation, and an operation reward corresponding to the time step is determined; training the agent based on the operation reward.

[0007] In another aspect, an agent training device is provided by embodiments of the present application, and the device comprises: A generating module is configured to, for a time step in an expert trajectory, generate an extended interface operation corresponding to the time step, the time step in the expert trajectory corresponding to a reference interface operation performed in an interface when performing a training task, and the extended interface operation corresponding to the same time step being different from the reference interface operation; an obtaining module configured to obtain a predicted interface operation performed by the agent at different time steps when the agent performs the training task, the agent being used to perform an interaction task in an interface; a determining module configured to, for a same time step, determine an operation reward corresponding to the time step, with the predicted interface operation being targeted at the extended interface operation corresponding to the time step and the reference interface operation corresponding to the time step; a training module configured to train the agent based on the operation reward.

[0008] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory having stored therein a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned agent training method.

[0009] In another aspect, a computer readable storage medium is provided, the computer readable storage medium having stored therein a computer program, the computer program being loaded and executed by a processor to implement the above-mentioned agent training method.

[0010] The technical solutions provided by the embodiments of the present application can bring the following beneficial effects: By pre-generating, when training the agent, an extended interface operation different from the reference interface operation corresponding to each time step in the expert trajectory, the operation reward can be determined by simultaneously taking, in the subsequent process of training the agent, the reference interface operation corresponding to the time step and the extended interface operation corresponding to the time step as the target of the predicted interface operation generated by the agent, and the agent can be trained, so that the agent can learn the extended interface operation in addition to the reference interface operation in the expert trajectory, the dependence of the agent on single training data is broken, overfitting of the agent is avoided, and the generalization ability of the agent is improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of an agent training method provided by an example embodiment of the present application; Figure 2 is a schematic diagram of an expert trajectory provided by an example embodiment of the present application; Figure 3 is a schematic diagram of an extended interface operation provided by an example embodiment of the present application; Figure 4 is a schematic diagram of a predicted interface operation provided by an example embodiment of the present application; Figure 5 is a schematic diagram of an agent being trained provided by an example embodiment of the present application; Figure 6 is a schematic diagram of a training task execution state being reset provided by an example embodiment of the present application; Figure 7 is a flowchart of an embodiment of the method for training an agent provided by another example embodiment of the present application; Figure 8 is a flowchart of an embodiment of the method for training an agent provided by another example embodiment of the present application; Figure 9 is a structural schematic diagram of an embodiment of the apparatus for training an agent provided by an example embodiment of the present application; Figure 10 is a structural block diagram of an embodiment of the computer device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0013] The example embodiments will be described in detail herein below with reference to the drawings. The following description relates to the drawings, in which like numerals refer to like elements or to similar elements, unless otherwise indicated. The embodiments described in the following example embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0014] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0015] It should be understood that although the terms "first," "second," etc. can be used herein to describe various information, these terms are not intended to denote a particular order or priority among the information. These terms are used to distinguish one from another only. For example, without departing from the scope of the present application, a first parameter can be termed a second parameter, and similarly, a second parameter can be termed a first parameter. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0016] It should be noted that the application can display a prompt interface, a pop-up window or output voice prompt information before collecting user, processor, computer device and other related data, and in the process of collecting user related data. The prompt interface, pop-up window or voice prompt information is used to prompt the user that the application is currently collecting his related data, so that the application only starts to perform the related steps of obtaining user related data after obtaining the confirmation operation of the user to the prompt interface or pop-up window. Otherwise (i.e. without obtaining the confirmation operation of the user to the prompt interface or pop-up window), end the related steps of obtaining user related data, i.e. do not obtain the related data of the user. In other words, all user data collected by the application is collected with the consent and authorization of the user, and the collection, use and processing of related user data need to comply with relevant national and regional laws, regulations and standards.

[0017] Firstly, the terms involved in the embodiments of the application are introduced.

[0018] Graphical user interface agent: an artificial intelligence system based on multi-modal visual model driving, which can automatically reason and execute the interaction of graphical user interface, simulate the operation of human user such as clicking, inputting, dragging, reading interface information, etc. to complete the work task required by human. Its core functions are: understanding the task target through the input natural language request, realizing natural language interaction; analyzing multi-modal information such as interface screenshot, graphical user interface elements and reasoning action, realizing multi-modal perception and reasoning; executing action through application graphical user interface, such as opening application, batch editing, data processing, realizing task automation. The execution of the task needs the help of computer operation tools, so that the GUI agent can operate the graphical user interface like a person.

[0019] Expert trajectory: refers to a series of operations generated on the target interface, which has a high reliability of standard operation sequence. Specifically, a complete expert trajectory includes a plurality of time steps arranged in time sequence, each time step corresponds to the interface at that time and the interface operation performed under the interface. Among them, the interface operation must define its operation type and operation position, so as to ensure the reproducibility and measurability of the expert trajectory.

[0020] Teacher model: the teacher model is a well-trained and well-performing deep learning model, which usually has a large number of parameters and complex structure. It can capture rich features and knowledge, and is often used to guide the learning process of simpler student model (Student Model). The teacher model can be used to transfer the knowledge learned by it to the student model to improve the generalization ability and performance of the student model.

[0021] Chain of Thought (CoT): refers to the step-by-step reasoning process exhibited by a model before generating a final answer or conclusion, simulating human reasoning. This process breaks down complex reasoning tasks into a series of coherent and interpretable intermediate steps, improving the accuracy and reliability of the model in handling reasoning problems. Specifically, a complete thought chain is composed of multiple reasoning steps arranged in logical order. The thought chain exists in the form of text, and each reasoning step is based on the state of the previous step and the original problem description, producing a new intermediate conclusion by applying specific reasoning rules. These intermediate conclusions are successively passed and iterated, ultimately pointing to a clear final answer.

[0022] An agent refers to an entity that can perceive the environment and autonomously perform specific tasks by taking actions. Common agents include GUI agents that can simulate human operations in GUI, such as clicking, inputting, and reading interface information, to complete complex tasks. In related technologies, when training an agent, the expert trajectory is usually considered as the target of the agent, the reward is calculated according to the achievement of the target, and the agent is trained through the reward. However, the above training method of giving reward according to the expert trajectory will cause overfitting of the agent and reduce the generalization ability of the agent.

[0023] To solve the above problems, the embodiment of the present application provides an agent training method, which generates an extended interface operation different from the reference interface operation corresponding to each time step in the expert trajectory in advance when training the agent. Subsequently, in the process of training the agent, the reference interface operation and the extended interface operation corresponding to the time step can be used as the target of the predicted interface operation generated by the agent to determine the operation reward and train the agent, so that the agent can learn the extended interface operation in addition to the reference interface operation in the expert trajectory. The dependence of the agent on single training data is broken, overfitting of the agent is avoided, and the generalization ability of the agent is improved.

[0024] The solution provided in this application can be used in computer devices that require intelligent agent training. This device can be a terminal or a server. The terminal can be an electronic device such as a mobile phone, tablet computer, in-vehicle terminal (vehicle infotainment system), wearable device, or PC (Personal Computer). The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. In the following embodiments, for ease of description, the example of an intelligent agent training method being executed by a computer device is used for illustration.

[0025] Figure 1 This is a flowchart of an agent training method provided in an exemplary embodiment of this application. The method can be executed by the aforementioned computer device, and includes at least one of the following steps.

[0026] Step 110: For each time step in the expert trajectory, generate the extended interface operation corresponding to the time step. The time step in the expert trajectory corresponds to the reference interface operation executed in the interface when performing the training task, and the extended interface operation corresponding to the same time step is different from the reference interface operation.

[0027] In some embodiments, the expert trajectory includes a series of time steps for performing a training task. Each time step corresponds to the current interface and a reference interface operation performed on the interface at that current moment. The reference interface operation is also referred to as the expert interface operation or expert real action. By sequentially performing the corresponding reference interface operation on the interface corresponding to each time step, the training task is finally completed.

[0028] Optionally, the expert trajectory can be generated by collecting data on interface operations and changes during the manual execution of training tasks.

[0029] Alternatively, the expert trajectory can also be generated by collecting the interface operations and interface changes when the expert model performs the training task.

[0030] like Figure 2 The figure shows a schematic diagram of an expert trajectory provided by an exemplary embodiment of this application. The expert trajectory shown in the figure includes time step 1, time step 2 and time step 3. Time step 1 corresponds to the current interface 1 and reference interface operation 1, time step 2 corresponds to the current interface 2 and reference interface operation 2, and time step 3 corresponds to the current interface 3 and reference interface operation 3.

[0031] After the reference interface operation 1 is performed on the interface 1, the interface corresponding to the current time instant is changed to the interface 2; after the reference interface operation 2 is performed on the interface 2, the interface corresponding to the current time instant is changed to the interface 3; after the reference interface operation 3 is performed on the interface 3, the training task is completed.

[0032] Illustratively, in a training task of purchasing a black water cup in an online shopping application (APP), the interface corresponding to the first time step is the main interface of the APP, and the corresponding reference interface operation is clicking the search bar and inputting “black water cup”; the interface corresponding to the second time step is a search result list interface, and the corresponding reference interface operation is clicking the third item in the search result list; the interface corresponding to the third time step is a black water cup detail page interface, and the corresponding reference interface operation is clicking the “buy now” button; the interface corresponding to the fourth time step is a purchase interface, and the corresponding reference interface operation is clicking the “pay now” button to make payment. The task of purchasing the black water cup can be completed by sequentially performing the corresponding reference interface operations on the interfaces corresponding to the respective time steps.

[0033] Since the way to complete the training task is usually not unique, in order to avoid the agent taking the expert trajectory as the only training data, causing the agent to overfit, an extended interface operation is generated based on the interface corresponding to the time step in the expert trajectory and the training task.

[0034] The training task is used to determine the goal to be achieved, and the interface corresponding to the current time step can be used to determine the starting point of performing the training task. Different execution ways of the training task can be extended based on the interface corresponding to the time step in the expert trajectory and the training task.

[0035] Illustratively, for the above-mentioned training task of purchasing a black water cup, one extended interface operation corresponding to the first time step is clicking the household supplies column in the main page. Among them, clicking the household supplies column can find the black water cup in the opened household supplies detailed interface, which is another path to complete the training task. The two extended interface operations corresponding to the second time step are sliding the screen downward and clicking the lower right corner of the product picture. Among them, sliding the screen downward can be used to display other water cup information of the current interface, and provides support for clicking the third item in the search result list when the product display is not complete; clicking the lower right corner of the product picture and clicking the third item in the search result list can both be used to open the product detail page.

[0036] In some embodiments, the number of generated extended interface operations corresponding to the time step can be one or more, and the number of extended interface operations corresponding to different time steps in the expert trajectory can be the same or different. The embodiments of the present application do not limit this.

[0037] As Figure 3As shown in the figure, the figure shows the schematic diagram of the extended interface operation provided by one of the illustrative embodiments of the present application. The figure shows the correspondence between the extended interface operation and the time step in the expert trajectory. Time step 1 corresponds to extended interface operation 1, time step 2 corresponds to extended interface operation 2 and extended interface operation 3, and time step 3 corresponds to extended interface operation 4. Among them, time step 1 and time step 3 only correspond to one extended interface operation, and time step 2 corresponds to two extended interface operations.

[0038] In step 120, the predicted interface operation executed by the agent at different time steps when the agent executes the training task is obtained, and the agent is used to execute the interaction task in the interface.

[0039] In some embodiments, the agent can correctly recognize the current task execution scene based on the interface corresponding to the current time step and the training task, and infer to generate the predicted interface operation executed at the current time step.

[0040] Illustratively, when the agent executes the above-mentioned training task of buying a black water cup, in the search result list interface corresponding to the second time step, it is judged that the current page is not fully displayed and more goods need to be displayed by sliding down, and the predicted interface operation of sliding the screen downward is generated.

[0041] The agent generates the corresponding predicted interface operation at each time step, and there is a correspondence between the predicted interface operation and the time step.

[0042] As Figure 4 As shown in the figure, the figure shows the schematic diagram of the predicted interface operation provided by one of the illustrative embodiments of the present application. The figure shows the correspondence between the predicted interface operation and the time step in the expert trajectory, time step 1 corresponds to predicted interface operation 1, time step 2 corresponds to predicted interface operation 2, and time step 3 corresponds to predicted interface operation 3.

[0043] In step 130, for the same time step, the operation reward corresponding to the time step is determined by taking the extended interface operation corresponding to the time step and the reference interface operation as the target of the predicted interface operation.

[0044] When training the agent, in order to learn the reference interface operation in the expert trajectory and the extended interface operation other than the reference interface operation at the same time, the computer device determines the operation reward by taking the extended interface operation and the reference interface operation as the target of the predicted interface operation generated by the agent.

[0045] Optionally, the operation reward can be determined by taking the reference interface operation corresponding to the time step as the target, or the operation reward can be determined by taking the extended interface operation corresponding to the time step as the target.

[0046] Optionally, the target of the predicted interface operation can be selected sequentially according to the selection priority of the extended interface operation and the reference interface operation. For example, if the selection priority of the reference interface operation is higher than that of the extended interface operation, the computer device first uses the reference interface operation as the target of the predicted interface operation, and then uses the extended interface operation as the target of the predicted interface operation.

[0047] By using the extended interface operation and reference interface operation corresponding to the time step as the target for predicting the interface operation, compared to only using the reference interface operation corresponding to the time step as the target for predicting the interface operation, the agent can simultaneously learn the reasoning logic for executing the reference interface operation and the reasoning logic for executing the extended interface operation at the current time step. This avoids the agent overlearning the reference interface operation at the current time step, thereby avoiding overfitting and improving the agent's generalization ability.

[0048] Rewards are the core feedback signals used in reinforcement learning to evaluate and guide model behavior. Model training is achieved by updating model parameters with the goal of maximizing the reward. In this embodiment, the reward is manifested as an operational reward. After obtaining the operational reward, the agent's policy parameters are updated to maximize the obtained operational reward, thereby training the agent.

[0049] In some embodiments, when training an agent, it uses time steps as units, taking the extended interface operation and reference interface operation corresponding to the same time step as the target for predicting the interface operation, and determines the operation reward corresponding to the time step. After obtaining the operation reward corresponding to each time step, the operation reward can be used to optimize the parameters of the agent and improve the performance of the agent.

[0050] like Figure 5 The figure illustrates a schematic diagram of a training agent provided in an illustrative embodiment of this application. As shown, the operation reward 1 corresponding to time step 1 is determined based on the reference interface operation 1, extended interface operation 1, and prediction interface operation 1 corresponding to time step 1. The operation reward 2 corresponding to time step 2 is determined based on the reference interface operation 2, extended interface operation 2, extended interface operation 3, and prediction interface operation 2 corresponding to time step 2. The operation reward 3 corresponding to time step 3 is determined based on the reference interface operation 3, extended interface operation 4, and prediction interface operation 3 corresponding to time step 3. After obtaining the operation rewards corresponding to each time step, the agent can be trained based on these operation rewards.

[0051] Step 140: Train the agent based on operation rewards.

[0052] In some embodiments, after determining the operation reward corresponding to each time step, the computer device calculates the total operation reward based on the operation reward corresponding to each time step, and trains the agent based on the total operation reward.

[0053] Illustratively, the computer device adds the operation rewards corresponding to each time step to obtain a total operation reward, and trains the agent by using a policy gradient algorithm in order to maximize the total operation reward. Specifically, the computer device samples and calculates a log probability gradient of the total operation reward with respect to a policy parameter of the agent, and takes a product of the total operation reward and the log probability gradient as an estimation value of a policy gradient. Subsequently, the policy gradient estimation is used to update the policy parameter of the agent by using a gradient ascent method, so as to realize the training of the agent.

[0054] To sum up, in the embodiments of the present application, when training the agent, for each time step in the expert trajectory, an extended interface operation different from the reference interface operation corresponding to the time step is generated in advance, and in the subsequent process of training the agent, the reference interface operation and the extended interface operation corresponding to the time step can be used as the target of the predicted interface operation generated by the agent to determine the operation reward and train the agent, so that the agent can learn the extended interface operation in addition to the reference interface operation in the expert trajectory, break the dependence of the agent on single training data, avoid overfitting of the agent, and improve the generalization ability of the agent.

[0055] The agent training method provided in the above embodiments can be used to train GUI agents, game artificial intelligence (AI) agents, customer service agents, and other agents having the function of automatically performing specific tasks on an interface, and the embodiments of the present application do not limit this.

[0056] In some embodiments, for a time step in the expert trajectory, the extended interface operation corresponding to the time step is generated by performing the following operations: I. Generating candidate interface operations corresponding to each time step in the expert trajectory by executing the training task through a teacher model.

[0057] The teacher model refers to a high-performance deep learning model that is trained in advance and has good execution effect in the face of the task applied to the graphical user interface. The candidate interface operations corresponding to each time step can be generated through the teacher model.

[0058] In some embodiments, the computer device inputs the training task and the interface screenshot of the interface corresponding to the time step in the expert trajectory into the teacher model to obtain the candidate interface operation corresponding to the time step output by the teacher model.

[0059] The computer device inputs the interface screenshot corresponding to the time step in the training task and the expert trajectory to the teacher model. The teacher model determines the target to be achieved according to the training task, determines the interface corresponding to the current time step according to the interface screenshot, and further determines the task execution scene of the training task, and infers the candidate interface operation performed on the interface corresponding to the current time step.

[0060] In some embodiments, the number of candidate interface operations corresponding to the time step generated by the teacher model can be one or more, and the number of candidate interface operations corresponding to different time steps generated by the teacher model can be the same or different, which is not limited in the embodiments of the present application.

[0061] In some embodiments, the teacher model can be a multi-modal large model. Illustratively, the computer device uses Qwen2.5-VL-32B as the teacher model, and inputs the current task instruction and the corresponding interface screenshot as input. By setting a higher sampling temperature (Temperature), such as setting Temperature to 1.0, and performing multiple independent inference sampling, diverse candidate interface operations can be generated for each time step, improving the diversity of the generated extended interface operations. The specific type of the teacher model is not limited in the embodiments of the present application.

[0062] By inputting the training task and the interface screenshot corresponding to the time step in the expert trajectory to the teacher model, the teacher model can determine the task execution scene of the training task according to the training task and the interface screenshot, improve the correctness of the teacher model's cognition of the task execution scene of the training task, and improve the efficiency and correctness of the teacher model in generating candidate interface operations.

[0063] II. Filtering illegal interface operations in the candidate interface operations to obtain extended interface operations, the illegal interface operations including interface operations that do not conform to the input data format specification and / or do not belong to the legal interface operation set.

[0064] In some embodiments, the teacher model cannot directly perform interface operations on the graphical user interface, and the generated candidate interface operations exist in the form of text, and the data format of the input data receivable by the agent is limited. In order to make the extended interface operation directly usable by the agent, the computer device needs to first filter the candidate interface operations generated by the teacher model, and exclude the interface operations that do not conform to the input data format specification of the agent.

[0065] In some embodiments, the agent can receive input data in JSON format, and remove the candidate interface operations generated by the teacher model that do not conform to the JSON format specification. Illustratively, the candidate interface operation corresponding to the JSON format with punctuation errors is removed.

[0066] In some embodiments, the teacher model receives the training task and the candidate interface operation generated by the interface screenshot. Due to the difference in platforms where the graphical user interface is located, part of the candidate interface operation may not be suitable for the graphical user interface required by the training task, and the candidate interface operation that is not legal for the graphical user interface required by the training task needs to be removed.

[0067] Illustratively, the training task needs to be completed in the graphical user interface of the mobile device platform, and the candidate interface operation generated by the teacher model includes the "right-click" operation, while the mobile device platform does not support mouse operation, and the candidate interface operation corresponding to the "right-click" operation is removed.

[0068] In some embodiments, the candidate interface operation generated by the teacher model is the same as the reference interface operation in the expert trajectory, and the computer device removes the candidate interface operation that is the same as the expert trajectory in the candidate interface operation.

[0069] By filtering the illegal interface operation in the candidate interface operation, the executability of the generated extended interface operation in the graphical user interface required by the training task is ensured, and at the same time, the generated extended interface operation is ensured to be suitable as input data of the intelligent agent, and the correctness of generating the extended interface operation is improved.

[0070] In some embodiments, when the teacher model generates the extended interface operation, the corresponding extended thinking chain text is also generated, and the extended thinking chain text is used to represent the reasoning process of the teacher model when generating the extended interface operation, and the thinking chain text exists in the form of text.

[0071] Illustratively, when the teacher model generates the extended interface operation of sliding the screen down, the corresponding extended thinking chain text "because the current page is not fully displayed, the page needs to be scrolled down" is also generated; when the teacher model generates the extended interface operation of clicking the lower right corner of the product picture, the corresponding extended thinking chain text "clicking the lower right corner of the picture can also enter the product detail page" is also generated.

[0072] In some embodiments, the intelligent agent may perform a different predicted interface operation at some time steps than the corresponding reference interface operation, so that the interface after performing the predicted interface operation is different from the interface after performing the reference interface operation, causing the interface of the intelligent agent at the current time to be different from the interface corresponding to the time step, and the learning of the reference interface operation corresponding to the time step cannot continue.

[0073] In order to ensure the continuity of the training state of the intelligent agent and make full use of the reference extended operation corresponding to each time step, the present application embodiment is provided with a correction mechanism, which resets the task execution state of the training task from the second time step, and modifies the current interface to the interface after performing the reference interface operation corresponding to the previous time step.

[0074] In some embodiments, the predicted interface operation performed by the agent at different time steps when performing the training task is obtained, including: I. At the first time step, the predicted interface operation performed by the agent when performing the training task is obtained.

[0075] At the first time step, the task execution state of the training task at this time is in an initial state, and the agent starts to perform the training task to generate the predicted interface operation corresponding to the first time step.

[0076] The agent generates the predicted interface operation corresponding to the first time step according to the training task and the interface corresponding to the first time step. The agent determines the target to be achieved according to the training task, determines the task execution scene of the training task according to the interface corresponding to the first time step, and infers the predicted interface operation performed on the interface corresponding to the current time step.

[0077] II. At the i-th time step, the task execution state of the training task is reset based on the reference interface operation corresponding to the (i-1)-th time step, and the predicted interface operation performed by the agent when performing the training task is obtained, i being an integer greater than 1.

[0078] At the i-th time step, the task execution state of the training task at this time is determined by the corresponding predicted interface operation performed at the (i-1)-th time step.

[0079] In some embodiments, the corresponding predicted interface operation performed at the (i-1)-th time step matches the reference interface operation corresponding to the (i-1)-th time step. In other embodiments, the corresponding predicted interface operation performed at the (i-1)-th time step matches one of the extended interface operations corresponding to the (i-1)-th time step. In other embodiments, the corresponding predicted interface operation performed at the (i-1)-th time step does not match the reference interface operation corresponding to the (i-1)-th time step or any of the extended interface operations corresponding to the (i-1)-th time step.

[0080] The predicted interface operation corresponding to the (i-1)-th time step does not match the reference interface operation, and the agent performs the predicted interface operation corresponding to the (i-1)-th time step and the interface obtained after performing the corresponding reference interface operation are different.

[0081] In order to enable the agent to continue to learn the reference interface operation corresponding to the i-th time step in the expert trajectory at the i-th time step, the computer device resets the task execution state of the training task based on the reference interface operation corresponding to the (i-1)-th time step at the i-th time step, and modifies the current interface to the interface after performing the reference interface operation corresponding to the (i-1)-th time step.

[0082] As Figure 6As shown in the figure, it shows a schematic diagram of resetting the training task execution state provided by one illustrative embodiment of the present application. As shown in the figure, there are other time steps before the first time step, after the agent performs the predicted interface operation 1 at the first time step, the current interface changes to interface 4, in order to continue to learn the reference interface operation corresponding to the second time step, the computer device resets the current interface to interface 2; after performing the predicted interface operation 2 at the second time step, the current interface changes to interface 5, in order to continue to learn the reference interface operation corresponding to the third time step, the computer device resets the current interface to interface 3.

[0083] At the i-th time step, the computer device resets the task execution state of the training task based on the reference interface operation corresponding to the i-1-th time step. That is, from the second time step, in the current time step, the computer device resets the current interface to the interface after performing the reference interface operation corresponding to the previous time step.

[0084] In some embodiments, the predicted interface operation and the reference interface operation corresponding to the same time step match, that is, the interface after the computer device performs the predicted interface operation corresponding to the current time step is the same as the interface after performing the reference interface operation corresponding to the current time step, and the task execution state of the training task is still reset.

[0085] In other embodiments, the predicted interface operation and the reference interface operation corresponding to the same time step match, and the task execution state of the training task is not reset.

[0086] The embodiments of the present application do not limit whether to reset the task execution state of the training task in the case that the predicted interface operation and the reference interface operation corresponding to the same time step match.

[0087] After resetting the task execution state of the training task, the agent generates the predicted interface operation corresponding to the i-th time step according to the training task and the interface corresponding to the i-th time step. The agent determines the target to be achieved according to the training task, determines the task execution scene of the training task according to the interface corresponding to the i-th time step, and infers the predicted interface operation to be performed on the interface corresponding to the current time step.

[0088] By resetting the current task execution state to the interface state after performing the reference interface operation corresponding to the previous time step from the second time step, the agent can continue to learn the reference interface operation corresponding to the subsequent time step in the case that the interface after performing the predicted interface operation is different from the interface corresponding to the same time step in the expert trajectory, improves the continuity of the agent training, and can avoid stopping learning the expert trajectory in advance due to the current interface state exceeding the interface state in the expert trajectory, and improves the utilization rate of the reference interface operation corresponding to the time step in the expert trajectory.

[0089] In some embodiments, the computer device first targets the reference interface operation as the predicted interface operation, and then targets the extended interface operation as the predicted interface operation in the case that the reference interface operation does not match the predicted interface operation, determines the operation reward, and makes the agent take the reference interface operation corresponding to the time step in the expert trajectory as the primary learning target and reduce the learning of the corresponding extended interface operation.

[0090] Figure 7 FIG. 13 is a flowchart illustrating a method for training an agent according to another illustrative embodiment of the present disclosure, which can be executed by the computer device described above. As shown in the figure, the step 130 described above can be implemented as steps 132, 134, 136 and 138.

[0091] Step 132: Matching the reference interface operation and the predicted interface operation corresponding to the time step.

[0092] In some embodiments, for the same time step, when the reference interface operation and the extended interface operation are targeted as the predicted interface operation to determine the operation reward corresponding to the time step, the selection priority of the reference interface operation is higher than that of the extended interface operation, the computer device first targets the reference interface operation as the predicted interface operation, and then targets the extended interface operation as the predicted interface operation.

[0093] The reference interface operation in the expert trajectory is an interface operation formulated by an expert in the field, which has a high confidence level. When the reference interface operation and the extended interface operation are targeted to determine the operation reward, the reference interface operation is first targeted as the predicted interface operation to determine whether it matches the reference interface operation with a high confidence level.

[0094] In some embodiments, matching the reference interface operation and the predicted interface operation corresponding to the time step comprises: I. Matching the operation type and the operation position of the reference interface operation and the predicted interface operation corresponding to the time step.

[0095] In some embodiments, the reference interface operation and the predicted interface operation are both interface operations performed on a graphical user interface. Illustratively, the reference interface operation is represented as a right-click of a mouse on a position with screen coordinates [720, 1080] in the graphical user interface, where the right-click of the mouse is the operation type and [720, 1080] is the operation position; and the predicted interface operation is represented as a left-click of a mouse on a position with screen coordinates [720, 1000] in the graphical user interface, where the left-click of the mouse is the operation type and [720, 1000] is the operation position.

[0096] In some embodiments, the computer device matches the reference interface operation and the predicted interface operation, specifically, matching the operation type and the operation position of the reference interface operation and the predicted interface operation. The operation type is the type of the interface operation to be performed, and the operation position is the position of the object on which the interface operation is performed.

[0097] II. In the case that the operation type of the reference interface operation and the predicted interface operation matches, and the distance between the operation positions of the reference interface operation and the predicted interface operation is less than the distance threshold, the computer device determines that the reference interface operation and the predicted interface operation match.

[0098] The computer device matches the reference interface operation and the predicted interface operation corresponding to the time step. In the case that the operation type matches, and the distance between the operation positions is less than the distance threshold, the computer device determines that the types of the interface operations performed by the reference interface operation and the predicted interface operation match, and the objects on which the interface operations are performed also match, and determines that the reference interface operation and the predicted interface operation match.

[0099] In some embodiments, the interface effects implemented by interface operations of different operation types are different. Only in the case that the operation types of the reference interface operation and the predicted interface operation are the same, it is determined that the operation types of the reference interface operation and the predicted interface operation match. Illustratively, the interface effects implemented by clicking the left mouse button on the desktop and clicking the right mouse button are different. In the case that the operation types of the reference interface operation and the predicted interface operation are both left mouse button, it is determined that the operation types of the reference interface operation and the predicted interface operation match.

[0100] In some other embodiments, the interface effects implemented by interface operations of different operation types are the same. Only in the case that the interface effects implemented by the operation types of the reference interface operation and the predicted interface operation are the same, it is determined that the operation types of the reference interface operation and the predicted interface operation match. Illustratively, in the display commodity list interface of an application program, clicking the commodity with the left mouse button and the right mouse button implements the interface effect of opening the detail page of the commodity. In the display commodity list interface, it is determined that clicking the commodity with the left mouse button and clicking the commodity with the right mouse button are operation types that match.

[0101] It is worth noting that the specific way of determining whether the operation types match is not limited in the embodiments of the present application.

[0102] In some embodiments, there is a judgment range for the object of the interface operation when the interface operation is performed. For example, for a circular button with a diameter of 100 pixels in the graphical user interface, as long as the click position of the button is within the pixel range, it is considered that the button is clicked. Therefore, there is a judgment range for the object of the interface operation. After the operation type of the reference interface operation and the predicted interface operation is determined, it is necessary to determine whether the operation position of the predicted interface operation is within the allowed range of the operation position of the reference interface operation, so as to determine whether the object of the reference interface operation and the object of the predicted interface operation are the same.

[0103] In some embodiments, the position judgment range of the object of the interface operation is a circular range. For example, the position judgment range of the object of the interface operation is the circular pixel area of the circular button with a diameter of 100 pixels. The operation position is a specific pixel position, and whether the operation position is within the circular pixel area can be determined by judging whether the distance between the two operation positions is less than the distance threshold. The operation positions of the reference interface operation and the predicted interface operation are specific pixel positions, and whether the operation positions of the reference interface operation and the predicted interface operation are within the allowed range can be determined by calculating the distance between the pixel positions of the two operation positions and determining that the operation positions of the reference interface operation and the predicted interface operation are within the allowed range when the distance is less than the distance threshold.

[0104] For example, the Euclidean distance between the operation positions of the reference interface operation and the predicted interface operation is calculated, and the calculation formula of the Euclidean distance is as follows:

[0105] wherein, , is the coordinate of the first operation position, , is the coordinate of the second operation position.

[0106] In the case where the Euclidean distance is less than the distance threshold, it is determined that the operation position of the predicted interface operation is within the allowed range of the operation position of the reference interface operation.

[0107] In some embodiments, the distance threshold is different for different operation types. For example, the distance threshold for the left mouse click is 20 pixels, which is used to accurately determine the click target; the distance threshold for the sliding action is 50 pixels, which is used to determine the direction and does not require high accuracy of the starting position.

[0108] In other embodiments, the distance threshold is the same for different operation types. The embodiments of the present application do not limit this.

[0109] In some embodiments, the position determination range of the object of the interface operation is a polygon range other than a circular range. A bounding box corresponding to the reference interface operation is determined, and the computer device determines whether the operation position of the predicted interface operation is within the allowed range of the operation position of the reference interface operation by determining whether the operation position of the predicted interface operation is within the bounding box corresponding to the reference interface operation.

[0110] It is worth noting that the present application does not limit the specific manner of determining whether the operation position of the predicted interface operation is within the allowed range of the operation position of the reference interface operation.

[0111] In some embodiments, the computer device determines that the reference interface operation and the predicted interface operation match when the operation types of the reference interface operation and the predicted interface operation match. That is, it is not necessary to determine whether the distance between the operation positions of the reference interface operation and the predicted interface operation is less than the distance threshold, but only necessary to determine that the operation types match, i.e., the reference interface operation and the predicted interface operation match.

[0112] The present application does not limit the manner of determining whether the reference interface operation and the predicted interface operation match.

[0113] When the operation types match and the distance between the operation positions is less than the distance threshold, the types of the interface operations performed by the reference interface operation and the predicted interface operation match, and the objects of the interface operations performed are the same. In this case, the reference interface operation and the predicted interface operation are determined to match, thereby improving the efficiency and accuracy of determining whether the reference interface operation and the predicted interface operation match.

[0114] In step 134, the reference reward is determined as the operation reward corresponding to the time step when the reference interface operation and the predicted interface operation match.

[0115] When the reference interface operation and the predicted interface operation match, the strategy of the agent for performing the corresponding predicted interface operation at the current time step conforms to the expert strategy for performing the corresponding reference interface operation, which to some extent reflects the effectiveness of the agent's imitation learning at the current time step. Therefore, the reference reward is determined as the operation reward corresponding to the current time step.

[0116] In some embodiments, the reference reward is a basic reward value awarded when the predicted interface operation matches the reference interface operation. The reward determined when the predicted interface operation and the extended interface operation match is less than the reference reward, which is used to ensure that the agent will learn the reference interface operation corresponding to the time step in the expert trajectory as the primary learning goal.

[0117] Step 136, in the case that the reference interface operation and the predicted interface operation do not match, matching the predicted interface operation corresponding to the time step with each extended interface operation.

[0118] In the case that the reference interface operation and the predicted interface operation do not match, the strategy of the agent performing the predicted interface operation at the current time step is different from the expert strategy of performing the reference interface operation at the current time step corresponding to the interface, but since the way of performing the training task is not unique, the computer device matches the predicted interface operation corresponding to the time step with each extended interface operation to determine whether the strategy of the agent performing the predicted interface operation at the current time step conforms to the extended strategy of performing the corresponding extended interface operation.

[0119] In some embodiments, the number of extended interface operations corresponding to the time step can be one or more, and correspondingly, the number of corresponding extended interface operations matched with the predicted interface operation corresponding to the time step can also be one or more. The embodiments of the present application do not limit this.

[0120] In some embodiments, matching the predicted interface operation corresponding to the time step with each extended interface operation includes: I. For each extended interface operation corresponding to the time step, matching the extended interface operation and the predicted interface operation in operation type and operation position.

[0121] In some embodiments, the extended interface operation and the predicted interface operation are both interface operations performed on the graphical user interface. Illustratively, the extended interface operation is a right mouse click on a position with screen coordinates [720, 1080] in the graphical user interface, where the right mouse click is the operation type and [720, 1080] is the operation position; the predicted interface operation is a left mouse click on a position with screen coordinates [720, 1000] in the graphical user interface, where the left mouse click is the operation type and [720, 1080] is the operation position.

[0122] In some embodiments, the matching of the extended interface operation and the predicted interface operation by the computer device for each extended interface operation corresponding to the time step specifically embodies the matching between the operation type and the operation position of each extended interface operation corresponding to the time step and the predicted interface operation. The operation type is the type of interface operation to be performed, and the operation position is used to represent the position of the object of the performed interface operation, and the object of the interface operation is determined through the operation position.

[0123] II. In the case that the operation type of the extended interface operation and the predicted interface operation matches, and the distance between the operation position of the extended interface operation and the operation position of the predicted interface operation is less than the distance threshold, it is determined that the extended interface operation and the predicted interface operation match.

[0124] The computer device matches the extended interface operation and the predicted interface operation corresponding to the time step. In a case where the operation types are matched and the distance between the operation positions is less than the distance threshold, the types of the interface operations performed by the extended interface operation and the predicted interface operation are matched, the objects performing the interface operations are also matched, and it is determined that the extended interface operation and the predicted interface operation are matched.

[0125] For a specific manner in which the embodiments of the present application determine whether the operation types of the extended interface operation and the predicted interface operation are matched, refer to the manner in which it is determined whether the operation types of the reference interface operation and the predicted interface operation are matched.

[0126] In some embodiments, the manner in which the computer device determines whether the operation types of the extended interface operation and the predicted interface operation are matched can be the same as or different from the manner in which it is determined whether the operation types of the reference interface operation and the predicted interface operation are matched, and the embodiments of the present application do not make any limitation in this regard.

[0127] For a specific manner in which the embodiments of the present application determine whether the operation position of the predicted interface operation is within the allowable range of the operation position of the extended interface operation, refer to the manner in which it is determined whether the operation position of the predicted interface operation is within the allowable range of the operation position of the reference interface operation.

[0128] In some embodiments, the manner in which the computer device determines whether the operation position of the predicted interface operation is within the allowable range of the operation position of the extended interface operation can be the same as or different from the manner in which it is determined whether the operation position of the predicted interface operation is within the allowable range of the operation position of the reference interface operation, and the embodiments of the present application do not make any limitation in this regard.

[0129] In a case where the operation types are matched and the distance between the operation positions is less than the distance threshold, the types of the interface operations performed by the extended interface operation and the predicted interface operation are matched, and the objects performing the interface operations are also matched, and in this case, the extended interface operation and the predicted interface operation are determined to be matched, thereby improving the efficiency and accuracy of determining whether the extended interface operation and the predicted interface operation are matched.

[0130] In step 138, in a case where there is at least one extended interface operation matched with the predicted interface operation, the operation reward corresponding to the time step is determined based on the benchmark reward, wherein the operation reward determined based on the benchmark reward is less than the benchmark reward.

[0131] In some embodiments, in a case where there is at least one extended interface operation matched with the predicted interface operation, the policy of the interface operation decided by the agent at the current time step conforms to the expert policy corresponding to the same time step in the expert trajectory, which to some extent reflects the effectiveness of the agent in imitative learning at the current time step, and the operation reward corresponding to the time step is determined based on the benchmark reward.

[0132] The benchmark reward is a basic reward value awarded when the predicted interface operation matches the reference interface operation. In order to make the agent learn the reference interface operation corresponding to the time step in the expert trajectory as the primary learning goal, while indicating the reserved trust for the non-expert trajectory, in the case that the predicted interface operation corresponding to the current time step does not match the corresponding reference interface operation and matches the corresponding extended interface operation, the operation reward of the computer device determined based on the benchmark reward corresponding to the current time step is less than the benchmark reward.

[0133] In some embodiments, the predicted interface operation corresponds to predicted thought chain text, and the predicted thought chain text is used to represent the reasoning process of the agent generating the predicted interface operation. The extended interface operation corresponds to extended thought chain text, and the extended thought chain text is used to represent the reasoning process of generating the extended interface operation.

[0134] The agent has a corresponding reasoning process when generating the predicted interface operation. Illustratively, the agent generates the predicted interface operation of sliding the screen downward while generating the corresponding extended thought chain text "in order to view more goods, the screen needs to be slid downward". Similarly, the extended interface operation also corresponds to the extended thought chain text.

[0135] In some embodiments, in the case that there is at least one extended interface operation matching the predicted interface operation, determining the operation reward of the time step based on the benchmark reward includes: I. In the case that there is at least one extended interface operation matching the predicted interface operation, for each extended interface operation matching the predicted interface operation, determining the thought chain text similarity between the extended thought chain text corresponding to the extended interface operation and the predicted thought chain text.

[0136] In the case that there is at least one extended interface operation matching the predicted interface operation, the strategy of the agent performing the corresponding reference interface operation at the current time step conforms to the extended strategy of performing the corresponding extended interface operation. However, in order to avoid the agent being randomly generated predicted interface operation being identified as conforming to the extended thought, the strategy of the agent performing the corresponding predicted interface operation at the current time step is matched and verified with the extended strategy of performing the corresponding extended interface operation in the case of matching the interface operation.

[0137] In some embodiments, the predicted interface operation corresponds to predicted thought chain text, and the predicted thought chain text is used to represent the reasoning process of the agent generating the predicted interface operation. The extended interface operation corresponds to extended thought chain text, and the extended thought chain text is used to represent the reasoning process of generating the extended interface operation. Determining the thought chain text similarity between the extended thought chain text corresponding to the extended interface operation and the predicted thought chain text includes: I. Determine the extended thinking chain text vector corresponding to the extended interface operation, and determine the predicted thinking chain text vector corresponding to the predicted thinking chain text.

[0138] In some embodiments, the predicted thinking chain text is used to represent the reasoning process of the agent when generating the predicted interface operation, and the extended thinking chain text is used to represent the reasoning process when generating the extended interface operation. By using the predicted thinking chain text corresponding to the predicted interface operation at the current time step and the extended thinking chain text corresponding to the extended interface operation, the computer device can verify the matching of the strategy of the agent performing the corresponding predicted interface operation at the current time step and the extended strategy of performing the corresponding extended interface operation.

[0139] In some embodiments, the extended thinking chain text is output by a teacher model, and the teacher model outputs the corresponding extended thinking chain text when generating the extended interface operation at the time step.

[0140] Since the thinking chain text exists in the form of text, it is difficult to quantitatively calculate the similarity between texts. By obtaining the thinking chain text vector corresponding to the thinking chain text, the similarity of the thinking chain text can be calculated by calculating the vector similarity.

[0141] Before determining the thinking chain text similarity between the extended thinking chain text corresponding to the extended interface operation and the predicted thinking chain text, the computer device first needs to obtain the extended thinking chain text vector of the extended thinking chain text corresponding to the extended interface operation, and the predicted thinking chain text vector of the predicted thinking chain text corresponding to the predicted interface operation.

[0142] In some embodiments, the computer device uses an embedding model to convert the thinking chain text in the form of text into a thinking chain text vector. The embodiments of the present application do not limit the way of determining the thinking chain text vector corresponding to the thinking chain text.

[0143] II. Determine the thinking chain text similarity based on the cosine similarity between the extended thinking chain text vector and the predicted thinking chain text vector.

[0144] After the computer device determines the extended thinking chain text vector of the extended thinking chain text corresponding to the extended interface operation and the predicted thinking chain text vector of the predicted thinking chain text, the computer device calculates the thinking chain text similarity between the extended thinking chain text vector and the predicted thinking chain text vector.

[0145] In some embodiments, the computer device determines the thinking chain text similarity by calculating the cosine similarity between the extended thinking chain text vector and the predicted thinking chain text vector. The formula for calculating the cosine similarity is as follows:

[0146] wherein, is an extended thought chain text vector, is a predicted thought chain text vector.

[0147] The computer device takes the cosine similarity between the calculated extended thought chain text vector and the predicted thought chain text vector as the thought chain text similarity.

[0148] By obtaining the extended thought chain text vector of the extended interface operation corresponding to the extended thought chain text and the predicted thought chain text vector of the predicted thought chain text, calculating the cosine similarity between the extended thought chain text vector and the predicted thought chain text vector, and determining the thought chain text similarity, the efficiency and accuracy of determining the thought chain text similarity are improved.

[0149] It is worth noting that the embodiments of the present application do not limit the way of determining the thought chain text similarity between the extended thought chain text and the predicted thought chain text.

[0150] II. In the case where the thought chain text similarity between at least one extended thought chain text and the predicted thought chain text reaches a similarity threshold, the operation reward corresponding to the time step is determined based on the baseline reward.

[0151] The thought chain text similarity can express the similarity between the extended thought chain text vector and the predicted thought chain text vector. The higher the thought chain text similarity, the higher the similarity between the extended thought chain text and the predicted thought chain text, and the higher the similarity between the corresponding reasoning process of the extended thought chain text and the corresponding reasoning process of the predicted thought chain text.

[0152] In some embodiments, the computer device determines the operation reward corresponding to the current time step in the case where there is at least one extended interface operation matching the predicted interface operation. That is, the strategy of the agent performing the corresponding predicted interface operation at the current time step does not need to be the same as the strategy of performing the corresponding extended interface operation. As long as the corresponding predicted interface operation at the current time step matches the corresponding extended interface operation, it is considered that the agent has completed the learning of performing the extended interface operation at the current time step.

[0153] In some embodiments, the computer device considers that the extended thought chain text matches the predicted thought chain text in the case where the calculated cosine similarity between the extended thought chain text vector and the predicted thought chain text vector reaches a similarity threshold. Illustratively, in the case where the cosine similarity between the extended thought chain text vector and the predicted thought chain text vector reaches 0.8, it is considered that the extended thought chain text matches the predicted thought chain text.

[0154] In some embodiments, in the same time step, there is only one extended interface operation matching the predicted interface operation, and if the extended thought chain text corresponding to the extended interface operation reaches the similarity threshold with the predicted thought chain text of the predicted interface operation, it means that the strategy of the agent executing the corresponding predicted interface operation in the current time step conforms to the extended strategy of executing the corresponding extended interface operation, and the computer device determines the operation reward corresponding to the time step based on the benchmark reward.

[0155] In other embodiments, in the same time step, there are multiple extended interface operations matching the predicted interface operation, and only one extended interface operation needs to have the extended thought chain text corresponding to the extended interface operation reaching the similarity threshold with the predicted thought chain text of the predicted interface operation, which means that the strategy of the agent executing the corresponding predicted interface operation in the current time step conforms to the extended strategy of executing the corresponding extended interface operation, and the computer device determines the operation reward corresponding to the time step based on the benchmark reward.

[0156] By calculating the thought chain text similarity between the extended thought chain text corresponding to the extended interface operation and the predicted thought chain text in the case where there is at least one extended interface operation matching the predicted interface operation, and determining the operation reward in the case where the thought chain text similarity reaches the similarity threshold, while ensuring the correctness of the strategy of the agent executing the corresponding predicted interface operation in the current time step, the efficiency and correctness of the agent learning are improved.

[0157] In some embodiments, determining the operation reward corresponding to the time step based on the benchmark reward comprises: Determining the operation reward corresponding to the time step based on the benchmark reward and the conversion factor corresponding to the time step, the conversion factor including at least one of a penalty factor and a progress factor, the penalty factor being used to represent a penalty for executing the extended interface operation outside the reference interface operation, and the progress factor being used to represent the progress of the time step in the expert trajectory.

[0158] The reference interface operation in the expert trajectory has the characteristic of high reliability, while the extended interface operation, although generated based on the time step in the expert trajectory, has lower reliability than the reference interface operation. In order to learn the reference interface operation corresponding to the time step in the expert trajectory as the primary learning goal, while expressing the reserved trust for the non-expert trajectory, it is necessary to reduce the operation reward corresponding to the current time step determined in the case where the predicted interface operation matches the extended interface operation.

[0159] In some embodiments, the operation reward corresponding to the current time step is determined based on the benchmark reward and the conversion factor corresponding to the current time step, and the conversion factor is used to reduce the operation reward corresponding to the current time step.

[0160] In some embodiments, the conversion factor specifically embodies a penalty factor, which is used to represent a penalty for performing the extended interface operation other than the corresponding reference interface operation at the current time step, and the operation reward corresponding to the current time step is determined based on the benchmark reward and the penalty factor.

[0161] Illustratively, when the benchmark reward is 1.0 and the penalty factor is 0.6, the computer device takes the product of the benchmark reward and the penalty factor, i.e., 0.6, as the operation reward corresponding to the current time step.

[0162] In some other embodiments, the conversion factor specifically embodies a progress factor, which is used to represent the progress of the current time step in the expert trajectory. In a series of time steps in the expert trajectory, the closer to the end of the training task the interface operation is made at a later time step, and by using the progress factor, the value of the interface operation closer to the end of the training task is represented as higher. In the case that the predicted interface operation matches the extended interface operation, the computer device determines the operation reward corresponding to the current time step based on the benchmark reward and the progress factor.

[0163] Illustratively, when the benchmark reward is 1.0, the expert trajectory has a total of 10 time steps, and the current time step is the 3rd time step, the corresponding progress factor is 0.3. The product of the benchmark reward and the progress factor, i.e., 0.3, is taken as the operation reward corresponding to the current time step.

[0164] In some other embodiments, the conversion factor includes a penalty factor and a progress factor, and the operation reward corresponding to the current time step is determined based on the benchmark reward, the penalty factor, and the progress factor. The operation reward corresponding to the current time step determined based on the benchmark reward corresponds to a calculation formula, which is as follows:

[0165] wherein, is the benchmark reward, is the penalty factor, is the progress factor.

[0166] Illustratively, when the benchmark reward is 1.0, the penalty factor is 0.6, the expert trajectory has a total of 10 time steps, and the current time step is the 3rd time step, the corresponding progress factor is 0.3. The product of the benchmark reward, the penalty factor, and the progress factor, i.e., 0.18, is taken as the operation reward corresponding to the current time step.

[0167] In some other embodiments, the conversion factor is a value smaller than the benchmark reward, and the computer device takes the difference between the benchmark reward and the conversion factor as the operation reward corresponding to the current time step. Illustratively, when the benchmark reward is 1.0 and the penalty factor is 0.2, the difference between the benchmark reward and the penalty factor, i.e., 0.8, is taken as the operation reward corresponding to the current time step.

[0168] It is worth noting that the number of factors included in the conversion factor and the way of determining the operation reward based on the benchmark reward and the conversion factor are not limited in the embodiments of the present application.

[0169] By determining the operation reward corresponding to the current time step based on the benchmark reward and the conversion factor, the determined operation reward is less than the benchmark reward, and the reduction of the operation reward weakens the learning tendency of the agent for the non-expert path and strengthens the learning and imitation of the agent for the expert trajectory.

[0170] By first matching the reference interface operation and the predicted interface operation, and matching the extended interface operation and the predicted interface operation in the case that the reference interface operation and the predicted interface operation do not match, the agent focuses on learning the reference interface operation corresponding to the current time step in the expert trajectory, while reducing the learning of the corresponding extended interface operation, thereby improving the learning efficiency of the agent.

[0171] In summary, by first matching the reference interface operation corresponding to the time step and the corresponding extended interface operation, the reference interface operation with high reliability is taken as the primary learning target, which improves the correctness of the learning of the agent. In the case that the reference interface operation and the predicted interface operation do not match, the reference interface operation corresponding to the time step and the corresponding extended interface operation are matched again, and the extended interface operation corresponding to the time step is taken as the secondary learning target, which breaks the dependence of the agent on single training data, avoids overfitting of the agent, and improves the generalization ability of the agent.

[0172] In some embodiments, the predicted interface operation corresponds to a predicted thought chain text, and the predicted thought chain text is used to represent the reasoning process of the agent when generating the predicted interface operation. The extended interface operation corresponds to an extended thought chain text, and the extended thought chain text is used to represent the reasoning process when generating the extended interface operation.

[0173] In some embodiments, since the efficiency of training the agent only using the operation reward is low, the training efficiency of the agent is improved by introducing a supervision loss and training the agent using the supervision loss and the operation reward.

[0174] The computer device determines the supervision loss corresponding to the predicted interface operation in the case that there is at least one extended interface operation matching the predicted interface operation, and uses the extended thought chain text corresponding to the extended interface operation as the supervision of the predicted thought chain text for the extended interface operation matching the predicted interface operation.

[0175] In some embodiments, the computer device determines the supervision loss corresponding to the predicted interface operation based on the predicted thought chain text corresponding to the predicted interface operation and the extended thought chain text corresponding to the extended interface operation, in a case where there is at least one extended interface operation matching the predicted interface operation and the thought chain text similarity between the extended thought chain text and the predicted thought chain text reaches a similarity threshold.

[0176] In some other embodiments, the computer device determines the supervision loss corresponding to the predicted interface operation based on the predicted thought chain text corresponding to the predicted interface operation and the extended thought chain text corresponding to the extended interface operation, in a case where there is at least one extended interface operation matching the predicted interface operation. That is, the thought chain text similarity between the extended thought chain text and the predicted thought chain text is not required, which is not limited in the embodiments of the present application.

[0177] In some embodiments, the computer device calculates a cross-entropy loss based on the predicted thought chain text vector and the extended thought chain text vector, and takes the cross-entropy loss as the supervision loss corresponding to the corresponding predicted interface operation.

[0178] In some embodiments, the intelligent agent is trained based on the operation reward, including: training the intelligent agent based on the supervision loss corresponding to the predicted interface operation and the operation reward.

[0179] In some embodiments, a total loss is determined based on the supervision loss and the operation reward, and the intelligent agent is trained based on the total loss.

[0180] In some embodiments, the computer device determines the total loss as the sum of the negative value of the operation reward and the supervision loss. In some other embodiments, the computer device determines the total loss as the sum of the product of the negative value of the operation reward and a respective weight and the product of the supervision loss and a respective weight. The embodiments of the present application do not limit the way of determining the total loss based on the supervision loss and the operation reward.

[0181] In some embodiments, the computer device optimizes the total loss by a back propagation algorithm to update the policy parameters of the intelligent agent. In some other embodiments, the intelligent agent is trained only based on the operation reward. Illustratively, the computer device optimizes the operation reward by a policy gradient algorithm to update the policy parameters of the intelligent agent. The embodiments of the present application do not limit this.

[0182] In the embodiments, by determining the supervision loss corresponding to the predicted interface operation and training the intelligent agent based on the supervision loss and the operation reward, compared with the single reward optimization using only the operation reward, after adding the supervision loss, the intelligent agent can not only be guided to optimize the selection strategy of the interface operation, but also be guided to learn the reasoning logic of the extended path, thereby improving the performance and learning efficiency of the intelligent agent.

[0183] As Figure 8 shown, it shows a flowchart of an intelligent agent training method provided by another illustrative embodiment of the present application, which can be executed by the computer device described above. As shown, the method includes an offline data expansion phase and an online training loop phase, which are as follows: The following is the offline data expansion phase, which does not require interaction with the real environment.

[0184] Step 811, obtaining expert trajectory data.

[0185] The expert trajectory includes a series of time steps for performing the training task, and each time step corresponds to an interface at the current time and a reference interface operation performed on the interface at the current time.

[0186] Step 812, teacher model diversity expansion.

[0187] For each time step in the expert trajectory, the computer device inputs the training task and the interface screenshot of the interface corresponding to the time step to the teacher model. The teacher model determines the target to be achieved according to the training task, determines the interface corresponding to the current time step according to the interface screenshot, and further determines the task execution scene of the training task. The teacher model infers the candidate interface operation performed on the interface corresponding to the current time step, and the corresponding expansion thought chain text, which is the inference process of the teacher model generating the expanded interface operation.

[0188] Step 813, action cleaning and vectorization.

[0189] Unlike the reference interface operation corresponding to the time step in the expert trajectory, which is a high-reliability interface operation specified for the training task, some of the expanded interface operations generated by the teacher model may not be suitable for execution on the graphical user interface corresponding to the training task, or may not be suitable as input for the intelligent agent. After the teacher model generates the expanded interface operation, the computer device removes the expanded interface operations that cannot be used.

[0190] In some embodiments, the computer device calculates the expanded thought chain text vector corresponding to the expanded thought chain text in the offline data expansion phase, avoiding calculating the thought chain text vector corresponding to the thought chain text during training, which prolongs the training time.

[0191] Step 814, constructing an expanded reference set.

[0192] In some embodiments, after the teacher model generates the extended interface operations and corresponding extended thought chain texts for each time step in the expert trajectory, and calculates the extended thought chain text vectors corresponding to the extended thought chain texts, the computer device creates an extended reference set to store the extended interface operations, extended thought chain texts, and corresponding extended thought chain text vectors for each time step, so as to facilitate the agent to read relevant data during the online training loop phase.

[0193] In summary, by performing complex inference tasks and vectorizing thought chain text during the offline data expansion phase, the real-time interference of complex calculations on the training process during the online training loop phase is effectively avoided, thus accelerating the training efficiency of the agent.

[0194] The following is the online training loop phase, in which the agent needs to interact with the real environment.

[0195] Step 821: Obtain the time step in the expert trajectory.

[0196] The expert trajectory comprises multiple time steps, each corresponding to the current interface and the reference interface operations performed when executing the training task on that current interface. Learning is conducted on a time-step basis during agent training.

[0197] Step 822, Agent prediction.

[0198] Based on the interface and training task corresponding to the current time step, the intelligent agent can correctly recognize the current task execution scenario, reason and generate the predicted interface operation to be executed at the current time step, and at the same time generate the predicted thought chain text corresponding to the predicted interface operation.

[0199] Step 823, precise matching.

[0200] At the current time step, the agent first uses the reference interface operation as the target of the predicted interface operation, and matches the reference interface operation and the corresponding predicted interface operation at the current time step.

[0201] Step 824, obtain the baseline reward.

[0202] If the reference interface operation and the corresponding predicted interface operation match at the current time step, the agent's strategy of executing the corresponding predicted interface operation at the current time step conforms to the expert strategy of executing the corresponding reference interface operation, and the benchmark reward is used as the operation reward for the current time step.

[0203] Step 825, physical matching.

[0204] In the case of a failure in the precise matching, the computer device matches the predicted interface operation corresponding to the time step with each extended interface operation. The extended interface operation corresponding to the time step can include multiple, and needs to be matched with each extended interface operation respectively.

[0205] Step 826, semantic matching.

[0206] In the case of a successful physical matching, in order to ensure the correctness of the strategy of the agent performing the corresponding predicted interface operation in the current time step, the computer device matches the corresponding predicted thinking chain text and the extended thinking chain text for each extended interface operation matched with the predicted interface operation.

[0207] Step 827, obtain the discounted reward, and knowledge distillation.

[0208] In the case of a successful semantic matching, the strategy of the agent performing the corresponding predicted interface operation in the current time step conforms to the extended strategy of performing the corresponding extended interface operation, but in order to express the reserved trust for the non-expert trajectory, the obtained discounted reward is less than the benchmark reward obtained in the case of a precise matching hit.

[0209] In the case of a semantic matching, in order to speed up the training of the agent, the agent is subjected to knowledge distillation. The extended thinking chain text corresponding to the successful semantic matching is taken as supervision to determine the supervision loss of the predicted interface operation, and guide the agent to learn the reasoning logic of the teacher model.

[0210] Step 828, obtain the zero reward.

[0211] In the case of a failure in the physical matching or the semantic matching, the strategy of the agent performing the corresponding predicted interface operation in the current time step does not conform to the expert idea of performing the corresponding reference interface operation, nor does it conform to the extended idea of performing the corresponding extended interface operation, and the zero reward is obtained, indicating that the decision made by the agent in the current time step needs to be improved.

[0212] Step 829, reset the task execution state.

[0213] After the agent performs the predicted interface operation, since the predicted interface operation corresponding to the current time step can not match the corresponding reference interface operation, in order to be able to continue to learn the reference interface operation corresponding to the subsequent time step in the expert trajectory, the computer device updates the task execution state of the training task, and modifies the current interface to the interface after performing the reference interface operation corresponding to the previous time step.

[0214] Step 830, update the strategy.

[0215] After resetting the task execution state, the training at the current time step ends, and the agent needs to be optimized based on the operation reward and the supervision loss corresponding to the current time step, and the computer device updates the policy parameters of the agent.

[0216] After updating the policy, the training and optimization at the current time step end, and the training and optimization at the next time step start.

[0217] In summary, only the matching of the interface operation and the similarity calculation are performed in the online training loop phase, so that the calculation overhead of the online loop training phase is controlled at a low level, and the three matching of the accurate matching, the physical matching, and the semantic matching are sequentially performed, thereby improving the learning efficiency and accuracy of the agent.

[0218] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, refer to the method embodiments of the present application.

[0219] As shown in Figure 9 FIG. 1, which shows a structural schematic diagram of an agent training apparatus provided by an illustrative embodiment of the present application. The apparatus includes a generation module 910, an acquisition module 920, a determination module 930, and a training module 940.

[0220] The generation module 910 is configured to generate an extended interface operation corresponding to a time step in an expert trajectory for the time step, the time step in the expert trajectory corresponding to a reference interface operation performed in an interface when performing a training task, and the extended interface operation corresponding to the same time step being different from the reference interface operation; The acquisition module 920 is configured to acquire a predicted interface operation performed by an agent at different time steps when performing a training task, the agent being used to perform an interactive task in the interface; The determination module 930 is configured to determine an operation reward corresponding to the same time step, taking the extended interface operation and the reference interface operation corresponding to the time step as targets of the predicted interface operation; The training module 940 is configured to train the agent based on the operation reward.

[0221] In some embodiments, the determination module 930 is further configured to match the reference interface operation and the predicted interface operation corresponding to the time step; In the case where the reference interface operation and the predicted interface operation match, the reference reward is determined as the operation reward corresponding to the time step; In the case where the reference interface operation and the predicted interface operation do not match, the predicted interface operation corresponding to the time step is matched with each extended interface operation; In a case where there is at least one extended interface operation matching the predicted interface operation, the operation reward corresponding to the time step is determined based on the benchmark reward, wherein the operation reward determined based on the benchmark reward is less than the benchmark reward.

[0222] In some embodiments, the determining module 930 is further configured to perform operation type and operation position matching on the reference interface operation and the predicted interface operation corresponding to the time step. In a case where the operation type of the reference interface operation matches the operation type of the predicted interface operation, and the distance between the operation positions of the reference interface operation and the predicted interface operation is less than the distance threshold, it is determined that the reference interface operation and the predicted interface operation match. For each extended interface operation corresponding to the time step, operation type and operation position matching is performed on the extended interface operation and the predicted interface operation. In a case where the operation type of the extended interface operation matches the operation type of the predicted interface operation, and the distance between the operation positions of the extended interface operation and the predicted interface operation is less than the distance threshold, it is determined that the extended interface operation and the predicted interface operation match.

[0223] In some embodiments, the predicted interface operation corresponds to predicted thought chain text, the predicted thought chain text being used to represent the reasoning process of the agent generating the predicted interface operation, and the extended interface operation corresponds to extended thought chain text, the extended thought chain text being used to represent the reasoning process of generating the extended interface operation. The determining module 930 is further configured to, in a case where there is at least one extended interface operation matching the predicted interface operation, for each extended interface operation matching the predicted interface operation, determine the thought chain text similarity between the extended thought chain text corresponding to the extended interface operation and the predicted thought chain text. In a case where there is at least one thought chain text similarity between the extended thought chain text and the predicted thought chain text reaching the similarity threshold, the operation reward corresponding to the time step is determined based on the benchmark reward.

[0224] In some embodiments, the determining module 930 is further configured to determine an extended thought chain text vector of the extended thought chain text corresponding to the extended interface operation, and a predicted thought chain text vector of the predicted thought chain text. The thought chain text similarity is determined based on the cosine similarity between the extended thought chain text vector and the predicted thought chain text vector.

[0225] In some embodiments, the determining module 930 is further configured to determine the operation reward corresponding to the time step based on the benchmark reward and a conversion factor corresponding to the time step, the conversion factor including at least one of a penalty factor and a progress factor, the penalty factor being used to represent the punishment for executing the extended interface operation outside the reference interface operation, and the progress factor being used to represent the progress of the time step in the expert trajectory.

[0226] In some embodiments, the predicted interface operation corresponds to predicted thought chain text, and the predicted thought chain text is used to represent the reasoning process of the agent when generating the predicted interface operation; the extended interface operation corresponds to extended thought chain text, and the extended thought chain text is used to represent the reasoning process when the extended interface operation is generated. The determination module 930 is further configured to, in the case that there is at least one extended interface operation matching the predicted interface operation, for the extended interface operation matching the predicted interface operation, determine a supervision loss corresponding to the predicted interface operation, with the extended thought chain text corresponding to the extended interface operation as the supervision of the predicted thought chain text. The training module 940 is further configured to train the agent based on the supervision loss corresponding to the predicted interface operation and the operation reward.

[0227] In some embodiments, the generation module 910 is further configured to execute the training task by the teacher model, and generate the candidate interface operation corresponding to each time step in the expert trajectory. The illegal interface operation in the candidate interface operation is filtered to obtain the extended interface operation, and the illegal interface operation includes an interface operation that does not conform to the input data format specification and / or does not belong to the set of legal interface operations.

[0228] In some embodiments, the generation module 910 is further configured to input the training task and the interface screenshot of the interface corresponding to the time step in the expert trajectory into the teacher model to obtain the candidate interface operation output by the teacher model corresponding to the time step.

[0229] In some embodiments, the acquisition module 920 is further configured to, at the first time step, acquire the predicted interface operation executed by the agent when executing the training task. At the i-th time step, reset the task execution state of the training task based on the reference interface operation corresponding to the (i-1)-th time step, and acquire the predicted interface operation executed by the agent when executing the training task, i being an integer greater than 1.

[0230] In summary, by pre-generating the extended interface operation different from the reference interface operation corresponding to each time step in the expert trajectory when training the agent, the reference interface operation and the extended interface operation corresponding to the time step can be used as the target of the predicted interface operation generated by the agent in the subsequent process of training the agent to determine the operation reward and train the agent, so that the agent can learn the extended interface operation in addition to the reference interface operation in the expert trajectory, break the dependence of the agent on single training data, avoid overfitting of the agent, and improve the generalization ability of the agent.

[0231] Optionally, the method provided in the embodiments of the present application can be executed by a chip, which can be an AI chip, a GPU chip, a Neural network Processing Unit (NPU) chip, or the like, and the chip has the functions of model inference and model training. The embodiments of the present application do not limit this.

[0232] Please refer to Figure 10 which shows a structural block diagram of a computer device 1000 provided in an example embodiment of the present application. The computer device 1000 can be a terminal or a server.

[0233] Generally, the computer device 1000 includes a processor 1001 and a memory 1002.

[0234] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1001 can be implemented in at least one of a hardware form of a Digital Signal Processing (DSP), a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA). The processor 1001 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a Central Processing Unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a Graphics Processing Unit (GPU). The GPU is used to render and draw the content required to be displayed on the display screen. In some embodiments, the processor 1001 can further include an AI processor for processing machine learning-related computing operations.

[0235] In some embodiments, the processor 1001 can execute an agent training task by using the agent training method provided in the above embodiments.

[0236] The memory 1002 can include one or more computer-readable storage media that can be tangible and non-transitory. The memory 1002 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices.

[0237] In some embodiments, the computer device 1000 can further include a peripheral device interface 1003 and at least one peripheral device.

[0238] Those skilled in the art can understand that, Figure 10 The structure shown in the figure does not constitute a limitation on the computer device 1000, and can include more or fewer components than the figure, or combine certain components, or adopt a different component arrangement.

[0239] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program to instruct related hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0240] In another aspect, the embodiments of the present application provide a computer device, which comprises a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the agent training method provided in the embodiments of the present application.

[0241] In another aspect, the embodiments of the present application provide a computer readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the agent training method provided in the embodiments of the present application.

[0242] The above is only optional embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an intelligent agent, characterized in that, The method includes: For each time step in the expert trajectory, an extended interface operation corresponding to the time step is generated. The time step in the expert trajectory corresponds to the reference interface operation executed in the interface when the training task is performed, and the extended interface operation corresponding to the same time step is different from the reference interface operation. The AI ​​agent performs predicted interface operations at different time steps when executing the training task, and the AI ​​agent is used to perform interactive tasks in the interface. For the same time step, the operation reward corresponding to the time step is determined by taking the extended interface operation and the reference interface operation corresponding to the time step as the target of the predicted interface operation. The agent is trained based on the rewards given for the aforementioned operations.

2. The method according to claim 1, characterized in that, The step of determining the operation reward corresponding to the time step by using the extended interface operation and the reference interface operation corresponding to the time step as the target of the predicted interface operation includes: Match the reference interface operation and the prediction interface operation corresponding to the time step; When the reference interface operation and the prediction interface operation match, the baseline reward is determined as the operation reward corresponding to the time step; If the reference interface operation and the prediction interface operation do not match, the prediction interface operation corresponding to the time step is matched with each of the extended interface operations. In the case where there is at least one extended interface operation that matches the prediction interface operation, the operation reward corresponding to the time step is determined based on the benchmark reward, wherein the operation reward determined based on the benchmark reward is less than the benchmark reward.

3. The method according to claim 2, characterized in that, The matching of the reference interface operation and the prediction interface operation corresponding to the time step includes: Match the operation type and operation position of the reference interface operation and the prediction interface operation corresponding to the time step; If the operation types of the reference interface operation and the predicted interface operation match, and the distance between the operation positions of the reference interface operation and the predicted interface operation is less than a distance threshold, then the reference interface operation and the predicted interface operation are determined to be matched. The matching of the prediction interface operation corresponding to the time step with each of the extended interface operations includes: For each of the extended interface operations corresponding to the time step, the operation type and operation position of the extended interface operation and the prediction interface operation are matched. If the operation types of the extended interface operation and the predicted interface operation match, and the distance between the operation positions of the extended interface operation and the predicted interface operation is less than a distance threshold, then the extended interface operation and the predicted interface operation are determined to be a match.

4. The method according to claim 2, characterized in that, The prediction interface operation corresponds to a prediction thought chain text, which is used to characterize the reasoning process when the agent generates the prediction interface operation. The extended interface operation corresponds to an extended thought chain text, which is used to characterize the reasoning process when generating the extended interface operation. The step of determining the operation reward corresponding to the time step based on the baseline reward when at least one extended interface operation matches the prediction interface operation includes: In the case where there is at least one extended interface operation that matches the predicted interface operation, for each extended interface operation that matches the predicted interface operation, the similarity between the extended thought chain text corresponding to the extended interface operation and the thought chain text of the predicted thought chain text is determined. If at least one extended thought chain text has a similarity threshold with the predicted thought chain text, the operation reward corresponding to the time step is determined based on the baseline reward.

5. The method according to claim 4, characterized in that, The step of determining the similarity between the extended thought chain text corresponding to the extended interface operation and the predicted thought chain text includes: Determine the extended thought chain text vector of the extended thought chain text corresponding to the extended interface operation, and the predicted thought chain text vector of the predicted thought chain text. The text similarity of the thought chain is determined based on the cosine similarity between the extended thought chain text vector and the predicted thought chain text vector.

6. The method according to any one of claims 2 to 5, characterized in that, The step of determining the operation reward corresponding to the time step based on the benchmark reward includes: Based on the baseline reward and the conversion factor corresponding to the time step, the operation reward corresponding to the time step is determined. The conversion factor includes at least one of a penalty factor and a progress factor. The penalty factor is used to characterize the penalty for performing the extended interface operation other than the reference interface operation, and the progress factor is used to characterize the progress of the time step in the expert trajectory.

7. The method according to any one of claims 2 to 5, characterized in that, The prediction interface operation corresponds to a prediction thought chain text, which is used to characterize the reasoning process when the agent generates the prediction interface operation. The extended interface operation corresponds to an extended thought chain text, which is used to characterize the reasoning process when generating the extended interface operation. The method further includes: In the presence of at least one extended interface operation that matches the predicted interface operation, for the extended interface operation that matches the predicted interface operation, the extended thought chain text corresponding to the extended interface operation is used as the supervision of the predicted thought chain text, and the supervision loss corresponding to the predicted interface operation is determined. Training the agent based on the operation rewards includes: The agent is trained based on the supervised loss and the operation reward corresponding to the operation on the prediction interface.

8. The method according to any one of claims 1 to 5, characterized in that, The step of generating an extended interface operation corresponding to a time step in the expert trajectory includes: The training task is executed by the teacher model to generate candidate interface operations corresponding to each time step in the expert trajectory. The candidate interface operations are filtered to obtain the extended interface operations. The illegal interface operations include those that do not conform to the input data format specifications and / or do not belong to the set of legal interface operations.

9. The method according to claim 8, characterized in that, The step of executing the training task through the teacher model to generate candidate interface operations corresponding to the time steps in the expert trajectory includes: Input the screenshots of the interfaces corresponding to the time steps in the training task and the expert trajectory into the teacher model to obtain the candidate interface operations corresponding to the time steps output by the teacher model.

10. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the prediction interface operations performed by the agent at different time steps when executing the training task includes: In the first time step, the prediction interface operation performed by the agent when executing the training task is obtained; At time step i, based on the reference interface operation corresponding to time step i-1, the task execution state of the training task is reset, and the prediction interface operation executed by the agent when executing the training task is obtained, where i is an integer greater than 1.

11. An intelligent agent training device, characterized in that, The device includes: The generation module is configured to generate an extended interface operation corresponding to a time step in the expert trajectory. The time step in the expert trajectory corresponds to a reference interface operation executed in the interface when performing the training task, and the extended interface operation corresponding to the same time step is different from the reference interface operation. The acquisition module is configured to acquire the prediction interface operations performed by the agent at different time steps when the agent performs the training task, wherein the agent is used to perform interactive tasks in the interface; The determination module is configured to, for the same time step, use the extended interface operation and the reference interface operation corresponding to the time step as the target of the predicted interface operation, and determine the operation reward corresponding to the time step. The training module is configured to train the agent based on the operation rewards.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the agent training method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the agent training method as described in any one of claims 1 to 10.

Images