Method and apparatus for training a model

Abstract

The embodiments of the present disclosure disclose a method and apparatus for training a model. A specific implementation of the method includes: obtaining a sample state transition trajectory data set; performing the following training steps: selecting a target state value from the sample state transition trajectory data, and using the target state value, performing the following processing steps: inputting the target state value into the initial action selection model to obtain a target action value; inputting the target state value and the target action value into the environment model to obtain a simulation subsequent state value and a simulation reward value; using the simulation subsequent state value as the target state value, repeatedly performing the processing steps until the preset processing conditions are met, and obtaining simulation state transition trajectory data; forming a training sample set from the sample single-step state transition data and the simulation single-step state transition data; and using the training sample set to train the initial action selection model to obtain a trained action selection model. This implementation can train an action selection model with stronger generalization capabilities.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to methods and devices for training models and methods and devices for controlling intelligent agents. Background Art

[0002] Reinforcement learning is an agent learning in a trial-and-error manner, obtaining rewards by interacting with the environment, and then using the rewards to guide behavior, with the goal of enabling the agent to obtain the maximum reward.

[0003] Currently, in the research of reinforcement learning, due to many practical factors such as high experimental costs and safety, it is often impossible to directly interact with the real environment, and reinforcement learning can only be carried out using collected offline experience data. Summary of the Invention

[0004] The embodiments of the present disclosure provide methods and devices for training models and methods and devices for controlling intelligent agents.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for training a model, the method comprising: obtaining a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value; obtaining an initial action selection model and an environment model; and performing the following training steps based on the initial action selection model and the environment model: selecting a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and performing the following processing using the target state value. Processing steps: input the target state value into the initial action selection model to obtain the action value as the target action value; input the target state value and the obtained target action value into the environment model to obtain the simulation subsequent state value and the simulation reward value; use the obtained simulation subsequent state value as the target state value, repeat the processing steps until the preset processing conditions are met, and obtain the simulation state transfer trajectory data; form a training sample set with the sample single-step state transfer data in the sample state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data; use the formed training sample set to train the initial action selection model to obtain a trained action selection model.

[0006] In some embodiments, the method further includes: determining whether a preset training completion condition is met; in response to not meeting the preset training completion condition, using the trained action selection model as the initial action selection model and continuing to execute the training step.

[0007] In some embodiments, obtaining an environmental model includes: obtaining at least two environmental models; and inputting the target state value and the obtained target action value into the environmental model, and obtaining a simulated subsequent state value and a simulated reward value includes: inputting the target state value and the obtained target action value into at least two environmental models respectively, obtaining at least two candidate subsequent state values ​​and at least two candidate reward values; generating a simulated subsequent state value based on the at least two candidate subsequent state values, and generating a simulated reward value based on the at least two candidate reward values.

[0008] In some embodiments, obtaining the environment model includes: obtaining an initial environment model; and training the initial environment model using sample single-step state transition data in the sample state transition trajectory data set to obtain the environment model.

[0009] In some embodiments, combining the sample single-step state transfer data in the sample state transfer trajectory data and the simulated single-step state transfer data in the simulation state transfer trajectory data into a training sample set includes: selecting sample state transfer trajectory data that matches the simulation state transfer trajectory data from the sample state transfer trajectory data set; and combining the sample single-step state transfer data in the selected sample state transfer trajectory data and the simulated single-step state transfer data in the simulation state transfer trajectory data into a training sample set.

[0010] In some embodiments, selecting the sample state transition trajectory data that matches the obtained simulation state transition trajectory data from the sample state transition trajectory data set includes: selecting the sample state transition trajectory data whose trajectory starting point is the same as the trajectory starting point of the simulation state transition trajectory data from the sample state transition trajectory data set.

[0011] In some embodiments, selecting sample state transition trajectory data that matches the obtained simulation state transition trajectory data from the sample state transition trajectory data set includes: selecting sample state transition trajectory data from the sample state transition trajectory data set whose corresponding trajectory length is the same as the trajectory length corresponding to the obtained simulation state transition trajectory data.

[0012] In some embodiments, the initial action selection model is trained using the composed training sample set to obtain the trained action selection model, which includes: selecting training samples from the composed training sample set; determining the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training samples; using the determined difference as a penalty term for training the initial action selection model, training the initial action selection model, and obtaining the trained action selection model.

[0013] In the second aspect, an embodiment of the present disclosure provides a method for controlling an intelligent agent, the method comprising: obtaining a state value corresponding to a current environmental state; inputting the obtained state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate an action that the target intelligent agent can perform under the current environmental state, and the action selection model is a model trained using the method of any embodiment of the method described in the first aspect above; controlling the target intelligent agent to perform the action indicated by the obtained action value.

[0014] In a third aspect, an embodiment of the present disclosure provides a device for training a model, the device comprising: a first acquisition unit, configured to acquire a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value; a second acquisition unit, configured to acquire an initial action selection model and an environment model; a first execution unit, configured to perform the following training steps based on the initial action selection model and the environment model: selecting a sample state value as a target state value from the sample state transition trajectory data included in the sample state transition trajectory data set; , and using the target state value, perform the following processing steps: input the target state value into the initial action selection model to obtain the action value as the target action value; input the target state value and the obtained target action value into the environment model to obtain the simulation subsequent state value and the simulation reward value; use the obtained simulation subsequent state value as the target state value, repeat the processing steps until the preset processing conditions are met, and obtain the simulation state transfer trajectory data; form a training sample set with the sample single-step state transfer data in the sample state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data; use the formed training sample set to train the initial action selection model to obtain a trained action selection model.

[0015] In some embodiments, the device also includes: a determination unit configured to determine whether a preset training completion condition is met; a second execution unit configured to continue executing the training step using the trained action selection model as the initial action selection model in response to the preset training completion condition not being met.

[0016] In some embodiments, the second acquisition unit is further configured to: acquire at least two environmental models; and the first execution unit is further configured to: input the target state value and the obtained target action value into at least two environmental models respectively to obtain at least two candidate subsequent state values ​​and at least two candidate reward values; generate a simulated subsequent state value based on the at least two candidate subsequent state values, and generate a simulated reward value based on the at least two candidate reward values.

[0017] In some embodiments, the second acquisition unit includes: an acquisition module configured to acquire an initial environment model; a training module configured to train the initial environment model using sample single-step state transition data in the sample state transition trajectory data set to obtain an environment model.

[0018] In some embodiments, the first execution unit is further configured to: select sample state transition trajectory data that matches the simulation state transition trajectory data from the sample state transition trajectory data set; and form a training sample set with the sample single-step state transition data in the selected sample state transition trajectory data and the simulated single-step state transition data in the simulation state transition trajectory data.

[0019] In some embodiments, the first execution unit is further configured to: select, from the sample state transition trajectory data set, sample state transition trajectory data including a trajectory starting point that is the same as a trajectory starting point of the simulation state transition trajectory data.

[0020] In some embodiments, the first execution unit is further configured to: select, from the sample state transition trajectory data set, sample state transition trajectory data whose corresponding trajectory length is the same as the trajectory length corresponding to the obtained simulation state transition trajectory data.

[0021] In some embodiments, the first execution unit is further configured to: select a training sample from the composed training sample set; determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training sample; use the determined difference as a penalty term for training the initial action selection model, train the initial action selection model, and obtain a trained action selection model.

[0022] In a fourth aspect, an embodiment of the present disclosure provides a device for controlling an intelligent agent, the device comprising: a third acquisition unit, configured to acquire a state value corresponding to a current environmental state; an input unit, configured to input the acquired state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate an action that can be performed by the target intelligent agent under the current environmental state, and the action selection model is a model trained using the method of any embodiment of the method described in the first aspect above; and a control unit, configured to control the target intelligent agent to perform the action indicated by the acquired action value.

[0023] In the fifth aspect, an embodiment of the present disclosure provides a server, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method of any embodiment of the method described in the first and second aspects above.

[0024] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements a method of any embodiment of the methods described in the first and second aspects above.

[0025] The method and apparatus for training a model provided by the embodiments of the present disclosure obtain a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value, and then obtain an initial action selection model and an environmental model, and then perform the following training steps based on the initial action selection model and the environmental model: select the sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as the target state value, and perform the following processing steps using the target state value: input the target state value into the initial action selection model to obtain the action value as the target action value; input the target state value and the obtained target action value into the environmental model to obtain the simulated subsequent state value and the simulated reward value; and input the obtained simulated subsequent state value and the simulated reward value into the environmental model. The continuous state value is used as the target state value, and the processing steps are repeatedly executed until the preset processing conditions are met to obtain the simulation state transfer trajectory data; the sample single-step state transfer data in the sample state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data are composed of a training sample set; the initial action selection model is trained using the composed training sample set to obtain a trained action selection model, so that the simulation training samples corresponding to the simulation environment model can be obtained based on the sample state transfer trajectory data, the number of training samples used to train the action selection model is increased, and the full use of the sample state transfer trajectory data is achieved, which is helpful to use the obtained simulation training samples and the real training samples corresponding to the sample state transfer trajectory data to train an action selection model with stronger generalization ability, and then perform more accurate and reliable intelligent body control based on the action selection model.

[0026] The embodiments of the present disclosure provide a method and apparatus for controlling an intelligent agent, which obtains a state value corresponding to the current environmental state and then inputs the obtained state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate the action that the target intelligent agent can perform under the current environmental state. The action selection model is a model trained using the method of any embodiment of the method described in the first aspect above. Finally, the target intelligent agent is controlled to perform the action indicated by the obtained action value, thereby achieving more accurate and reliable intelligent agent control based on an action selection model with stronger generalization ability, thereby improving the accuracy and reliability of the intelligent agent control. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0028] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0029] Figure 2 is a flowchart of an embodiment of a method for training a model according to the present disclosure;

[0030] Figure 3 is a schematic diagram of an application scenario of a method for training a model according to an embodiment of the present disclosure;

[0031] Figure 4 is a flow chart of one embodiment of a method for controlling an intelligent agent according to the present disclosure;

[0032] Figure 5 is a schematic diagram of an application scenario of a method for controlling an intelligent agent according to an embodiment of the present disclosure;

[0033] Figure 6 is a structural diagram of an embodiment of an apparatus for training a model according to the present disclosure;

[0034] Figure 7 This is a schematic diagram of a structure of an embodiment of a device for controlling an intelligent agent according to the present disclosure.

[0035] Figure 8 It is a structural diagram of a computer system suitable for implementing a server of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0039] Figure 1An exemplary system architecture 100 is shown to which embodiments of the method for training a model, the apparatus for training a model, the method for controlling an agent, or the apparatus for controlling an agent of the present disclosure may be applied.

[0040] like Figure 1 As shown, system architecture 100 may include agents 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between agents 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0041] Agents 101, 102, 103 can interact with server 105 through network 104 to receive or send messages, etc. Agents 101, 102, 103 can be hardware or software. When agents 101, 102, 103 are hardware, they can be various intelligent electronic devices, including but not limited to smart phones, smart computers, smart robots, smart refrigerators, unmanned vehicles, drones, etc. When agents 101, 102, 103 are software, they can be installed in the intelligent electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0042] Server 105 can be a server that provides various services, such as a model training server that trains a model using the sample state transition trajectory data sets sent by agents 101, 102, and 103. The model training server can use the received sample state transition trajectory data sets and other data to train a model and obtain a trained model (e.g., an action selection model). Alternatively, server 105 can also be a backend server that controls agents 101, 102, and 103. The backend server can analyze and process the acquired data, such as the state value of the current environment state, and obtain processing results (e.g., action values), and then control agents 101, 102, and 103 based on the processing results.

[0043] It should be noted that the method for training a model provided in the embodiments of the present disclosure can be executed by the server 105, and accordingly, the device for training a model can be set in the server 105. In addition, the method for controlling an intelligent agent provided in the embodiments of the present disclosure can be executed by the server 105, and accordingly, the device for controlling an intelligent agent can be set in the server 105.

[0044] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., multiple software or software modules used to provide distributed services), or as a single software or software module. No specific limitations are given here.

[0045] It should be understood that Figure 1 The number of agents, networks and servers in the embodiment is only for illustration. Any number of agents, networks and servers may be used depending on the implementation requirements.

[0046] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for training a model according to the present disclosure. The method for training a model includes the following steps:

[0047] Step 201: Acquire a preset sample state transition trajectory data set.

[0048] In this embodiment, the execution subject of the method for training the model (eg Figure 1 The server shown in FIG. 1 can obtain a preset sample state transition trajectory data set remotely or locally via a wired or wireless connection. The sample state transition trajectory data set can include multiple sample state transition trajectory data. The sample state transition trajectory data can be data obtained from offline experience data. The sample state transition trajectory data can include at least one sample single-step state transition data. The sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value.

[0049] The sample state value can be used to characterize the sample state. The sample state specifically refers to the sample environment state. It should be noted that, in practice, various data can be used to characterize the sample state, such as an environmental image or an environmental temperature value. In this disclosure, data characterizing the sample state is referred to as the sample state value.

[0050] The sample action value can be used to represent the sample action performed by the sample agent in the sample state. The sample agent can be a predetermined variety of smart devices, such as a smart refrigerator, a smartphone, a smart car, and so on. In practice, various data can be used to represent the sample action performed by the sample agent in the sample state. For example, the sample action can be represented by the direction and distance traveled by the sample agent, or by the temperature rise of the sample agent. In this disclosure, the data representing the sample action performed by the sample agent is referred to as the sample action value.

[0051] The subsequent sample state value is used to represent the subsequent sample state to which the sample state transitions after the sample agent executes the sample action. For example, if the sample state value is -20 degrees Celsius and the sample action executed by the sample agent is to increase the temperature by 10 degrees Celsius, then the subsequent sample state value is -10 degrees Celsius. It will be understood that, similar to the sample state value, the present disclosure refers to data representing the subsequent sample state as the subsequent sample state value.

[0052] The sample reward value can be used to represent the sample reward obtained by the sample agent when performing the sample action in the sample state. The sample reward can be determined based on the quality of the environment state transition. For example, when the transition from the sample state to the subsequent sample state is in the direction of completing the task, a relatively large sample reward (corresponding to a large sample reward value) can be allocated to the sample action of the sample agent; when the transition from the sample state to the subsequent sample state is in the opposite direction of completing the task, a relatively small sample reward (corresponding to a small sample reward value) can be allocated to the sample action of the sample agent.

[0053] In this embodiment, the sample single-step state transition data corresponds to one state transition, and a plurality of continuous sample single-step state transition data can constitute sample state transition trajectory data, wherein the sample state transition trajectory data can record the trajectory of the state transition.

[0054] As an example, the sample state transition trajectory data can be (s1, a1, r1, s2, a2, r2, s3, a3, r3, s4), where si is the sample state value used to represent the sample state, ai is the sample action value used to represent the sample action, and ri is the sample reward value used to represent the sample reward, with i = 1, 2, 3, ... n. Furthermore, it can be seen that the sample state transition trajectory data records the sample state transition from s1 to s2, then from s2 to s3, and finally from s3 to s4, that is, the state transition trajectory s1-s2-s3-s4. Each state transition can correspond to a sample single-step state transition data. Therefore, the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3, a3, r3, s4) can include three sample single-step state transition data: (s1, a1, r1, s2), (s2, a2, r2, s3), and (s3, a3, r3, s4).

[0055] Step 202: Acquire an initial action selection model and an environment model.

[0056] In this embodiment, the execution subject can obtain an initial action selection model and an environment model. The initial action selection model can be a predetermined action selection model to be trained. Specifically, the initial selection model can be an untrained action selection model or a trained but incomplete action selection model. The action selection model can be used to select actions for the agent, and specifically can be used to characterize the correspondence between state values ​​and action values. In practice, the state value of the current state is input into the action selection model, and the action selection model can output an action value. The action indicated by the output action value is the action selected by the action selection model for the agent.

[0057] It should be noted that, in practice, an action selection model can be set for a certain agent or a certain type of agent, and the set action selection model can then select actions for the agent or the type of agent. In this disclosure, the agent (a certain agent or a certain type of agent) for which the initial action selection model is obtained can be referred to as the target agent.

[0058] In this embodiment, the environment model can be a simulated environment model obtained by simulating a real environment. Specifically, the environment model can be used to represent the correspondence between state values, action values, subsequent state values, and reward values. In practice, the state value of the current state and the action value corresponding to the action performed by the agent are input into the environment model, and the environment model can output the subsequent state value corresponding to the subsequent state and the reward value corresponding to the reward assigned to the action performed by the agent.

[0059] In this embodiment, various methods can be used to obtain the environment model. For example, a pre-simulated environment model can be obtained from a local or other electronic device that is connected to the communication.

[0060] In some optional implementations of this embodiment, the execution entity may obtain the environment model through the following steps: First, the execution entity may obtain an initial environment model. Then, the execution entity may train the initial environment model using the sample single-step state transition data in the sample state transition trajectory data set obtained in step 201 to obtain the environment model.

[0061] Specifically, the above-mentioned execution entity can use the sample state value and sample action value in the sample single-step state transition data as the input of the initial environment model, and use the subsequent sample state value and sample reward value in the sample single-step state transition data as the expected output of the initial environment model, and use the machine learning method to train the initial environment model until the preset conditions are met (for example, the number of training times reaches the preset number, or the loss value is less than or equal to the preset threshold), and obtain the trained initial environment model as the environment model.

[0062] This implementation uses the acquired sample state transition trajectory data to train the environment model, which can make the obtained environment model closer to the environment that generated the sample state transition trajectory data, and facilitate the subsequent use of the environment model and sample state transition trajectory data to more accurately and reasonably train the initial action selection model.

[0063] In this embodiment, the execution entity may obtain one environment model or at least two environment models.

[0064] In particular, when the execution subject obtains an environment model by using the sample state transition trajectory dataset to train the initial environment model as described in the optional implementation, the original environment model can be obtained, and then the original environment model can be initialized using different model parameters to obtain at least two initial environment models. The execution subject can then use the sample state transition trajectory dataset to train the at least two initial environment models to obtain at least two environment models. The original environment model can be a predetermined environment model to be trained.

[0065] It should be noted that step 202 can be performed before step 201 or after step 201. This disclosure only provides one execution method, which should not be regarded as a limitation of this disclosure.

[0066] Step 203: Perform a training step based on the initial action selection model and the environment model.

[0067] In this embodiment, based on the initial action selection model and the environment model obtained in step 202, the execution entity may perform the following training steps (steps 2031 to 2034):

[0068] Step 2031: Select a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and use the target state value to perform the following processing steps:

[0069] In the first step, the target state value is input into the initial action selection model to obtain the action value as the target action value.

[0070] Here, the target state value can be a state value used to expand the training sample. Specifically, the above-mentioned execution entity can use various methods to select the target state value from the sample state transition trajectory data. As an example, the above-mentioned execution entity can select the target state value from the sample state transition trajectory data by random selection; or, the above-mentioned execution entity can select the starting point of the state transition trajectory from the sample state transition trajectory data as the target state value. For example, if the sample state transition trajectory data is (s1, a1, r1, s2, a2, r2, s3, a3, r3, s4), and the corresponding state transition trajectory is s1-s2-s3-s4, then the starting point s1 of the trajectory can be selected as the target state value.

[0071] It can be understood that the initial action selection model is used to characterize the correspondence between state values ​​and action values. Therefore, by inputting the target state value into the initial action selection model, the action value corresponding to the target state value output by the initial action selection model can be obtained. Furthermore, the above-mentioned execution entity can use the action value output by the initial action selection model as the target action value corresponding to the target state value.

[0072] In the second step, the target state value and the obtained target action value are input into the environment model to obtain the simulation subsequent state value and simulation reward value.

[0073] In this embodiment, the obtained environment model is a simulation environment model obtained by simulating the real environment. Then, the target state value and target action value are input into the environment model to obtain the simulation subsequent state value and simulation reward value fed back by the simulation environment model.

[0074] In some optional implementations of this embodiment, when the execution entity obtains at least two environment models in step 202, the execution entity may obtain a simulated subsequent state value and a simulated reward value by performing the following steps: first, inputting the target state value and the obtained target action value into the at least two environment models, respectively, to obtain at least two candidate subsequent state values ​​and at least two candidate reward values, wherein each of the at least two environment models may output a candidate subsequent state value and a candidate reward value. Then, the execution entity may generate a simulated subsequent state value based on the at least two candidate subsequent state values, and generate a simulated reward value based on the at least two candidate reward values.

[0075] Specifically, the above-mentioned execution entity can use various methods to generate a simulated subsequent state value based on at least two candidate subsequent state values. For example, the above-mentioned execution entity can randomly select a candidate subsequent state value from at least two candidate subsequent state values ​​as the simulated subsequent state value; or, the above-mentioned execution entity can calculate the mean of at least two candidate subsequent state values ​​and use the calculation result as the simulated subsequent state value.

[0076] Similarly, the method for generating a simulated reward value based on at least two candidate reward values ​​may be the same as the above-mentioned method for generating a simulated subsequent state value based on at least two candidate subsequent state values, and will not be described in detail here.

[0077] This implementation method can generate simulated subsequent state values ​​and simulated reward values ​​for expanding training samples based on multiple simulation environment models. In this way, the randomness and diversity of the simulated subsequent state values ​​and simulation reward values ​​can be increased, which helps to expand the distribution space of training samples generated based on the simulated subsequent state values ​​and simulation reward values, and then use the generated training samples to train an action selection model with stronger generalization ability.

[0078] In step 2032, the obtained simulation subsequent state value is used as the target state value, and the processing steps are repeatedly executed until the preset processing conditions are met to obtain simulation state transfer trajectory data.

[0079] In this embodiment, the execution entity may use the subsequent state value of the simulation obtained in step 2031 as a new target state value, and repeatedly execute the above processing steps using the new target state value until a preset processing condition is satisfied, thereby obtaining simulation state transition trajectory data consisting of simulation single-step state transition data. The preset processing condition may be various predetermined conditions, such as the number of executions of the processing step being greater than or equal to a preset number of times, or the difference between the most recently obtained subsequent state value of the simulation and the most recently obtained target state value being less than or equal to a preset threshold (corresponding to convergence).

[0080] In this embodiment, the simulation state transition trajectory data consists of simulation single-step state transition data. The simulation single-step state transition data may include a target state value, a target action value, a simulation subsequent state value, and a simulation reward value. Each simulation single-step state transition data in the simulation state transition trajectory data corresponds to a state transition, and each execution of a processing step corresponds to a transition from the target state value to the simulation subsequent state value. Therefore, each execution of a processing step can correspond to a state transition, that is, to a piece of simulation single-step state transition data.

[0081] As an example, the preset processing condition is that the processing step is executed twice or more. When the processing step is executed for the first time, the target state value 1 can be input into the initial action selection model to obtain the target action value 1. The target action value 1 and the target state value 1 are then input into the environment model to obtain the simulation subsequent state value 2 and the simulation reward value 1. The execution entity can then use the simulation subsequent state value 2 as the new target state value 2 and execute the processing step a second time to obtain the target action value 2, the simulation subsequent state value 3, and the simulation reward value 2. At this time, it is determined that the number of executions of the processing step (twice) is greater than or equal to two times (that is, the preset processing conditions are met), and the execution of the processing step is stopped. The target state value 1, target action value 1, simulation reward value 1, target state value 2, target action value 2, simulation reward value 2 and simulation subsequent state value 3 obtained through the processing step are used to form simulation state transfer trajectory data (S1, A1, R1, S2, A2, R2, S3), where S1 represents the target state value 1; A1 represents the target action value 1; R1 represents the simulation reward value 1; S2 represents the target state value 2; A2 represents the target action value 2; R2 represents the simulation reward value 2; S3 represents the simulation subsequent state value 3.

[0082] It can be understood that in this example, the simulation state transition trajectory data (S1, A1, R1, S2, A2, R2, S3) records two state transitions, namely the transition from S1 to S2 (corresponding to the execution of the first processing step) and the transition from S2 to S3 (corresponding to the execution of the second processing step). Furthermore, the simulation state transition trajectory data (S1, A1, R1, S2, A2, R2, S3) includes two simulation single-step state transition data, namely (S1, A1, R1, S2) and (S2, A2, R2, S3).

[0083] Step 2033 : Combining the sample single-step state transition data in the sample state transition trajectory data and the simulation single-step state transition data in the simulation state transition trajectory data into a training sample set.

[0084] In this embodiment, based on the sample state transition trajectory data obtained in step 201 and the simulation state transition trajectory data obtained in step 2032, the above-mentioned execution entity can respectively use the sample single-step state transition data in the sample state transition trajectory data and the simulated single-step state transition data in the simulation state transition trajectory data as training samples, and then form a training sample set, wherein each training sample in the training sample set is a single-step state transition data (simulated single-step state transition data or sample state transition data).

[0085] Specifically, the execution entity may determine the sample single-step state transition data included in all the sample state transition trajectory data in the sample state transition trajectory data set as the training sample. Alternatively, the execution entity may select target state transition trajectory data from the sample state transition trajectory data set and further determine the sample single-step state transition data in the target state transition trajectory data as the training sample. Various methods may be used to select the target state transition trajectory data.

[0086] In some optional implementations of this embodiment, the above-mentioned execution entity may select sample state transition trajectory data that matches the simulation state transition trajectory data from the sample state transition trajectory data set as the target state transition trajectory data, and then the above-mentioned execution entity may form a training sample set with the sample single-step state transition data in the selected sample state transition trajectory data that matches the simulation state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data.

[0087] Here, the sample state transition trajectory data that matches the simulation state transition trajectory data may be sample state transition trajectory data that is similar to the simulation state transition trajectory data. Specifically, the execution entity may use various methods to select the sample state transition trajectory data that matches the simulation state transition trajectory data from the sample state transition trajectory data set.

[0088] In some optional implementations of this embodiment, the execution entity may select, from the sample state transition trajectory data set, sample state transition trajectory data whose trajectory starting point is the same as the trajectory starting point of the simulation state transition trajectory data as the sample state transition trajectory data that matches the simulation state transition trajectory data. Here, the trajectory starting point refers to the starting point of the state transition trajectory, that is, the initial state value during the state transition process.

[0089] As an example, the simulation state transition trajectory data is (S1, A1, R1, S2, A2, R2, S3); the sample state transition trajectory data set includes the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) and the sample state transition trajectory data (s7, a7, r7, s8, a8, r8, s9), and the state transition trajectory corresponding to the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) is s1-s2-s3; the state transition trajectory corresponding to the sample state transition trajectory data (s7, a7, r7, s8, a8, r8, s9) is The trace is s7-s8-s9; the state transition trajectory corresponding to the simulation state transition trajectory data (S1, A1, R1, S2, A2, R2, S3) is S1-S2-S3. Since S1 and s1 are the same state value, the trajectory starting point included in the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) is the same as the trajectory starting point of the simulation state transition trajectory data. Furthermore, the above-mentioned execution entity can select the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) as the sample state transition trajectory data that matches the simulation state transfer trajectory data.

[0090] In some optional implementations of this embodiment, the execution entity may further select, from the sample state transition trajectory data set, sample state transition trajectory data having a trajectory length identical to the trajectory length corresponding to the obtained simulation state transition trajectory data as the sample state transition trajectory data that matches the simulation state transition trajectory data. Here, the trajectory length refers to the length of the state transition trajectory.

[0091] As an example, the simulation state transition trajectory data is (S1, A1, R1, S2, A2, R2, S3); the sample state transition trajectory data set includes sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) and sample state transition trajectory data (s7, a7, r7, s8), and the state transition trajectory corresponding to the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) is s1-s2-s3; the state transition trajectory corresponding to the sample state transition trajectory data (s7, a7, r7, s8) is s7-s8; the state transition trajectory corresponding to the simulation state transition trajectory data (S1, A1, R1, S2, A2, R2, S3) is S1-S2- S3, it can be seen that the trajectory length of the state transition trajectory corresponding to the sample state transition trajectory data (s1, a1, r1, s2, a2, r2, s3) is 3; the trajectory length of the state transition trajectory corresponding to the sample state transition trajectory data (s7, a7, r7, s8) is 2; the trajectory length of the state transition trajectory corresponding to the simulation state transfer trajectory data is 3, which is the same as the trajectory length of the state transfer trajectory corresponding to the sample state transfer trajectory data (s1, a1, r1, s2, a2, r2, s3). Furthermore, the above-mentioned execution entity can select the sample state transfer trajectory data (s1, a1, r1, s2, a2, r2, s3) as the sample state transfer trajectory data that matches the simulation state transfer trajectory data.

[0092] In particular, to more rigorously select sample state transition trajectory data and more effectively train the initial action selection model, the execution entity may also select, from the sample state transition trajectory data set, sample state transition trajectory data that simultaneously satisfies the selection conditions specified in the two optional implementations as the sample state transition trajectory data that matches the simulation state transition trajectory data. It is understood that the selection conditions specified in the two optional implementations include: 1. The starting point of the included trajectory is the same as the starting point of the trajectory of the simulation state transition trajectory data; 2. The corresponding trajectory length is the same as the trajectory length corresponding to the obtained simulation state transition trajectory data.

[0093] Step 2034: Use the formed training sample set to train the initial action selection model to obtain a trained action selection model.

[0094] In this embodiment, the execution entity may use the training sample set obtained in step 2033 to train the initial action selection model to obtain a trained action selection model.

[0095] Specifically, the execution entity may use the training samples in the training sample set and adopt a reinforcement learning algorithm to train the initial action selection model to obtain a trained action selection model.

[0096] Specifically, the reinforcement learning algorithm defines a value function to represent the expected future rewards under the action selection strategy. The optimal strategy to be found (i.e., the trained action selection model) is the strategy that maximizes the expected value of future rewards at any time. In this embodiment, the above-mentioned execution entity can first adopt the existing actor-critic method to minimize the Bellman error on the single-step state transition data (i.e., the training sample) to learn an action-value function, and then perform strategy update by maximizing the action-value function (i.e., adjusting the parameters of the initial action selection model to obtain the trained action selection model).

[0097] In some optional implementations of this embodiment, in order to make the strategy learned by the reinforcement learning algorithm close to the strategy for generating the sample state transition trajectory data set, the above-mentioned execution entity may further train the initial action selection model through the following steps: First, the above-mentioned execution entity may select training samples from the composed training sample set. Then, the above-mentioned execution entity may determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training samples. Finally, the above-mentioned execution entity may use the determined difference as a penalty term for training the initial action selection model, train the initial action selection model, and obtain a trained action selection model.

[0098] Here, the execution entity may select training samples from the training sample set using various methods, such as a random selection method.

[0099] In this implementation, various methods may be used to determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training samples, for example, the difference may be determined by determining the KL divergence or the Wasserstein distance.

[0100] In practice, a penalty term, also called a regularization term, is added to an objective function (such as the action-value function mentioned above). When the objective function achieves an optimal solution (such as the maximum expected future value), the penalty term reaches its minimum value. For example, if the objective function is y = x, the goal is to maximize y, and the penalty term is z, then the penalty term can be added to the objective function to obtain a new objective function y = xz, so that the smaller z is, the larger y is. Consequently, when y reaches its maximum value, z can reach its minimum value.

[0101] This implementation uses the difference between the distribution of the initial action selection model and the distribution of the data in the selected training samples as a penalty term. This allows the trained action selection model to produce the highest future expected value while also approximating the learned action selection model to the strategy that generated the training samples. As can be appreciated, the strategy that generated the training samples is historically validated and offers greater security and stability. Therefore, by adding a penalty term, this implementation allows the learned strategy to approximate the strategy that generated the training samples, thereby achieving greater security and stability.

[0102] In some optional implementations of this embodiment, after executing step 2034, the execution subject may further perform the following steps: determining whether a preset training completion condition is satisfied. In response to determining that the preset training completion condition is not satisfied, the trained action selection model obtained in step 2034 is used as the initial action selection model, and the above training steps (steps 2031-2034) are continued.

[0103] Here, the preset training completion condition may be a condition pre-set for the training process of the action selection model. When the preset training completion condition is met, the action selection model training is completed. Specifically, the preset training completion condition may be various conditions, such as reaching a preset number of training times or exhausting the training samples in the training sample set.

[0104] Continue to see Figure 3 , Figure 3 It is a schematic diagram of an application scenario of the method for training a model according to this embodiment.

[0105] exist Figure 3 In the application scenario, server 301 can first obtain a preset sample state transition trajectory data set 302, where sample state transition trajectory data set 302 includes sample state transition trajectory data 3021 and sample state transition trajectory data 3022. Sample state transition trajectory data 3021 and 3022 each include at least one sample single-step state transition data. Each sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value.

[0106] Then, the server 301 may obtain the initial action selection model 303 and the environment model 304 .

[0107] Next, the server 301 may perform the following training steps based on the initial action selection model 303 and the environment model 304:

[0108] In the first step, a sample state value is selected from the sample state transition trajectory data included in the sample state transition trajectory data set 302 as the target state value 305, and the target state value 305 is used to perform the following processing steps: the target state value 305 is input into the initial action selection model 303 to obtain the action value as the target action value 306; the target state value 305 and the obtained target action value 306 are input into the environment model 304 to obtain the simulation subsequent state value 307 and the simulation reward value 308.

[0109] In the second step, the obtained simulation subsequent state value 307 is used as the target state value 305, and the processing steps are repeatedly executed until the preset processing conditions are met (for example, the processing steps are executed twice), and the simulation state transfer trajectory data 309 is obtained.

[0110] Then, the server 301 may combine the sample state transition trajectory data 3021 and / or the sample single-step state transition data in the sample state transition trajectory data 3022 and the simulated single-step state transition data in the simulated state transition trajectory data 309 into a training sample set 310 .

[0111] Finally, the server 301 may use the formed training sample set 310 to train the initial action selection model 303 to obtain a trained action selection model 311 .

[0112] The method provided by the above-mentioned embodiments of the present disclosure can obtain simulation training samples corresponding to the simulation environment model based on the sample state transition trajectory data, thereby increasing the number of training samples used to train the action selection model and achieving full utilization of the sample state transition trajectory data. This helps to utilize the obtained simulation training samples and the real training samples corresponding to the sample state transition trajectory data to train an action selection model with stronger generalization ability, thereby achieving more accurate and reliable intelligent body control based on the action selection model.

[0113] Continue to refer Figure 4 , shows a process 400 of an embodiment of a method for controlling an intelligent agent according to the present disclosure. The method for controlling an intelligent agent includes the following steps:

[0114] Step 401: Obtain the state value corresponding to the current environment state.

[0115] In this embodiment, the execution subject of the method for controlling the intelligent agent (eg Figure 1 The server shown in FIG. 1 can obtain the status value corresponding to the current environmental status remotely or locally through a wired connection or a wireless connection.

[0116] In this disclosure, data used to represent the current environmental state is referred to as a state value corresponding to the current environmental state. The data used to represent the current environmental state (i.e., the state value corresponding to the current environmental state) can be various data used to represent the current environmental state, such as an image of the current environment, the temperature of the current environment, etc.

[0117] Step 402: Input the acquired state value into a pre-trained action selection model to obtain an action value.

[0118] In this embodiment, based on the state value obtained in step 401, the execution entity may input the state value into a pre-trained action selection model to obtain an action value. The action value indicates an action that the target agent can perform under the current environmental state. The target agent may be an agent that the execution entity can control. The action selection model may be a model configured for the target agent and used to select an action for the target agent.

[0119] Specifically, the action selection model can be used to Figure 2 The model obtained by training the method of any embodiment in the corresponding embodiment. The specific training process can be referred to Figure 2 The corresponding embodiments will not be described in detail here.

[0120] Step 403: Control the target agent to execute the action indicated by the obtained action value.

[0121] In this embodiment, based on the action value obtained in step 402, the execution entity may control the target agent to perform the action indicated by the action value. Specifically, the execution entity may send an action instruction corresponding to the action value to the target agent to control the target agent to perform the action indicated by the action value.

[0122] As an example, if the action value is -10, the above-mentioned execution entity can send an action instruction to the target smart refrigerator to instruct the target smart refrigerator to lower the temperature by 10 degrees Celsius, so as to control the target smart refrigerator to perform the action of lowering the temperature by 10 degrees Celsius.

[0123] Continue to see Figure 5 , Figure 5 It is a schematic diagram of an application scenario of the method for controlling an intelligent agent according to this embodiment.

[0124] exist Figure 5 In the application scenario, the server 501 first obtains the state value 502 corresponding to the current environment state. Then, the server 501 can input the obtained state value 502 into the pre-trained action selection model 503 to obtain the action value 504, wherein the action value 504 can be used to indicate the action that the target agent 505 can perform under the current environment state. The action selection model 503 is based on Figure 2 Finally, the server 301 can send an action instruction 506 corresponding to the action value 504 to the target agent 505 to control the target agent 505 to perform the action indicated by the obtained action value 504.

[0125] The method provided by the above embodiments of the present disclosure can be based on Figure 2 In the corresponding embodiment, an action selection model with stronger generalization ability is trained, and then the action selection model with stronger generalization ability can be used to achieve more accurate and reliable intelligent agent control, thereby improving the accuracy and reliability of intelligent agent control.

[0126] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training a model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0127] like Figure 6 As shown, the device 600 for training the model of this embodiment includes: a first acquisition unit 601, a second acquisition unit 602 and a first execution unit 603. The first acquisition unit 601 is configured to acquire a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value and a sample reward value; the second acquisition unit 602 is configured to acquire an initial action selection model and an environment model; the first execution unit 603 is configured to perform the following training steps based on the initial action selection model and the environment model: select a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and use the target state value to execute The following processing steps are performed: the target state value is input into the initial action selection model to obtain the action value as the target action value; the target state value and the obtained target action value are input into the environment model to obtain the simulation subsequent state value and the simulation reward value; the obtained simulation subsequent state value is used as the target state value, and the processing steps are repeated until the preset processing conditions are met to obtain the simulation state transfer trajectory data; the sample single-step state transfer data in the sample state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data are combined into a training sample set; the initial action selection model is trained using the combined training sample set to obtain a trained action selection model.

[0128] In this embodiment, the first acquisition unit 601 of the device 600 for training the model can remotely or locally acquire a preset sample state transition trajectory data set via a wired connection or a wireless connection. The sample state transition trajectory data set may include multiple sample state transition trajectory data. The sample state transition trajectory data may be data acquired from offline experience data. The sample state transition trajectory data may include at least one sample single-step state transition data. The sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value.

[0129] The sample state value can be used to represent the sample state. The sample state specifically refers to the sample environment state. The sample action value can be used to represent the sample action performed by the sample agent in the sample state. The sample agent can be a predetermined variety of intelligent devices. The subsequent sample state value is used to represent the subsequent sample state to which the sample state transitions after the sample agent performs the sample action. The sample reward value can be used to represent the sample reward earned by the sample agent for performing the sample action in the sample state. The sample reward can be determined based on the quality of the environmental state transition.

[0130] In this embodiment, the sample single-step state transition data corresponds to one state transition, and a plurality of continuous sample single-step state transition data can constitute sample state transition trajectory data, wherein the sample state transition trajectory data can record the trajectory of the state transition.

[0131] In this embodiment, the second acquisition unit 602 may acquire an initial action selection model and an environment model. The initial action selection model may be a predetermined action selection model to be trained. The action selection model may be used to select actions for the agent, and specifically may be used to characterize the correspondence between state values ​​and action values. The environment model may be a simulated environment model obtained by simulating a real environment. Specifically, the environment model may be used to characterize the correspondence between state values, action values, and subsequent state values ​​and reward values.

[0132] In this embodiment, based on the initial action selection model and the environment model obtained by the second acquisition unit 602, the first execution unit 603 may perform the following training steps (steps 6031 to 6034):

[0133] Step 6031: Select a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and use the target state value to perform the following processing steps:

[0134] In the first step, the target state value is input into the initial action selection model to obtain the action value as the target action value.

[0135] In the second step, the target state value and the obtained target action value are input into the environment model to obtain the simulation subsequent state value and simulation reward value.

[0136] Step 6032: Use the obtained simulation subsequent state value as the target state value, and repeat the processing steps until the preset processing conditions are met to obtain simulation state transfer trajectory data.

[0137] In this embodiment, the first execution unit 603 may use the simulation subsequent state value obtained in step 6031 as a new target state value, and repeatedly execute the above processing steps using the new target state value until a preset processing condition is satisfied, thereby obtaining simulation state transition trajectory data consisting of simulation single-step state transition data. The preset processing condition may be various predetermined conditions.

[0138] In this embodiment, the simulation state transition trajectory data consists of simulation single-step state transition data. The simulation single-step state transition data may include a target state value, a target action value, a simulation subsequent state value, and a simulation reward value. Each simulation single-step state transition data in the simulation state transition trajectory data corresponds to a state transition, and each execution of a processing step corresponds to a transition from the target state value to the simulation subsequent state value. Therefore, each execution of a processing step can correspond to a state transition, that is, to a piece of simulation single-step state transition data.

[0139] Step 6033: The sample single-step state transition data in the sample state transition trajectory data and the simulation single-step state transition data in the simulation state transition trajectory data are combined into a training sample set.

[0140] In this embodiment, based on the sample state transition trajectory data obtained by the first acquisition unit 601 and the simulation state transition trajectory data obtained in step 6032, the first execution unit 603 can respectively use the sample single-step state transition data in the sample state transition trajectory data and the simulated single-step state transition data in the simulation state transition trajectory data as training samples, thereby forming a training sample set, wherein each training sample in the training sample set is a single-step state transition data (simulated single-step state transition data or sample state transition data).

[0141] Step 6034: Use the formed training sample set to train the initial action selection model to obtain a trained action selection model.

[0142] In this embodiment, the first execution unit 603 may use the training sample set obtained in step 6033 to train the initial action selection model to obtain a trained action selection model.

[0143] In some optional implementations of this embodiment, the device 600 also includes: a determination unit (not shown in the figure), configured to determine whether the preset training completion conditions are met; a second execution unit (not shown in the figure), configured to respond to the preset training completion conditions not being met, using the trained action selection model as the initial action selection model, and continuing to execute the training steps.

[0144] In some optional implementations of this embodiment, the second acquisition unit 602 is further configured to: acquire at least two environmental models; and the first execution unit 603 is further configured to: input the target state value and the obtained target action value into at least two environmental models respectively to obtain at least two candidate subsequent state values ​​and at least two candidate reward values; generate a simulated subsequent state value based on the at least two candidate subsequent state values, and generate a simulated reward value based on the at least two candidate reward values.

[0145] In some optional implementations of this embodiment, the second acquisition unit 602 includes: an acquisition module (not shown in the figure), configured to acquire an initial environment model; a training module (not shown in the figure), configured to train the initial environment model using sample single-step state transition data in the sample state transition trajectory data set to obtain an environment model.

[0146] In some optional implementations of this embodiment, the first execution unit 603 is further configured to: select sample state transition trajectory data that matches the simulation state transition trajectory data from the sample state transition trajectory data set; and form a training sample set with the sample single-step state transition data in the selected sample state transition trajectory data and the simulated single-step state transition data in the simulation state transition trajectory data.

[0147] In some optional implementations of this embodiment, the first execution unit 603 is further configured to: select, from the sample state transition trajectory data set, sample state transition trajectory data whose trajectory starting point is the same as that of the simulation state transition trajectory data.

[0148] In some optional implementations of this embodiment, the first execution unit 603 is further configured to: select, from the sample state transition trajectory data set, sample state transition trajectory data whose corresponding trajectory length is the same as the trajectory length corresponding to the obtained simulation state transition trajectory data.

[0149] In some optional implementations of this embodiment, the first execution unit 603 is further configured to: select a training sample from the composed training sample set; determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training sample; use the determined difference as a penalty item for training the initial action selection model, train the initial action selection model, and obtain a trained action selection model.

[0150] It is understood that the units described in the device 600 are similar to those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be repeated here.

[0151] The device 600 provided by the above-mentioned embodiment of the present disclosure can obtain simulation training samples corresponding to the simulation environment model based on the sample state transition trajectory data, thereby increasing the number of training samples used to train the action selection model and achieving full utilization of the sample state transition trajectory data. This helps to utilize the obtained simulation training samples and the real training samples corresponding to the sample state transition trajectory data to train an action selection model with stronger generalization ability, thereby performing more accurate and reliable intelligent body control based on the action selection model.

[0152] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for controlling an intelligent agent. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0153] like Figure 7 As shown, the apparatus 700 for controlling an intelligent agent in this embodiment includes: a third acquisition unit 701, an input unit 702, and a control unit 703. The third acquisition unit 701 is configured to acquire a state value corresponding to the current environment state; the input unit 702 is configured to input the acquired state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate the action that the target intelligent agent can perform in the current environment state. The action selection model is based on Figure 2 The model is obtained by training the method of any embodiment in the corresponding embodiment; the control unit 703 is configured to control the target intelligent agent to perform the action indicated by the obtained action value.

[0154] In this embodiment, the third acquisition unit 701 of the apparatus 700 for controlling an intelligent agent may acquire a state value corresponding to the current environmental state remotely or locally through a wired connection or a wireless connection.

[0155] In this disclosure, data used to represent the current environmental state is referred to as a state value corresponding to the current environmental state. The data used to represent the current environmental state (i.e., the state value corresponding to the current environmental state) can be various data used to represent the current environmental state, such as an image of the current environment, the temperature of the current environment, etc.

[0156] In this embodiment, based on the state value obtained by the third acquisition unit 701, the input unit 702 can input the state value into a pre-trained action selection model to obtain an action value. The action value is used to indicate the action that the target agent can perform under the current environmental state. The target agent can be an agent that can be controlled by the above-mentioned device 700. The action selection model can be a model set for the target agent and used to select an action for the target agent.

[0157] Specifically, the action selection model can be used to Figure 2 The model obtained by training the method of any embodiment in the corresponding embodiment. The specific training process can be referred to Figure 2 The corresponding embodiments will not be described in detail here.

[0158] In this embodiment, the control unit 703 may control the target agent to perform the action indicated by the action value based on the action value obtained by the input unit 702. Specifically, the control unit 703 may send an action instruction corresponding to the action value to the target agent to control the target agent to perform the action indicated by the action value.

[0159] It is understood that the units described in the device 700 are similar to those in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 700 and the units included therein, and will not be repeated here.

[0160] The apparatus 700 provided by the above embodiment of the present disclosure can be based on Figure 2 In the corresponding embodiment, an action selection model with stronger generalization ability is trained, and then the action selection model with stronger generalization ability can be used to achieve more accurate and reliable intelligent agent control, thereby improving the accuracy and reliability of intelligent agent control.

[0161] Reference below Figure 8 , which shows a server (eg Figure 1 A schematic diagram of the structure of the server in (800). Figure 8 The server shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0162] like Figure 8As shown, the server 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the server 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0163] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the server 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The server 800 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0164] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0165] It should be noted that the computer-readable medium described in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0166] The above-mentioned computer-readable medium may be included in the above-mentioned server; or it may exist independently without being assembled into the server. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the server, the server: obtains a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value; obtains an initial action selection model and an environmental model; based on the initial action selection model and the environmental model, performs the following training steps: selects a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and uses the target state value, perform the following processing steps: input the target state value into the initial action selection model to obtain the action value as the target action value; input the target state value and the obtained target action value into the environment model to obtain the simulation subsequent state value and the simulation reward value; use the obtained simulation subsequent state value as the target state value, repeat the processing steps until the preset processing conditions are met, and obtain the simulation state transfer trajectory data; form a training sample set with the sample single-step state transfer data in the sample state transfer trajectory data and the simulation single-step state transfer data in the simulation state transfer trajectory data; use the formed training sample set to train the initial action selection model to obtain a trained action selection model.

[0167] In addition, when the above one or more programs are executed by the server, the server can also: obtain the state value corresponding to the current environment state; input the obtained state value into the pre-trained action selection model to obtain the action value, wherein the action value is used to indicate the action that the target agent can perform in the current environment state, and the action selection model is used to use Figure 2 A model obtained by training the method of any one of the corresponding embodiments; controlling the target intelligent agent to perform the action indicated by the obtained action value.

[0168] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The units described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring a sample state transition trajectory data set."

[0171] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims

1. A method for training a model, comprising: Obtain a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value; wherein the sample state value is used to represent the sample environment state; the sample action value is used to represent the sample action performed by the sample intelligent device under the sample environment state; the subsequent sample state value is used to represent the subsequent sample environment state to which the sample environment state is converted after the sample intelligent device performs the sample action; the sample reward value is used to represent the sample reward obtained by the sample intelligent agent when performing the sample action under the sample environment state; the sample reward is determined based on the quality of the environment state transition; Obtain an initial action selection model and an environment model; wherein the environment model is used to represent the correspondence between state values, action values, and subsequent state values ​​and reward values; Based on the initial action selection model and the environment model, the following training steps are performed: Selecting a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and using the target state value, performing the following processing steps: inputting the target state value into an initial action selection model to obtain an action value as a target action value; inputting the target state value and the obtained target action value into the environment model to obtain a simulation subsequent state value and a simulation reward value; The obtained simulation subsequent state value is used as the target state value, and the processing steps are repeatedly performed until the preset processing conditions are met to obtain simulation state transfer trajectory data; The sample single-step state transfer data in the sample state transfer trajectory data and the simulated single-step state transfer data in the simulation state transfer trajectory data form a training sample set, wherein the trajectory starting point of the sample state transfer trajectory data is the same as the trajectory starting point of the simulation state transfer trajectory data, and / or the trajectory length of the sample state transfer trajectory data is the same as the trajectory length of the simulation state transfer trajectory data; The initial action selection model is trained using the formed training sample set to obtain an action selection model for controlling the smart device after training; wherein the action selection model is used to characterize the correspondence between the action value and the state value of the environmental state, the action value is used to indicate the action selected for the smart device, the environmental state includes at least one of an environmental image and an environmental temperature, and the smart device includes at least one of a smart refrigerator, a smart phone, and a smart car.

2. The method according to claim 1, wherein The method further comprises: Determine whether the preset training completion conditions are met; In response to the preset training completion condition not being met, the trained action selection model is used as the initial action selection model, and the training step is continued.

3. The method according to claim 1, wherein The acquiring of the environment model comprises: obtaining at least two environment models; and Inputting the target state value and the obtained target action value into the environment model to obtain the simulation subsequent state value and the simulation reward value includes: Inputting the target state value and the obtained target action value into the at least two environment models respectively to obtain at least two candidate subsequent state values ​​and at least two candidate reward values; A simulated subsequent state value is generated based on the at least two candidate subsequent state values, and a simulated reward value is generated based on the at least two candidate reward values.

4. The method according to claim 1, wherein The acquiring of the environment model comprises: Obtain an initial environment model; The initial environment model is trained using the sample single-step state transition data in the sample state transition trajectory data set to obtain an environment model.

5. The method according to any one of claims 1 to 4, wherein: The training of the initial action selection model using the formed training sample set to obtain the trained action selection model includes: Selecting a training sample from the formed training sample set; Determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training sample; The determined difference is used as a penalty term for training the initial action selection model, and the initial action selection model is trained to obtain a trained action selection model.

6. A method for controlling an intelligent agent, comprising: Obtaining a state value corresponding to a current environmental state; wherein the current environmental state includes at least one of a current environmental image and a current environmental temperature; Inputting the acquired state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate an action that a target intelligent agent can perform under the current environmental state, wherein the target intelligent agent includes a smart device, and the action selection model is a model trained using the method of any one of claims 1 to 5 for controlling the smart device; wherein the smart device includes at least one of a smart refrigerator, a smart phone, and a smart car; The target agent is controlled to perform an action indicated by the obtained action value.

7. An apparatus for training a model, comprising: A first acquisition unit is configured to acquire a preset sample state transition trajectory data set, wherein the sample state transition trajectory data includes at least one sample single-step state transition data, and the sample single-step state transition data includes a sample state value, a sample action value, a subsequent sample state value, and a sample reward value; wherein the sample state value is used to represent the sample environment state; the sample action value is used to represent the sample action performed by the sample intelligent device under the sample environment state; the subsequent sample state value is used to represent the subsequent sample environment state to which the sample environment state is converted after the sample intelligent device performs the sample action; the sample reward value is used to represent the sample reward obtained by the sample intelligent agent when performing the sample action under the sample environment state, and the sample reward is determined based on the quality of the environment state transition; A second acquisition unit is configured to acquire an initial action selection model and an environment model; wherein the environment model is used to represent the correspondence between state values, action values, and subsequent state values ​​and reward values; The first execution unit is configured to perform the following training steps based on the initial action selection model and the environment model: Selecting a sample state value from the sample state transition trajectory data included in the sample state transition trajectory data set as a target state value, and using the target state value, performing the following processing steps: inputting the target state value into an initial action selection model to obtain an action value as a target action value; inputting the target state value and the obtained target action value into the environment model to obtain a simulation subsequent state value and a simulation reward value; The obtained simulation subsequent state value is used as the target state value, and the processing steps are repeatedly performed until the preset processing conditions are met to obtain simulation state transfer trajectory data; The sample single-step state transfer data in the sample state transfer trajectory data and the simulated single-step state transfer data in the simulation state transfer trajectory data form a training sample set, wherein the trajectory starting point of the sample state transfer trajectory data is the same as the trajectory starting point of the simulation state transfer trajectory data, and / or the trajectory length of the sample state transfer trajectory data is the same as the trajectory length of the simulation state transfer trajectory data; The initial action selection model is trained using the formed training sample set to obtain an action selection model for controlling the smart device after training; wherein the action selection model is used to characterize the correspondence between the action value and the state value of the environmental state, the action value is used to indicate the action selected for the smart device, the environmental state includes at least one of an environmental image and an environmental temperature, and the smart device includes at least one of a smart refrigerator, a smart phone, and a smart car.

8. The device according to claim 7, wherein The device further comprises: a determining unit configured to determine whether a preset training completion condition is satisfied; The second execution unit is configured to, in response to not meeting the preset training completion condition, use the trained action selection model as the initial action selection model and continue to execute the training step.

9. The device according to claim 7, wherein The second acquiring unit is further configured to: obtaining at least two environment models; and The first execution unit is further configured to: Inputting the target state value and the obtained target action value into the at least two environment models respectively to obtain at least two candidate subsequent state values ​​and at least two candidate reward values; A simulated subsequent state value is generated based on the at least two candidate subsequent state values, and a simulated reward value is generated based on the at least two candidate reward values.

10. The device according to claim 7, wherein The second acquiring unit includes: an acquisition module configured to acquire an initial environment model; The training module is configured to train the initial environment model using the sample single-step state transition data in the sample state transition trajectory data set to obtain an environment model.

11. The device according to any one of claims 7 to 10, wherein: The first execution unit is further configured to: Selecting a training sample from the formed training sample set; Determine the difference between the distribution obeyed by the initial action selection model and the distribution obeyed by the data in the selected training sample; The determined difference is used as a penalty term for training the initial action selection model, and the initial action selection model is trained to obtain a trained action selection model.

12. A device for controlling an intelligent agent, comprising: A third acquisition unit is configured to acquire a state value corresponding to a current environmental state; wherein the current environmental state includes at least one of a current environmental image and a current environmental temperature; An input unit configured to input the acquired state value into a pre-trained action selection model to obtain an action value, wherein the action value is used to indicate an action that a target intelligent agent can perform under the current environmental state, wherein the target intelligent agent includes a smart device, and the action selection model is a model trained using the method of any one of claims 1 to 5 for controlling the smart device; wherein the smart device includes at least one of a smart refrigerator, a smart phone, and a smart car; The control unit is configured to control the target agent to perform an action indicated by the obtained action value.

13. A server comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images