Intelligent agent training method, device, storage medium and electronic device

Abstract

The present disclosure relates to a training method, device, storage medium and electronic device for an intelligent agent, and to the field of terminal technology. The method comprises: obtaining environmental state information from an environment engine, wherein the environment engine comprises a pre-established virtual environment, wherein the virtual environment comprises an object corresponding to at least one object, and an intelligent agent object corresponding to an intelligent agent, and the environmental state information is used to indicate the state of each object object and intelligent agent object in the virtual environment. According to the environmental state information and the control model included in the intelligent agent, control information is determined, and the control information is sent to the environment engine so that the environment engine controls the object objects and intelligent agent objects in the virtual environment according to the control information. Updated environmental state information is obtained from the environment engine, and a target reward is determined according to the updated environmental state information. The control model is trained according to the target reward. The present disclosure can reduce the complexity of intelligent agent training and improve the accuracy of intelligent agent training.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular, to a training method, device, storage medium and electronic device for an intelligent agent. Background Art

[0002] With the rapid development of artificial intelligence, intelligent agents have begun to be widely used in people's daily lives. Usually, it is necessary to train the intelligent agent in advance so that the trained intelligent agent can accurately perform the preset tasks. At present, when training an intelligent agent, it is usually necessary to obtain environmental images through an image acquisition device set on the intelligent agent, and then identify the picture or video to extract environmental information, so as to train the intelligent agent. However, this training method is highly complex and computationally intensive, so it has high requirements on hardware and takes a long time to train. At the same time, the accuracy of extracting environmental information from pictures or videos often depends on the accuracy of the recognition algorithm, which limits the accuracy of the intelligent agent training. Summary of the invention

[0003] The purpose of the present disclosure is to provide a training method, device, storage medium and electronic device for an intelligent agent, so as to reduce the complexity of the training method for the intelligent agent.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training an agent is provided, the method comprising:

[0005] Acquire environmental state information from an environmental engine, wherein the environmental engine includes a pre-established virtual environment, wherein the virtual environment includes an object corresponding to at least one object and an agent object corresponding to an agent, wherein the environmental state information is used to indicate a state of each of the object object and the agent object in the virtual environment;

[0006] Determine control information according to the environment state information and the control model included in the agent, and send the control information to the environment engine, so that the environment engine controls the agent object in the virtual environment according to the control information;

[0007] Acquire updated environment state information from the environment engine, and determine a target reward based on the updated environment state information;

[0008] The control model is trained according to the target reward.

[0009] Optionally, the method further comprises:

[0010] Repeat the steps of acquiring the environment state information from the environment engine to training the control model according to the target reward until the control model meets a preset convergence condition.

[0011] Optionally, determining control information according to the environment state information and a control model included in the agent, and sending the control information to the environment engine includes:

[0012] Performing semantic recognition on the environment state information to obtain semantic information contained in the environment state information, wherein the semantic information includes: the position and attribute of each of the object and the agent in the virtual environment;

[0013] Using the semantic information as input to an action network in the control model to obtain a control instruction output by the action network;

[0014] The control instruction is encapsulated as the control information, and the control information is sent to the environment engine, wherein the control information includes at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the intelligent body.

[0015] Optionally, the performing semantic recognition on the environmental status information to obtain semantic information contained in the environmental status information includes:

[0016] Performing semantic recognition on the environmental state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part; or

[0017] Semantic recognition is performed on the environmental state information to obtain an identifier of each sub-area in a preset area, wherein the preset area is an area in the virtual environment that satisfies a preset position relationship with the intelligent object, and the sub-area is an area obtained by dividing the preset area according to a preset size. The identifier of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

[0018] Optionally, determining a target reward according to the updated environmental state information includes:

[0019] Using the updated environmental state information as input to an evaluation network in the control model to obtain the target reward output by the evaluation network;

[0020] The step of training the control model according to the target reward comprises:

[0021] The action network and the evaluation network in the control model are trained according to the target reward.

[0022] Optionally, the method further comprises:

[0023] The environment engine is controlled to initialize the virtual environment, and environmental status information of the virtual environment after the initialization is obtained.

[0024] Optionally, obtaining environment status information from the environment engine includes:

[0025] After sending the historical control information to the environment engine through a preset execution interface, obtaining the environment state information from the environment engine through the execution interface, the historical control information being determined by the agent based on previous environment state information;

[0026] If the environmental status information cannot be obtained through the execution interface, the environmental status information is obtained through a preset observation interface.

[0027] According to a second aspect of an embodiment of the present disclosure, there is provided a training device for an intelligent agent, the device comprising:

[0028] An acquisition module, configured to acquire environment state information from an environment engine, wherein the environment engine includes a pre-established virtual environment, wherein the virtual environment includes an object corresponding to at least one object and an agent object corresponding to an agent, wherein the environment state information is used to indicate a state of each of the object object and the agent object in the virtual environment;

[0029] A control module, configured to determine control information according to the environment state information and a control model included in the agent, and send the control information to the environment engine, so that the environment engine controls the agent object in the virtual environment according to the control information;

[0030] A determination module, configured to obtain updated environment state information from the environment engine and determine a target reward according to the updated environment state information;

[0031] A training module is used to train the control model according to the target reward.

[0032] Optionally, the steps of obtaining the environment state information from the environment engine to training the control model according to the target reward are repeated until the control model meets a preset convergence condition.

[0033] Optionally, the control module includes:

[0034] An environment processing submodule, configured to perform semantic recognition on the environment state information to obtain semantic information contained in the environment state information, wherein the semantic information includes: the position and attributes of each of the object and the agent in the virtual environment;

[0035] A reinforcement learning submodule, used for taking the semantic information as input of an action network in the control model to obtain a control instruction output by the action network;

[0036] The environment processing submodule is also used to encapsulate the control instructions into the control information and send the control information to the environment engine. The control information includes: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the intelligent body.

[0037] Optionally, the environment processing submodule is used to:

[0038] Performing semantic recognition on the environmental state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part; or

[0039] Semantic recognition is performed on the environmental state information to obtain an identifier of each sub-area in a preset area, wherein the preset area is an area in the virtual environment that satisfies a preset position relationship with the intelligent object, and the sub-area is an area obtained by dividing the preset area according to a preset size. The identifier of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

[0040] Optionally, the determining module is used to:

[0041] Using the updated environmental state information as input to an evaluation network in the control model to obtain the target reward output by the evaluation network;

[0042] The training module is used to:

[0043] The action network and the evaluation network in the control model are trained according to the target reward.

[0044] Optionally, the control module is further used for:

[0045] The environment engine is controlled to initialize the virtual environment, and environmental status information of the virtual environment after the initialization is obtained.

[0046] Optionally, the acquisition module is used to:

[0047] After sending the historical control information to the environment engine through a preset execution interface, obtaining the environment state information from the environment engine through the execution interface, the historical control information being determined by the agent based on previous environment state information;

[0048] If the environmental status information cannot be obtained through the execution interface, the environmental status information is obtained through a preset observation interface.

[0049] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect of the present disclosure are implemented.

[0050] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0051] a memory having a computer program stored thereon;

[0052] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods described in the first aspect of the present disclosure.

[0053] Through the above technical solution, the present disclosure first obtains environmental state information from the environment engine, and then determines control information based on the environmental state information and the control model included in the agent, and sends the control information to the environment engine, so that the environment engine controls the object objects and agent objects in the virtual environment according to the control information. Among them, the environment engine includes a pre-established virtual environment, the virtual environment includes an object object corresponding to at least one object, and an agent object corresponding to the agent, and the environmental state information is used to indicate the state of each object object and agent object in the virtual environment. Further, the updated environmental state information is obtained from the environment engine, and the target reward is determined based on the updated environmental state information, and finally the control model is trained based on the target reward. The present disclosure uses the environment engine to establish a virtual environment that can simulate the real world, and trains the control model of the agent by interacting with the environment engine, which can reduce the complexity of agent training and improve the accuracy of agent training.

[0054] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0056] Figure 1 is a flow chart of a method for training an intelligent agent according to an exemplary embodiment;

[0057] Figure 2 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment;

[0058] Figure 3is a flow chart of another method for training an intelligent agent according to an exemplary embodiment;

[0059] Figure 4 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment;

[0060] Figure 5 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment;

[0061] Figure 6 is a block diagram of a training device for an intelligent agent according to an exemplary embodiment;

[0062] Figure 7 is a block diagram of another intelligent agent training device according to an exemplary embodiment;

[0063] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0066] Figure 1 is a flow chart of a method for training an intelligent agent according to an exemplary embodiment. Figure 1 As shown, the method may include:

[0067] Step 101, obtaining environmental status information from an environmental engine, wherein the environmental engine includes a pre-established virtual environment, wherein the virtual environment includes an object corresponding to at least one object and an agent object corresponding to an agent, and the environmental status information is used to indicate the status of each object object and agent object in the virtual environment.

[0068] For example, a corresponding virtual environment can be established in the environment engine in advance according to the actual scene in the real world, wherein the virtual environment includes an object object corresponding to at least one object, and an agent object corresponding to the agent. The object objects in the virtual environment correspond to the objects in the actual scene one by one, and the shape, size, and position of the object objects in the virtual environment and the corresponding objects in the actual scene are the same. The agent object in the virtual environment corresponds to the agent in the actual scene, and the shape, size, and position of the agent object in the virtual environment and the agent in the actual scene are the same. It can be understood that the virtual environment is the embodiment of the actual scene in the environment engine. The environment engine can be, for example, UE4 (English: Unreal Engine 4, Chinese: Unreal Engine), Unity3D, etc., and the present disclosure does not make specific limitations on this. The agent can interact with the environment engine through a preset protocol to train the agent to perform a specified task, wherein the preset protocol can be, for example, a gRPC (English: Remote Procedure Calls, Chinese: Remote Procedure Call) protocol, and the specified task can be, for example, grabbing a target object, hitting a target object, moving a target object, etc.

[0069] First, the environment state information can be directly obtained from the virtual environment pre-established by the environment engine, wherein the environment state information is used to indicate the state of each object and agent object in the virtual environment, for example, it may include the position, angle, attribute, etc. of each object and agent object. In this way, the agent does not need to collect pictures or videos in the actual scene through an image acquisition device, nor does it need to recognize pictures or videos to obtain the environment state information. By interacting with the environment engine, the environment state information can be accurately obtained, reducing the complexity of obtaining the environment state information.

[0070] Step 102, determining control information according to the environment state information and the control model included in the intelligent agent, and sending the control information to the environment engine, so that the environment engine controls the intelligent agent object in the virtual environment according to the control information.

[0071] For example, after obtaining the environment state information, the environment state information can be input into the control model included in the agent, and the control model can process the environment state information to output the control information. After that, the control information can be sent to the environment engine, and the environment engine can control the agent object in the virtual environment to perform corresponding actions according to the control information. When the agent object performs the corresponding action, it may affect the state of the object in the virtual environment. For example, the agent object can move the target object so that the coordinates of the target object change. Among them, the control information can be used to indicate the movement speed of the agent object, the specified position to which the agent object is to move, the angle of the agent object, the angle of the limb joints of the agent object, and the angle of the finger joints, etc., wherein the movement speed may include linear velocity and angular velocity. Taking the agent object performing a grasping task as an example, the control information can instruct the environment engine to control the agent object to move to a specified position close to the target object at a specified movement speed in the virtual environment, and then adjust the angle of the agent object for grasping. Then adjust the angle of the arm joint of the agent object so that the hand of the agent object is close to the target object, and finally adjust the angle of the finger joint of the agent object to grasp the target object. Among them, the target object is the target of the grasping task, that is, the object that the intelligent agent needs to grasp in the actual scene corresponds to the object in the virtual environment.

[0072] Step 103, obtaining updated environment state information from the environment engine, and determining a target reward according to the updated environment state information.

[0073] Step 104, training the control model according to the target reward.

[0074] For example, after the environment engine controls the intelligent object to complete the action indicated by the control information in the virtual environment, the virtual environment will change compared to before the action is performed, that is, the environment state information will be updated, so the updated environment state information can be obtained from the environment engine, and the target reward obtained by the current execution of the action can be determined based on the updated environment state information. Taking the training of the intelligent object to perform the grasping task as an example, if the updated environment state information indicates that the distance between the intelligent object and the target object is reduced, or the intelligent object grasps the target object, etc., then a positive target reward can be obtained. If the updated environment state information indicates that the distance between the intelligent object and the target object is increased, or the intelligent object encounters an obstacle object, or the joint of the intelligent object exceeds the preset motion limit, etc., then a negative target reward can be obtained. Finally, according to the target reward obtained, the control model can be trained in a reinforcement learning (English: Reinforcement Learning, abbreviated: RL) manner, that is, according to whether the target reward is positive or negative, the parameters in the control model are adjusted, so that the intelligent object can accurately and efficiently perform the specified task. The obstacle objects are other object objects in the virtual environment except the target object object, that is, in the actual scene, other objects except the objects that the intelligent agent needs to grasp correspond to the object objects in the virtual environment.

[0075] It should be noted that the present disclosure can be Figure 1 The steps 101 to 104 are performed in the order shown in the figure. Alternatively, step 102 may be performed first, and then step 103, step 104, and step 101 are performed in the order shown in the figure. The present disclosure does not make any specific limitation on this.

[0076] In summary, the present disclosure first obtains environmental state information from the environment engine, then determines control information based on the environmental state information and the control model included in the agent, and sends the control information to the environment engine, so that the environment engine controls the object objects and agent objects in the virtual environment according to the control information. Among them, the environment engine includes a pre-established virtual environment, the virtual environment includes an object object corresponding to at least one object, and an agent object corresponding to the agent, and the environmental state information is used to indicate the state of each object object and agent object in the virtual environment. Further, the updated environmental state information is obtained from the environment engine, and the target reward is determined based on the updated environmental state information, and finally the control model is trained based on the target reward. The present disclosure uses the environment engine to establish a virtual environment that can simulate the real world, and trains the control model of the agent by interacting with the environment engine, which can reduce the complexity of agent training and improve the accuracy of agent training.

[0077] Figure 2is a flow chart of another method for training an intelligent agent according to an exemplary embodiment. Figure 2 As shown, the method may also include:

[0078] Step 105, repeatedly executing the steps of obtaining the environment state information from the environment engine and training the control model according to the target reward until the control model meets the preset convergence condition.

[0079] For example, steps 101 to 104 may be repeatedly performed to train the control model multiple times until the control model meets a preset convergence condition, so that the intelligent agent can perform the specified task accurately and efficiently.

[0080] Figure 3 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment. Figure 3 As shown, step 102 can be implemented by the following steps:

[0081] Step 1021 , semantic recognition is performed on the environment state information to obtain semantic information contained in the environment state information, where the semantic information includes: the position and attributes of each object and agent object in the virtual environment.

[0082] Step 1022, using the semantic information as input to the action network in the control model to obtain control instructions output by the action network.

[0083] Step 1023, encapsulate the control instruction into control information, and send the control information to the environment engine, the control information includes: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the intelligent body.

[0084] For example, the data in the environment engine usually has a fixed format, and the control model of the intelligent agent may not be able to recognize the environmental state information directly obtained from the environment engine. Therefore, after obtaining the environmental state information, the environmental state information can be semantically recognized to obtain semantic information that can be recognized by the control model, where the semantic information may include: the position and attributes of each object and intelligent agent object in the virtual environment.

[0085] The pose of an object may include: center point, angle, and bounding box information. Among them, the center point is used to indicate the position of the object. For a smaller object, the center point can indicate the target point that the object can grasp. The angle is used to indicate the spatial angle of the object relative to the intelligent object. Each object can have one or more bounding boxes, which are used to indicate the appearance shape of the object. The number of bounding boxes of the object is determined by its structure and degree of curvature. For example, the object corresponding to the mug can use two bounding boxes to represent the handle and the body of the cup respectively, and the object corresponding to the banana can use three bounding boxes to represent each arc segment respectively. The properties of the object may include, for example, whether it is movable, whether it is a target object, whether a collision occurs, whether it is grabbed, etc. For example, a json object can be used to represent the pose and properties of an object.

[0086]

[0087]

[0088] Among them, objects is the object corresponding to "Desk", the object's ID is 0, the position coordinates are (0, 0, 0), the angle relative to the intelligent object is 0, and the object is represented by a bounding box. The bounding box information is (35, 63, 92) and (-193, -44, 0), (35, 63, 92) and (-193, -44, 0), which are the coordinates of the two relative vertices of the bounding box respectively. The properties of the object are: non-target object, immovable, no collision, and not grasped.

[0089] The pose of the intelligent object may include: the position of each joint, the angle of each joint, and the position of the intelligent object. According to the pose of the intelligent object and the pose of the object, the positional relationship with the object and whether there is a collision with the object can be determined. The properties of the intelligent object may include multiple states: waiting for control information, executing actions according to control information, completing a specified task, and colliding with an object. For example, a json object may be used to represent the pose and properties of the intelligent object.

[0090]

[0091]

[0092] Among them, joints is the json object corresponding to a joint "Back_Z_Anchorn" of the intelligent body. The ID of the joint is 1, the position coordinates are (59.82463073730469, 10.000000953674316, 88.20069122314453), the angle relative to the intelligent body object is (3.000000106112566e-06, 1.4000000192027073e-05, 0), and the state of the joint is open.

[0093] Furthermore, the semantic information (i.e., the position and attributes of each object and agent in the virtual environment) can be input into the action network in the control model, and the action network can output corresponding control instructions according to the input semantic information. Among them, the control model may include an action network and an evaluation network, the action network can be used to generate corresponding control instructions according to the environmental state information, and the evaluation network can be used to generate corresponding target rewards according to the updated environmental state information according to the preset reward rules. Afterwards, the control instructions can be encapsulated into control information that can be recognized by the environment engine, and the control information can be sent to the environment engine. Among them, the control information may include: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the agent, the control speed may include linear velocity and angular velocity, and the target joint can be understood as the joint of the agent object that the environment engine is to control to perform the specified task as indicated by the control instruction. Taking the example of an intelligent object performing a grasping task, the target object to be grasped is an apple object, the control speed of the intelligent object is v, the control angle is α, the target joint of the intelligent object is the finger joint, the control angle of the finger joint is β, and the control position coordinate is A, the environment engine can control the intelligent object to move to A at a speed of v, and control the finger joints of the intelligent object to grasp the apple object at an angle β.

[0094] In an application scenario, an implementation of step 1021 may be:

[0095] Semantic recognition is performed on the environment state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part. Or,

[0096] Semantic recognition is performed on the environmental state information to obtain the identification of each sub-area in the preset area. The preset area is the area in the virtual environment that satisfies the preset position relationship with the intelligent object. The sub-area is the area obtained by dividing the preset area according to the preset size. The identification of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

[0097] For example, in one implementation, semantic recognition can be performed on the environmental state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part. Among them, the target part can be understood as the part used by the intelligent object to perform the specified task. Taking the grasping task as an example, the target part can be the hand. Specifically, the center position and axis position of the target object can be obtained based on the bounding box coordinates of the target object, and the center position of the target part of the intelligent object can be obtained based on the coordinates of the designated joints of the intelligent object. Taking the target part as the hand as an example, the corresponding designated joints can be the second joint of the thumb and the second joint of the middle finger. The center position of the hand can be obtained based on the coordinates of the second joint of the thumb and the coordinates of the second joint of the middle finger.

[0098] In another implementation, a preset area that satisfies a preset position relationship with the agent object can be established in the virtual environment, and the preset area can be divided into multiple sub-areas according to the preset size. Among them, the preset area can be a three-dimensional cube, a three-dimensional cylinder, a three-dimensional sphere, etc. Taking the preset area as a three-dimensional cube and the sub-area as a three-dimensional small cube as an example, a 50*50*50 three-dimensional cube can be established at a specified position directly in front of the agent, and the three-dimensional cube can be divided into 12,500 1*1*1 three-dimensional small cubes. After obtaining the environmental state information, the environmental state information can be semantically recognized to obtain the identification of each sub-area in the preset area. Specifically, the bounding box information of all object objects can be input into the preset area, and then the identification of each sub-area is determined according to whether each sub-area contains a target object object, an obstacle object object and an agent object. For example, the identification of the sub-area containing the target object object can be 1, the identification of the sub-area containing the obstacle object object can be -1, the identification of the sub-area containing the agent object can be 0.5, and the identification of the empty sub-area (i.e., the sub-area that does not contain any of the target object object, the obstacle object object, and the agent object) can be 0.

[0099] In another application scenario, an implementation of step 103 may be:

[0100] The updated environment state information is used as the input of the evaluation network in the control model to obtain the target reward output by the evaluation network.

[0101] Accordingly, one implementation of step 104 may be:

[0102] The action network and the critic network in the control model are trained according to the target reward.

[0103] For example, after obtaining the updated environment state information from the environment engine, the updated environment state information can be input into the evaluation network in the control model, and the evaluation network can process the environment state information according to the preset reward rules, thereby outputting the target reward corresponding to the current execution action. Afterwards, the action network and the evaluation network in the control model can be trained in a reinforcement learning manner according to the target reward. Specifically, the parameters in the evaluation network can be updated according to the target reward first, and then the parameters in the action network can be updated according to the updated evaluation network, so that the action network can output more accurate control instructions corresponding to the environment state information.

[0104] Taking the task of training an intelligent agent to move a target object to a target position as an example, the reward rule of the evaluation network can be: if the updated environmental state information indicates that the distance between the intelligent agent and the target object decreases, or the intelligent agent hits the target object, or the distance between the target object and the target position decreases, etc., then the evaluation network can obtain a positive target reward based on the updated environmental state information. If the updated environmental state information indicates that the distance between the intelligent agent and the target object increases, or the distance between the target object and the target position increases, or the intelligent agent hits an obstacle object, etc., then the evaluation network can obtain a negative target reward based on the updated environmental state information.

[0105] Figure 4 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment. Figure 4 As shown, the method may also include:

[0106] Step 106: Control the environment engine to initialize the virtual environment and obtain environment status information of the initialized virtual environment.

[0107] For example, before training the intelligent agent object, or after the intelligent agent object completes the training, the environment engine can be controlled to initialize the virtual environment to restore the virtual environment to an initial state, where the initial state can be understood as a state of the virtual environment that is pre-established and corresponds to an actual scene in the real world. Afterwards, the environment state information of the virtual environment after initialization can be obtained to control the intelligent agent object to restart training.

[0108] Figure 5 is a flow chart of another method for training an intelligent agent according to an exemplary embodiment. Figure 5 As shown, step 101 can be implemented by the following steps:

[0109] Step 1011, after sending the historical control information to the environment engine through the preset execution interface, the environment state information is obtained from the environment engine through the execution interface, and the historical control information is determined by the intelligent agent according to the previous environment state information.

[0110] Step 1012: If the environmental status information cannot be obtained through the execution interface, the environmental status information is obtained through a preset observation interface.

[0111] For example, the intelligent agent can interact with the environment engine through a preset interface, and the preset interface may include an execution interface, an observation interface, and an initialization interface. Among them, control information can be sent to the environment engine through the execution interface, and after the intelligent agent object completes the action indicated by the control information, the updated environment state information can be obtained through the execution interface. The environment engine can be controlled to initialize the virtual environment through the initialization interface, and after the initialization is completed, the environment state information after initialization can be obtained through the initialization interface. If the execution interface fails to obtain the updated environment state information, the updated environment state information can be obtained again through the observation interface. Similarly, if the initialization interface fails to obtain the environment state information after initialization, the environment state information after initialization can be obtained again through the observation interface.

[0112] Specifically, before obtaining environmental state information from the environment engine, the historical control information can be determined based on the previous environmental state information, and the historical control information can be sent to the environment engine through a preset execution interface. The environment engine can control the physical object and the agent object to perform the action indicated by the historical control information, and then the agent can obtain the environmental state information from the environment engine through the execution interface. Among them, the historical control information is determined by the agent based on the previous environmental state information in the last execution of the task before the current execution of the task. If the environmental state information fails to be obtained through the execution interface, the environmental state information can be obtained again through the preset observation interface. In this way, after the environmental state information fails to be obtained through the execution interface, the environmental state information can be obtained again through the preset observation interface, which avoids the problem of being unable to obtain the environmental state information and can improve the efficiency and accuracy of agent training.

[0113] In summary, the present disclosure first obtains environmental state information from the environment engine, then determines control information based on the environmental state information and the control model included in the agent, and sends the control information to the environment engine, so that the environment engine controls the object objects and agent objects in the virtual environment according to the control information. Among them, the environment engine includes a pre-established virtual environment, the virtual environment includes an object object corresponding to at least one object, and an agent object corresponding to the agent, and the environmental state information is used to indicate the state of each object object and agent object in the virtual environment. Further, the updated environmental state information is obtained from the environment engine, and the target reward is determined based on the updated environmental state information, and finally the control model is trained based on the target reward. The present disclosure uses the environment engine to establish a virtual environment that can simulate the real world, and trains the control model of the agent by interacting with the environment engine, which can reduce the complexity of agent training and improve the accuracy of agent training.

[0114] Figure 6 is a block diagram of a training device for an intelligent agent according to an exemplary embodiment. Figure 6 As shown, the device comprises:

[0115] The acquisition module 201 is used to obtain environmental status information from the environmental engine, which includes a pre-established virtual environment. The virtual environment includes an object corresponding to at least one object and an agent object corresponding to the agent. The environmental status information is used to indicate the status of each object object and agent object in the virtual environment.

[0116] The control module 202 is used to determine control information according to the environment state information and the control model included in the intelligent agent, and send the control information to the environment engine so that the environment engine controls the intelligent agent object in the virtual environment according to the control information.

[0117] The determination module 203 is used to obtain updated environment state information from the environment engine and determine the target reward according to the updated environment state information.

[0118] The training module 204 is used to train the control model according to the target reward.

[0119] In one application scenario, the steps of obtaining environmental state information from an environmental engine and training a control model according to a target reward are repeated until the control model meets a preset convergence condition.

[0120] Figure 7 is a block diagram of another intelligent agent training device according to an exemplary embodiment. Figure 7 As shown, the control module 202 includes:

[0121] The environment processing submodule 2021 is used to perform semantic recognition on the environment state information to obtain the semantic information contained in the environment state information, and the semantic information includes: the position and attributes of each object and intelligent agent object in the virtual environment.

[0122] The reinforcement learning submodule 2022 is used to use the semantic information as the input of the action network in the control model to obtain the control instructions output by the action network.

[0123] The environment processing submodule 2021 is also used to encapsulate the control instructions into control information and send the control information to the environment engine. The control information includes: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the intelligent body.

[0124] In another application scenario, the environment processing submodule 2021 is used to:

[0125] Semantic recognition is performed on the environment state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part. Or,

[0126] Semantic recognition is performed on the environmental state information to obtain the identification of each sub-area in the preset area. The preset area is the area in the virtual environment that satisfies the preset position relationship with the intelligent object. The sub-area is the area obtained by dividing the preset area according to the preset size. The identification of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

[0127] In another application scenario, the determination module 203 is used to:

[0128] The updated environment state information is used as the input of the evaluation network in the control model to obtain the target reward output by the evaluation network.

[0129] Accordingly, the training module 204 is used to:

[0130] The action network and the critic network in the control model are trained according to the target reward.

[0131] In another application scenario, the control module 202 is further used for:

[0132] The control environment engine initializes the virtual environment and obtains environmental status information of the initialized virtual environment.

[0133] In another application scenario, the acquisition module 201 is used to:

[0134] After the historical control information is sent to the environment engine through a preset execution interface, the environment state information is obtained from the environment engine through the execution interface. The historical control information is determined by the intelligent agent based on the previous environment state information.

[0135] If the environment status information cannot be obtained through the execution interface, the environment status information is obtained through the preset observation interface.

[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] In summary, the present disclosure first obtains environmental state information from the environment engine, then determines control information based on the environmental state information and the control model included in the agent, and sends the control information to the environment engine, so that the environment engine controls the object objects and agent objects in the virtual environment according to the control information. Among them, the environment engine includes a pre-established virtual environment, the virtual environment includes an object object corresponding to at least one object, and an agent object corresponding to the agent, and the environmental state information is used to indicate the state of each object object and agent object in the virtual environment. Further, the updated environmental state information is obtained from the environment engine, and the target reward is determined based on the updated environmental state information, and finally the control model is trained based on the target reward. The present disclosure uses the environment engine to establish a virtual environment that can simulate the real world, and trains the control model of the agent by interacting with the environment engine, which can reduce the complexity of agent training and improve the accuracy of agent training.

[0138] Figure 8 FIG. 1 is a block diagram of an electronic device 300 according to an exemplary embodiment. Figure 8 As shown, the electronic device 300 may include: a processor 301 , a memory 302 . The electronic device 300 may also include one or more of a multimedia component 303 , an input / output (I / O) interface 304 , and a communication component 305 .

[0139] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the training method of the above-mentioned intelligent agent. The memory 302 is used to store various types of data to support the operation of the electronic device 300, and these data may include, for example, instructions for any application or method used to operate on the electronic device 300, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 302 or sent through the communication component 305. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 304 provides an interface between the processor 301 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 305 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0140] In an exemplary embodiment, the electronic device 300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned training method of the intelligent agent.

[0141] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned agent training method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 302 including program instructions, and the above-mentioned program instructions can be executed by the processor 301 of the electronic device 300 to complete the above-mentioned agent training method.

[0142] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned intelligent agent training method when executed by the programmable device.

[0143] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0144] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0145] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for training an intelligent agent, characterized in that: The method comprises: Acquire environmental state information from an environmental engine, wherein the environmental engine includes a pre-established virtual environment, wherein the virtual environment includes an object corresponding to at least one object and an agent object corresponding to an agent, wherein the environmental state information is used to indicate a state of each of the object object and the agent object in the virtual environment; Determine control information according to the environment state information and the control model included in the agent, and send the control information to the environment engine, so that the environment engine controls the agent object in the virtual environment according to the control information; Acquire updated environment state information from the environment engine, and determine a target reward based on the updated environment state information; training the control model according to the target reward; The step of determining control information according to the environment state information and a control model included in the agent, and sending the control information to the environment engine comprises: Performing semantic recognition on the environment state information to obtain semantic information contained in the environment state information, wherein the semantic information includes: the position and attribute of each of the object and the agent in the virtual environment; Using the semantic information as input to an action network in the control model to obtain a control instruction output by the action network; Encapsulating the control instruction into the control information, and sending the control information to the environment engine, the control information including: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the agent; The performing semantic recognition on the environmental status information to obtain semantic information contained in the environmental status information includes: Performing semantic recognition on the environmental state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part of the intelligent object; or Semantic recognition is performed on the environmental state information to obtain an identifier of each sub-area in a preset area, wherein the preset area is an area in the virtual environment that satisfies a preset position relationship with the intelligent object, and the sub-area is an area obtained by dividing the preset area according to a preset size. The identifier of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

2. The method according to claim 1, characterized in that The method further comprises: Repeat the steps of acquiring the environment state information from the environment engine to training the control model according to the target reward until the control model meets a preset convergence condition.

3. The method according to claim 1 or 2, characterized in that: The determining the target reward according to the updated environmental state information includes: Using the updated environmental state information as input to an evaluation network in the control model to obtain the target reward output by the evaluation network; The step of training the control model according to the target reward comprises: The action network and the evaluation network in the control model are trained according to the target reward.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: The environment engine is controlled to initialize the virtual environment, and environmental status information of the virtual environment after the initialization is obtained.

5. The method according to claim 1 or 2, characterized in that: The obtaining of environment status information from the environment engine includes: After sending the historical control information to the environment engine through a preset execution interface, obtaining the environment state information from the environment engine through the execution interface, the historical control information being determined by the agent based on previous environment state information; If the environmental status information cannot be obtained through the execution interface, the environmental status information is obtained through a preset observation interface.

6. A training device for an intelligent agent, characterized in that: The device comprises: An acquisition module, configured to acquire environment state information from an environment engine, wherein the environment engine includes a pre-established virtual environment, wherein the virtual environment includes an object corresponding to at least one object and an agent object corresponding to an agent, wherein the environment state information is used to indicate a state of each of the object object and the agent object in the virtual environment; A control module, configured to determine control information according to the environment state information and a control model included in the agent, and send the control information to the environment engine, so that the environment engine controls the agent object in the virtual environment according to the control information; A determination module, configured to obtain updated environment state information from the environment engine and determine a target reward according to the updated environment state information; A training module, used for training the control model according to the target reward; The control module comprises: An environment processing submodule, configured to perform semantic recognition on the environment state information to obtain semantic information contained in the environment state information, wherein the semantic information includes: the position and attributes of each of the object and the agent in the virtual environment; A reinforcement learning submodule, used for taking the semantic information as input of an action network in the control model to obtain control instructions output by the action network; The environment processing submodule is further used to encapsulate the control instruction into the control information and send the control information to the environment engine, wherein the control information includes: at least one of the control speed, control angle, control position coordinates and control angle of the target joint of the intelligent body; The environment processing submodule is used for: Performing semantic recognition on the environmental state information to obtain the center position and axis position of the target object in the virtual environment, the bounding box coordinates of the obstacle object in the virtual environment, and the center position of the target part of the intelligent object; or Semantic recognition is performed on the environmental state information to obtain an identifier of each sub-area in a preset area, wherein the preset area is an area in the virtual environment that satisfies a preset position relationship with the intelligent object, and the sub-area is an area obtained by dividing the preset area according to a preset size. The identifier of each sub-area is used to indicate the type of object contained in the sub-area, and the types include: target object object, obstacle object object, and intelligent object.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5.

Images