Value network training method, action planning method and related device

Abstract

The invention provides a value network training method, an action planning method and a related device. The value network comprises a transformer feature extraction module. The value network training method comprises the following steps: acquiring a target tetrad from an experience pool; determining a first return value corresponding to the target tetrad according to the first reward value in the target tetrad; inputting the first state data in the target tetrad into the value network of the current training period, extracting a state feature vector corresponding to the first state data through a transformer feature extraction module in the value network of the current training period, and obtaining a second return value output by the value network of the current training period; and according to the first return value and the second return value, adjusting network parameters of the value network of the current training period, so that the second return value approaches the first return value, and obtaining a target value network. The technical scheme can adapt to motion planning for the mobile robot in a dense scene.

Description

Technical Field

[0001] The present application relates to the field of robot navigation, and in particular to a value network training method, an action planning method and related devices. Background Art

[0002] With the rapid development of science and technology, mobile robots are widely used in fields such as unmanned delivery, which greatly facilitates people's lives. Mobile robot navigation technology is one of the key technologies for the application of mobile robots in various scenarios. Excellent navigation algorithms can not only improve the working efficiency of mobile robots, but also reduce energy consumption and costs.

[0003] Traditional robot navigation algorithms include dynamic window approach (DWA), optimal reciprocal collision avoidance (ORCA) algorithm, etc. Traditional robot navigation algorithms can only be applied to relatively open scenes, and are not applicable to dense scenes (such as crowded stations, shopping malls, etc.). Summary of the invention

[0004] The present application provides a value network training method, an action planning method and related devices to perform action planning for a mobile robot in a dense scene.

[0005] In a first aspect, a value network training method is provided, wherein the value network is used to output a reward value according to a state of a target, and the value network includes a transformer feature extraction module; the method includes:

[0006] Acquire a target quadruple from an experience pool, the experience pool comprising a plurality of quadruple collected in advance, each quadruple comprising first state data, first action data, second state data and a first reward value, the first state data comprising state data of all targets in a navigation simulation environment at a first moment, the first action data being used to indicate an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data comprising state data of all targets at a second moment, the second moment being a moment next to the first moment, and the first reward value being a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment;

[0007] Determining a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple;

[0008] Inputting the first state data in the target quadruple into the value network of the current training cycle, extracting the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtaining the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data;

[0009] According to the first reward value and the second reward value, the network parameters of the value network of the current training cycle are adjusted so that the second reward value approaches the first reward value, thereby obtaining a target value network.

[0010] In this technical solution, after obtaining the target quadruple from the experience pool, the first reward value corresponding to the target quadruple is first determined according to the first reward value in the target quadruple; then the first state data in the target quadruple is input into the value network, and the state feature vector of the first state data in the target quadruple is extracted by the transformer feature extraction module in the value network, and the second reward value output by the value network based on the state feature vector of the first state data is obtained; finally, according to the first reward value and the second reward value, the network parameters of the value network are adjusted so that the second reward value is close to the first reward value; by adjusting the network parameters of the value network so that the second reward value is close to the first reward value, the value network can learn the navigation obstacle avoidance strategy; by setting the transformer feature extraction module in the value network, the navigation environment can be extracted. The multi-head attention mechanism in the transformer feature extraction module can better express global information and more accurately express the interaction behavior between the first mobile target and other targets in the navigation environment. The value network can more accurately output a more accurate reward value based on the learned interaction behavior between the first mobile target and other targets, so that the action planner in the action planning model containing the value network can complete precise action planning according to the more accurate reward value. Since the action planner in the action planning model completes action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, so it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0011] In combination with the first aspect, in a possible implementation method, the first state data includes a first state vector and a second state vector, the first state vector is used to represent the state of the first moving target at the first moment, and the second state vector is used to represent the state of all targets other than the first moving target at the first moment; the value network also includes a vector alignment module and a vector splicing module; the first state data in the target quadruple is input into the value network of the current training cycle, and the state feature vector corresponding to the first state data is extracted through the transformer feature extraction module in the value network of the current training cycle, including: inputting the first state vector and the second state vector into the vector alignment module in the value network of the current training cycle, and respectively aligning the first state vector and the second state vector through the vector alignment module in the value network of the current training cycle. Perform vector mapping to obtain a first initial vector corresponding to the first state vector and a second initial vector corresponding to the second state vector, wherein the dimension of the first initial vector is the same as the dimension of the second initial vector; input the first initial vector and the second initial vector into a vector splicing module in a value network of a current training cycle, and splice the first initial vector and the second initial vector row by row through the vector splicing module in the value network of the current training cycle to obtain a state input matrix corresponding to the first state data, wherein the number of rows of the state input matrix is ​​(n+1), where n is the number of the other targets; input the state input matrix into a transformer feature extraction module in a value network of the current training cycle, and perform state feature extraction on the state input matrix through the transformer feature extraction module in the value network of the current training cycle to obtain the state feature vector. By setting a vector alignment module in the value network to first align the state vectors of the first mobile target and other targets in the navigation simulation environment, initial vectors with the same dimension are obtained, and then the initial vectors corresponding to the first mobile target and other targets are spliced ​​row by row through the vector splicing module to obtain the state input matrix corresponding to the first state data for state feature extraction. This can not only represent the relationship between any two targets in the navigation environment, but also adapt to different numbers of targets, and has wider adaptability.

[0012] In combination with the first aspect, in a possible implementation, the value network further includes multiple layers of fully connected layers connected in sequence; the obtaining of the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data includes: inputting the state feature vector corresponding to the first state data into the multiple layers of fully connected layers in the value network of the current training cycle, obtaining the reward value output by the last fully connected layer in the value network of the current training cycle as the second reward value. The second reward value is obtained by sequentially connecting multiple layers of fully connected layers, and the implementation is simple.

[0013] In combination with the first aspect, in a possible implementation manner, the transformer feature extraction module includes multiple transformer feature extraction structures.

[0014] In combination with the first aspect, in a possible implementation method, determining the first reward value corresponding to the target quadruple based on the first reward value in the target quadruple includes: judging whether the moving target state corresponding to the second state data in the target quadruple is a terminal state, the moving target state being the state of the first moving target; if the moving target state corresponding to the second state data in the target quadruple is a terminal state, determining the first reward value in the target quadruple as the first reward value; if the moving target state corresponding to the second state data in the target quadruple is not a terminal state, determining the third reward value based on the second state data in the target quadruple, and determining the first reward value based on the first reward value and the third reward value.

[0015] In combination with the first aspect, in a possible implementation method, determining the third reward value based on the second state data in the target quadruple includes: inputting the second state data in the target quadruple into the value network of the previous training cycle of the current training cycle, and determining the reward value output by the value network of the previous training cycle as the third reward value.

[0016] In combination with the first aspect, in a possible implementation method, determining the first reward value based on the first reward value and the third reward value includes: calculating the product of the third reward value and the attenuation coefficient to obtain the attenuated reward value corresponding to the third reward value; and determining the sum of the first reward value and the attenuated reward value as the first reward value.

[0017] In combination with the first aspect, in a possible implementation method, before obtaining the target quadruple from the experience pool, it also includes: obtaining the tth action data, the tth action data is used to characterize the action performed by the first mobile target under the tth state data, t is a positive integer greater than or equal to 0; controlling the first mobile target in the tth state data to perform the action corresponding to the tth action data in the navigation simulation environment, and obtaining the (t+1)th state data and the tth reward value output by the navigation simulation environment; taking the tth state data, tth action data, (t+1)th state data and tth reward value as a quadruple, adding it to the experience pool, and taking the (t+1)th state data as the tth state data, and returning to execute the step of obtaining the tth action data.

[0018] In combination with the first aspect, in a possible implementation method, obtaining the tth action data includes: randomly sampling actions for the first moving target in the tth state data to obtain random action data of the first moving target in the tth state data; if the random probability is less than or equal to a preset probability threshold, determining the random action data as the tth action data; if the random probability is greater than the preset probability threshold, inputting the tth state data into an action planning model including the value network, and determining the planned action data output by the action planning model as the tth action data.

[0019] In a second aspect, an action planning method is provided, comprising:

[0020] Acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target;

[0021] The target state data is input into the action planning model to obtain the current planned action data of the first mobile target, the action planning model includes a target value network and an action planner, the target value network is trained by the method of the first aspect above, the target value network is used to output a reward value according to the state of the target, and the action planner is used to output the planned action data according to the reward value output by the target value network.

[0022] In this technical solution, after acquiring the target state data, the target state data is input into the action planning model to obtain the current planned action data of the first mobile target in the current state, so as to realize the action planning of the mobile target; since the action planner in the action planning model completes the action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, so it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0023] In a third aspect, a value network training device is provided, wherein the value network is used to output a reward value according to a state of a target, and the value network includes a transformer feature extraction module;

[0024] The device comprises:

[0025] A quadruple acquisition module, used for acquiring a target quadruple from an experience pool, wherein the experience pool includes a plurality of quadruple collected in advance, each quadruple includes first state data, first action data, second state data and a first reward value, wherein the first state data includes state data of all targets in a navigation simulation environment at a first moment, the first action data is used for indicating an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data includes state data of all targets at a second moment, the second moment being a moment next to the first moment, and the first reward value is a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment;

[0026] A first reward value determination module, used to determine a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple;

[0027] A second reward value determination module is used to input the first state data in the target quadruple into the value network of the current training cycle, extract the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtain the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data;

[0028] A parameter adjustment module is used to adjust the network parameters of the value network of the current training cycle according to the first reward value and the second reward value, so that the second reward value is close to the first reward value, and obtain a target value network.

[0029] In a fourth aspect, an action planning device is provided, comprising:

[0030] A state acquisition module, used to acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target;

[0031] An action planning module is used to input the target state data into an action planning model to obtain the current planned action data of the first mobile target. The action planning model includes a target value network and an action planner. The target value network is trained by the method of the first aspect above. The target value network is used to output a reward value according to the state of the target. The action planner is used to output the planned action data according to the reward value output by the target value network.

[0032] In a fifth aspect, a computer device is provided, comprising a memory and one or more processors, wherein the memory is connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the computer device implements the value network training method of the first aspect or the action planning method of the second aspect.

[0033] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the value network training method of the first aspect or the action planning method of the second aspect.

[0034] The present application can achieve the following technical effects: by adjusting the network parameters of the value network so that the second reward value approaches the first reward value, the value network in the action planning model can learn the navigation obstacle avoidance strategy; by setting a transformer feature extraction module in the value network to extract the state feature vector of the state data of the target in the navigation environment, the multi-head attention mechanism in the transformer feature extraction module can better express the global information, and can more accurately express the interaction behavior between the first mobile target and other targets in the navigation environment. The value network of the action planning model can more accurately output a more accurate reward value based on the learned interaction behavior between the first mobile target and other targets, so that the action planner in the action planning model containing the value network can complete precise action planning according to the more accurate reward value; because the action planner in the action planning model completes action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, and thus it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the obstacle avoidance strategy learning network framework provided for this application;

[0036] Figure 2 and Figure 3A schematic diagram of the structure of the action planning model provided in the embodiment of the present application;

[0037] Figure 4 A flowchart of a value network training method provided in an embodiment of the present application;

[0038] Figure 5 A flowchart of an action planning method provided in an embodiment of the present application;

[0039] Figure 6 It is a structural schematic diagram of a value network training device provided in an embodiment of the present application;

[0040] Figure 7 It is a structural schematic diagram of an action planning device provided in an embodiment of the present application;

[0041] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0043] The technical solution of the present application is applicable to the path planning scenario of mobile devices, where the mobile device is a device that can move autonomously in the environment, and the mobile device can specifically be a mobile robot, an unmanned car, an unmanned luggage cart, etc. In the path planning scenario of mobile devices, when the environment in which the mobile device is located is relatively simple, that is, when there are fewer pedestrians and other mobile devices in the environment in which the mobile device is located, traditional navigation algorithms such as the DWA algorithm and the ORCA algorithm can be used for path planning to achieve obstacle avoidance navigation. When planning actions for mobile devices, these traditional algorithms usually do not consider the intentions of pedestrians and interactions with pedestrians, and only prevent collisions through passive reactions, and use artificially defined functions to ensure safety; in dense navigation scenarios, since there are many pedestrians and other mobile devices in dense navigation scenarios, using these traditional algorithms to plan actions for mobile devices will make the movement of the mobile device unnatural, with left and right swaying, and even collisions with pedestrians. Therefore, traditional robot navigation algorithms are not suitable for dense navigation scenarios.

[0044] In view of this, the present application proposes a navigation and obstacle avoidance solution based on reinforcement learning. By constructing an action planning model composed of a value network and an action planner, a navigation simulation environment is used to simulate the scenario of a mobile device navigating in a dense crowd, and a navigation reward value is obtained. The value network of the action planning model is trained through the reward value, so that the action planning model completes environmental modeling and learning, learns the optimal obstacle avoidance strategy, and then completes the action planning based on the optimal obstacle avoidance strategy.

[0045] To facilitate understanding, the obstacle avoidance strategy learning network framework and action planning model involved in this application are first introduced.

[0046] See also Figure 1 , Figure 1 The following is a schematic diagram of the framework of the obstacle avoidance strategy learning network provided in this application. Figure 1 As shown, the framework of the obstacle avoidance strategy learning network includes a navigation simulation environment, a trainer and an action planning model. The navigation simulation environment is a pre-set scene for simulating the navigation of a mobile device in a dense scene. The navigation simulation environment is provided with a target motion strategy. The target motion strategy refers to the motion strategy of other targets in the navigation simulation environment except the first moving target. The target motion strategy can be an ORCA strategy or a Social force strategy, etc. The first moving target refers to the target that needs to be action planned. The first moving target can be the aforementioned mobile robot, unmanned car, unmanned luggage cart, etc., and other targets except the first moving target can be pedestrians and dynamic obstacles. When the action data corresponding to the action performed by the first moving target at the target time is input into the navigation simulation environment, the simulator in the navigation simulation environment will control the first moving target to perform the corresponding action, and output the status of all targets (including the first moving target and other targets except the first moving target) in the navigation simulation environment at the next moment of the target time (i.e., the moment after the first moving target performs the action) and the operation status of all targets after the first moving target performs the action, such as whether the first moving target collides with other targets, whether the first moving target reaches the target position, etc.; and the simulator in the navigation simulation environment will also output the reward value obtained by the first moving target for performing the action, and the reward value is obtained based on the reward function, and the reward function is as follows:

[0047]

[0048] Among them, t is the serial number of the target time, the target time can be any time, a t is the action data at the target time, used to represent the action performed by the first moving target at the target time, s t is the state data at the target time, which is used to represent the state of all targets in the navigation simulation environment at the target time. t represents the minimum distance between the first moving target and other targets in the navigation simulation environment at the target time, p t represents the position of the first moving target at the target time, p g Indicates the target position that the first moving target needs to reach; R t The first moving target is at the target time s t Execute a in the corresponding state t The reward value obtained by the corresponding action.

[0049] The input and output of the navigation simulation environment can be composed of a quaternion (s t , a t , R t ,s t+1 ), s t+1 The state data of the next moment of the target moment is used to represent the state of all targets in the navigation simulation environment at the next moment of the target moment. By inputting action data into the navigation simulation environment multiple times, multiple quadruple groups can be obtained, and multiple quadruple groups form an experience pool. The trainer is used to obtain the quadruple group (s t , a t , R t ,s t+1 ) as training data, and use the training data to train the action planning model. The action planning model can be based on the s in the quaternary t+1 , output a t+1 , a t+1 Indicates the action performed by the mobile device at the next moment of the target moment, a t+1 It can also be used as the input of the navigation simulation environment to obtain new quadruple and add it to the experience pool. The trainer continuously obtains quadruple from the experience pool and iteratively adjusts the action planning model, so that the action planning model can learn the obstacle avoidance strategy and complete the action planning.

[0050] The structure of the action planning model is as follows: Figure 2 As shown, the action planning model includes a value network 10 and an action planner 20. The value network 10 is used to output a reward value according to the state of the target. The input of the value network is a first state vector and a second state vector centered on the first moving target. The first state vector is used to represent the state of the first moving target at the target time. The first state vector d g represents the distance between the first moving target and the target position at the target time, v ref represents the reference speed of the first moving target at the target time, θ represents the angle between the orientation of the first moving target at the target time and the X-axis, where the X-axis is the line between the mobile device and the target position, r represents the radius of the first moving target, and v x and v y They represent the velocity components of the first moving target in two directions at the target time. The second state vector is used to characterize the state of other targets except the first moving target at the target time. p x and p y represents the position component of other targets relative to the first moving target at the target time, v x and v yrepresents the velocity component of other targets relative to the first moving target at the target time, r h Indicates the radius of other targets, d a represents the distance between other targets and the first moving target at the target time, r+r h is the radius of the first moving target and other targets. There can be multiple second state vectors, the number of which is n, where n is the number of other targets in the navigation simulation environment. The first state vector and the second state vector are both generated by the state data s of the target at the moment. t Converted.

[0051] like Figure 2 As shown, the value network 10 includes a vector alignment module 101, a vector concatenation module 102, a transformer feature extraction module 103 and a multi-layer fully connected layer 104 connected in sequence.

[0052] The input of the vector alignment module 101 is the first state vector and the second state vector. The vector alignment module 101 is used to perform vector mapping on the first state vector and the second state vector respectively to obtain a first initial vector corresponding to the first state vector and a second initial vector corresponding to the second state vector, and the dimension of the first initial vector is the same as the dimension of the second initial vector. In a specific implementation, the vector alignment module can be an embedding module, and the embedding module is used to map the first state vector and the second state vector into a 64-dimensional vector, that is, the dimensions of the first initial vector and the second initial vector are both 64-dimensional.

[0053] The output of the vector alignment module 101 is the input of the vector splicing module 102. The vector splicing module 102 is used to splice the first initial vector and the second initial vector row by row to obtain a state input matrix, and the number of rows of the state input matrix is ​​(n+1). For example, the dimensions of the first initial vector and the second initial vector are both 64 dimensions, and there are 5 pedestrians in the navigation simulation environment, then the first initial vector and the second initial vector are spliced ​​row by row to obtain a 6*64 state input matrix.

[0054] By setting a vector alignment module in the value network to first align the state vectors of the first mobile target and other targets in the navigation simulation environment, initial vectors with the same dimension are obtained, and then the initial vectors corresponding to the first mobile target and other targets are spliced ​​row by row through the vector splicing module to obtain the state input matrix corresponding to the first state data for state feature extraction. This can not only represent the relationship between any two targets in the navigation environment, but also adapt to different numbers of targets, and has wider adaptability.

[0055] The output of the vector concatenation module 102 is the input of the transformer feature extraction module 103, which is used to extract state features from the state input matrix to obtain a state feature vector, wherein the transformer feature extraction module 102 may include multiple transformer feature extraction structures. For example, Figure 3 As shown, the feature extraction module 102 may include two transformer feature extraction structures. Each transformer feature extraction structure is composed of a multi-head attention layer, an addition and normalization layer, a fully connected neural network layer, and an addition and normalization layer. Since the transformer feature extraction structure has a multi-head attention layer, the state feature vector finally extracted can better express the state of the target and the interaction between the targets, which helps to learn a better navigation and obstacle avoidance strategy.

[0056] The output of transformer feature extraction module 103 is the input of multi-layer fully connected layer 104. Multi-layer fully connected layer 103 is used to output reward value according to state feature vector. Figure 3 As shown, the multi-layer fully connected layer may include 3 fully connected layers, and the feature dimensions of the 3 fully connected layers may be 1000, 50, and 1 respectively.

[0057] The action planner 20 is used to output the planned action data according to the reward value output by the value network 10, wherein the action planner selects a state transition distribution that satisfies the value network measurement strategy π(s)=argmax in the action space according to the reward value and the linear assumption. a (R(s t ,a t )+γV(s - ;s - ~p(s - |s t ,a t ) corresponding to the planned action, as the action data a at the target time t Among them, s - is the state of the first moving target at time (t+Δt), γ is the attenuation coefficient, p(s - |s t , a t ) is the state transition distribution probability from time t (i.e., the target time) to time (t+Δt) (i.e., the next time after the target time). For example, the action space may include 80 actions, and the 80 actions are to discretize the linear speed into 5 types in an exponential array manner, and to discretize the direction into 16 types uniformly in the range of 0 to 2π, and then combine a linear speed and a direction into one action, and obtain the action data a t =[ut ,θ t ],u t represents the linear velocity of the forward motion, θ t Indicates direction.

[0058] The action planner obtains the action data a at the target time based on the reward value and state transition distribution t The specific method is as follows:

[0059] When the state data s at the target time is obtained t Afterwards, the action planner calculates the state data s at the target time. t And state transition distribution, determine N candidate state data k=1,2…,N, where the N candidate state data correspond to the N action data in the action space respectively, and the N candidate state data are the first moving target in the state data s t The state data after executing N action data in the action space are respectively executed, where N is the total number of actions in the action space.

[0060] N candidate state data Input them into the value network 10 respectively to obtain N reward values ​​V output by the value network 10 k , k = 1, 2, ..., N. It should be understood that the N candidate state data Inputting into the value network 10 respectively means inputting the first state vector and the second state vector corresponding to each candidate state data into the value network 10 respectively, and the first state vector and the second state vector corresponding to each candidate state data are respectively converted from each candidate state data.

[0061] The N reward values ​​(V 1 , V 2 , …, V N ) to determine the maximum return value max(V 1 , V 2 , …, V N ); Set the maximum return value max(V 1 , V 2 , …, V N ) The corresponding action data is determined as the action data a at the target time t Assume that N return values ​​(V 1 , V 2 , …, V N ) K (1≤K≤N) is the maximum value, V K Candidate status data The corresponding reward value is the Kth action data in the action space, which is determined as the action data a at the target moment. t .

[0062] This application aims to Figure 2 and Figure 3 The value network shown is trained so that the value network can complete the modeling of the navigation simulation environment and the obstacle avoidance strategy learning, and then the action planner in the action planning model can complete the action planning according to the reward value output by the value network. Among them, training the value network refers to adjusting the network parameters of the value network, and the network parameters of the value network include the parameters of the vector alignment module, the transformer feature extraction module and the multi-layer fully connected layer.

[0063] The technical solution of this application is described in detail below.

[0064] See also Figure 4 , Figure 4 A flow chart of a value network training method provided in an embodiment of the present application, wherein the value network belongs to an action planning model, and the action planning model and the value network are as follows Figure 2 and Figure 3 As shown, Figure 4 As shown, the method comprises the following steps:

[0065] S301, set i to 0.

[0066] Here, the training of the value network can be divided into multiple training cycles in sequence, and i is the sequence number of the current training cycle.

[0067] S302, determine whether i is equal to M.

[0068] Here, M is the preset maximum number of training cycles.

[0069] When i is not equal to M, it means that the last training cycle has not been reached, and it is necessary to enter the next training cycle and execute step S303; when i is equal to M, it means that the last training cycle has been reached and the training process ends.

[0070] S303, set c to 0.

[0071] Here, c is the number of newly collected quadruples. For the definition of quadruples, please refer to the above description.

[0072] S304, determine whether c is equal to C.

[0073] Here, C is the update frequency pre-set for the action planning model, that is, the number of newly collected quadruplets between two training cycles.

[0074] When c is equal to C, it means that the number of newly collected quadruple groups in the experience pool has reached the requirement for entering a new training cycle, and a new training cycle can be entered, and step S312 is executed; when c is not equal to C, it means that the number of newly collected quadruple groups in the experience pool does not meet the requirement for entering a new training cycle, and it is necessary to continue to collect quadruple groups and put them into the experience pool, and step S305 is executed.

[0075] S305, increase t by 1.

[0076] S306, determine whether t is equal to L.

[0077] Here, L is the maximum number of trajectory steps of the first moving target, that is, the maximum number of steps the first moving target takes to move to the target position.

[0078] When t is equal to L, it means that the first moving target has not reached the target position after the maximum number of steps of the trajectory, and the trajectory needs to be replanned, and step S307 is executed; when t is not equal to L, it means that the mobile device has not reached the maximum number of steps of the trajectory, and the action can continue to be planned along the current moving trajectory of the first moving target, and S308 is executed.

[0079] S307, set t to 0.

[0080] S308, obtaining the tth action data.

[0081] Here, the t-th action data is used to represent the action performed by the first moving target under the t-th state data, and the t-th state data is the s-th state data introduced above. t , the t-th action data is the a introduced above t , about s t and a t For the specific meaning of , please refer to the aforementioned description of the quadruple, which will not be repeated here.

[0082] The t-th action data can be obtained through the following steps A1-A3:

[0083] A1. Randomly sample actions for the first moving target in the tth state data to obtain random action data of the first moving target in the tth state data.

[0084] A2. If the random probability is less than or equal to a preset probability threshold, the random action data of the first moving target under the tth state data is determined as the tth action data.

[0085] Here, the random probability is the probability obtained by random sampling in a probability distribution of 0 to 1. The preset probability threshold is a preset probability threshold for determining whether to insert a random action. Exemplarily, the preset probability threshold may be 0.1.

[0086] A3. If the random probability is less than or equal to the preset probability threshold, the t-th state data is input into the action planning model, and the planned action data output by the action planning model is determined as the t-th action data.

[0087] Here, after the t-th state data is input into the action planning model, the planned action data output by the action planning model can be expressed as a t = argmax a (R(s t ,a t )+γV(s - ;s - ~p(s - |s t ,a t ). The principle of the action planning model outputting the planned action data can be referred to above. Figure 2 and Figure 3 The introduction of the action planning model will not be repeated here.

[0088] S309, controlling the first moving target in the tth state data to perform an action corresponding to the tth action data in the navigation simulation environment, and obtaining the (t+1)th state data and the tth reward value output in the navigation simulation environment.

[0089] Here, the (t+1)th state data output in the navigation simulation environment is the s described above. t+1 , the tth reward value is the R introduced above t .

[0090] S310, taking the tth state data, the tth action data, the (t+1)th state data and the tth reward value as a four-tuple, and adding it to the experience pool.

[0091] Here, the quaternion is the aforementioned (s t , a t , R t ,s t+1 ).

[0092] S311, add 1 to c and return to execute S304.

[0093] S312, obtaining a target quadruple from the experience pool.

[0094] Here, multiple quadruple groups can be randomly selected from the experience pool as target quadruple groups. Each quadruple group is represented by (s t , a t , R t ,s t+1 ) form. The t of each quadruple may be different.

[0095] S313: Determine a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple.

[0096] Here, the first reward value in the target quaternary can be the aforementioned R t , the first return value can be expressed as y it .

[0097] The first reward value corresponding to the target quadruple can be determined through the following steps B1-B3:

[0098] B1. Determine whether the moving target state corresponding to the second state data in the target quadruple is a terminal state.

[0099] Here, the moving target state is the state of the first moving target, and the second state data in the target quadruple can represent s t+1 , the moving target state is s t+1 The converted The termination state refers to the state in which the first moving target cannot continue to move in the navigation simulation environment. The termination state may include the following situations: (1) the first moving target moves to the target position; (2) the first moving target collides with other targets in the simulated navigation environment; (3) t is equal to L, that is, the path traversed by the first moving target reaches the maximum number of steps. Among them, it can be judged whether the moving target state corresponding to the second state data in the target quadruple is the termination state according to the running status of all targets after the first moving target output in the navigation simulation environment executes the action corresponding to the action data in the target quadruple.

[0100] B2. If the moving target state corresponding to the second state data in the target quadruple is a terminal state, the first reward value in the target quadruple is determined as the first return value.

[0101] Here the first return value y it =R t .

[0102] B3. If the moving target state corresponding to the second state data in the target quadruple is not the terminal state, determine the third reward value according to the second state data in the target quadruple, and determine the first reward value according to the first reward value and the third reward value.

[0103] The second state data in the target quadruple can be input into the value network of the previous training cycle of the current training cycle, and the reward value output by the value network of the previous training cycle is determined as the third reward value. The value network of the current training cycle can be expressed as V i , the value network of the previous training cycle of the current training cycle can be expressed as V i-1After the second state data in the target quadruple is input into the value network of the previous training cycle of the current training cycle, the reward value output by the value network of the previous training cycle of the current training cycle can be expressed as V i-1 (s t+1 ), that is, the third return value can be expressed as V i-1 (s t+1 ). The principle of the network outputting the reward value of the previous stage of the current training cycle can be referred to above Figure 2 and Figure 3 The description of the introduction to the value network will not be repeated here.

[0104] After obtaining the third reward value, the product of the third reward value and the attenuation coefficient can be calculated to obtain the attenuated reward value corresponding to the third reward value; the sum of the first reward value and the attenuated reward value is determined as the first reward value, that is, the first reward value is y it＝ R t +γV i-1 (s t+1 ).

[0105] S314, input the first state data in the target quadruple into the value network of the current training cycle, extract the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtain the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data.

[0106] Here, the first state data is the aforementioned s t The value of the network in the current training cycle is represented by V i , the second reward value output by the network in the current training cycle can be expressed as V i (s t ). The principle of the network outputting the second reward value at the current stage can be referred to the aforementioned Figure 2 and Figure 3 The description of the introduction to the value network will not be repeated here.

[0107] S315, adjusting the network parameters of the value network of the current training cycle according to the first reward value and the second reward value, so that the second reward value is close to the first reward value, and obtaining an action planning model.

[0108] Here, the loss of the value network can be determined according to the first reward value and the second reward value. The loss of the value network is expressed as [y it -V i (s t )] 2 ; Gradient update is used according to the loss of the value network to adjust the network parameters of the value network in the current training cycle to obtain the action planning model.

[0109] After steps S312 to S315, the training of the action planning model in the current training cycle is completed and the next training cycle is entered. After step S315, step S302 is executed.

[0110] S316, add 1 to i and return to step S302.

[0111] In the above steps S312 to S315, after obtaining the target quadruple from the experience pool, first determine the first reward value corresponding to the target quadruple according to the first reward value in the target quadruple; then input the first state data in the target quadruple into the value network, extract the state feature vector of the first state data in the target quadruple through the transformer feature extraction module in the value network, and obtain the second reward value output by the value network based on the state feature vector of the first state data; finally, adjust the network parameters of the value network according to the first reward value and the second reward value, so that the second reward value is close to the first reward value; by adjusting the network parameters of the value network so that the second reward value is close to the first reward value, the value network in the action planning model can learn the navigation obstacle avoidance strategy; by setting the transformer feature extraction module in the value network The module is used to extract the state feature vector of the state data of the target in the navigation environment. The multi-head attention mechanism in the transformer feature extraction module can better express the global information and more accurately express the interaction behavior between the first mobile target and other targets in the navigation environment. The value network of the action planning model can more accurately output a more accurate reward value based on the learned interaction behavior between the first mobile target and other targets, so that the action planner in the action planning model can complete precise action planning according to the more accurate reward value. Since the action planner in the action planning model completes action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, so it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0112] Through the above Figure 4 After the target value network is trained by the technical solution, the action planning model can be used to plan actions for the mobile device.

[0113] See also Figure 5 , Figure 5 A flow chart of an action planning method provided in an embodiment of the present application, which can be applied on a mobile device; Figure 5 As shown, the method comprises the following steps:

[0114] S401, obtaining target status data.

[0115] Here, the target state data includes the current state data of the first mobile target in the current environment and the current state data of the mobile targets other than the first mobile target in the current environment. For the concept of the first mobile target, please refer to the above description. The form and content of the target state data may refer to the above description. t .

[0116] S402, inputting the target state data into the motion planning model to obtain the current planned motion data of the first moving target.

[0117] The action planning model here includes a target value network and an action planner. The target value network is obtained by training the corresponding technical solutions in the above 4. The structure of the action planning model can be found in the above Figure 2 and Figure 3 The form and content of the current planned action data can be referred to the above a t The action planning model processes the target state number to obtain the current planned action data of the first moving target. Figure 2 and Figure 3 The introduction of the action planning model will not be repeated here.

[0118] exist Figure 5 In the corresponding technical solution, after obtaining the target state data, the target state data is input into the action planning model to obtain the current planned action data of the first mobile target in the current state, so as to realize the action planning of the mobile target; since the action planner in the action planning model completes the action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, so it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0119] The method of the present application is introduced above, and the device of the present application is introduced below.

[0120] See also Figure 6 , Figure 6 is a schematic diagram of a value network training device provided in an embodiment of the present application. The value network can be as follows Figure 2 and Figure 3 As shown. Figure 6 As shown, the value network training device 50 includes:

[0121] A quadruple acquisition module 501 is used to acquire a target quadruple from an experience pool, wherein the experience pool includes a plurality of quadruple collected in advance, each quadruple includes first state data, first action data, second state data and a first reward value, wherein the first state data includes state data of all targets in a navigation simulation environment at a first moment, the first action data is used to indicate an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data includes state data of all targets at a second moment, the second moment is a moment next to the first moment, and the first reward value is a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment;

[0122] A first reward value determination module 502, configured to determine a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple;

[0123] A second reward value determination module 503 is used to input the first state data in the target quadruple into the value network of the current training cycle, extract the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtain the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data;

[0124] The parameter adjustment module 504 is used to adjust the network parameters of the value network of the current training cycle according to the first reward value and the second reward value, so that the second reward value is close to the first reward value, and obtain the target value network.

[0125] In a possible design, the first state data includes a first state vector and a second state vector, the first state vector is used to represent the state of the first moving target at the first moment, and the second state vector is used to represent the state of all targets except the first moving target at the first moment; the above-mentioned second reward value determination module 503 is specifically used to: input the first state vector and the second state vector into the vector alignment module in the value network of the current training cycle, and perform vector mapping on the first state vector and the second state vector respectively through the vector alignment module in the value network of the current training cycle to obtain a first initial vector corresponding to the first state vector and a second initial vector corresponding to the second state vector, the first initial The dimension of the initial vector is the same as the dimension of the second initial vector; the first initial vector and the second initial vector are input into the vector concatenation module in the value network of the current training cycle, and the first initial vector and the second initial vector are concatenated row by row through the vector concatenation module in the value network of the current training cycle to obtain a state input matrix corresponding to the first state data, and the number of rows of the state input matrix is ​​(n+1), where n is the number of the other targets; the state input matrix is ​​input into the transformer feature extraction module in the value network of the current training cycle, and the state feature extraction is performed on the state input matrix through the transformer feature extraction module in the value network of the current training cycle to obtain the state feature vector.

[0126] In one possible design, the above-mentioned second reward value determination module 503 is specifically used to: input the state feature vector corresponding to the first state data into the multi-layer fully connected layer in the value network of the current training cycle, and obtain the reward value output by the last fully connected layer in the value network of the current training cycle as the second reward value.

[0127] In one possible design, the transformer feature extraction module includes multiple transformer feature extraction structures.

[0128] In one possible design, the above-mentioned first reward value determination module 502 is specifically used to: determine whether the moving target state corresponding to the second state data in the target quadruple is a terminal state, and the mobile target state is the state of the first mobile target; if the moving target state corresponding to the second state data in the target quadruple is a terminal state, then the first reward value in the target quadruple is determined as the first reward value; if the moving target state corresponding to the second state data in the target quadruple is not a terminal state, then determine the third reward value according to the second state data in the target quadruple, and determine the first reward value according to the first reward value and the third reward value.

[0129] In one possible design, the above-mentioned first reward value determination module 502 is specifically used to: input the second state data in the target quadruple into the value network of the previous training cycle of the current training cycle, and determine the reward value output by the value network of the previous training cycle as the third reward value.

[0130] In one possible design, the first reward value determination module 502 is specifically used to: calculate the product of the third reward value and the attenuation coefficient to obtain the attenuated reward value corresponding to the third reward value; and determine the sum of the first reward value and the attenuated reward value as the first reward value.

[0131] In a possible design, the above-mentioned value network training device 50 also includes an experience pool construction module 505, which is used to obtain the tth action data, where the tth action data is used to characterize the action performed by the first mobile target under the tth state data, and t is a positive integer greater than or equal to 0; control the first mobile target in the tth state data to perform the action corresponding to the tth action data in the navigation simulation environment, and obtain the (t+1)th state data and the tth reward value output by the navigation simulation environment; take the tth state data, tth action data, (t+1)th state data and tth reward value as a four-tuple, add it to the experience pool, and take the (t+1)th state data as the tth state data, and return to execute the step of obtaining the tth action data.

[0132] In a possible design, the above-mentioned experience pool construction module 505 is specifically used to: randomly sample actions for the first moving target in the tth state data to obtain random action data of the first moving target in the tth state data; if the random probability is greater than a preset probability threshold, the random action data is determined as the tth action data; if the random probability is less than or equal to the preset probability threshold, the tth state data is input into the action planning model containing the value network, and the planned action data output by the action planning model is determined as the tth action data.

[0133] It should be noted that Figure 6 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0134] After obtaining the target quadruple from the experience pool, the above-mentioned device first determines the first reward value corresponding to the target quadruple according to the first reward value in the target quadruple; then inputs the first state data in the target quadruple into the value network, extracts the state feature vector of the first state data in the target quadruple through the transformer feature extraction module in the value network, and obtains the second reward value output by the value network based on the state feature vector of the first state data; finally, according to the first reward value and the second reward value, adjusts the network parameters of the value network so that the second reward value is close to the first reward value; by adjusting the network parameters of the value network so that the second reward value is close to the first reward value, the value network can automatically learn the navigation obstacle avoidance strategy; by setting the transformer feature extraction module in the value network to extract the navigation environment The multi-head attention mechanism in the transformer feature extraction module can better express the global information, so as to more accurately express the interaction behavior between the first mobile target and other targets in the navigation environment. The value network can more accurately output a more accurate reward value based on the learned interaction behavior between the first mobile target and other targets, so that the action planner in the action planning model containing the value network can complete precise action planning according to the more accurate reward value; since the action planner in the action planning model completes action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, and thus it can be adapted to action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0135] See also Figure 7 , Figure 7 It is a structural diagram of an action planning device provided in an embodiment of the present application.

[0136] like Figure 7 As shown, the action planning device 60 includes:

[0137] A state acquisition module 601 is used to acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target;

[0138] The action planning module 602 is used to input the target state data into the action planning model to obtain the current planned action data of the first mobile target. The action planning model includes a target value network and an action planner. The target value network is obtained by the aforementioned Figure 4 The technical solution shown is trained to obtain that the target value network is used to output a reward value according to the state of the target, and the action planner is used to output planned action data according to the reward value output by the target value network.

[0139] After acquiring the target state data, the above-mentioned device inputs the target state data into the action planning model to obtain the current planned action data of the first mobile target in the current state, and can realize the action planning of the mobile target; because the action planner in the action planning model completes the action planning according to the reward value output by the value network, the state data of all targets are taken into account when performing action planning, and thus it can be adapted to perform action planning for mobile robots in dense scenes, reduce navigation time in dense scenes, and reduce the probability of collision.

[0140] See also Figure 8 , Figure 8 7 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, wherein the computer device 70 includes a processor 701 and a memory 702. The memory 702 is connected to the processor 701, for example, via a bus.

[0141] The processor 701 is configured to support the computer device 70 to perform the corresponding functions in the method in the above method embodiment. The processor 701 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0142] The memory 702 is used to store program codes, etc. The memory 702 may include a volatile memory (VM), such as a random access memory (RAM); the memory 702 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 702 may also include a combination of the above-mentioned types of memories.

[0143] In one possible scenario, the processor 701 may call the program code to perform the following operations:

[0144] Acquire a target quadruple from an experience pool, the experience pool comprising a plurality of quadruple collected in advance, each quadruple comprising first state data, first action data, second state data and a first reward value, the first state data comprising state data of all targets in a navigation simulation environment at a first moment, the first action data being used to indicate an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data comprising state data of all targets at a second moment, the second moment being a moment next to the first moment, and the first reward value being a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment;

[0145] Determining a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple;

[0146] Inputting the first state data in the target quadruple into the value network of the current training cycle, extracting the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtaining the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data;

[0147] According to the first reward value and the second reward value, the network parameters of the value network of the current training cycle are adjusted so that the second reward value approaches the first reward value, thereby obtaining a target value network.

[0148] In another possible scenario, the processor 701 may call the program code to perform the following operations:

[0149] Acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target;

[0150] The target state data is input into the action planning model to obtain the current planned action data of the first mobile target, wherein the action planning model includes a target value network and an action planner, wherein the target value network is obtained by the aforementioned Figure 4 The technical solution shown is trained to obtain that the target value network is used to output a reward value according to the state of the target, and the action planner is used to output planned action data according to the reward value output by the target value network.

[0151] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0152] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0153] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A value network training method, characterized in that: The value network is used to output a reward value according to the state of the target, and the value network includes a transformer feature extraction module; The method comprises: Acquire a target quadruple from an experience pool, the experience pool comprising a plurality of quadruple collected in advance, each quadruple comprising first state data, first action data, second state data and a first reward value, the first state data comprising state data of all targets in a navigation simulation environment at a first moment, the first action data being used to indicate an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data comprising state data of all targets at a second moment, the second moment being a moment next to the first moment, and the first reward value being a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment; Determining a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple; Inputting the first state data in the target quadruple into the value network of the current training cycle, extracting the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtaining the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data; According to the first reward value and the second reward value, the network parameters of the value network of the current training cycle are adjusted so that the second reward value approaches the first reward value, thereby obtaining a target value network.

2. The method according to claim 1, characterized in that The first state data includes a first state vector and a second state vector, the first state vector is used to represent the state of the first moving target at the first moment, and the second state vector is used to represent the states of all targets except the first moving target at the first moment; the value network also includes a vector alignment module and a vector splicing module; The step of inputting the first state data in the target quadruple into the value network of the current training cycle, and extracting the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle includes: Inputting the first state vector and the second state vector into a vector alignment module in a value network of a current training cycle, and performing vector mapping on the first state vector and the second state vector respectively through the vector alignment module in the value network of the current training cycle to obtain a first initial vector corresponding to the first state vector and a second initial vector corresponding to the second state vector, wherein the dimension of the first initial vector is the same as the dimension of the second initial vector; Inputting the first initial vector and the second initial vector into a vector concatenation module in a value network of a current training cycle, and concatenating the first initial vector and the second initial vector row by row through the vector concatenation module in the value network of the current training cycle to obtain a state input matrix corresponding to the first state data, wherein the number of rows of the state input matrix is ​​(n+1), and n is the number of the other targets; The state input matrix is ​​input into the transformer feature extraction module in the value network of the current training cycle, and the state feature extraction is performed on the state input matrix through the transformer feature extraction module in the value network of the current training cycle to obtain the state feature vector.

3. The method according to claim 2, characterized in that The value network also includes multiple fully connected layers connected in sequence; The obtaining of the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data includes: The state feature vector corresponding to the first state data is input into the multi-layer fully connected layer in the value network of the current training cycle, and the reward value output by the last fully connected layer in the value network of the current training cycle is obtained as the second reward value.

4. The method according to claim 1, characterized in that The transformer feature extraction module includes multiple transformer feature extraction structures.

5. The method according to any one of claims 1 to 4, characterized in that: The determining, according to the first reward value in the target quadruple, a first reward value corresponding to the target quadruple includes: Determine whether the moving target state corresponding to the second state data in the target quadruple is a termination state, the moving target state being the state of the first moving target; If the moving target state corresponding to the second state data in the target quadruple is a terminal state, determining the first reward value in the target quadruple as the first reward value; If the moving target state corresponding to the second state data in the target quadruple is not a terminal state, a third reward value is determined according to the second state data in the target quadruple, and the first reward value is determined according to the first reward value and the third reward value.

6. The method according to claim 5, characterized in that The determining of the third reward value according to the second state data in the target quadruple includes: The second state data in the target quadruple is input into the value network of the previous training cycle of the current training cycle, and the reward value output by the value network of the previous training cycle is determined as the third reward value.

7. The method according to claim 5, characterized in that The determining the first reward value according to the first reward value and the third reward value includes: Calculate the product of the third reward value and the attenuation coefficient to obtain the attenuated reward value corresponding to the third reward value; The sum of the first reward value and the decay reward value is determined as the first reward value.

8. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the target quadruple from the experience pool, the method further includes: Acquire t-th action data, where the t-th action data is used to represent an action performed by the first moving target under the t-th state data, where t is a positive integer greater than or equal to 0; Controlling the first mobile target in the tth state data to perform an action corresponding to the tth action data in the navigation simulation environment, and obtaining the (t+1)th state data and the tth reward value output by the navigation simulation environment; The tth state data, tth action data, (t+1)th state data and tth reward value are taken as a four-tuple, added to the experience pool, and the (t+1)th state data is taken as the tth state data, and the step of obtaining the tth action data is returned to execute.

9. The method according to claim 8, characterized in that The obtaining of the t-th action data comprises: Randomly sampling actions for a first moving target in the tth state data to obtain random action data of the first moving target in the tth state data; If the random probability is less than or equal to the preset probability threshold, the random action data is determined as the t-th action data; If the random probability is greater than a preset probability threshold, the t-th state data is input into the action planning model including the value network, and the planned action data output by the action planning model is determined as the t-th action data.

10. An action planning method, characterized in that: include: Acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target; The target state data is input into the action planning model to obtain the current planned action data of the first mobile target, the action planning model includes a target value network and an action planner, the target value network is trained by the method described in any one of claims 1-9, the target value network is used to output a reward value according to the state of the target, and the action planner is used to output the planned action data according to the reward value output by the target value network.

11. A value network training device, characterized in that: The value network is used to output a reward value according to the state of the target, and the value network includes a transformer feature extraction module; The device comprises: A quadruple acquisition module, used for acquiring a target quadruple from an experience pool, wherein the experience pool includes a plurality of quadruple collected in advance, each quadruple includes first state data, first action data, second state data and a first reward value, wherein the first state data includes state data of all targets in a navigation simulation environment at a first moment, the first action data is used for indicating an action performed by a first mobile target among all targets in the navigation simulation environment at the first moment, the second state data includes state data of all targets at a second moment, the second moment being a moment next to the first moment, and the first reward value is a reward value corresponding to the action performed by the first mobile target in the navigation simulation environment at the first moment; A first reward value determination module, used to determine a first reward value corresponding to the target quadruple according to the first reward value in the target quadruple; A second reward value determination module is used to input the first state data in the target quadruple into the value network of the current training cycle, extract the state feature vector corresponding to the first state data through the transformer feature extraction module in the value network of the current training cycle, and obtain the second reward value output by the value network of the current training cycle based on the state feature vector corresponding to the first state data; A parameter adjustment module is used to adjust the network parameters of the value network of the current training cycle according to the first reward value and the second reward value, so that the second reward value is close to the first reward value, and obtain a target value network.

12. An action planning device, characterized in that: include: A state acquisition module, used to acquire target state data, wherein the target state data includes current state data of a first mobile target in a current environment and current state data of other targets in the current environment except the first mobile target; An action planning module is used to input the target state data into an action planning model to obtain the current planned action data of the first mobile target, wherein the action planning model includes a target value network and an action planner, wherein the target value network is trained by the method described in any one of claims 1 to 9, wherein the target value network is used to output a reward value according to the state of the target, and the action planner is used to output the planned action data according to the reward value output by the target value network.

13. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 9 or claim 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 9 or claim 10.

Images