An autonomous navigation method for intelligent agents to avoid collision in dynamic scenes

Abstract

The present invention discloses an intelligent agent autonomous navigation method for avoiding collision in a dynamic scene, including constructing a state feature vector descriptor of an intelligent agent and an obstacle in a dynamic environment; building a Trans+GNN network model for representing and extracting the state features of the intelligent agent and the environment, and obtaining the spatiotemporal state information generated in the process of interaction between the intelligent agent and the environment; using a value-based deep reinforcement learning method to train the model, the intelligent agent learns through trial and error in a dynamic environment, combined with a reward function, with the goal of maximizing the reward, to obtain an optimal navigation strategy for the intelligent agent to perform navigation and path planning tasks in a dynamic environment. It can comprehensively represent and extract the spatial state information in the environment and the historical trajectory information generated in the process of interaction between the intelligent agent and the environment, that is, the spatiotemporal information, thereby generating a safe and efficient navigation strategy, greatly improving the ability of the intelligent agent to navigate and plan paths in a complex dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent body navigation, and in particular relates to an intelligent body autonomous navigation method for avoiding collision in dynamic scenes. Background Art

[0002] With the rapid development of robotics technology, various types of robots are entering people's daily lives and playing an important role in logistics, transportation, warehouse management, autonomous driving and other application scenarios. Intelligent agent autonomous navigation technology allows robots to complete navigation and path planning tasks safely and efficiently, which is the basis for robots to realize other functions. Therefore, the research on intelligent agent autonomous navigation methods has important practical significance.

[0003] Traditional navigation methods are mainly based on the perception of the environment and obstacles, and use large-scale fast search algorithms to plan a feasible path to the navigation target point. This type of method regards the environment and obstacles as static, so it performs poorly when dealing with navigation tasks in complex and dynamic environments. With the development of artificial intelligence technology, methods based on deep reinforcement learning have been widely used in intelligent agent autonomous navigation tasks, and have achieved satisfactory navigation results in large-scale dynamic environments. This type of method mainly uses neural networks to extract the characteristics of the environment and obstacles. The intelligent agent learns navigation strategies by interacting with the environment and obtains the optimal navigation strategy by maximizing rewards.

[0004] However, in the currently commonly used deep reinforcement learning-based intelligent agent autonomous navigation methods, how to effectively represent the state, extract key information and use it to learn the optimal navigation strategy remains a challenge. Summary of the invention

[0005] In view of the above technical problems, the present invention provides an intelligent agent autonomous navigation method for avoiding collision in dynamic scenes.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] An intelligent agent autonomous navigation method for avoiding collision in a dynamic scene, the method comprising the following steps:

[0008] S100: Construct state feature vector descriptors of the agent and obstacles in a dynamic environment to describe the motion state and spatial position information of the agent and environment in the dynamic scene;

[0009] S200: Build a Trans+GNN network model to represent and extract the state characteristics of the agent and the environment, and obtain the spatiotemporal state information generated during the interaction between the agent and the environment;

[0010] S300: The Trans+GNN network model is trained using a value-based deep reinforcement learning method. The intelligent agent learns through trial and error in a dynamic environment. Combined with the reward function, the optimal navigation strategy is obtained with the goal of maximizing the reward. This strategy is used for the intelligent agent to perform navigation and path planning tasks in a dynamic environment.

[0011] Preferably, the state feature vector descriptors of the agent and obstacles in the dynamic environment in S100 are specifically:

[0012] ;

[0013] ;

[0014] In the formula express The state of the agent at any moment, express In the moment environment obstacle status; Represents the position coordinates, Indicates speed, Indicates its own radius; represents the direction angle of the agent, Represents the target point of the agent's navigation.

[0015] Preferably, S200 includes:

[0016] S210: Use feature extraction network to , and The states of the agent and obstacles at three moments are extracted to obtain preliminary features ;

[0017] S220: Using the parallel network structure of Transformer network and graph neural network GNN, the initial features Further feature extraction obtains time state features and spatial state characteristics , the two are spliced ​​and fused to obtain features with spatiotemporal state information ;

[0018] S230: The feature containing the spatiotemporal state information Input into the value prediction network to obtain the predicted value , and further generate navigation strategies , used for intelligent agents to perform navigation and path planning tasks in dynamic environments.

[0019] Preferably, the input of the feature extraction network in S210 is time, Time and The state of the agent and obstacles at each moment, represents the input data, where "3" represents 3 moments, "N" represents the number of obstacles, and "L" represents the length of the state sequence of the agent and obstacles. The feature extraction network includes a linear layer and an activation function Gelu, which can convert the input data Transform to a higher dimension for feature extraction, and its output is the preliminary feature , the expression is as follows:

[0020] ;

[0021] In the formula represents a linear layer with activation function Gelu, is the weight matrix of the linear layer.

[0022] Preferably, the Transformer network includes a multi-head self-attention layer, a feedforward network and a layer normalization. In S220, the Transformer network is used to perform the initial feature Perform feature extraction to obtain time state features , specifically:

[0023] ;

[0024] In the formula, represents a multi-head self-attention layer, represents the feed-forward network layer, Representation layer normalization.

[0025] Preferably, the graph neural network includes a graph convolution layer, a ReLU function, a global average pooling layer and a linear layer. In S220, the graph neural network GNN is used to perform the initial feature Perform feature extraction to obtain spatial state features , specifically:

[0026] ;

[0027] in, represents a linear layer with activation function ReLU, is the adjacency matrix of the graph, is the number of nodes, express The first node, Represent the weight matrices of the three-layer graph convolutional layer in turn, Represents the weight matrix of the Linear layer.

[0028] Preferably, the value prediction network includes a linear layer and GeLU, and S230 will contain features of spatiotemporal state information Input into the value prediction network to obtain the predicted value , specifically:

[0029] ;

[0030] In the formula, represents a linear layer with activation function Gelu, Represents the weight matrices of the three linear layers in sequence.

[0031] Preferably, in S300, a value-based deep reinforcement learning method is used to train the Trans+GNN network model, including:

[0032] S310: Initialize the value function V and the target value function V', initialize the experience playback memory and the total number of training times;

[0033] S320: The agent executes the current action a, calculates the value function V, and updates the state s, stores the state-value correspondence data into the experience replay memory, randomly samples small batches of data from the experience replay memory, and uses the gradient descent method to optimize the value function V;

[0034] S330: When the agent reaches the target point or times out, the target value function V' is updated, and when the number of training times reaches the total number of training times, the training of the Trans+GNN network model is completed.

[0035] Preferably, the reward function expression in S300 is:

[0036] ;

[0037] In the formula, Indicates that the agent is The state of the moment, Indicates that the agent is The action of the moment, is the minimum distance between the agent and the obstacle during navigation, is the minimum distance threshold set, is the distance between the agent and the navigation target point.

[0038] Preferably, in S300 The maximum value function at the moment is expressed as , the optimal navigation strategy is expressed as , The expression is as follows:

[0039] ;

[0040] in, express The state of obstacles at all times, Indicates from time The number of decisions from the state to the final state; is the discount factor, with a range of ;

[0041] Suppose that in a short period of time If the speed of the obstacles in the environment remains unchanged at the previous moment, the state of the agent and the obstacles at the next moment is predicted. , optimal navigation strategy It is expressed as:

[0042] .

[0043] The above-mentioned method for autonomous navigation of an intelligent agent to avoid collision in a dynamic scenario provides a network model (Trans+GNN) of parallel Transformer network and graph neural network GNN to represent state characteristics. The Transformer network is good at processing sequence data and can capture the historical trajectory information of the interaction between the intelligent agent and the environment; GNN is good at processing spatial relationships and can accurately understand the spatial layout of the environment where the intelligent agent is located. It can comprehensively and effectively represent the feature information of space and time, and at the same time use value-based reinforcement learning methods to find the optimal navigation strategy, which can enable the intelligent agent to navigate safely and efficiently in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of an intelligent agent autonomous navigation method for avoiding collision in a dynamic scene in one embodiment of the present invention;

[0045] Figure 2 This is a Trans+GNN network structure diagram proposed in one embodiment of the present invention;

[0046] Figure 3 This is a flow chart of Trans+GNN network training in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0048] In one embodiment, Figure 1 As shown, an intelligent agent autonomous navigation method for avoiding collision in a dynamic scene, the method comprising the following steps:

[0049] S100: Construct state feature vector descriptors of the agent and obstacles in a dynamic environment to describe the motion state and spatial position information of the agent and the environment in the dynamic scene.

[0050] Specifically, the state feature well describes the motion state and position information of the agent and obstacles in the current scene, and also implies the information of their relative spatial positions. This step ensures the effectiveness of the subsequent feature extraction.

[0051] S200: Build a Trans+GNN network model to represent and extract the state characteristics of the agent and the environment, and obtain the spatiotemporal state information generated during the interaction between the agent and the environment.

[0052] Specifically, the Trans+GNN network fully utilizes and integrates the Transformer network's ability to process time series data and the graph neural network GNN's ability to extract spatial information. Therefore, it can effectively extract the key spatiotemporal state information generated during the interaction between the intelligent agent and the environment and obtain spatiotemporal features.

[0053] S300: The Trans+GNN network model is trained using a value-based deep reinforcement learning method. The intelligent agent learns through trial and error in a dynamic environment. Combined with the reward function, the optimal navigation strategy is obtained with the goal of maximizing the reward. This strategy is used for the intelligent agent to perform navigation and path planning tasks in a dynamic environment.

[0054] Specifically, the optimal navigation strategy can be used for intelligent agents to perform safe and efficient navigation tasks in complex dynamic environments.

[0055] The Trans+GNN network model proposed in the present invention fuses the Transformer network and the GNN network in parallel to utilize the Transformer's ability to process time series data and the GNN's ability to extract spatial information, thereby comprehensively representing and extracting the spatial state information in the environment and the historical trajectory information generated during the interaction between the agent and the environment. Therefore, the present invention provides a feasible solution for effectively representing the state and extracting key information in feature extraction, thereby improving the safety and efficiency of the agent's navigation and path planning in complex dynamic environments.

[0056] The goal of the intelligent agent navigation task in a complex dynamic environment is: when the intelligent agent receives the coordinates of the navigation target point, it starts from the starting position and follows the autonomously planned path to reach the target point as quickly as possible without collision. The motion state of obstacles in the environment (such as pedestrians, vehicles, etc.) changes in real time. During the navigation process, considering that both the intelligent agent and the obstacles themselves have certain physical shapes, the intelligent agent should maintain a certain distance from the obstacles during the obstacle avoidance process to ensure safety.

[0057] Furthermore, S100 specifically includes constructing state feature vector descriptors for the intelligent agent and obstacles respectively in a dynamic scenario, taking into account information such as the motion state, spatial position and physical parameters of the intelligent agent and obstacles at any time, so as to ensure effective feature information extraction in subsequent steps.

[0058] In one embodiment, the state feature vector descriptor of the agent and the obstacle in the dynamic environment in S100 is specifically:

[0059] ;

[0060] ;

[0061] In the formula express The state of the agent at any moment, express In the moment environment obstacle status; Represents the position coordinates, Indicates speed, Indicates its own radius; represents the direction angle of the agent, Represents the target point of the agent's navigation.

[0062] Furthermore, the agent The action at a given moment can be expressed as a discrete variable express, ,in Indicates the speed of the agent, the value range is ; Indicates the speed direction angle of the intelligent agent, the value range is .

[0063] In one embodiment, S200 includes:

[0064] S210: Use feature extraction network to , and The states of the agent and obstacles at three moments are extracted to obtain preliminary features ;

[0065] S220: Using the parallel network structure of Transformer network and graph neural network GNN, the initial features Further feature extraction obtains time state features and spatial state characteristics , the two are spliced ​​and fused to obtain features with spatiotemporal state information ;

[0066] S230: The feature containing the spatiotemporal state information Input into the value prediction network to obtain the predicted value , and further generate navigation strategies , used for intelligent agents to perform navigation and path planning tasks in dynamic environments.

[0067] In one embodiment, the input of the feature extraction network in S210 is time, Time and The state of the agent and obstacles at each moment, represents the input data, where "3" represents 3 moments, "N" represents the number of obstacles, and "L" represents the length of the state sequence of the agent and obstacles. The feature extraction network includes a linear layer and an activation function Gelu, which can convert the input data Transform to a higher dimension for feature extraction, and its output is the preliminary feature , the expression is as follows:

[0068] ;

[0069] In the formula represents a linear layer with activation function Gelu, is the weight matrix of the linear layer.

[0070] In one embodiment, the feature with spatiotemporal state information in S22 , which is expressed as follows:

[0071] ;

[0072] In the formula, Represents the temporal state characteristics of the Transformer network output, Represents the spatial state features of the graph neural network output.

[0073] Specifically, Transformer is good at processing time series data. Through the multi-head self-attention mechanism, it calculates the correlation between the time series such as the position and speed of dynamic obstacles at different times, and effectively extracts key time information such as the motion trajectory and speed change of dynamic obstacles; by introducing a residual structure, it alleviates the gradient vanishing problem and helps the network to train better; by introducing a feedforward network, it uses nonlinear transformations to further enhance the expressiveness of the model.

[0074] In one embodiment, the Transformer network includes a multi-head self-attention layer, a feed-forward network, and a layer normalization. In S220, the Transformer network is used to perform the initial feature Perform feature extraction to obtain time state features , specifically:

[0075] ;

[0076] In the formula, represents a multi-head self-attention layer, represents the feed-forward network layer, Representation layer normalization.

[0077] Furthermore, the graph neural network (GNN) captures the spatial distribution relationship between agents and obstacles in the environment by stacking multiple graph convolution layers. It models the environment as a graph, where nodes represent obstacles in the environment and edges represent the spatial relationship between them. The graph convolution layer can aggregate the information of adjacent nodes of each node, extract and integrate the local spatial information around the node; the activation function ReLU introduces nonlinear transformation to improve the expressiveness of the model; global average pooling can aggregate the node information output by multiple graph convolution layers.

[0078] In one embodiment, the graph neural network includes a graph convolution layer, a ReLU function, a global average pooling layer, and a linear layer. In S220, the graph neural network GNN is used to perform the initial feature Perform feature extraction to obtain spatial state features , specifically:

[0079] ;

[0080] in, represents a linear layer with activation function ReLU, is the adjacency matrix of the graph, is the number of nodes, express The first node, Represent the weight matrices of the three-layer graph convolutional layer in turn, Represents the weight matrix of the Linear layer.

[0081] In one embodiment, the value prediction network includes a linear layer and GeLU, and S230 includes features of spatiotemporal state information. Input into the value prediction network to obtain the predicted value , specifically:

[0082] ;

[0083] In the formula, represents a linear layer with activation function Gelu, Represents the weight matrices of the three linear layers in sequence.

[0084] Furthermore, we use a value-based deep reinforcement learning method, taking the state feature vector of the environment as input and the value function As the optimization target. In each round of training, the agent performs actions, and the environment updates the state, feedback rewards and value functions After training is completed, according to the updated maximum value function The optimal navigation strategy can be obtained.

[0085] In one embodiment, in S300, a value-based deep reinforcement learning method is used to train the model, including:

[0086] S310: Initialize the value function and the target value function, initialize the experience playback memory and the total number of training times;

[0087] S320: The agent executes the current action, calculates the value function, and updates the state, stores the state-value correspondence data into the experience replay memory, randomly samples small batches of data from the experience replay memory, and optimizes the value function using the gradient descent method;

[0088] S330: When the agent reaches the target point or times out, the target value function is updated, and when the number of training times reaches the total number of training times, the training of the Trans+GNN network model is completed.

[0089] Specifically, the value function is used in reinforcement learning to estimate the expected long-term cumulative reward (value) of the agent under a given state (state-action pair). The target value function is an expression used to calculate the update target during the update of the value function, which is used to guide the update of the value function. The state of the input at any time , output value , and further generate navigation strategies , used for intelligent agents to perform navigation and path planning tasks in dynamic environments.

[0090] In one embodiment, the reward function expression in S300 is:

[0091] ;

[0092] In the formula, Indicates that the agent is The state of the moment, Indicates that the agent is The action of the moment, is the minimum distance between the agent and the obstacle during navigation, is the minimum distance threshold set, is the distance between the agent and the navigation target point.

[0093] Specifically, when the agent successfully reaches the target point without collision, the reward is positive; when the agent collides or the distance between it and the obstacle is too small, the reward is negative; in other cases, the reward is .

[0094] In the value-based reinforcement learning method, the optimal strategy is directly related to the maximum function. In one embodiment, S300 The maximum value function at the moment is expressed as , the optimal navigation strategy is expressed as , The expression is as follows:

[0095] ;

[0096] in, express The state of obstacles at all times, Indicates from time The number of decisions from the state to the final state; is the discount factor, with a range of ;

[0097] Suppose that in a short period of time If the speed of the obstacles in the environment remains unchanged at the previous moment, the state of the agent and the obstacles at the next moment is predicted. , optimal navigation strategy It is expressed as:

[0098] .

[0099] The above-mentioned method for autonomous navigation of an intelligent agent to avoid collision in a dynamic scenario provides a network model (Trans+GNN) of parallel Transformer network and graph neural network GNN to represent state characteristics. The Transformer network is good at processing sequence data and can capture the historical trajectory information of the interaction between the intelligent agent and the environment; GNN is good at processing spatial relationships and can accurately understand the spatial layout of the environment where the intelligent agent is located. It can comprehensively and effectively represent the feature information of space and time, and at the same time use value-based reinforcement learning methods to find the optimal navigation strategy, which can enable the intelligent agent to navigate safely and efficiently in complex dynamic environments.

[0100] The above is a detailed introduction to the intelligent autonomous navigation method for avoiding collision in a dynamic scene provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An intelligent agent autonomous navigation method for avoiding collision in dynamic scenes, characterized in that: The method comprises the following steps: S100: Construct state feature vector descriptors of the agent and obstacles in a dynamic environment to describe the motion state and spatial position information of the agent and environment in the dynamic scene; S200: Build a Trans+GNN network model to represent and extract the state characteristics of the agent and the environment, and obtain the spatiotemporal state information generated during the interaction between the agent and the environment; S200 includes: S210: The feature extraction network includes a linear layer and an activation function Gelu. The feature extraction network is used to extract , and The states of the agent and obstacles at three moments are extracted to obtain preliminary features ; S220: The Transformer network includes a multi-head self-attention layer, a feedforward network, and layer normalization. The graph neural network includes a graph convolution layer, a ReLU function, a global average pooling, and a linear layer. The Transformer network and the graph neural network GNN are connected in parallel to perform preliminary feature extraction. Further feature extraction obtains time state features and spatial state characteristics , the two are spliced ​​and fused to obtain features with spatiotemporal state information ; S230: The feature containing the spatiotemporal state information Input into the value prediction network to obtain the predicted value , and further generate navigation strategies , used for intelligent agents to perform navigation and path planning tasks in dynamic environments; S300: The Trans+GNN network model is trained using a value-based deep reinforcement learning method. The intelligent agent learns through trial and error in a dynamic environment. Combined with the reward function, the optimal navigation strategy is obtained with the goal of maximizing the reward. This strategy is used for the intelligent agent to perform navigation and path planning tasks in a dynamic environment.

2. The method according to claim 1, characterized in that The state feature vector descriptors of the agent and obstacles in the dynamic environment in S100 are: ； ； In the formula express The state of the agent at each moment, express In the moment environment obstacle status; Represents the position coordinates, Indicates speed, Represents its own radius; represents the direction angle of the agent, Represents the target point of the agent's navigation.

3. The method according to claim 2, characterized in that The input of the feature extraction network in S210 is time, Time and The state of the agent and obstacles at each moment, represents the input data, where "3" represents 3 moments, "N" represents the number of obstacles, and "L" represents the length of the state sequence of the agent and obstacles. The feature extraction network will input the data Transform to a higher dimension for feature extraction, and its output is the preliminary feature , the expression is as follows: ； In the formula represents a linear layer with activation function Gelu, is the weight matrix of the linear layer.

4. The method according to claim 3, characterized in that: In S220, the Transformer network is used to analyze the initial features. Perform feature extraction to obtain time state features , specifically: ； In the formula, represents a multi-head self-attention layer, represents the feed-forward network layer, Representation layer normalization.

5. The method according to claim 3, characterized in that: S220 uses graph neural network GNN to analyze the initial features Perform feature extraction to obtain spatial state features , specifically: ； in, represents a linear layer with activation function ReLU, is the adjacency matrix of the graph, is the number of nodes, express The first node, Represent the weight matrices of the three-layer graph convolutional layer in turn, Represents the weight matrix of the Linear layer.

6. The method according to claim 5, characterized in that The value prediction network includes linear layers and GeLU, and S230 will include features of spatiotemporal state information Input into the value prediction network to obtain the predicted value , specifically: ； In the formula, represents a linear layer with activation function Gelu, Represents the weight matrices of the three linear layers in sequence.

7. The method according to claim 6, characterized in that In S300, the Trans+GNN network model is trained using a value-based deep reinforcement learning method, including: S310: Initialize the value function V and the target value function V', initialize the experience playback memory and the total number of training times; S320: The agent executes the current action a, calculates the value function V, and updates the state s, stores the state-value correspondence data into the experience replay memory, randomly samples small batches of data from the experience replay memory, and uses the gradient descent method to optimize the value function V; S330: When the agent reaches the target point or times out, the target value function V' is updated, and when the number of training times reaches the total number of training times, the training of the Trans+GNN network model is completed.

8. The method according to claim 7, characterized in that The reward function expression in S300 is: ； In the formula, Indicates that the agent is The state of the moment, Indicates that the agent is The action of the moment, is the minimum distance between the agent and the obstacle during navigation, is the minimum distance threshold set, is the distance between the agent and the navigation target point.

9. The method according to claim 8, characterized in that S300 The maximum value function at the moment is expressed as , the optimal navigation strategy is expressed as , The expression is as follows: ； in, express The state of obstacles at all times, Indicates from time The number of decisions from the state to the final state; is the discount factor, with a range of ; Suppose that in a short period of time If the speed of the obstacles in the environment remains unchanged at the previous moment, the state of the agent and the obstacles at the next moment is predicted. , optimal navigation strategy It is expressed as: ； in, Indicates the next moment The value function of .

Images