Demand-driven air-ground cooperative robust tracking method

Abstract

The invention relates to a demand-driven air-ground cooperative robust tracking method, and belongs to the field of heterogeneous robot cooperative adaptive tracking. In the air-ground cooperative adaptive tracking process, the decision strategy of the air-ground robot is influenced by the own demand and the target demand, and the air-ground robot can autonomously predict the target strategy by referring to the ERG demand theory and formulate the corresponding strategy according to the target strategy. On the basis of the ERG demand mechanism, when the target changes the escape strategy, the air-ground robot still can accurately predict the target intention strategy, and finally the air-ground robot team can realize robust and accurate tracking under the condition that the target strategy changes.

Description

Technical Field

[0001] The present invention belongs to the field of heterogeneous robot collaborative adaptive tracking, and in particular relates to a demand-driven air-ground collaborative robust tracking method. Background Art

[0002] Robots are known as the "crown jewel of the manufacturing industry". Robots with intelligence and autonomy as their main features are receiving more and more attention from countries around the world. However, the overall functions, task execution efficiency and fault tolerance of a single robot are relatively limited, making it difficult to adapt to complex tasks and dynamic environments. Therefore, many researchers have begun to turn their attention to multi-robot cluster systems with higher task execution efficiency and environmental adaptability. The heterogeneous multi-robot collaborative system composed of aerial robots and ground robots is a typical multi-robot cluster system. It combines the complementary advantages of aerial groups and ground groups, expands the two-dimensional plane of ground robot movement to three-dimensional space, and utilizes the advantages of aerial robots' flexibility and wide viewing angles, combined with the high load and strong computing and reasoning capabilities of ground robots. It has shown strong advantages in task execution efficiency, environmental adaptability and fault tolerance, and has successfully replaced humans to complete search and rescue, autonomous exploration, patrol and maintenance tasks in dangerous, dirty and extreme temperature environments.

[0003] However, facing complex tasks and dynamic environments, the current air-ground collaborative system still faces challenges such as low intelligence and weak autonomy. Based on this, applying artificial intelligence technology and natural group intelligence theory to air-ground collaborative systems is an effective solution to enhance individual intelligence and group intelligence. Although artificial intelligence (AI) technology represented by deep learning has been widely used in the fields of robot perception, decision-making and control in recent years, due to the high real-time requirements of air-ground robots, AI algorithms are often difficult to deploy directly on air-ground terminals, and a more efficient communication mechanism needs to be designed. At the same time, inspired by the activities of natural group intelligence, researchers at home and abroad have begun to introduce the study of natural group intelligence in the study of multi-robot systems. However, due to the high heterogeneity of the capabilities, structures and action spaces of air-ground robots, the research on multi-robot collaboration based on natural group intelligence still needs to consider additional challenges such as system complexity, dynamics, time-varying and coupling when applied to the field of air-ground collaboration.

[0004] There are many research works on the problem of air-ground collaborative tracking. The earliest works started from the perspective of planning and tracked targets in a static environment based on classical control theory. These studies often ignored the information source of the perception layer and relied on pre-processed data. And they set the same model for air-ground robots, that is, they adopted isomorphic strategies to solve the problem of air-ground heterogeneous robot collaboration. However, this approach does not fully utilize the complementary advantages of air and ground robots. Multi-Agent Deep Reinforcement Learning (MADRL) has been widely used in robot collaboration for a long time, especially in decision-making and planning tasks. To solve the problem of unstable environment, most works follow the centralized training distributed execution (CTDE) framework. This approach requires global information to be obtained through real-time communication during training. However, this situation is not common in the real world due to high bandwidth transmission requirements. In addition, the decision-making of each agent based on local observations during execution also limits the ability of multiple agents to cooperate. Recent studies aim to improve the efficiency of cooperation through effective communication between agents. They focus on reducing the consumption of communication resources during the collaboration process, mainly by selecting communication objects, learning communication content, and managing the frequency of communication. Although existing work has been able to persistently track targets with fixed strategies in a single environment, it is unable to adapt to changes in the target strategy, making the tracking effect less robust. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to avoid the shortcomings of the prior art, the present invention provides a demand-driven air-ground collaborative robust tracking method, which is used to achieve collaborative tracking between multiple intelligent agents, and adaptively adjust according to target requirements and their own needs to improve the robustness of tracking.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A demand-driven air-ground collaborative robust tracking method, characterized by comprising:

[0009] The aerial robot obtains a first semantic map and a first target direction in real time and inputs them into a first observation encoder; the first observation encoder obtains a first environmental feature based on the first semantic map and the first target direction and inputs it into a first demand predictor; the first demand predictor obtains a first target predicted demand based on the first environmental feature and inputs it into a demand alignment module;

[0010] The ground robot acquires the second semantic map and the second target direction in real time and inputs them into the second observation encoder; the second observation encoder acquires the second environmental feature based on the second semantic map and the second target direction and inputs it into the second demand predictor; the second demand predictor acquires the second target prediction demand based on the second environmental feature and inputs it into the demand alignment module;

[0011] The demand alignment module obtains a target demand based on the first target predicted demand and the second target predicted demand;

[0012] The target demand and the aerial robot's own demand are input into a first decision module, and the first decision module outputs instructions to adjust the aerial robot's continuous actions;

[0013] The target demand and the ground robot's own demand are input into a second decision module, and the second decision module outputs instructions to adjust the ground robot's continuous actions.

[0014] A further technical solution of the present invention is as follows: the first observation encoder and the second encoder have the same structure and adopt a CNN-LSTM framework, including:

[0015] The first semantic map is input into the CNN module for encoding to obtain a hidden vector. The hidden vector and the target direction are combined by MLP to obtain encoding information. The encoding information is input into the LSTM module to obtain environmental features, as shown in the following formula:

[0016]

[0017] in, Indicates robot R i The cell state at time step t, Indicates aerial robot R i The cell state at time step t-1, represents the environmental characteristics at time step t, represents the environmental characteristics at time step t-1, Indicates coded information; the subscript i is A or G, representing an aerial robot or a ground robot, respectively.

[0018] A further technical solution of the present invention: the first demand forecaster has the same structure as the first demand forecaster, and adopts three MLP neural networks to respectively predict three target demands; the three target demands include survival needs, relationship needs and growth and development needs.

[0019] A further technical solution of the present invention: the demand alignment module adopts an MLP neural network.

[0020] A further technical solution of the present invention is as follows: the first decision module and the second decision module have the same structure, including an actor and a critic, the actor is used to output continuous actions, and the critic is used to estimate the quality of the actions.

[0021] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0022] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0023] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0024] The beneficial effects of the present invention are:

[0025] The present invention provides a demand-driven air-ground collaborative robust tracking method. In the process of air-ground collaborative adaptive tracking, the decision-making strategy of the air-ground robot will be affected by its own needs and target needs. By drawing on the ERG demand theory, the air-ground robot can autonomously predict the target strategy and formulate corresponding strategies based on it. Based on this ERG demand mechanism, when the target changes its escape strategy, the air-ground robot can still accurately predict the target intention strategy, and finally the air-ground robot team can achieve robust and accurate tracking when the target strategy changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0027] Figure 1 It is a process diagram of the method framework in the example of the present invention.

[0028] Figure 2 It is a conceptual diagram of the theory of human social needs in the example of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0032] The present invention utilizes the following principle: in the process of air-ground collaborative adaptive tracking, the decision-making strategy of the air-ground robot will be affected by its own needs and the target needs. By drawing on the ERG demand theory, the air-ground robot can autonomously predict the target strategy and formulate corresponding strategies based on it. Based on this ERG demand mechanism, when the target changes its escape strategy, the air-ground robot can still accurately predict the target intention strategy, and finally the air-ground robot team can achieve robust and accurate tracking when the target strategy changes.

[0033] 1) Establishing a model for collaborative tracking of targets by air-ground robots

[0034] The air-to-ground robot needs to continuously and stably monitor dynamic targets under the constraints of the air-to-ground robot's field of view and communication distance, ensure that the target is always within the field of view of the air-to-ground robot, and minimize communication resources as much as possible. The tracking process can be seen as two air-to-ground heterogeneous intelligent agents (the aerial robot R A , ground robot R G ) Collaborative pursuit of target R T , the objective function is:

[0035]

[0036] in, Indicates that within time T, R A and R G With R T The sum of the distances between is defined as:

[0037]

[0038] Where dist(·,·) represents the Euclidean distance between two robots at time t.A , π G and π T Respectively represent the strategies of the aerial robot, the ground robot, and the target robot, θ A ,θ G and θ T Respectively represent its strategy parameters. Function are all calculation results at time t. Ensure Aerial Robot R A The visual range c1 must always cover the target R T , Ensure Aerial Robot R A and ground robot R G The distance between them must be less than their maximum communication distance c2, that is, there is a connection between the two. Ensure ground robot R G The distance to the obstacle must be greater than its safety distance c3. It means that the goal is to avoid being tracked by the air-to-ground robot as much as possible, and the entire tracking process is a zero-sum game.

[0039] 2) Reinforcement Learning Modeling

[0040] Based on the above-established air-ground robot collaborative tracking target model, after analyzing the problem, the entire air-ground collaborative tracking process can be modeled as a decentralized partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP) in Represent the state space, observation space, action space, immediate reward and state transition probability respectively. At each time step (superscript t represents time), the aerial robot R A With ground robot R G Get the observation status from the environment respectively And based on their respective strategies π A ,π G Take Action Then the environment state will be determined according to the state transition probability Update to the next state s t +1 At the same time, the air-ground robot obtains immediate rewards from the environment Aerial Robot R A The goal is to maximize the expected reward To learn strategies, ground robots R G Same reason.

[0041] Air-Ground Robot Observation: Aerial Robot R A Get the top-down semantic map and its relationship with the target R T The relative direction of each semantic graph is represented by a one-hot vector to represent free space, obstacles, and other objects. G Observation and aerial robot R A Similar, but with a wider field of view than Aerial Robot R A To make the tracking process more challenging, the target R T All observations within the entire environment can be obtained.

[0042] High-level actions of air-ground robots: Air-ground robots all use continuous action space a i . Ground Robot R G The McNamee wheels are used, so the direction can be changed freely. The speed of the ground robot is c,G It is a three-dimensional vector, including the angular velocity around the yaw axis and the linear velocity along the x and y axes. A The action space of the ground robot R G Similarly, both air-to-ground robots use high-level strategies, and low-level strategies are controlled by PID controllers.

[0043] Heterogeneous robot communication: Air-to-ground robots communicate through mobile self-organizing networks, and point-to-point information sharing can be carried out between robots.

[0044] Reward function: According to the ERG (Existence Relatedness Growth) demand theory, the goal strategy of the air-ground robot is affected by its own needs and the needs of others. The air-ground robot should maximize its own needs as much as possible, that is, its own needs are the intrinsic motivation for the behavior decision of the air-ground robot. It pursues the maximization of the reward function. In particular, they are defined as:

[0045]

[0046] Among them, μ1,μ2>0,γ1,γ2>0,ε1,ε2>0 are hyperparameters. It represents the survival needs reward. The air-to-ground robot will develop other needs based on its own survival needs. When the distance between the air-to-ground robot and the obstacle is less than its safe distance, it will be punished. The specific definition is:

[0047]

[0048] Where n is the number of obstacles in the environment. Represents a mutual relationship reward. The air-ground robot will maintain a close relationship within its communication range. When the air-ground robot is within its communication range, a reward is given. The specific definition is:

[0049]

[0050] Among them, d safety is the minimum safe distance between air-ground robots. It represents the growth and development needs reward. The goal of the air-to-ground robot is to chase the target, so it is necessary to ensure that the target is within its field of view as much as possible. When the distance between the robot and the target exceeds its maximum field of view, it is meaningless, which is defined as:

[0051]

[0052] Among them, dist(R i ,R T ) represents the real-time distance between the air-ground robot and the target, D obs Indicates the maximum observation distance of the robot, d * Indicates the optimal observation distance.

[0053] At the same time, the behavior of the target is also driven by its internal needs, so various levels of demand rewards are set for it. It is worth noting that the survival reward of the target is the same as that of the air-ground robot, and both must ensure that it does not collide with obstacles. The definitions of mutual relationship rewards and development demand rewards are as follows:

[0054]

[0055] The target's mutual relationship requirement is to escape the tracking of the air-ground robot. The farther away from the air-ground robot, the more satisfied the requirement is, and the greater the reward is. The target's growth and development requirement is to reach the destination as much as possible. The closer to the destination, the greater the reward is. The tracking-escaping process between the target and the air-ground robot is a zero-sum game, so the overall reward of the target can be defined as:

[0056]

[0057] Among them, μ3,γ3,ε3>0 are hyperparameters

[0058] 3) Independent reinforcement learning framework design

[0059] Each air-to-ground robot adopts a personalized model. A Includes observation encoder module and demand prediction module.

[0060] The observation encoder module adopts the CNN-LSTM framework and consists of two parts: first, visual information That is, the semantic map will be input into the CNN module to encode the hidden vector features, the hidden vector features and the target direction The encoded information is then merged into the MLP to obtain the encoded information, and then passed into the LSTM module to obtain the environmental features, as shown in the formula:

[0061]

[0062] in, Indicates aerial robot R A The cell state at time step t, the environment characteristics Can implicitly represent the characteristics of the target, To encode information.

[0063] The demand forecasting module is used to predict target demand. It uses three MLP neural networks to predict three target demands respectively.

[0064] The demand alignment module is used to unify the target demand predicted by the aerial robot and the target demand predicted by the ground robot.

[0065] Environmental characteristics The unified target requirements are passed into the decision module, and the decision module finally has a continuous actor for outputting continuous actions and a critic for estimating the quality of the action.

[0066] With Aerial Robot R A Similarly, the ground robot R G The visual input is sent to the CNN and MLP modules for encoding features and target demand prediction. Then the behavior is decided based on the environmental features and the predicted target demand. The reinforcement learning module then uses the target demand and its own observations to output continuous actions.

[0067] During the training process, each robot is trained using the proximal policy optimization algorithm.

[0068] Based on the above-mentioned air-ground robot collaborative tracking target model, reinforcement learning modeling is performed on the air-ground robot collaborative tracking target model, and an independent reinforcement learning framework is designed for the reinforcement learning model. The present invention provides a demand-driven air-ground collaborative robust tracking method. First, the air-ground robots respectively obtain their first-person observation states, and use the ERG (Existence Relatedness Growth) demand prediction module to predict the future demand of the target. Then, taking into account the heterogeneous characteristics of the air-ground robots, the target demands predicted by the air-ground robots will undergo a process of value unification; then the unified target demands will be input into their respective decision-making modules, and the reinforcement learning modules will determine their own actions based on the observation information and target demands. Figure 1 As shown, specifically including:

[0069] The aerial robot obtains a first semantic map and a first target direction in real time and inputs them into a first observation encoder; the first observation encoder obtains a first environmental feature based on the first semantic map and the first target direction and inputs it into a first demand predictor; the first demand predictor obtains a first target predicted demand based on the first environmental feature and inputs it into a demand alignment module;

[0070] The ground robot acquires the second semantic map and the second target direction in real time and inputs them into the second observation encoder; the second observation encoder acquires the second environmental feature based on the second semantic map and the second target direction and inputs it into the second demand predictor; the second demand predictor acquires the second target prediction demand based on the second environmental feature and inputs it into the demand alignment module;

[0071] The demand alignment module obtains a target demand based on the first target predicted demand and the second target predicted demand;

[0072] The target demand and the aerial robot's own demand are input into a first decision module, and the first decision module outputs instructions to adjust the aerial robot's continuous actions;

[0073] The target demand and the ground robot's own demand are input into a second decision module, and the second decision module outputs instructions to adjust the ground robot's continuous actions.

[0074] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A demand-driven air-ground collaborative robust tracking method, characterized in that: include: The aerial robot obtains the first semantic map and the first target direction in real time and inputs them into a first observation encoder; The first observation encoder obtains a first environmental feature based on the first semantic map and the first target direction and inputs the first demand predictor; the first demand predictor obtains a first target predicted demand based on the first environmental feature and inputs the first target predicted demand into a demand alignment module; The ground robot acquires the second semantic map and the second target direction in real time and inputs them into the second observation encoder; the second observation encoder acquires the second environmental feature based on the second semantic map and the second target direction and inputs it into the second demand predictor; the second demand predictor acquires the second target prediction demand based on the second environmental feature and inputs it into the demand alignment module; The demand alignment module obtains a target demand based on the first target predicted demand and the second target predicted demand; The target demand and the aerial robot's own demand are input into a first decision module, and the first decision module outputs instructions to adjust the aerial robot's continuous actions; The target demand and the ground robot's own demand are input into a second decision module, and the second decision module outputs instructions to adjust the ground robot's continuous actions.

2. A demand-driven air-ground collaborative robust tracking method according to claim 1, characterized in that: include: The first observation encoder has the same structure as the second encoder, using a CNN-LSTM framework, including: The first semantic map is input into the CNN module for encoding to obtain a hidden vector. The hidden vector and the target direction are combined by MLP to obtain encoding information. The encoding information is input into the LSTM module to obtain environmental features, as shown in the following formula: in, Indicates robot R i The cell state at time step t, Indicates aerial robot R i The cell state at time step t-1, represents the environmental characteristics at time step t, represents the environmental characteristics at time step t-1, Indicates coded information; the subscript i is A or G, representing an aerial robot or a ground robot, respectively.

3. The demand-driven air-ground collaborative robust tracking method according to claim 1, characterized in that: The first demand forecaster has the same structure as the first demand forecaster, and adopts three MLP neural networks to respectively predict three target demands; the three target demands include survival demands, relationship demands and growth and development demands.

4. The demand-driven air-ground collaborative robust tracking method according to claim 1, characterized in that: The demand alignment module adopts an MLP neural network.

5. The demand-driven air-ground collaborative robust tracking method according to claim 1, characterized in that: The first decision module has the same structure as the second decision module, including an actor and a critic, wherein the actor is used to output continuous actions, and the critic is used to estimate the quality of the actions.

6. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

7. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

8. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images