Reinforcement learning-based denial environment collaborative autonomous perception and navigation method

Abstract

The invention relates to a collaborative autonomous perception and navigation method in a denial environment, which comprises the following steps: constructing a self-organizing robust communication system to realize dynamic networking and data sharing among multiple unmanned aerial vehicles in the denial environment, and adopting a self-adaptive network protocol to realize automatic configuration of a communication network under network topology change or communication interference; based on a multi-sensor fusion sensing system, environment information and motion information of the unmanned aerial vehicle are processed through a data fusion algorithm; a single unmanned aerial vehicle constructs a reward function based on unmanned aerial vehicle energy consumption, an exploration area, a path overlapping degree and a path danger degree, executes autonomous perception and exploration in an unknown environment and completes mapping; the distributed multiple unmanned aerial vehicles carry out individual communication and game playing in the unmanned aerial vehicle cluster according to map information, respective positions and working states, task distribution is achieved, and path planning and action decision making are completed. According to the invention, autonomy and collaboration of the unmanned aerial vehicle in a denial environment are improved, and an effective solution is provided for navigation in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of robot autonomous perception and navigation technology, and in particular to a method for collaborative autonomous perception and navigation in a denied environment based on reinforcement learning. Background Art

[0002] Autonomous navigation technology plays a vital role in the field of robotics and unmanned systems. In recent years, autonomous navigation methods based on deep learning and reinforcement learning have gradually become a research hotspot. Reinforcement learning continuously adjusts strategies through interaction with the environment to achieve optimal decisions, and has shown great potential in autonomous navigation. Reinforcement learning is a machine learning method that learns optimal strategies by interacting with the environment. Its core idea is that the agent adjusts its strategy according to the rewards fed back by the environment by performing actions in the environment to maximize the cumulative rewards. Compared with traditional path planning methods, reinforcement learning does not rely on accurate environmental models and can adapt to dynamically changing environments. In recent years, deep reinforcement learning has combined the powerful representation ability of deep learning, making reinforcement learning have made breakthrough progress in high-dimensional perception and complex decision-making tasks.

[0003] Denied environments refer to areas that are strongly interfered with or completely blocked from external signals (such as GPS). Such environments are common in military applications, post-disaster search and rescue, underground mines, and space exploration. In denied environments, traditional navigation methods such as GPS navigation cannot be used, and map-based navigation methods may also fail due to dynamic changes in the environment. Therefore, studying autonomous navigation methods in denied environments has important practical significance and challenges.

[0004] At present, autonomous navigation technology based on reinforcement learning has achieved many results in laboratory environments, but it still faces many challenges in practical applications. First, the reinforcement learning training process requires a large amount of interaction data, which may be difficult to obtain in practical applications. Secondly, navigation tasks in complex environments place higher requirements on the robustness and real-time performance of the algorithm. In addition, communication and coordination issues in multi-agent collaboration are also key problems that need to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for collaborative autonomous perception and navigation in a denied environment based on reinforcement learning.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A collaborative autonomous perception and navigation method in a denied environment, the method steps comprising:

[0008] Build a self-organizing robust communication system to achieve dynamic networking and data sharing among multiple drones in a denied environment, and use adaptive network protocols to achieve automatic configuration of the communication network under network topology changes or communication interference;

[0009] A multi-sensor fusion perception system integrating lidar, visual sensor and inertial measurement unit is used to capture and process environmental information and the UAV's own motion information in real time through data fusion algorithm;

[0010] Based on the sensors equipped, a single drone uses a reinforcement learning algorithm to build a reward function based on the drone's energy consumption, exploration area, path overlap, and path danger level, and autonomously perceives and explores in an unknown environment and completes mapping.

[0011] Distributed multi-UAVs use reinforcement learning algorithms to communicate and negotiate among individuals in a UAV cluster of specified specifications based on map information, their respective locations, and working status, to achieve task allocation, path planning, and action decision-making.

[0012] As a preferred technical solution, the self-organizing robust communication system regards the drones as independent communication nodes and constructs a drone self-organizing network:

[0013] When UAV i The adjacent drone U j The signal strength between ij When the strength threshold is met, the two drones establish stable communication; the edge of the network topology structure contains the signal strength Q between the two drones. ij and the gradient of signal intensity over time G ij ;

[0014] When the signal strength meets the strength threshold, a strong communication edge is formed between the two drone nodes. ij When the strength is less than the strength threshold and greater than the connectable strength, a weak communication edge is formed between the two drones;

[0015] The drones in the subgraphs connected by strong edges in the topological network transmit the topological network information of the current subgraph to each other, clarifying their respective identities and topological connection relationships;

[0016] Each drone predicts the changes in the topological network at future moments based on the signal strength gradient, monitors the neighboring nodes connected to it by weak edges, and deletes or converts the weak edges into strong edges when the communication strength changes.

[0017] As a preferred technical solution, the adaptive network protocol is specifically: the drone senses the usage of the surrounding wireless spectrum and automatically adjusts its communication frequency according to the occupancy of the spectrum:

[0018] Using Q learning algorithm to realize route selection and route establishment, the minimum route hop factor J is:

[0019]

[0020] Among them, hop min is the minimum number of hops for breadth-first search, hip max The maximum number of hops allowed for routing;

[0021] The node load is evaluated using the queuing delay of packets within the network node. The M / M / 1 queuing model is used for modeling. The expected value of the number of packets in the network layer queue is:

[0022]

[0023] Among them, p i represents the probability when the number of packets in the queue is i, λ represents the rate at which the node receives packets, μ represents the rate at which the node processes packets, and the average queuing demonstration of packets is:

[0024]

[0025] When the rate μ of processing data packets at each node is constant, the higher the rate at which the node receives data packets, the longer the queuing delay of the data packets and the higher the load of the node;

[0026] Therefore, the node receiving rate λ can be used to measure the load of network nodes. The estimated value of the idleness factor of a node in the network relative to its neighbor node i is as follows:

[0027]

[0028] Among them, Z i , Z max is the current load and maximum load of network node i;

[0029] According to the signal strength Q, the hop factor J and the idleness factor F, the instantaneous reward R obtained by the current node i from the node j is set as:

[0030] R(i,j)=ω Q Q(i,j)+ω J J(i,j)+ω F F(i,j)

[0031] Among them, ω Q ,ω J and ω F is the weight of signal strength, hop factor and idleness factor.

[0032] As a preferred technical solution, in the self-organizing robust communication system:

[0033] Each drone stores a copy of critical data and is able to synchronize information with other drones;

[0034] Before the end of data sharing, the drones that establish communication with each other have relative motion range constraints to avoid communication interruption;

[0035] All drones must pass authentication when joining the network to prevent unauthorized access; the system uses data encryption technology, and communications between drones use digital signature technology to verify the integrity and source of the data.

[0036] As a preferred technical solution, the multi-sensor fusion perception system synchronizes and calibrates multi-sensor data as follows:

[0037] Jointly calibrate the LiDAR, visual sensor, and IMU to obtain the pose transformation relationship between the sensors and ensure spatial consistency of the data coordinates of each sensor;

[0038] Pre-integrate the IMU to obtain the relative position transformation relationship of the UAV at the previous and next moments, and provide the UAV motion parameters at a short moment for the laser radar and visual sensor;

[0039] Using the pose changes obtained by the IMU, the interpolation method is used to generate the corresponding pose for each point in the lidar point cloud and correct the motion distortion;

[0040] Capture environmental images through visual sensors and obtain feature points in the images using feature extraction algorithms;

[0041] Using the short-term motion parameters provided by the IMU, for the image feature point F captured by the camera, a vector pointing from the origin of the camera coordinate system to the spatial position corresponding to the feature point is obtained. The strip area through which the line segment is projected by this vector in the camera image space is taken as the area where the feature point F may exist, and the image feature points in the previous and next frames of the visual sensor are matched.

[0042] As a preferred technical solution, the single drone mapping is specifically as follows:

[0043] The laser inertial navigation odometer is constructed by using the fusion system of laser and IMU, and the visual odometer is constructed by using the fusion system of visual sensor and IMU to estimate the motion trajectory of the UAV respectively; the two types of posture data are combined to construct the posture optimization problem and obtain the optimal posture estimation;

[0044] Based on the optimal pose estimation, a point cloud map is created using laser point cloud and image feature points;

[0045] The bag-of-words data consisting of line and surface features captured by the laser point cloud and feature points of the visual sensor image are used as feature information of the local map for loop closure detection;

[0046] Compress the map data and use plane patches to represent the dense point cloud structure data;

[0047] The key data of shared loop detection is retained, including the line and surface features of the laser point cloud and the visual image features that can characterize the differences in locations, forming path loops in the multi-machine map, allowing loop detection to be performed across drones, and the constructed maps to be shared and integrated among multiple drones.

[0048] As a preferred technical solution, the single drone adopts DDPG deep reinforcement learning to enable the drone to learn the local exploration capability of unknown space. The reward function includes energy factor E, area factor S, overlap factor C, and danger factor D, which are constructed as follows:

[0049] R(t)=ω E ·E+ω S ·S+ω C ·C+ω D ·D

[0050] Among them, ω E ,ω S ,ω C ,ω D is the weight parameter of each factor;

[0051]

[0052] S=S t -S t-1

[0053]

[0054] D=||p t -o t || 2 ·I(||p t -o t || 2 <TH D )

[0055] Among them, E max is the total energy storage of the UAV, E t S is the current remaining energy of the drone; t-1 is the area of ​​the map at the previous moment, S t is the current mapping area; p t is the current position of the UAV, p i For historical exploration trajectory location points, TH C Stabilize the sensing radius of the drone;t This is the obstacle point closest to the drone at this time, TH D The safe operating radius of the drone.

[0056] As a preferred technical solution, the distributed multi-UAVs convert natural language tasks into specific exploration and navigation tasks through a large model and specify the number and capability indicators of the UAVs participating in the task according to the capabilities of the UAVs, the priority of the tasks and the environmental conditions, so as to realize intelligent decomposition of the tasks and transmit them to the distributed multi-UAVs through a communication system;

[0057] The large model selects a base model and fine-tunes the base model with the capabilities of a certain number of drones, task priorities and environmental condition data, as well as corresponding task decomposition example data sets to obtain relevant capabilities.

[0058] As an optimal technical solution, after receiving the task, the distributed multi-UAVs use a reinforcement learning algorithm to communicate and negotiate among individuals in a UAV cluster of specified specifications based on map information, their respective positions and working status. After the task planning converges, the UAVs reach a consensus to achieve distributed task planning.

[0059] As a preferred technical solution, the distributed task planning is realized by using a reinforcement learning algorithm, as follows:

[0060] Combining each UAV U and each exploration direction target V in pairs can form the decision space {U i ,V j};

[0061] Construct a strategy value function W to evaluate the expected benefits of each drone's strategy under the state, so as to construct an m×n strategy value matrix; the strategy value function W includes movement cost and state cost. When the map is partially unknown, the actual distance of the drone to the target exploration area is often difficult to calculate accurately. Only the search-based planning path length H between the drone's spatial position and the boundary of the unknown area in the known map area is considered as the movement cost; the operation of the drone is constrained by energy P, and only drones with sufficient energy can complete the complete exploration task;

[0062] W=ω H ·H+ω P ·f(P)

[0063]

[0064] Among them, ω H ,ω P is the weight of the path length and energy constraint, P TH It is the upper limit threshold of energy required for the drone to perform a single mission;

[0065] The drones are given sequential codes. Each drone sends its own information and the information received from the lower-order drones to the highest-order drone that it can communicate with. Finally, the information is aggregated to the highest-order drone in the drone group. After the information is aggregated, the strategy value matrix is ​​used to solve the optimal allocation strategy P. * :

[0066]

[0067]

[0068] x ij =1or 0

[0069] When the jth target is assigned to the i-th aircraft, x ij is 1, otherwise, x ij is 0;

[0070] When the optimal allocation strategy is obtained, the highest-order UAV forwards the task to the lower-order UAVs, and the task can be executed after the task is forwarded.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1) The self-organizing robust communication system constructed by the present invention can flexibly and efficiently complete self-organizing adaptive networking in a denied environment, thereby supporting multiple drones to explore the environment more freely and flexibly. Self-organizing networks and multi-hop communication technologies enable drones to maintain stable communication connections in complex environments, ensuring efficient transmission and sharing of data. Dynamic routing protocols and spectrum management technologies enable the system to adaptively adjust communication paths and frequencies, adapt to environmental changes, and maintain the continuity and stability of communications. Redundant design and fault-tolerant technology improve the stability and reliability of the system in the face of node failures or environmental interference, ensuring the smooth progress of tasks. Distributed computing and edge computing technologies enable computing tasks to be completed close to the data source, reducing data transmission delays and improving the response speed of the system. Encrypted communication and identity authentication technologies ensure the security of data transmission within the system and prevent information leakage and unauthorized access.

[0073] 2) The multi-sensor fusion perception system constructed by the present invention fuses the perception data of three sensors: lidar, visual sensor and IMU. The perception data of different modalities complement each other, which enhances the drone's perception ability and accuracy of the environment. The fusion of multiple modal data also improves the accuracy of positioning and navigation. Through multi-sensor data fusion, the system can rely on the data of other sensors to maintain normal operation when a single sensor fails or the data quality decreases, thereby improving the robustness of perception. Multi-sensor fusion enables the system to adapt to various complex environments, including indoors and outdoors, lighting changes, dynamic obstacles, etc., improving the environmental adaptability of the drone.

[0074] 3) The single UAV autonomous perception and mapping system constructed by the present invention uses reinforcement learning to enable the UAV to make autonomous decisions based on environmental feedback, thereby finding the best path in a complex environment, avoiding obstacles, and effectively covering the exploration area, so that the UAV can learn the ability to autonomously explore in an unknown environment, thereby adapting to various complex and dynamic environments, and improving exploration efficiency and perception accuracy. During the exploration process, the UAV uses SLAM technology to generate explicit maps, and uses efficient map representation and storage methods, which improves the efficiency of map use and improves the efficiency of data sharing and collaborative work between multiple UAVs.

[0075] 4) The distributed multi-UAV task planning and navigation system constructed by the present invention uses a large model-driven reinforcement learning algorithm to comprehensively evaluate the capabilities of the UAVs, the priority of the tasks, and the environmental conditions. The UAVs can achieve close collaboration through a self-organizing communication network, thereby achieving optimal dynamic allocation of tasks, improving resource utilization efficiency, and improving the coordination and consistency of task execution. At the same time, the UAVs can quickly adjust their strategies based on real-time environmental feedback, respond to environmental changes and task requirements in real time, improve the flexibility and adaptability of task execution, and reduce the time and cost of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of a method for collaborative autonomous perception and navigation in a denied environment based on reinforcement learning according to the present invention;

[0077] Figure 2 A schematic diagram of a self-organizing robust communication topology network of the present invention;

[0078] Figure 3 This is a schematic diagram for optimizing the scale estimation of image feature points using the short-term motion parameters provided by IMU;

[0079] Figure 4 Schematic diagram of using visual sensors and laser sensors to estimate the pose of a drone. DETAILED DESCRIPTION

[0080] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0081] Example 1

[0082] A collaborative autonomous perception and navigation method in a denied environment based on reinforcement learning, specifically including the following key subsystems:

[0083] Self-organizing robust communication system. The present invention constructs a self-organizing robust communication system that can realize dynamic networking and data sharing among multiple drones in a denied environment. The system adopts an adaptive network protocol and can automatically reconfigure under network topology changes or communication interference to ensure the continuity and reliability of data transmission. In addition, the system also integrates efficient data encoding and error correction mechanisms to improve the accuracy and integrity of data transmission.

[0084] Multi-sensor fusion perception system. Aiming at the special needs of the denial environment, the present invention designs a multi-sensor fusion perception system, which integrates three sensors: lidar, visual sensor and inertial measurement unit (IMU). Through the data fusion algorithm, the system can capture and process environmental information and the UAV's own motion information in real time. In order to improve the accuracy and robustness of perception, the present invention also introduces multi-sensor data synchronization and calibration technology, as well as adaptive feature extraction and information fusion strategies.

[0085] Single UAV autonomous perception and mapping system. Using the equipped sensors, the present invention constructs a single UAV autonomous perception and mapping system based on reinforcement learning. The system realizes autonomous perception and autonomous exploration of the UAV in an unknown environment through a reinforcement learning algorithm. During the exploration of the UAV, SLAM technology is used to generate an explicit map to record the structure and characteristics of the environment, and an efficient map representation and storage method is adopted. In addition, the system can also share and integrate the constructed map with other UAVs to achieve a wider range of environmental understanding and collaborative navigation.

[0086] Distributed multi-UAV mission planning and navigation system. The present invention also proposes a distributed multi-UAV mission planning and navigation solution based on reinforcement learning and driven by a large model. Information sharing is achieved through a self-organizing communication network, and a large model is used to achieve intelligent decomposition of tasks based on the capabilities of the UAVs, the priorities of the tasks, and the environmental conditions. The reinforcement learning algorithm is used to achieve path planning and action decision-making. During the navigation process, the system can also respond to environmental changes and mission requirements in real time, and achieve formation, obstacle avoidance, and path following through collaborative control strategies.

[0087] Specifically, each part is implemented as follows:

[0088] like Figure 1 As shown in the figure, the self-organizing robust communication system regards UAVs as independent communication nodes and builds a UAV self-organizing network (UANET); UANET has the advantages of self-organization, strong flexibility, strong scalability, and fast networking speed, and can meet the communication requirements of multi-UAV systems;

[0089] Distributed hash table (DHT) technology is used to quickly map the network topology to ensure that each drone can understand the existence and status of other drones. i , and its neighboring drone U j The signal strength between ij , when Q ij When the strength threshold is met, the two drones can establish stable communication; the edge of the network topology structure contains the signal strength Q between the two drones. ij , and the gradient of signal intensity over time G ij :

[0090]

[0091] When the signal strength meets the strength threshold, a communication “strong edge” is formed between the two drone nodes. ij When the strength is less than the threshold and greater than the connectable strength, a communication “weak edge” is formed between the two drones, such as Figure 2 As shown;

[0092] In a short moment, drones in the subgraphs connected by strong edges in the topological network transmit the topological network information of the subgraph to each other, clarifying their respective identities and topological connection relationships; at the same time, each drone predicts the changes in the topological network at future moments based on the signal strength gradient, monitors the neighboring nodes connected to it by "weak edges", and deletes or converts the weak edges into strong edges when the communication strength changes, thereby realizing dynamic topological network management;

[0093] A topology-based routing protocol is used to adapt to the characteristics of rapid movement and dynamic topology changes of multi-UAV systems. In a denial environment, in order to deal with communication interference and spectrum congestion, communication conditions may change at any time. Therefore, UAVs must have dynamic routing and spectrum management capabilities. UAVs can sense the use of the surrounding wireless spectrum and automatically adjust their communication frequency according to the occupancy of the spectrum, so that UAVs can predict communication interference and adjust routes in real time.

[0094] Using the Q learning algorithm to implement route selection and route establishment, first calculate the minimum route hop factor J:

[0095]

[0096] Among them, hop min is the minimum number of hops for breadth-first search, hop max The maximum number of hops allowed for routing; the queuing delay of data packets in network nodes is used to evaluate node load, and the M / M / 1 queuing model is used for modeling. The expected value of the number of data packets in the network layer queue is:

[0097]

[0098] Among them, p i represents the probability when the number of packets in the queue is i, λ represents the rate at which the node receives packets, and μ represents the rate at which the node processes packets. Then the average queuing demonstration of packets is:

[0099]

[0100] When the rate μ of processing data packets at each node is constant, the higher the rate at which the node receives data packets, the longer the queuing delay of the data packets and the higher the load of this node. Therefore, the node receiving rate i can be used to measure the load of the network node. The estimated value of the idleness factor of a node in the network relative to its neighbor node i is defined as follows:

[0101]

[0102] Among them, Z i , Z max is the current load and maximum load of network node i;

[0103] Based on the signal strength, hop factor and idleness factor, the instantaneous reward that the current node i obtains from node j is defined as:

[0104] R(i,j)=ω Q Q(i,j)+ω J J(i,j)+ω F F(i,j)

[0105] Among them, ω Q ,ω J and ω F is the weight of signal strength, hop factor and idleness factor.

[0106] The data sharing mechanism is an important part of the self-organizing robust communication system. The system adopts redundant design and fault-tolerant processing mechanism to ensure the accuracy and integrity of data transmission. In order to improve the robustness of data storage and sharing, each drone is able to store a copy of key data and synchronize information with other drones when necessary.

[0107] To ensure that data can be shared smoothly between drones, there is a relative motion range constraint for drones that establish communication with each other before data sharing ends to avoid communication interruption; the relative motion range refers to the area where the communication signal strength between two drones can be maintained above a certain value. In the absence of obstacles, the communication signal strength is positively correlated with the distance, and ensuring the straight-line distance between the two drones can ensure stable communication. In actual operation, the two drones can be kept relatively still to achieve stable communication.

[0108] Communication security is an important aspect of the design of this system. All drones must pass strict identity authentication when joining the network to prevent unauthorized access. The system uses data encryption technology to ensure the security of data during transmission. Communication between drones uses digital signature technology to verify the integrity and source of the data.

[0109] Multi-sensor fusion perception system, as follows:

[0110] First, the laser radar, visual sensor and IMU are jointly calibrated to obtain the posture transformation relationship between the sensors and achieve the spatial consistency of the data coordinates of each sensor;

[0111] Pre-integrate the IMU to obtain the relative position transformation relationship of the UAV at the previous and next moments, and provide the UAV motion parameters at a short moment for the laser radar and visual sensor;

[0112] The high frame rate posture change obtained by IMU is used to correct the motion distortion of the lidar point cloud to obtain a more accurate lidar point cloud. During the movement of the drone, the coordinate systems of different point clouds in the same frame do not coincide with each other. The posture change provided by IMU is used to correct the point cloud position. For a frame of point cloud P, it contains points {p1, p2, p3, ..., p n}, and corresponds to a series of IMU pose data {t1, t2, t3, …, t m}, use the interpolation method to generate the corresponding pose for each point in the point cloud, and get the point and pose pair (p i ,t i ),i∈[1,n], and further obtain the corrected point:

[0113]

[0114] The environment image is captured by the visual sensor, and the feature points in the image are obtained using the feature extraction algorithm. The short-term motion parameters provided by the IMU are used to improve the matching speed and accuracy of the image feature points in the previous and next frames of the visual sensor, helping to achieve pose estimation and feature point triangulation. Figure 3As shown, for the image feature point F captured by the camera at time t1, it corresponds to a vector from the origin of the camera coordinate system to the spatial position corresponding to the feature point. In the IMU coordinate system:

[0115]

[0116] At time t2, the vector in the IMU coordinate system is:

[0117]

[0118] Then the vector in the camera coordinate system is:

[0119]

[0120] This vector projects a line segment in the camera image space. The strip area where the line segment passes is the area where the feature point F may exist, which reduces the search space of the feature point.

[0121] The projection relationship between the laser point cloud and the environment image is used to perform depth matching, optimize the scale estimation of image feature points, and eliminate the scale drift of visual matching.

[0122] A single drone conducts autonomous perception, exploration and mapping. The specific plan is as follows:

[0123] The laser inertial navigation odometer is constructed by using the fusion system of laser and IMU, and the visual odometer is constructed by using the fusion system of visual sensor and IMU to estimate the motion trajectory of the UAV respectively; the two pose data are combined to construct the pose optimization problem and obtain the optimal pose estimation; Figure 4 As shown in the figure, the UAV goes through a series of pose points {x1, x2, x3, …} during operation. The visual sensor and laser sensor construct pose nodes based on their own perception data to form a pose graph, and the graph optimization algorithm is used to realize the pose estimation of the UAV.

[0124] Based on the optimal pose estimation, a point cloud map is created using laser point cloud and image feature points;

[0125] The bag-of-words data consisting of line and surface features captured by the laser point cloud and feature points of the visual sensor image is used as the feature information of the local map to realize loop closure detection;

[0126] The map data is compressed and the dense point cloud structure data is represented by plane patches to reduce the data volume and improve data sharing efficiency. At the same time, key data for loop detection are shared, including line and surface features of the laser point cloud that can characterize the differences in locations and visual image features. Using these shared feature data, the drone can determine whether the current location has been explored based on its own observation data, forming a path loop in the multi-drone map, so that loop detection can be performed across drones, thereby reducing mapping errors within the multi-drone system and obtaining consistent map representation.

[0127] Using DDPG deep reinforcement learning, the drone learns the local exploration capability of unknown space. The reward function focuses on four aspects: drone energy consumption, exploration area, path overlap, and path danger level, corresponding to energy factor (E), area factor (S), overlap factor (C), and danger factor (D) respectively:

[0128]

[0129] S=S t -S t-1

[0130]

[0131] D=||p t -o t || 2 ·I(||p t -o t || 2 <TH D )

[0132] Among them, E max is the total energy storage of the UAV, E t S is the current remaining energy of the drone; t-1 is the area of ​​the map at the previous moment, S t is the current mapping area; p t is the current position of the UAV, p i For historical exploration trajectory location points, TH C Stabilize the sensing radius of the drone; t This is the obstacle point closest to the drone at this time, TH D is the safe working radius of the drone; construct the reward function based on these constraints:

[0133] R(t)=ω E ·E+ω S ·S+ω C ·C+ω D ·D

[0134] Among them, ω E,ω S ,ω C ,ω D is the weight parameter of each factor;

[0135] When a single drone faces an unknown environment, it uses its own sensors to create a map and uses reinforcement learning to plan exploration paths based on the current map. It then autonomously explores and maps the environment under energy constraints, cost constraints, and exploration direction constraints.

[0136] Distributed multi-UAV mission planning and navigation, specifically as follows: a single UAV has autonomous perception, exploration and mapping capabilities, and uses a self-organizing robust communication system to achieve information interaction and data sharing between UAVs;

[0137] The big model establishes communication with the operator, converts natural language tasks into specific exploration and navigation tasks, and specifies the number and capability indicators of the UAVs participating in the task according to the UAVs' capabilities, task priorities and environmental conditions, realizes intelligent decomposition of tasks, and transmits them to the multi-UAV system through the communication system. In order to make the big model have relevant capabilities, it is necessary to select a base model with good performance, and prepare a certain number of UAVs' capabilities, task priorities and environmental conditions data as well as corresponding task decomposition example data sets to fine-tune the base model.

[0138] After receiving the task, the multi-UAV system uses the reinforcement learning algorithm to communicate and negotiate among individuals in the UAV cluster of specified specifications based on map information, their respective positions and working status. After the task planning converges, the UAVs reach a consensus and realize distributed task planning.

[0139] The drone group contains multiple drones {U1,U2,U3,…,U m}, the task contains multiple exploration direction targets {V1,V2,V3,…,V n}, each UAV and each target can form a decision space for task allocation {U i ,V j}; Construct a strategy value function W to evaluate the expected benefits of each drone's strategy under the state, so as to construct an m×n strategy value matrix; The strategy value function W includes movement cost and state cost. When the map is partially unknown, the actual distance of the drone to the target exploration area is often difficult to calculate accurately. Only the search-based planning path length H between its own spatial position and the boundary of the unknown area in the known map area is considered as the movement cost; The operation of the drone is constrained by energy P, and only drones with sufficient energy can complete the complete exploration task;

[0140] W=ω H ·H+ωP ·f(P)

[0141]

[0142] Among them, ω H ,ω P is the weight of the path length and energy constraint, P TH It is the upper limit threshold of energy required for the drone to perform a single mission;

[0143] A single drone can only directly obtain the strategic value calculation results between itself and each task, and the calculation results of each drone need to be summarized to obtain the overall information; the drones are given sequence codes in advance, and each drone sends its own information and the information received from the lower-order drones to the highest-order drone that it can communicate with, and finally the information is summarized to the drone with the highest order in the drone group; after the information is summarized, the optimal allocation strategy P is solved using the strategic value matrix * , the mathematical model of this problem is described as follows:

[0144]

[0145] x ij =1or 0

[0146] When the jth target is assigned to the i-th aircraft, x ij is 1, otherwise, x ij is 0; when the optimal allocation strategy is obtained, the highest-order UAV forwards the task to the lower-order UAV, and the task can be executed after the task is forwarded.

[0147] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A collaborative autonomous perception and navigation method in a denied environment, characterized in that: The method steps include: Build a self-organizing robust communication system to achieve dynamic networking and data sharing among multiple drones in a denied environment, and use adaptive network protocols to achieve automatic configuration of the communication network under network topology changes or communication interference; A multi-sensor fusion perception system integrating lidar, visual sensor and inertial measurement unit is used to capture and process environmental information and the UAV's own motion information in real time through data fusion algorithm; Based on the sensors equipped, a single drone uses a reinforcement learning algorithm to build a reward function based on the drone's energy consumption, exploration area, path overlap, and path danger level, and autonomously perceives and explores in an unknown environment and completes mapping. Distributed multi-UAVs use reinforcement learning algorithms to communicate and negotiate among individuals in a UAV cluster of specified specifications based on map information, their respective locations, and working status, to achieve task allocation, path planning, and action decision-making.

2. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: The self-organizing robust communication system regards the drones as independent communication nodes and builds a self-organizing network of drones: When UAV i The adjacent drone U j The signal strength between ij When the strength threshold is met, the two drones establish stable communication; The edge of the network topology contains the signal strength Q between the two drones. ij and the gradient of signal intensity over time G ij ; When the signal strength meets the strength threshold, a strong communication edge is formed between the two drone nodes. ij When the strength is less than the strength threshold and greater than the connectable strength, a weak communication edge is formed between the two drones; The drones in the subgraphs connected by strong edges in the topological network transmit the topological network information of the current subgraph to each other, clarifying their respective identities and topological connection relationships; Each drone predicts the changes in the topological network at future moments based on the signal strength gradient, monitors the neighboring nodes connected to it by weak edges, and deletes or converts the weak edges into strong edges when the communication strength changes.

3. The method for collaborative autonomous perception and navigation in a denied environment according to claim 2, characterized in that: The adaptive network protocol is specifically as follows: the drone senses the usage of the surrounding wireless spectrum and automatically adjusts its communication frequency according to the occupancy of the spectrum: Using Q learning algorithm to realize route selection and route establishment, the minimum route hop factor J is: Among them, hop min is the minimum number of hops for breadth-first search, hop max The maximum number of hops allowed for routing; The node load is evaluated using the queuing delay of packets within the network node. The M / M / 1 queuing model is used for modeling. The expected value of the number of packets in the network layer queue is: Among them, p i represents the probability when the number of packets in the queue is i, λ represents the rate at which the node receives packets, μ represents the rate at which the node processes packets, and the average queuing demonstration of packets is: When the rate μ of processing data packets at each node is constant, the higher the rate at which the node receives data packets, the longer the queuing delay of the data packets and the higher the load of the node; Therefore, the node receiving rate λ can be used to measure the load of network nodes. The estimated value of the idleness factor of a node in the network relative to its neighbor node i is as follows: Among them, Z i , Z max is the current load and maximum load of network node i; According to the signal strength Q, the hop factor J and the idleness factor F, the instantaneous reward R obtained by the current node i from the node j is set as: R(i,j)=ω Q ·Q(i,j)+ω J ·J(i,j)+ω F ·F(i,j) Among them, ω Q ,ω J and ω F is the weight of signal strength, hop factor and idleness factor.

4. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: In the self-organizing robust communication system: Each drone stores a copy of critical data and is able to synchronize information with other drones; Before the end of data sharing, the drones that establish communication with each other have relative motion range constraints to avoid communication interruption; All drones must pass authentication when joining the network to prevent unauthorized access; the system uses data encryption technology, and communications between drones use digital signature technology to verify the integrity and source of the data.

5. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: The multi-sensor fusion perception system synchronizes and calibrates multi-sensor data as follows: Jointly calibrate the LiDAR, visual sensor, and IMU to obtain the pose transformation relationship between the sensors and ensure spatial consistency of the data coordinates of each sensor; Pre-integrate the IMU to obtain the relative position transformation relationship of the UAV at the previous and next moments, and provide the UAV motion parameters at a short moment for the laser radar and visual sensor; Using the pose changes obtained by the IMU, the interpolation method is used to generate the corresponding pose for each point in the lidar point cloud and correct the motion distortion; Capture environmental images through visual sensors and obtain feature points in the images using feature extraction algorithms; Using the short-term motion parameters provided by the IMU, for the image feature point F captured by the camera, a vector pointing from the origin of the camera coordinate system to the spatial position corresponding to the feature point is obtained. The strip area through which the line segment is projected by this vector in the camera image space is taken as the area where the feature point F may exist, and the image feature points in the previous and next frames of the visual sensor are matched.

6. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: The single drone mapping is specifically as follows: The laser inertial navigation odometer is constructed by using the fusion system of laser and IMU, and the visual odometer is constructed by using the fusion system of visual sensor and IMU to estimate the motion trajectory of the UAV respectively; the two types of posture data are combined to construct the posture optimization problem and obtain the optimal posture estimation; Based on the optimal pose estimation, a point cloud map is created using laser point cloud and image feature points; The bag-of-words data consisting of line and surface features captured by the laser point cloud and feature points of the visual sensor image are used as feature information of the local map for loop closure detection; Compress the map data and use plane patches to represent the dense point cloud structure data; The key data of shared loop detection is retained, including the line and surface features of the laser point cloud and the visual image features that can characterize the differences in locations, forming path loops in the multi-machine map, allowing loop detection to be performed across drones, and the constructed maps to be shared and integrated among multiple drones.

7. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: The single drone adopts DDPG deep reinforcement learning to enable the drone to learn the local exploration ability of unknown space. The reward function includes energy factor E, area factor S, overlap factor C, and danger factor D, which are constructed as follows: R(t)=ω E ·E+ω S ·S+ω C ·C+ω D ·D Among them, ω E ,ω s ,ω C ,ω D is the weight parameter of each factor; S=S t -S t-1 D=||p t -o t || 2 ·I(||p t -o t || 2 <TH D ) Among them, E max is the total energy storage of the UAV, E t S is the current remaining energy of the drone; t-1 is the area of ​​the map at the previous moment, S t is the current mapping area; p t is the current position of the UAV, p i For historical exploration trajectory location points, TH C Stabilize the sensing radius of the drone; t This is the obstacle point closest to the drone at this time, TH D The safe operating radius of the drone.

8. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: The distributed multi-UAVs convert natural language tasks into specific exploration and navigation tasks through a large model and specify the number and capability indicators of the UAVs participating in the task according to the capabilities of the UAVs, the priorities of the tasks and the environmental conditions, thereby realizing intelligent decomposition of the tasks and transmitting them to the distributed multi-UAVs through a communication system; The large model selects a base model and fine-tunes the base model with the capabilities of a certain number of drones, task priorities and environmental condition data, as well as corresponding task decomposition example data sets to obtain relevant capabilities.

9. The method for collaborative autonomous perception and navigation in a denied environment according to claim 1, characterized in that: After receiving the task, the distributed multi-UAVs use the reinforcement learning algorithm to communicate and negotiate among individuals in a UAV cluster of specified specifications based on map information, their respective positions and working status. After the task planning converges, the UAVs reach a consensus to realize distributed task planning.

10. The method for collaborative autonomous perception and navigation in a denied environment according to claim 9, characterized in that: The distributed task planning is implemented by using the reinforcement learning algorithm as follows: Combining each UAV U and each exploration direction target V in pairs can form the decision space {U i ,V j }; Construct a strategy value function W to evaluate the expected benefits of each drone's strategy under the state, so as to construct an m×n strategy value matrix; the strategy value function W includes movement cost and state cost. When the map is partially unknown, the actual distance of the drone to the target exploration area is often difficult to calculate accurately. Only the search-based planning path length H from the drone's spatial position to the boundary of the unknown area is considered as the movement cost within the known map area; The operation of the drone is subject to the energy P constraint, and only drones with sufficient energy can complete the complete exploration mission; W=ω H ·H+ω P ·f(P) Among them, ω H ,ω P is the weight of the path length and energy constraint, P TH It is the upper limit threshold of energy required for the drone to perform a single mission; The drones are given sequential codes. Each drone sends its own information and the information received from the lower-order drones to the highest-order drone that it can communicate with. Finally, the information is aggregated to the highest-order drone in the drone group. After the information is aggregated, the strategy value matrix is ​​used to solve the optimal allocation strategy P. * : x ij =1or 0 When the jth target is assigned to the i-th aircraft, x ij is 1, otherwise, x ij is 0; When the optimal allocation strategy is obtained, the highest-order UAV forwards the task to the lower-order UAVs, and the task can be executed after the task is forwarded.

Images