A reinforcement learning-based method for space-ground collaborative deployment of multiple detectors in deep space catalogs

Abstract

The present invention provides a method for the coordinated deployment of multiple detectors in deep space and star catalogs based on reinforcement learning, which belongs to the field of planetary surface detection technology. The method of the present invention uses a flyer and a rover as detectors to collaboratively detect unknown areas in deep space and star catalogs. Three types of communication channels are established between the detectors, and the communication radius between different detectors is calculated. Constraints are established in terms of communication range, perception range, coverage mode and energy. The optimization goal is to maintain communication connectivity between detectors and maximize the total coverage area of ​​the detectors under limited energy. The flyer and the rover are used as two intelligent agents, and the optimal deployment strategy is determined based on the multi-agent proximal strategy optimization algorithm. The present invention optimizes the regional coverage strategy, realizes efficient allocation of resources, improves the detection efficiency and accuracy of the flyer and the detector in extreme terrain and complex environments, reduces energy consumption while ensuring coverage efficiency, and improves the overall energy utilization efficiency of the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep space planetary surface detection, and in particular to a method for space-ground collaborative deployment of multiple deep space planetary surface detectors based on reinforcement learning. Background Art

[0002] Deep space exploration is at the forefront of scientific and technological development and the foundation for future deep space operations. Its importance is self-evident. Due to the unique characteristics of the deep space environment, intelligent agents face far less difficulty operating in deep space scenarios like the lunar surface or Mars than humans. Deep space exploration should be developed with autonomy, large-scale development, and intelligence as its primary characteristics. This means that swarms of probes can autonomously group, command, and explore. Therefore, it is necessary to design an intelligent algorithm-driven method for coordinated air-ground deployment to achieve these tasks.

[0003] As a widely used unmanned platform, the flyover is seen as an ideal choice to replace humans in high-risk tasks in harsh environments. With the increasing maturity of related technologies, the flyover has now demonstrated a wide range of application potential in agricultural production, traffic control, disaster relief, military reconnaissance and mobile communications, becoming a key trend in the current field of unmanned platform research. Among them, tasks such as mapping, monitoring, and detecting unknown targets require coverage of the entire area. The intelligent technology of the flyover is crucial to its function, especially the intelligent coverage technology. How to cover the target area efficiently and safely has become a research focus. However, in the deep space star exploration scenario, there are also some defects when using only the flyover for coverage tasks, such as fast energy consumption.

[0004] Generally speaking, when a swarm of drones is deployed to complete an area coverage mission, each drone is first precisely assigned a task through a task decision method. Subsequently, each drone plans its own coverage path based on the assigned task to ensure complete coverage of the entire target area. Cluster coverage planning strategies primarily include centralized planning and distributed planning. Centralized planning can develop a more optimal deployment plan for the cluster based on global information. However, as the number of drones increases and the coverage area expands, the computational complexity increases dramatically, limiting the scalability of centralized planning. In contrast, distributed planning exhibits greater flexibility and scalability. It adapts to all environmental changes through local information and communication with neighboring nodes, thereby achieving a more optimal state for the cluster system. However, the drawback is that it may be difficult to achieve a globally optimal coverage deployment. These traditional optimization algorithms primarily optimize the motion trajectories of intelligent agents to minimize the time it takes to cover the target area or maximize coverage. However, they struggle to handle large-scale problems, lack real-time performance, and exhibit certain limitations. Summary of the Invention

[0005] In response to the above-mentioned existing problems, in order to solve the problem of unknown area coverage in planetary exploration scenarios, especially the problem of exploring large-scale unknown areas, the present invention discloses a deep space star catalog multi-detector air-ground collaborative deployment method based on reinforcement learning. The air-ground collaborative system composed of multiple detectors (including flyers and rovers) explores the target area, and maintains communication connectivity during multiple detection periods and maximizes the coverage area under limited energy conditions, which improves real-time performance and reliability compared with existing technologies.

[0006] The present invention provides a method for collaborative deployment of multiple detectors in deep space and on the ground based on reinforcement learning. In a deep space, a flyer and a rover collaborate to detect unknown target areas. The planet has atmospheric conditions suitable for flyers to fly. The flyer performs detection in the air, while the rover performs detection on the planet's surface. The method includes the following steps:

[0007] Step 1: Model the communication channel of the deep space star catalog; the communication channels are divided into three categories: air-to-air (A2A), air-to-ground (A2G), and ground-to-ground (G2G); model the three types of communication channels respectively, and determine the communication radius between different detectors; including the communication radius between two flybys Communication radius between the leaper and the patroller Communication radius between two patrollers

[0008] Step 2: Determine the constraints and optimization objectives for the flyer and rover to collaboratively detect the target area. The constraints include the following:

[0009] (1) Communication range constraint: Leapers and patrollers are detectors. Each detector is required to maintain normal communication with at least one other detector. The constraints are set as follows:

[0010]

[0011] Among them, d gg d ag d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers;

[0012] (2) Perception range constraint: The perception coverage of the leaper and the patroller is represented by a circular area. Let the perception range radius of the leaper and the patroller be R respectively. a and R g ;

[0013] (3) Coverage mode constraint: Initially, all patrollers start from the same position, and the flyer is carried by the patroller and moves. After a certain period of movement, the flyer separates from the patroller. After the flyer reaches the designated position and hovers, it detects and collects information about the current sensing coverage area. The coverage area of ​​the flyer is only valid at the current moment. The patroller collects information while moving, and all areas where the patroller has moved are covered areas that have been detected.

[0014] (4) Energy-limited constraint: set the total energy consumption of each detector to be less than its own total energy;

[0015] The optimization goal is to maintain inter-detector communication connectivity and maximize the total detector coverage area under limited energy. The target area to be detected is represented as a 2D square grid. The coverage of each grid is indicated by a value: 0 indicates no coverage, and a value greater than 0 indicates coverage.

[0016] Step 3: Calculate the optimization objective of step 2 based on the Multi-Agent Proximal Policy Optimization (MA-PPO) algorithm to optimize the deployment coverage plan of the detector.

[0017] In the multi-agent proximal policy optimization algorithm, the state is set to include the coordinates and movement directions of all detectors, as well as the ratio of the current coverage target area. Two agents are designed, corresponding to the flyer and the patroller, and their actions are set to the movement direction. The reward function is set to be the weighted sum of the reward related to coverage area and the reward related to communication connectivity. The detector deployment coverage plan is iteratively optimized based on the reward function.

[0018] Compared with the prior art, the advantages and positive effects of the present invention are:

[0019] (1) Because the patrollers and flyers have different detection ranges and the resolution of the information they collect is also different, the flyer can collect information about areas that the patrollers cannot reach, and the patroller can collect information about areas where the flyer's line of sight is blocked. Therefore, the flyer and patroller can perform different detection functions. The present invention uses two types of detectors to give full play to the functional advantages of each and efficiently complete complex mission objectives.

[0020] (2) The method of the present invention has the advantage of information sharing. The flyer can quickly obtain star catalog information during flight. Within the system, information sharing can be achieved through communication between the flyer and the rover, between the flyers, and between the rover. This allows a comprehensive understanding of the current detection situation and avoids missing the target area. This information sharing and collaboration is conducive to improving the efficiency of task completion and can further enhance the efficiency and accuracy of the entire task.

[0021] (3) The method of the present invention has the advantage of detection efficiency. Although the traditional coverage algorithm can achieve coverage of the target area, it is difficult to achieve the best coverage deployment strategy when the number of detectors increases. The method of the present invention introduces the MA-PPO algorithm to dynamically adjust the coverage strategy of the leaper and the patroller according to the regional environmental information and the real-time changing coverage situation. The leaper and the patroller interact with the environment to obtain the best movement strategy, avoid repeated coverage, effectively reduce the detection time, and improve the task efficiency of the entire system. The solution obtained by the method of the present invention maximizes the coverage area under limited energy conditions, while maintaining communication between all leapers and patrollers, effectively overcoming the limitations of traditional optimization algorithms in dealing with large-scale problems, improving detection efficiency and ensuring communication between detectors.

[0022] (4) The method of the present invention has a cost advantage. The flyover has a simple structure and low cost. Compared with detection using only patrollers, air-ground collaborative detection does not incur high costs due to equipment failure or damage. In addition, the flyover has a wide detection range. When performing target area detection tasks, the number of detectors required for collaborative detection is less than using only patrollers, and the energy consumption is also lower.

[0023] (5) The method of the present invention has application advantages. The method of the present invention is applicable to various scenarios with complex environments and limited detector energy, providing an effective technical solution for the field of planetary exploration. The method of the present invention can effectively improve the detection efficiency and accuracy of the space-ground collaborative system in planetary exploration scenarios, while optimizing resource allocation, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a flow chart of a method for coordinated coverage of deep space star surfaces according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of an air-ground collaborative coverage scenario according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a coverage information map according to an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the specific deployment and coverage process of detectors according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more apparent, embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. The accompanying drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and the accompanying description are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments.

[0029] The present invention uses a method for collaborative detection using a flyover and a rover in a planetary exploration scenario. The energy of the flyover mainly comes from solar panels. When the flyover is used alone to perform a detection mission, its own energy is difficult to supply the long-term flight of the flyover. When only the rover is used for detection, information on extreme terrain cannot be collected and the coverage is limited. Therefore, an air-ground collaborative system is introduced into the planetary exploration scenario, and two detectors, a flyover and a rover, are used to perform tasks. This not only solves the problems of long time and high cost when only the rover is used for detection, but also makes up for the shortcomings of easy omissions when only the flyover is used for detection. The method of the present invention not only improves the ability to complete complex mission objectives through the air-ground collaborative system, but also further improves the efficiency and accuracy of the entire mission.

[0030] like Figure 1 As shown, the deep space catalog multi-detector air-ground collaborative deployment method based on reinforcement learning provided by an embodiment of the present invention performs regional exploration through the collaboration of a flyer and a rover, and mainly includes 8 steps S1 to S8.

[0031] S1: Model the communication channel when the air-ground collaborative system is working. Figure 2 As shown, the air-ground collaborative coverage scenario in this embodiment of the present invention includes three channel models: the A2A channel between flyers, the A2G channel between flyers and rovers, and the G2G channel between rovers. This embodiment of the present invention uses Mars as the target planet for exploration. Mars has atmospheric conditions suitable for flyers and meets the basic environmental requirements for flyers to perform exploration missions. Wireless signals experience different fading in different channels, so each of these three channel types is modeled as follows, as shown in steps S101-S103.

[0032] S101: A2A channel modeling. When the intercom communicates, the signal propagates in a straight line. The path loss is the free space loss and is also affected by the atmosphere. The total loss of the A2A channel is L a2a is the free space path loss L f and atmospheric absorption loss L a The sum is as follows:

[0033] L a2a =L f +L a ·d

[0034] Wherein, the free space path loss L f The unit is dB, L f is calculated as follows:

[0035] L f =20log 10 d+20log 10 f+32.45

[0036] The unit of the communication signal frequency f is MHz, and the unit of the distance d between the two jumpers is km.

[0037] Atmospheric absorption loss L a The unit is dB / Km, L a is the sum of the absorption losses of water vapor and oxygen, as shown below:

[0038]

[0039] in, is the water vapor absorption loss, It is the oxygen absorption loss.

[0040] S102: A2G channel modeling. The communication between the flyer and the rover is greatly affected by dust, and the atmospheric absorption loss can be ignored. The total loss of the A2G channel is L a2g is the path loss PL and dust loss L d The sum of, such as:

[0041] L a2g =PL+L d ·d

[0042] Among them, the unit of path loss PL is dB, and the unit of dust loss L is d The unit is dB / Km, d is the distance between the flyer and the patroller, the unit is Km.

[0043] When the jumper and the patroller communicate, as the jumper continues to move, there may be obstacles blocking the connection between the transmitter and receiver, that is, a non-line-of-sight environment. Since A2G communication exists in both line-of-sight (LOS) and non-line-of-sight (NLOS) situations, a probabilistic path loss model with different occurrence probabilities of line-of-sight and non-line-of-sight can be used to model them, as shown in the following formula:

[0044] PL=P LOS PL LOS +P NLOS PL NLOS

[0045] Among them, P LOS represents the probability of line-of-sight channel, PL LOS represents the path loss of the line-of-sight channel, P NLOS represents the probability of non-line-of-sight channel, PL NLOS Indicates the path loss of the non-line-of-sight channel.

[0046] Dust loss L d The calculation is as follows:

[0047]

[0048] Where λ is the wavelength of the communication signal, is the average dust particle radius, N T is the sand particle density, θ is the elevation angle of the propagation path, ε' and ε" are the real and imaginary parts of the average dielectric constant of the dust particles, respectively.

[0049] S103: G2G channel modeling. The loss of the G2G channel is the diffraction loss of the obstacles. In the actual detection environment, the distribution of obstacles on the surface of Mars is irregular, usually with circular and blade-shaped obstacles alternating, so a mixed obstacle model needs to be considered. The alternating method is used to calculate the loss L of the G2G channel. g2g as follows:

[0050]

[0051] Among them, L D1 is the diffraction loss of the first obstacle, which is the blade-shaped obstacle, a is the distance from the transmitter to the first obstacle, b is the distance between the two obstacles (i.e. the first obstacle and the second obstacle), c is the distance from the second obstacle to the receiver, d = a + b + c is the distance between the transmitter and the receiver, L D2 is the diffraction loss of the second obstacle, which is a cylindrical obstacle, and is calculated as follows:

[0052] L D2 =L D (b,c,H′ c2 )+T

[0053] Where T is the additional loss related to the propagation environment, L D is the knife-edge diffraction loss under the same conditions, H′ c1 , H′ c2 is the clearance, that is, the height of the top of the obstacle relative to the direct line of sight between the transmitter and the receiver.

[0054] S2: Obtain the maximum effective communication distance for the flyer and rover during joint detection. Based on the communication channel model, we can determine the communication losses between detectors and the effective communication range between detectors. This can be used to determine the communication limitations of the detectors during mission execution and improve detection accuracy.

[0055] According to the Friis transmission equation:

[0056]

[0057] Among them, P r is the received power, P t is the transmission power, G t is the antenna gain at the transmitting end, G ris the antenna gain at the receiving end, λ is the wavelength, and d is the distance. This equation is for free-space propagation, which assumes ideal propagation without environmental interference. Replacing the free-space loss with the total loss, L, yields the following signal propagation formula:

[0058] P r =P t +G t +G r -L

[0059] Where L is the loss.

[0060] Then, based on the loss of the three types of channels and the power and gain of the detectors, the maximum effective communication distance between the leaper and the patroller, that is, the communication radius between the two detectors, is obtained. In the embodiment of the present invention, the communication distance between the two detectors, that is, the communication radius, is determined based on the power and gain of the two detectors and the lower limit of the received power. The communication radius between the two leapers is calculated to be The communication radius between the flyer and the patroller is The communication radius between two patrollers is

[0061] S3: Model the coverage environment and represent the map area to be covered as a 2D square area.

[0062] In an embodiment of the present invention, the moving speeds of the patrollers and the flyer are set to constant values, and the flight altitudes of all flyers are consistent and fixed. There is no lift-off and descent process involved, and the coverage scene is considered to be a 2-dimensional square area with a size of z×z, which is the coverage target of the detector. A 2-dimensional coordinate system is established for the square map area to be covered. Each flyer is carried by the patroller, and initially all patrollers are in the same position, starting from the coordinate (0,0), and the flyer and the patroller move together. After the patroller carries the flyer for a period of time, the flyer separates from the patroller and begins to cover.

[0063] Assume that all detectors constitute a set N D ∈{i=1,2,...,N}, each patrol vehicle and each flyer in the set can perform the coverage task, and N is the total number of detectors.

[0064] S4: Design constraints for the flyer and the patroller, including communication range constraints, perception range constraints, coverage mode constraints, and energy constraints. During the execution of tasks in the embodiment of the present invention, the specific constraints imposed on the flyer and the patroller are as follows:

[0065] S401: Communication range constraint. Since the communication range between the flyer and the patroller is limited, each detector must maintain normal communication with at least one other detector to ensure that the entire multi-detector network maintains communication connection. Suppose the communication radius under the three channel models is When the distance between the detectors exceeds R connect When the communication is disconnected, R connect The corresponding communication radius value is determined based on the specific type of detectors that communicate with each other. For a patroller, it must maintain communication with any patroller or flyer; for a flyer, it must maintain communication with any flyer or patroller. The communication range constraint is as follows:

[0066]

[0067] Among them, d gg , d ag , d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers.

[0068] S402: Perception range constraint. The perception coverage of the camera equipment on the detector is limited. The coverage of the leaper and the rover is set into circles of different sizes. The perception range radius of the leaper and the rover is marked as R a and R g .

[0069] S403: Coverage mode constraint. Since the flyer flies at a high speed, it is set to start collecting information only after it reaches a certain position and hovers, covering the area at its current position. The patroller, on the other hand, moves at a slower speed and can collect information while moving. All areas it has moved to are considered covered areas.

[0070] S404: Energy Constraint. Both detectors should minimize energy loss during operation, that is, the coverage time should be as short as possible. The energy consumption of the i-th detector in time slot t is expressed as Set the total energy consumption of the detector to be less than its own total energy.

[0071] S5: Set optimization goals based on constraints to maintain inter-cluster communication connectivity and maximize the total coverage area of ​​the flyers and patrollers under limited energy conditions.

[0072] From S4, we can see that the detectors are subject to four constraints. Among them, the communication range constraint affects the communication connectivity between detectors (whether it is disconnected), the perception range and coverage mode constraints affect the coverage area and method, and the energy limited constraint affects the movable time of the detector. Therefore, the optimization goal is set to maintain communication between detectors during the coverage period and achieve the maximum coverage area under a limited number of steps.

[0073] S6: Design a coverage information map, grid the area, and use a value for each grid to represent the coverage status of the grid.

[0074] Divide the map area into multiple small square areas of size n×n, and the entire target area has a total of Each grid is identified by a numerical value k to represent its own coverage. The coverage of the entire target map area can be represented as a matrix G:

[0075]

[0076] Among them, k uv is the grid value of the uth row and vth column, u, v = 1, 2, ... m. The matrix G is used to record the coverage information of all detectors, and the value of k is 0, i or i + c, i∈N D , c is a constant, i represents the patroller number, and c represents the leaper. Initially, the coverage mark value of all grids is 0, indicating that they are not covered; a k value of i represents that they have been covered by patroller i; a k value of i+c represents that the grid is the overlapping coverage area of ​​the leaper and patroller i. Figure 3 As shown, the coverage information map provided by the embodiment of the present invention describes the coverage of the current target area through the matrix G, where white represents uncovered, that is, the value of the corresponding grid is 0, and black represents covered, that is, the value of the corresponding grid is not 0.

[0077] It should be noted that the values ​​in matrix G are only used to indicate whether the grid area is covered, and the coverage area of ​​the leaper is only valid at the current moment. When the leaper stops, the value of the covered grid is added to the leaper's fixed value c. When the leaper flies away, the value of the previous grid is subtracted from c. According to the coverage constraint, every area covered by the patroller from the initial moment is valid, and the k value remains unchanged. However, only the coverage area of ​​the leaper in the current time step is valid. The matrix G at each moment will update the grid value covered by the leaper in the previous time step.

[0078] S7: Use the MA-PPO algorithm to optimize the detector deployment plan. This step designs the state space, action space, and reward function.

[0079] S701. Design the state space. To avoid duplicate coverage and reduced coverage efficiency during the coverage process of all detectors, real-time perception of the target area is required. This means that each detector needs to know not only its own coverage status, but also the coverage status of all other detectors. This requires that the detectors communicate their location information, that is, the coordinates of all detectors. Therefore, the state space is set to the coordinate information and movement direction of all detectors, as shown in the following formula:

[0080] S=[(x1,y1,θ1),(x2,y2,θ2),...,(x N ,y N ,θ N )]

[0081] Where x is the horizontal coordinate, y is the vertical coordinate, and θ is the angle, or direction of movement. For the flyby, the distance from the planetary surface is ignored, and only the coordinate position in the plane coordinate system is considered. To ensure that the probe can obtain coverage area information at each step, the ratio of the total coverage area to the target area is added to the state space, as follows:

[0082] S=[(x1,y1,θ1),...,(x N ,y N ,θ N ),(S a +S g ) / a 2 ]

[0083] Among them, S a is the coverage area of ​​the leaper, S g is the coverage area of ​​the patroller, a 2 is the total area of ​​the target area that needs to be detected.

[0084] S702. Design the action space. Two different detectors correspond to two different types of agents when learning with the MA-PPO algorithm. These two agents have different coverage areas, representing circles of different radii. To facilitate convergence, the actions of both agents are set to continuous, i.e., moving through an angle of [0, 360°].

[0085] S703. Design a reward function. Based on three optimization goals: maximizing detector coverage of the target area, maintaining inter-detector connectivity, and minimizing energy consumption, design a reward function. Energy consumption is proportional to time consumption, as follows:

[0086]

[0087] Among them, l i represents the energy consumption per unit time of detector i, t i represents the working time of detector i, represents the energy consumption of detector i.

[0088] That is, the shorter the coverage completion time is, the lower the energy consumption is. The total energy consumption E from the beginning to the completion of coverage is as follows:

[0089]

[0090] Among them, the subscripts a and g represent the flyer and rover respectively. represents the energy consumption of leaper a, l a , t a Respectively represent the energy consumption per unit time and working time of leaper a, represents the energy consumption of patroller g, l g , t g Represents the energy consumption per unit time and working time of patroller g, i∈N D , a∈N a , g∈N g , N a 、N g They are the leaper collection and the patroller collection.

[0091] Therefore, optimizing energy consumption is about optimizing the time required to complete each round of coverage. The MA-PPO algorithm already includes this optimization, and does not require a reward function to implement it. Therefore, the reward function r only needs to include the optimization objectives of coverage area and communication, as shown in the following formula:

[0092]

[0093] in, Used to adjust the weights of coverage area and communication connectivity, It means that it does not care whether the communication is connected, but only cares about the size of the coverage area. This means that coverage area is as important as communication connectivity. Indicates that only the communication connectivity is concerned. s For a reward function related to the coverage area, in order to better compare the coverage of the current time step with that of the previous time step, we cannot simply use the total coverage area as the reward function, but use the following formula:

[0094] r s =(S a +S g )-(S a '+S' g )

[0095] Among them, S a ′ and S′ g Represents the total coverage area of ​​the leaper and the patroller in the previous time step. a With S g The calculation method is the total number of grids covered by the leaper and the patroller, which are count a_cover and count g_cover , and the product of the area of ​​a single grid is as follows:

[0096] S a =n 2*count a_cover ,S g =n 2 *count g_cover

[0097] The area covered by the leaper and the patroller is counted as the patroller's coverage area, and the leaper will subtract this part of the coverage area.

[0098] r t is a reward function related to communication connectivity. The reward is positive when the entire detector network can communicate normally. To ensure full connectivity, if only one detector is disconnected, the reward is negative, as shown in the following formula:

[0099]

[0100] Through reasonable adjustments The size of can be used to adjust the weights of the two parts so that the detectors remain connected during collaborative coverage and achieve the expected coverage effect.

[0101] S8: Use the MA-PPO algorithm to optimize the detector deployment coverage plan.

[0102] When the total coverage area of ​​the detector reaches the target value or the energy is exhausted, a coverage is completed, and the optimal deployment coverage plan of the detector is calculated using the MA-PPO algorithm. At the initial moment of the coverage task, the positions of all detectors are consistent, with coordinates (0,0). Suppose the moving direction of the i-th detector at time t is Converted to radians, it is [0,2π). The moving speeds of the flyer and the rover are v a and v g , is a constant value, and the moving distance in each time interval [t, t+1] is Δd a and Δd g , all detectors change their positions by changing their moving directions.

[0103] If the coordinate of detector i at the current moment is p = (x, y), then the coordinate p′ at the next moment is as follows:

[0104]

[0105] Among them, Δd is the moving distance of the detector per unit time. If the leaper takes the value of Δd a , if the patrol value is Δd g .

[0106] After setting the initial values ​​of each detector in each round, the environment is established and coverage is started, and the MA-PPO algorithm is used for target optimization. Figure 4As shown, the specific coverage process of the flyer and the patroller provided by the embodiment of the present invention mainly includes the following steps S801 to 806.

[0107] S801. Initialize all flyers and patrollers, with their coordinates at (0,0) and their initial angles at random numbers in [0,2π).

[0108] S802. The patroller starts to move with the flyer; the patroller completes detection while moving, and the flyer only detects when it is in a hovering state. The flyer eventually stops at a certain position, at which time the coverage position of the flyer is finally determined.

[0109] S803. When the current coverage range of the leaper is equal to its own maximum perception range, that is, when the coverage ranges of the leaper and other leapers do not overlap, the leaper detaches from the patroller and hovers at the current position to cover, and the patroller continues to move to cover; if the leaper's coverage area has not reached the maximum when the patroller moves a preset number of steps, such as 50 time steps, the leaper detaches from the patroller and starts to move autonomously, and the leaper and the patroller each cover, and the leaper hovers until it reaches its own maximum perception range.

[0110] S804. Before the flyer begins covering the area at each time step, it must first clear the coverage area from the previous time step. If the coverage area from the previous time step contains areas that were covered by both flyer i and patrollers, the grid values ​​in that area are all set to i+c, indicating that they were covered by patrollers. In the current time step, the grid values ​​in that area are set to i. If the covered area is entirely covered by the flyer alone, the grid values ​​are all set to 0, indicating that they are not covered. The flyer begins covering the current time step after clearing the information from the previous time step.

[0111] S805. Coverage is complete when the total coverage area reaches 70% or energy is exhausted. In this embodiment of the present invention, for patrollers, energy exhaustion means reaching 300 time steps, while for flyers, energy exhaustion means reaching 100 autonomous steps after detaching from the patroller. The total coverage area at the time of completion is the final coverage area for that round. In this embodiment of the present invention, reaching 70% of the total coverage area completes the coverage mission for that round.

[0112] S806. In this embodiment of the present invention, the number of iterations of the MA-PPO algorithm is set to 1000, and the MA-PPO algorithm is used for training according to the set reward function to obtain the optimal coverage solution, which enables all flyers to maintain normal communication with patrollers while maximizing the coverage area under limited energy conditions.

[0113] Through the above embodiments, in response to the problem of detectors detecting and collecting unknown environments in planetary exploration scenarios, the present invention successfully implements a method for the coordinated deployment of flyers and rover based on the MA-PPO algorithm, introduces an air-ground collaborative detection mechanism, realizes information sharing and collaboration, improves the efficiency of the mission and the coverage rate of the target area, while reducing energy consumption and maintaining normal communication of the entire system. The method of the present invention ensures that the air-ground collaborative system can efficiently complete information collection in the extraterrestrial body detection mission, improves the efficiency and coverage rate of regional coverage. At the same time, by using reinforcement learning algorithms to intelligently adjust the coverage strategies of flyers and rover, the energy consumption of the detector in the process of performing the mission is significantly reduced, and the energy utilization efficiency of the entire system is improved.

Claims

1. A deep space multi-detector air-ground collaborative deployment method based on reinforcement learning, characterized in that: The method comprises the following steps: Step 1: In the deep space star chart, the flyer and the rover cooperate to detect unknown target areas. The flyer detects in the air, and the rover detects on the surface of the planet. The communication channels during the detection process are modeled. The communication channels are divided into three categories: air-to-air, air-to-ground, and ground-to-ground. The three types of communication channels are modeled separately, and the communication radius between different detectors is determined, including the communication radius between the two flyers. Communication radius between the leaper and the patroller Communication radius between two patrollers Step 2: Determine the constraints and optimization objectives for the flyer and rover to collaboratively explore the target area; First, identify the constraints, including: (1) Communication range constraint: Leapers and patrollers are detectors. Each detector is required to maintain normal communication with at least one other detector. The constraints are set as follows: Among them, d gg d ag d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers; (2) Perception range constraint: The perception coverage of the leaper and the patroller is represented by a circular area. Let the perception range radius of the leaper and the patroller be R respectively. a and R g ; (3) Coverage mode constraint: Initially, all patrollers start from the same position, and the leaper is carried by the patroller and moves. After a certain period of movement, the leaper separates from the patroller. After the leaper reaches the designated position and hovers, it detects the current sensing coverage range. The coverage area of ​​the leaper is only valid at the current moment. The patroller detects while moving, and the areas where the patroller moves are all covered areas that have been detected. (4) Energy-limited constraint: set the total energy consumption of each detector to be less than its own total energy; Secondly, the optimization goal is to maintain communication connectivity between detectors and maximize the total coverage area of ​​the detectors under limited energy. Then, the target area to be detected is represented as a 2D square area and gridded. The coverage of each grid is marked with a value, where a value of 0 indicates no coverage and a value greater than 0 indicates coverage. Step 3: Use the multi-agent proximal strategy optimization algorithm to calculate the optimization goal of step 2 and determine the deployment coverage plan of the detector; In the multi-agent proximal strategy optimization algorithm, the set state includes the coordinates and movement directions of all detectors, as well as the ratio of the current coverage target area; two agents are designed corresponding to the leaper and the patroller, and the actions of the two agents are set as the movement direction; the reward function is set as the weighted sum of the rewards related to the coverage area and the rewards related to the communication connectivity; and the detector deployment coverage plan is iteratively optimized according to the reward function.

2. The method according to claim 1, characterized in that In step 2, the current coverage of the target area is represented by a matrix G, where each element in the matrix G identifies the coverage of a grid; initially, the identification values ​​of all grids are 0; when a grid is covered by patroller i, the identification value of the grid is i; when a grid is repeatedly covered by patroller i and a leaper, the identification value of the grid is i+c, where c represents the leaper and is a fixed value; the matrix G is updated at the current moment by subtracting c from the identification value of the grid covered by the leaper at the previous moment.

3. The method according to claim 1 or 2, characterized in that The step 3 includes: (1) The detectors need to communicate their positions and obtain coverage information. For the flyer, only the coordinate position in the plane coordinate system is considered. The state space S is set as follows: S=[(x1,y1,θ1),...,(x N ,y N ,i N ),(S a +S g ) / a 2 ] Among them, x1, y1 are the plane coordinates of detector 1, θ1 is the moving direction of detector 1; N ,y N is the plane coordinate of the detector N, θ N The moving direction of detector N; N is the number of detectors; S a is the coverage area of ​​the leaper; S g is the coverage area of ​​the patroller; a 2 is the total area of ​​the target area that needs to be detected; (2) The actions of the two agents are set as moving directions, which take values ​​between [0, 360°) and are continuous actions; (3) Design a reward function based on the optimization goal; optimize energy consumption by optimizing the time step required to complete coverage in each round, and optimize communication connectivity and coverage area by setting the reward function r; Reward Function in is the weight for adjusting coverage area and communication connectivity, r s is the reward related to the coverage area, calculated as: r s =(S a +S g )-(S′ a +S′ g );S a 、S g is the coverage area of ​​the leaper and patroller at the current time step, S′ a , S′ g is the coverage area of ​​the leaper and the patroller in the previous time step; the area covered by the leaper and the patroller is counted as the patroller's coverage area, and the leaper will subtract the area covered by the repeated coverage; r t Rewards related to communication connectivity are calculated as: When the entire detector network can communicate normally, the reward is r t is a positive value, as long as one detector is disconnected, the reward r t is -1.

4. The method according to claim 1 or 2, characterized in that In step 3, the multi-agent proximal strategy optimization algorithm is used to optimize the deployment coverage plan, including: (3.1) At the beginning of each round, the position coordinates of all jumpers and detectors are set to (0,0), and the moving direction is randomly selected in [0,2π); (3.2) The patroller carries the leaper and moves. When the coverage of the leaper does not overlap with that of other leapers, the leaper detaches from the patroller and hovers at the current position to cover the area, while the patroller continues to move to cover the area. If the patroller's movement steps reach the preset time step, the leaper detaches from the patroller and starts to move autonomously until the coverage of the leaper does not overlap with that of other leapers, and the patroller continues to move. (3.3) Before the leaper starts covering at each time step, it first clears the coverage area information of the previous time step and then starts covering the current time step; (3.4) When the total coverage area of ​​the detector reaches the set target value or the energy of the detector is exhausted, one coverage is completed; (3.5) The detector deployment coverage plan is trained according to the set number of iteration rounds and reward function, and the optimal detector deployment plan is output.

5. The method according to claim 1, wherein In step 1, modeling the communication channel during the detection process includes the following: (1.1) For the air-to-air channel model between hops, the total channel loss L a2a The calculation is as follows: L a2a =L f +L a ·d Wherein, the free space path loss L f =20log 10 d+20log 10 f+32.45; d is the distance between the inter-flyers; atmospheric absorption loss f is the communication signal frequency, is the absorption loss of water vapor, is the oxygen absorption loss; (1.2) For the air-to-ground channel model between the flyer and the rover, the total channel loss L a2g The calculation is as follows: L a2g =PL+L d ·d Where PL is the path loss, L d is the dust loss, d is the distance between the flyer and the rover; Dust loss Where λ is the wavelength of the communication signal, N T is the density of sand particles, is the average dust particle radius, θ is the elevation angle of the propagation path, ε' and ε" are the real and imaginary parts of the average dielectric constant of dust particles, respectively; (1.3) For the ground-to-ground channel model between patrollers, the channel loss is the diffraction loss of obstacles. The alternating method is used to calculate the channel loss L between patrollers. g2g as follows: Among them, L D1 is the diffraction loss of the first obstacle, i.e., the blade-shaped obstacle; a is the distance from the transmitter to the first obstacle; b is the distance between the first obstacle and the second obstacle; c is the distance from the second obstacle to the receiver; d = a + b + c is the distance between the transmitter and the receiver; L D2 is the diffraction loss of the second obstacle, i.e. the cylindrical obstacle, L D2 =L D (b,c,H′ c2 )+T, T is the additional loss related to the propagation environment, L D is the diffraction loss of a knife-shaped obstacle under the same conditions, H′ c1 , H′ c2 are the heights of the top of the obstacle relative to the line of sight between the transmitter and the receiver.

Images