Deep space star catalogue multi-detector air-ground collaborative deployment method based on reinforcement learning

Abstract

The invention provides a deep space star catalogue multi-detector air-ground collaborative deployment method based on reinforcement learning, and belongs to the technical field of planet surface detection. According to the method, a leap device and a patroller serve as detectors to cooperatively detect an unknown area of a deep space star catalogue, three types of communication channels are established between the detectors, and the communication radius between the different detectors is calculated; the method comprises the following steps: establishing constraints in the aspects of communication range, sensing range, coverage mode and energy, and setting an optimization target to keep communication connection between detectors under limited energy and maximize the total coverage area of the detectors; a leap device and a patroller are used as two agents, and an optimal deployment strategy is determined based on a multi-agent near-end strategy optimization algorithm. According to the invention, the method optimizes the regional coverage strategy, achieves the efficient distribution of resources, improves the detection efficiency and accuracy of a leap device and a detector in extreme terrains and complex environments, reduces the energy consumption while guaranteeing the coverage efficiency, and improves the overall energy utilization efficiency of the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep space planetary surface detection, and in particular to a method for space-ground coordinated deployment of multiple deep space star surface detectors based on reinforcement learning. Background Art

[0002] Deep space exploration is a frontier field of scientific and technological development and the foundation for other deep space operations in the future. Its importance is self-evident. Due to the particularity of the deep space environment, it is much easier for intelligent agents to operate in deep space scenes such as the moon or Mars than for humans. Deep space exploration should be developed with autonomy, large scale, and intelligence as its main characteristics, that is, the detector group is autonomously grouped, commanded, and detected. Therefore, it is necessary to design an air-ground collaborative deployment method driven by intelligent algorithms to achieve the above tasks.

[0003] As a widely used unmanned platform, the flyover is seen as an ideal choice to replace humans in high-risk and harsh environment tasks. With the increasing maturity of related technologies, the flyover has shown wide application potential in agricultural production, traffic control, disaster relief, military reconnaissance and mobile communications, and has become a key trend in the current unmanned platform research field. Among them, tasks such as surveying and mapping, monitoring, and detecting unknown targets all require coverage of the entire area. The intelligent technology of the flyover is crucial to its function, especially the intelligent coverage technology. How to cover the target area efficiently and safely has become the focus of research. However, in the deep space star exploration scenario, there are also some defects when only using the flyover for coverage tasks, such as fast energy consumption.

[0004] Generally speaking, when a swarm of swarmers is deployed to complete a regional coverage mission, each swarmer is first assigned an accurate task through a task decision method, and then the respective coverage paths are planned according to the assigned tasks to ensure that the entire target area is covered. Cluster coverage planning strategies mainly include centralized planning and distributed planning. Centralized planning can formulate a better deployment plan for the cluster based on global information. However, with the increase in the number of swarmers and the expansion of the coverage area, the amount of calculation will increase dramatically, which limits the scalability of centralized planning. In contrast, distributed planning shows better flexibility and scalability. It adapts to all environmental changes through local information and communication with neighboring nodes, thereby achieving a better state of the cluster system, but the disadvantage is that it may be difficult to achieve the global optimal coverage deployment. These traditional optimization algorithms mainly optimize the motion trajectory of the intelligent agent to achieve the shortest coverage time or maximize the coverage rate of the target area, but they are difficult to handle large-scale problems, lack real-time and intelligence, and have certain limitations. Summary of the invention

[0005] In view of the above-mentioned existing problems, in order to solve the problem of unknown area coverage in planetary exploration scenarios, especially the problem of large-scale unknown area exploration, the present invention discloses a deep space star catalog multi-detector air-ground collaborative deployment method based on reinforcement learning. An air-ground collaborative system composed of multiple detectors (including flybys and rovers) is used to explore the target area, thereby maintaining communication connectivity during multiple detection periods and maximizing the coverage area under limited energy conditions, which improves real-time and reliability compared with the existing technology.

[0006] The present invention provides a deep space star table multi-detector air-ground collaborative deployment method based on reinforcement learning. In the deep space star table, a flyer and a rover collaborate to detect unknown target areas. The planet has atmospheric conditions suitable for the flyer to fly. The flyer detects in the air and the rover detects on the surface of the planet. The method includes the following steps:

[0007] Step 1: Model the communication channel of the deep space star catalog; the communication channels are divided into three categories: air-to-air (A2A), air-to-ground (A2G), and ground-to-ground (G2G); model the three types of communication channels respectively, and determine the communication radius between different detectors; including the communication radius between two flybys Communication radius between the flyer and the patroller Communication radius between two patrollers

[0008] Step 2: Determine the constraints and optimization objectives for the flyer and the patroller to collaboratively detect the target area; the determined constraints include the following:

[0009] (1) Communication range constraint: The flyer and the patroller are detectors, and each detector is required to maintain normal communication with at least one other detector. The constraints are set as follows:

[0010]

[0011] Among them, d gg ,d ag ,d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers;

[0012] (2) Perception range constraint: The perception coverage range of the leaper and the patroller is represented by a circular area. Suppose the perception range radius of the leaper and the patroller is R respectively. a and R g ;

[0013] (3) Coverage mode constraint: Initially, all patrollers start from the same position, and the flyer is carried by the patroller. After moving for a certain period of time, the flyer separates from the patroller. After the flyer reaches the designated position and hovers, it detects and collects information about the current sensing coverage range. The coverage area of ​​the flyer is only valid at the current moment. The patroller collects information while moving, and all areas where the patroller has moved are covered areas that have been detected.

[0014] (4) Energy-limited constraint: Set the total energy consumption of each detector to be less than its own total energy;

[0015] The optimization goal is to maintain communication connectivity between detectors and maximize the total coverage area of ​​the detectors under limited energy. The target area to be detected is represented as a 2D square area and gridded. The coverage of each grid is marked with a value, where a value of 0 indicates no coverage and a value greater than 0 indicates coverage.

[0016] Step 3: Calculate the optimization target of step 2 based on the Multi-Agent Proximal Policy Optimization (MA-PPO) algorithm to optimize the deployment coverage plan of the detector.

[0017] In the multi-agent proximal strategy optimization algorithm, the set state includes the coordinates and movement directions of all detectors, as well as the ratio of the current coverage target area; two agents are designed corresponding to the leaper and the patroller, and the actions of the two agents are set to the movement direction; the reward function is set to the weighted sum of the reward related to the coverage area and the reward related to the communication connectivity. The detector deployment coverage plan is iteratively optimized according to the reward function.

[0018] Compared with the prior art, the advantages and positive effects of the present invention are:

[0019] (1) Since the patroller and the flyer have different detection ranges and different information resolutions, the flyer can collect information about areas that the patroller cannot reach, and the patroller can collect information about areas where the flyer's line of sight is blocked, so the flyer and the patroller can perform different detection functions. The present invention uses two types of detectors to give full play to their respective functional advantages and efficiently complete complex mission objectives.

[0020] (2) The method of the present invention has the advantage of information sharing. The flyer can quickly obtain star catalog information through flight. The flyer and the rover, the flyers, and the rover can all share information through communication within the system, so as to fully grasp the current detection situation and avoid missing the target area. This information sharing and coordination is conducive to improving the efficiency of task completion and can further improve the efficiency and accuracy of the entire task.

[0021] (3) The method of the present invention has the advantage of detection efficiency. Although the traditional coverage algorithm can achieve coverage of the target area, it is difficult to achieve the optimal coverage deployment strategy when the number of detectors increases. The method of the present invention introduces the MA-PPO algorithm to dynamically adjust the coverage strategy of the flyer and the patroller according to the regional environmental information and the real-time changing coverage situation. The flyer and the patroller interact with the environment to obtain the optimal movement strategy, avoid repeated coverage, effectively reduce the detection time, and improve the task efficiency of the entire system. The solution obtained by the method of the present invention maximizes the coverage area under limited energy conditions while maintaining communication between all flyers and patrollers, effectively overcoming the limitations of traditional optimization algorithms in dealing with large-scale problems, improving detection efficiency and ensuring communication between detectors.

[0022] (4) The method of the present invention has cost advantages. The flyover device has a simple structure and low cost. Compared with the detection using only patrol devices, air-ground collaborative detection will not incur high costs due to equipment failure or damage. In addition, the flyover device has a wide detection range. When performing the target area detection task, the number of detectors required for collaborative detection is less than that using only patrol devices, and the energy consumption is also lower.

[0023] (5) The method of the present invention has application advantages. The method of the present invention is applicable to various scenarios with complex environments and limited detector energy, and provides an effective technical solution for the field of planetary exploration. The method of the present invention can effectively improve the detection efficiency and accuracy of the air-ground collaborative system in planetary exploration scenarios, while optimizing resource allocation, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of a method for coordinated coverage of deep space star tables and ground surfaces according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of an air-ground collaborative coverage scenario according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a coverage information map according to an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the specific deployment and coverage process of the detectors according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Through the above-mentioned drawings, clear embodiments of the present disclosure have been shown, and a more detailed description will be given below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments.

[0029] The present invention uses a method for collaborative detection using a flyover and a rover in a planetary exploration scenario. The energy of the flyover mainly comes from solar panels. When the flyover is used alone to perform a detection mission, its own energy is difficult to supply the long-term flight of the flyover. When only the rover is used for detection, information on extreme terrain cannot be collected and the coverage is limited. Therefore, an air-ground collaborative system is introduced into the planetary exploration scenario, and two types of detectors, a flyover and a rover, are used to perform tasks. This can not only solve the problems of long time and high cost when only the rover is used for detection, but also make up for the shortcomings of easy omissions when only the flyover is used for detection. The method of the present invention not only improves the ability to complete complex mission objectives through the air-ground collaborative system, but also further improves the efficiency and accuracy of the entire mission.

[0030] like Figure 1 As shown, the deep space catalog multi-detector air-ground collaborative deployment method based on reinforcement learning provided by an embodiment of the present invention conducts regional exploration through the collaboration of a flyer and a rover, and mainly includes 8 steps S1 to S8.

[0031] S1: Model the communication channel when the air-ground collaborative system is working. Figure 2 As shown, the air-ground collaborative coverage scenario of the embodiment of the present invention includes three channel models: A2A channel between flyers, A2G channel between flyers and patrollers, and G2G channel between patrollers. The embodiment of the present invention uses Mars as the target planet for exploration. Mars has atmospheric conditions suitable for flyers to fly and can meet the basic environmental requirements for flyers to perform exploration tasks. The wireless signals are subject to different fading under different channels, and the three types of channels are modeled separately, as shown in the following steps S101-S103.

[0032] S101: A2A channel modeling. When the flyer communicates, the signal propagates in a straight line, and the path loss is the free space loss, which is also affected by the atmosphere. The total loss of the A2A channel is L a2a is the free space path loss L f and atmospheric absorption loss L a The sum is as follows:

[0033] L a2a =L f +L a ·d

[0034] Where, the free space path loss L f The unit is dB, L f The calculation method is as follows:

[0035] L f =20log 10 d+20log 10 f+32.45

[0036] The unit of the communication signal frequency f is MHz, and the unit of the distance d between the two jumpers is km.

[0037] Atmospheric absorption loss L a The unit is dB / Km, L a It is the sum of the absorption loss of water vapor and oxygen, as shown below:

[0038]

[0039] in, is the absorption loss of water vapor, It is the oxygen absorption loss.

[0040] S102: A2G channel modeling. When the flyer and the patroller communicate, they are greatly affected by the dust. At this time, the atmospheric absorption loss can be ignored. The total loss of the A2G channel is L a2g is the path loss PL and the dust loss L d The sum of, such as:

[0041] L a2g =PL+L d ·d

[0042] Among them, the unit of path loss PL is dB, and the unit of dust loss L d The unit is dB / Km, d is the distance between the flyer and the patroller, in Km.

[0043] When the leaper communicates with the patroller, as the leaper continues to move, there may be obstacles blocking the connection between the sender and the receiver, that is, a non-line-of-sight environment. Since A2G communication has both line-of-sight (LOS) and non-line-of-sight (NLOS) conditions, it can be modeled using a probabilistic path loss model that includes line-of-sight and non-line-of-sight conditions with different occurrence probabilities, as shown in the following formula:

[0044] PL=P LOS PL LOS +P NLOS PL NLOS

[0045] Among them, P LOS represents the probability of line-of-sight channel, PL LOS represents the path loss of the line-of-sight channel, P NLOS represents the probability of non-line-of-sight channel, PL NLOS Represents the path loss of the non-line-of-sight channel.

[0046] Dust loss L d The calculation is as follows:

[0047]

[0048] Where λ is the wavelength of the communication signal, is the average dust particle radius, N T is the density of sand particles, θ is the elevation angle of the path propagation, ε' and ε" are the real and imaginary parts of the average dielectric constant of sand and dust particles, respectively.

[0049] S103: G2G channel modeling. The loss of the G2G channel is the diffraction loss of the obstacle. In the actual detection environment, the distribution of obstacles on the surface of Mars is irregular, usually circular and blade-shaped obstacles exist alternately, so a mixed obstacle model needs to be considered. The alternating method is used to calculate the loss L of the G2G channel. g2g as follows:

[0050]

[0051] Among them, L D1 is the diffraction loss of the first obstacle, i.e. the blade-shaped obstacle, a is the distance from the transmitter to the first obstacle, b is the distance between the two obstacles (i.e. the first obstacle and the second obstacle), c is the distance from the second obstacle to the receiver, d = a + b + c is the distance between the transmitter and the receiver, L D2 is the diffraction loss of the second obstacle, which is a cylindrical obstacle, and is calculated as follows:

[0052] L D2 =L D (b,c,H′ c2 )+T

[0053] Where T is the additional loss related to the propagation environment, L D is the knife-edge diffraction loss under the same conditions, H′ c1 , H′ c2 is the clearance, that is, the height of the top of the obstacle relative to the direct line of sight between the transmitter and the receiver.

[0054] S2: Obtain the maximum effective communication distance of the flyer and the patroller during joint detection. Based on the communication channel model, the loss of communication between detectors can be obtained, and the effective communication range between detectors can also be determined to determine the communication limitations of the detectors when performing tasks and improve detection accuracy.

[0055] According to the Friis transmission equation:

[0056]

[0057] Among them, P r is the received power, P t is the transmission power, G t is the antenna gain at the transmitting end, G ris the antenna gain at the receiving end, λ is the wavelength, and d is the distance. This equation is the free space propagation formula, which is the propagation situation under ideal conditions without environmental interference. By converting the free space loss in the formula into the total loss L, the signal propagation formula is as follows:

[0058] P r =P t +G t +G r -L

[0059] Where L is the loss.

[0060] Then, according to the loss of the three types of channels and the power and gain of the detectors, the maximum effective communication distance between the leaper and the patroller, that is, the communication radius between the two detectors, is obtained. In the embodiment of the present invention, the communication distance between the two detectors, that is, the communication radius, is determined based on the power and gain of the two specific detectors and the lower limit of the received power. The communication radius between the two leapers is calculated to be The communication radius between the flyer and the patroller is The communication radius between two patrollers is

[0061] S3: Model the coverage environment and represent the map area to be covered as a 2D square area.

[0062] The embodiment of the present invention sets the moving speed of the patroller and the flyer to a constant value, and the flight height of all flyers is consistent and fixed. There is no lift-off and descent process involved. The coverage scene is considered to be a 2D square area with a size of z×z, which is the coverage target of the detector. A 2D coordinate system is established for the square map area to be covered. Each flyer is carried by a patroller. Initially, all patrollers are in the same position, starting from the coordinate (0,0), and the flyer and the patroller move together. After the patroller carries the flyer for a period of time, the flyer separates from the patroller and begins to cover.

[0063] Assume that all detectors constitute a set N D ∈{i=1,2,...,N}, each patrol vehicle and each flyer in the set can perform the coverage task, and N is the total number of detectors.

[0064] S4: Design constraints for the flyer and the patroller, including communication range constraints, perception range constraints, coverage mode constraints, and energy constraints. During the execution of tasks in the embodiment of the present invention, the specific constraints imposed on the flyer and the patroller are as follows:

[0065] S401: Communication range constraint. Since the communication range between the flyer and the patroller is limited, each detector must maintain normal communication with at least one other detector to ensure that the entire multi-detector network maintains communication connection. Suppose the communication radius under the three channel models is When the distance between the detectors exceeds R connect When the communication is disconnected, R connect The corresponding communication radius value is determined according to the specific type of the two communicating detectors. For a patroller, it is necessary to maintain communication with any patroller or flyer; for a flyer, it is necessary to maintain communication with any flyer or patroller. The communication range constraint is as follows:

[0066]

[0067] Among them, d gg , d ag , d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers.

[0068] S402: Perception range constraint. The perception coverage of the camera equipment on the detector is limited. The coverage of the leaper and the patroller is set to circles of different sizes. The perception range radii of the leaper and the patroller are marked as R and R respectively. a and R g .

[0069] S403: Coverage mode constraint. Since the flying vehicle has a high flying speed, it is set to start collecting information only after it reaches a certain position and hovers, covering the area at its current position. The patrol vehicle has a slow speed and can collect information while moving. All areas that have been moved are covered areas.

[0070] S404: Energy Constraint. Both detectors should minimize energy consumption during operation, that is, the coverage time should be as short as possible. The energy consumption of the i-th detector in time slot t is expressed as Set the detector's total energy consumption to be less than its own total energy.

[0071] S5: Set optimization goals according to constraints to maintain inter-cluster communication connectivity and maximize the total coverage area of ​​the flyers and patrollers under limited energy conditions.

[0072] From S4, we can know that the detector is subject to four constraints. Among them, the communication range constraint affects the communication connectivity between detectors (whether it is disconnected), the perception range and coverage mode constraints affect the coverage area and method, and the energy limited constraint affects the movable time of the detector. Therefore, the optimization goal is set to maintain communication between detectors during the coverage period and achieve the maximum coverage area under a limited number of steps.

[0073] S6: Design a coverage information map, grid the area, and use a value for each grid to represent the coverage status of the grid.

[0074] Divide the map area into multiple small square areas of size n×n, then the entire target area has There are square grids with a side length of n. All grids are marked with a numerical value k to represent their own coverage. The coverage of the entire target map area can be represented as a matrix G:

[0075]

[0076] Among them, k uv is the grid value of the uth row and vth column, u,v=1,2,…m. The matrix G is used to record the coverage information of all detectors, and the value of k is 0, i or i+c, i∈N D , c is a constant, i represents the patroller number, and c represents the leaper. Initially, the coverage mark value of all grids is 0, indicating that they are not covered; k value is i, indicating that they have been covered by patroller i; k value is i+c, indicating that the grid is the overlapping coverage area of ​​the leaper and patroller i. Figure 3 As shown, the coverage information map provided by the embodiment of the present invention describes the coverage of the current target area through a matrix G, wherein white represents uncovered, that is, the value of the corresponding grid is 0, and black represents covered, that is, the value of the corresponding grid is not 0.

[0077] It should be noted that the values ​​in the matrix G are only used to indicate whether the grid area is covered, and the coverage area of ​​the leaper is only valid at the current moment. When the leaper stays, the value of the covered grid is added with the fixed value c of the leaper. When the leaper flies away, the value of the previous grid is subtracted from c. According to the coverage constraint, every area covered by the patroller from the initial moment is valid, and the k value remains unchanged. However, only the coverage area of ​​the leaper in the current time step is valid. The matrix G at each moment will update the grid values ​​covered by the leaper in the previous time step.

[0078] S7: Use the MA-PPO algorithm to optimize the detector deployment plan. This step designs the state space, action space and reward function.

[0079] S701. Design state space. In the process of all detectors covering, in order to avoid repeated coverage and reduce coverage efficiency, it is necessary to be able to perceive the target area in real time, which means that each detector not only needs to know its own coverage, but also the coverage of all other detectors. This requires the detectors to communicate position information, that is, the coordinates of all detectors, so the state space is set to the coordinate information and movement direction of all detectors, as shown in the following formula:

[0080] S=[(x1,y1,θ1),(x2,y2,θ2),...,(x N ,y N ,θ N )]

[0081] Among them, x is the horizontal coordinate, y is the vertical coordinate, and θ is the angle, that is, the moving direction. For the flyby, ignore its distance from the planetary surface and only consider its coordinate position in the plane coordinate system. In order for the detector to obtain coverage area information at each step, the ratio of the total coverage area to the target area is added to the state space, as follows:

[0082] S=[(x1,y1,θ1),...,(x N ,y N ,θ N ),(S a +S g ) / a 2 ]

[0083] Among them, S a is the coverage area of ​​the leaper, S g is the coverage area of ​​the patroller, a 2 is the total area of ​​the target area that needs to be detected.

[0084] S702. Design action space. Two different detectors correspond to two different types of agents when learning with the MA-PPO algorithm. The coverage of these two agents is different, which are circles with different radii. In order to make the training results more convergent, the actions of the two types of agents are set as continuous actions, that is, the moving angle is [0,360°).

[0085] S703. Design a reward function. Design a reward function based on three optimization goals: maximizing the target area covered by detectors, maintaining communication connectivity between detectors, and minimizing energy consumption. Among them, energy consumption is proportional to time consumption, as follows:

[0086]

[0087] Among them, l i represents the energy consumption per unit time of detector i, t i represents the working time of detector i, represents the energy consumption of detector i.

[0088] That is, the shorter the coverage completion time is, the smaller the energy consumption is. The total energy consumption E from the beginning to the completion of coverage is as follows:

[0089]

[0090] The subscripts a and g represent the flyer and rover, respectively. represents the energy consumption of leaper a, l a ,t a Respectively represent the energy consumption per unit time and working time of leaper a, represents the energy consumption of patroller g, l g ,t g Respectively represent the energy consumption per unit time and working time of patrol g, i∈N D , a∈N a , g∈N g , N a 、N g They are the leaper collection and the patroller collection.

[0091] Therefore, optimizing energy consumption is to optimize the time steps required to complete each round of coverage. The MA-PPO algorithm itself already includes this part of the optimization, and there is no need to use a reward function to implement it. Therefore, the reward function r only needs to include the optimization objectives of coverage area and communication, as shown in the following formula:

[0092]

[0093] in, Used to adjust the weights of coverage area and communication connectivity, It means that it does not care whether the communication is connected, but only cares about the size of the coverage area. This means that the coverage area is as important as the communication connectivity. Indicates that only the communication connectivity is concerned. s For a reward function related to the coverage area, in order to better compare the coverage of the current time step with that of the previous time step, the total coverage area cannot be simply used as the reward function, but the following formula is used:

[0094] r s =(S a +S g )-(S a ′+S′ g )

[0095] Among them, S a ′ and S′ g Respectively represent the total coverage area of ​​the flyer and the patroller in the previous time step. a With S g The calculation method is the total number of grids covered by the leaper and the patroller, which are count a_cover and count g_cover , multiplied by the area of ​​a single grid is as follows:

[0096] S a =n 2*count a_cover ,S g =n 2 *count g_cover

[0097] The area covered by the leaper and the patroller is counted as the patroller coverage area, and the leaper will subtract this part of the coverage area.

[0098] r t is a reward function related to communication connectivity. When the entire detector network can communicate normally, the reward is positive. In order to ensure full connectivity, as long as one detector is disconnected, the reward is negative, as shown in the following formula:

[0099]

[0100] Through reasonable adjustments The size of can be used to adjust the weights of the two parts so that the detectors remain connected during collaborative coverage and achieve the expected coverage effect.

[0101] S8: Use the MA-PPO algorithm to optimize the deployment and coverage plan of the detector.

[0102] When the total coverage area of ​​the detector reaches the target value or the energy is exhausted, the coverage is completed. The optimal deployment coverage plan of the detector is calculated using the MA-PPO algorithm. At the initial moment of the coverage task, the positions of all detectors are consistent, with coordinates (0,0). Suppose the moving direction of the i-th detector at time t is Converted to radians, it is [0,2π). The moving speeds of the flyer and the rover are v a and v g , is a constant value, and the moving distance in each time interval [t, t+1] is Δd a and Δd g , all detectors change their positions by changing their moving directions.

[0103] If the coordinate of detector i at the current moment is p = (x, y), then the coordinate p' at the next moment is as follows:

[0104]

[0105] Where Δd is the distance the detector moves per unit time. If the leaper takes the value Δd a , if the patroller takes the value Δd g .

[0106] After setting the initial values ​​of each detector in each round, the environment is established to start coverage and use the MA-PPO algorithm for target optimization. Figure 4As shown, the specific coverage process of the leaper and the patroller provided in the embodiment of the present invention mainly includes the following steps S801 to 806.

[0107] S801. Initialize all flyers and patrollers, with their coordinates at (0,0) and their initial angles at random numbers in [0,2π).

[0108] S802. The patroller starts to move with the flyer; the patroller completes detection while moving, and the flyer performs detection only when it is in a hovering state. The flyer eventually stops at a certain position, at which time the coverage position of the flyer is finally determined.

[0109] S803. When the current coverage of the leaper is equal to its own maximum perception range, that is, when the coverage ranges of the leaper and other leapers do not overlap, the leaper detaches from the patroller and hovers at the current position to cover, and the patroller continues to move to cover; if the leaper's coverage area has not reached the maximum when the patroller's movement steps reach the preset number of steps, such as 50 time steps, the leaper detaches from the patroller and starts to move autonomously, the leaper and the patroller each cover, and the leaper hovers until it reaches its own maximum perception range.

[0110] S804. Before the leaper covers in each time step, it is necessary to clear the coverage area of ​​the previous time step. If there is an area in the coverage area of ​​the previous step that has been repeatedly covered by leaper i and patroller, that is, the grid values ​​of the area are all changed to i+c, indicating that it has been covered by the patroller, and the grid values ​​of the area are changed to i in the current time step; if the covered area is all covered by the leaper alone, the grid values ​​are all changed to 0, indicating that it is not covered. The leaper starts covering the current time step after clearing the information of the previous time step.

[0111] S805. Coverage is completed when the total coverage area reaches 70% or the energy is exhausted. In the embodiment of the present invention, for the patroller, energy exhaustion means that the time step reaches 300, and for the flyer, the number of autonomous movement steps after leaving the patroller reaches 100. The total coverage area when coverage is completed is the final coverage area of ​​the round. In the embodiment of the present invention, the total coverage area reaches 70% to complete the coverage task of this round.

[0112] S806. In the embodiment of the present invention, the number of iteration rounds of the MA-PP0 algorithm is set to 1000, and the MA-PPO algorithm is used for training according to the set reward function to obtain the best coverage plan, so that all flyers and patrollers can maintain normal communication and the coverage area can be maximized under limited energy conditions.

[0113] Through the above embodiments, in order to solve the problem of detecting and collecting unknown environments by detectors in planetary exploration scenarios, the present invention successfully implements a method for the coordinated deployment of flyers and rover based on the MA-PPO algorithm, introduces an air-ground collaborative detection mechanism, realizes information sharing and collaboration, improves the efficiency of the mission and the coverage of the target area, while reducing energy consumption and maintaining normal communication of the entire system. The method of the present invention ensures that the air-ground collaborative system can efficiently complete information collection in the extraterrestrial body detection mission, improves the efficiency and coverage of regional coverage. At the same time, by using a reinforcement learning algorithm to intelligently adjust the coverage strategy of the flyer and the rover, the energy consumption of the detector during the mission is significantly reduced, and the energy utilization efficiency of the entire system is improved.

Claims

1. A method for space-ground collaborative deployment of multiple detectors in deep space star catalog based on reinforcement learning, characterized in that: The method comprises the following steps: Step 1: In the deep space star chart, the flyby and the rover cooperate to detect unknown target areas. The flyby detects in the air, and the rover detects on the surface of the planet. The communication channels in the detection process are modeled; the communication channels are divided into three categories: air-to-air, air-to-ground, and ground-to-ground; the three types of communication channels are modeled respectively, and the communication radius between different detectors is determined, including the communication radius between the two flybys. Communication radius between the leaper and the patroller Communication radius between two patrollers Step 2: Determine the constraints and optimization objectives for the flyer and rover to collaboratively detect the target area; First, identify the constraints, including: (1) Communication range constraint: The flyer and the patroller are detectors, and each detector is required to maintain normal communication with at least one other detector. The constraints are set as follows: Among them, d gg ,d ag ,d aa They are the distance between patrollers, the distance between leapers and patrollers, and the distance between leapers; (2) Perception range constraint: The perception coverage of the leaper and the patroller is represented by a circular area. Suppose the perception range radii of the leaper and the patroller are R a and R g ; (3) Coverage mode constraint: Initially, all patrollers start from the same position, and the leaper is carried by the patroller. After moving for a certain period of time, the leaper separates from the patroller. After the leaper reaches the designated position and hovers, it detects the current sensing coverage range. The coverage area of ​​the leaper is only valid at the current moment. The patroller detects while moving, and the areas where the patroller moves are all covered areas that have been detected. (4) Energy-limited constraint: Set the total energy consumption of each detector to be less than its own total energy; Secondly, the optimization goal is to maintain the communication connectivity between detectors and maximize the total coverage area of ​​the detectors under limited energy. Then, the target area to be detected is represented as a 2D square area and gridded. The coverage of each grid is marked with a value. A value of 0 indicates no coverage, and a value greater than 0 indicates coverage. Step 3: Use the multi-agent proximal strategy optimization algorithm to calculate the optimization target of step 2 and determine the deployment coverage plan of the detector; In the multi-agent proximal strategy optimization algorithm, the set state includes the coordinates and moving directions of all detectors, and the ratio of the current coverage target area; two agents are designed corresponding to the leaper and the patroller, and the actions of the two agents are set as the moving direction; the reward function is set as the weighted sum of the reward related to the coverage area and the reward related to the communication connectivity; and the detector deployment coverage plan is iteratively optimized according to the reward function.

2. The method according to claim 1, characterized in that In the step 2, the coverage of the target area currently detected is represented by a matrix G, and each element in the matrix G identifies the coverage of a grid; initially, the identification values ​​of all grids are 0; when a grid is covered by patroller i, the identification value of the grid is i; when a grid is repeatedly covered by patroller i and a leaper, the identification value of the grid is i+c, where c represents the leaper and is a fixed value; the matrix G is updated at the current moment, and the identification value of the grid covered by the leaper at the previous moment is subtracted from c.

3. The method according to claim 1 or 2, characterized in that: The step 3 comprises: (1) The detectors need to communicate their positions and obtain coverage information. For the flyer, only the coordinate position in the plane coordinate system is considered. The state space S is set as follows: S=[(x1,y1,θ1),...,(x N ,y N ,i N ),(S a +S g ) / a 2 ] Among them, x1, y1 are the plane coordinates of detector 1, θ1 is the moving direction of detector 1; N ,y N is the plane coordinate of detector N, θ N The moving direction of detector N; N is the number of detectors; S a is the coverage area of ​​the leaper; S g is the coverage area of ​​the patroller; a 2 is the total area of ​​the target area that needs to be detected; (2) The actions of the two agents are set as moving directions, with values ​​between [0, 360°), which are continuous actions; (3) Design a reward function based on the optimization goal; optimize energy consumption by optimizing the time step required to complete coverage in each round, and optimize communication connectivity and coverage area by setting the reward function r; Reward Function in is the weight for adjusting coverage area and communication connectivity, r s is the reward related to the coverage area, calculated as: r s =(S a +S g )-(S′ a +S′ g );S a , S g is the coverage area of ​​the leaper and patroller at the current time step, S′ a , S′ g is the coverage area of ​​the leaper and the patroller in the previous time step; the area covered repeatedly by the leaper and the patroller is counted as the coverage area of ​​the patroller, and the leaper will subtract the area covered repeatedly; r t Rewards related to communication connectivity are calculated as: When the entire detector network can communicate normally, the reward is r t is a positive value. As long as one detector is disconnected, the reward r t is -1.

4. The method according to claim 1 or 2, characterized in that: In step 3, the multi-agent proximal strategy optimization algorithm is used to optimize the deployment coverage solution, including: (3.1) At the beginning of each round, the position coordinates of all leapers and detectors are set to (0,0), and the moving direction is randomly selected in [0,2π); (3.2) The patroller carries the leaper and moves. When the coverage of the leaper does not overlap with that of other leapers, the leaper detaches from the patroller and hovers at the current position to cover the patroller, and the patroller continues to move to cover the patroller. If the number of steps the patroller moves reaches the preset time step, the leaper detaches from the patroller and starts to move autonomously until the coverage of the leaper does not overlap with that of other leapers, and the patroller continues to move. (3.3) Before the leaper starts covering at each time step, the coverage area information of the previous time step is cleared first, and then the coverage of the current time step is started; (3.4) When the total coverage area of ​​the detector reaches the set target value or the energy of the detector is exhausted, one coverage is completed; (3.5) The detector deployment coverage plan is trained according to the set number of iteration rounds and reward function, and the optimal detector deployment plan is output.

5. The method according to claim 1, characterized in that In the step 1, modeling the communication channel in the detection process includes the following: (1.1) For the air-to-air channel modeling between the hops, the total channel loss L a2a The calculation is as follows: L a2a =L f +L a ·d Where, the free space path loss L f =20log 10 d+20log 10 f+32.45; d is the distance between the inter-flyers; atmospheric absorption loss f is the communication signal frequency, is the absorption loss of water vapor, The oxygen absorption loss; (1.2) For the air-to-ground channel model between the flyer and the rover, the total channel loss L a2g The calculation is as follows: L a2g =PL+L d ·d Where PL is the path loss, L d is the dust loss, d is the distance between the flyer and the patroller; Dust loss Where λ is the wavelength of the communication signal, N T is the density of sand particles, is the average dust particle radius, θ is the elevation angle of the path propagation, ε' and ε" are the real and imaginary parts of the average dielectric constant of the dust particles, respectively; (1.3) The ground-to-ground channel between patrollers is modeled. The channel loss is the diffraction loss of obstacles. The alternating method is used to calculate the channel loss L between patrollers. g2g as follows: Among them, L D1 is the diffraction loss of the first obstacle, i.e. the blade-shaped obstacle, a is the distance from the transmitter to the first obstacle, b is the distance between the first obstacle and the second obstacle, c is the distance from the second obstacle to the receiver, d = a + b + c is the distance between the transmitter and the receiver, L D2 is the diffraction loss of the second obstacle, i.e. the cylindrical obstacle, L D2 =L D (b,c,H′ c2 )+T, T is the additional loss related to the propagation environment, L D is the diffraction loss of a knife-shaped obstacle under the same conditions, H′ c1 , H′ c2 are the heights of the top of the obstacle relative to the line of sight between the transmitter and the receiver.

Images