A multi-UAV air-sea collaborative target monitoring method integrating A-star algorithm and reinforcement learning

Abstract

The present invention discloses a multi-UAV air-sea collaborative target monitoring method integrating A-star algorithm and reinforcement learning, which relates to the field of path planning technology, solves the technical problem that the efficiency and accuracy of air-sea collaborative monitoring are not high enough, and its technical solution integrates the path planning algorithm of A-star algorithm and reinforcement learning to guide the full coverage path planning of UAVs and the point-to-point path planning of surface ships, so as to improve the efficiency of task execution, and effectively reduce the risk of navigation to ensure the safe and efficient execution of tasks; by using sonar arrays to monitor underwater conditions at the same time, the depth and breadth of monitoring are broadened, and potential anomalies are effectively prevented. The present invention fully combines precise scanning in the air and on the sea surface with deep underwater survey, which can effectively improve the comprehensiveness and accuracy of monitoring of vast sea areas and underwater targets.

Description

Technical Field

[0001] The present application relates to the field of path planning technology, and in particular to a multi-UAV air-sea collaborative target monitoring method that integrates A-star algorithm and reinforcement learning. Background Art

[0002] In the current context of globalization and rapid development of the marine economy, the strategic significance and practical value of air-sea collaborative monitoring are becoming increasingly prominent. The rapid development of Internet of Things technology has provided powerful data collection and processing capabilities for air-sea collaborative monitoring. By deploying a wide range of sensor networks and efficient communication technologies, the Internet of Things can not only achieve large-scale real-time data collection, but also process and analyze data in real time. Through comprehensive information support and processing capabilities, a solid technical foundation has been laid for the realization of efficient air-sea collaborative monitoring. In addition, Internet of Things technology also supports the automated management and maintenance of monitoring equipment to ensure the continuous operation and reliability of the system, which is of great significance to improving the intelligence and automation level of the monitoring system.

[0003] Path planning algorithm is an important technology to achieve efficient monitoring tasks. Among the traditional path planning algorithms, the A-star algorithm is efficient and easy to implement in determining the shortest path. However, in the face of real-time changing environments and complex task requirements, such as the identification of dynamic obstacles and multi-target path planning, its limitations begin to emerge. For the complex marine environment involved in air-sea collaborative monitoring, how to make adjustments based on traditional path planning algorithms and seek methods that are more adaptable to dynamic environments and can optimize complex decision-making processes needs to be solved, so as to ensure that monitoring equipment such as drones can maintain efficient operation in complex environments. Summary of the invention

[0004] The present application provides a multi-UAV air-sea collaborative target monitoring method that integrates the A-star algorithm and reinforcement learning. Its technical purpose is to achieve efficient monitoring and precise target positioning in complex sea environments, improve the ability to respond to maritime emergencies, and promote the intelligence and automation of sea-air collaborative monitoring activities.

[0005] The above technical objectives of this application are achieved through the following technical solutions:

[0006] A multi-UAV air-sea collaborative target monitoring method integrating A-star algorithm and reinforcement learning, comprising:

[0007] When the shore base detects an abnormal situation during the signal monitoring process, it determines area A with the abnormal situation as the center, and performs grid modeling on area A. Based on the grid modeling, it performs risk zoning for the sea and air conditions in area A; with the high-risk position in the risk zoning as the center, it divides area A into multiple uniform sub-areas A. i; Obtain the optimal order of traversing all sub-areas through the A-star algorithm and save the corresponding first A-star path; Perform reinforcement learning training on the first A-star path through the first reinforcement learning method to obtain the reinforced path selection; Then dynamically adjust the reinforced path selection to obtain the final path planning, and the detection drone performs full coverage scanning according to the final path planning;

[0008] After the detection drone detects an abnormal target on the sea surface, it marks its specific location and transmits the marked location back to the shore base. The shore base then synchronizes the marked location to the mission-type vessel, and the mission-type vessel and functional vessel go to the marked location to start work; among them, the mission-type vessel plans the navigation path to the marked location according to the A-star algorithm and the second reinforcement learning method;

[0009] After arriving at the marked location, the mission-type vessel assigns tasks to the functional vessel, and the functional vessel deploys a sonar array at the selected location to monitor underwater abnormalities through the sonar array. At the same time, the functional drone continuously monitors the abnormal area and surrounding conditions.

[0010] When the sonar array, functional drones and ships no longer identify abnormal situations, the mission-type ship decides whether to stop the action or contact the shore to adjust resource input to complete the target detection.

[0011] Furthermore, the method of obtaining the optimal order of traversing all sub-areas through the A-star algorithm and saving the corresponding A-star path includes: calculating the shortest path between the center points of adjacent sub-areas through the A-star algorithm, obtaining a minimum spanning tree based on the shortest path through the minimum cost spanning tree algorithm, determining the optimal order of traversing all sub-areas according to the minimum spanning tree, and saving the corresponding A-star path according to the optimal order of traversal.

[0012] Further, the first A-star path is subjected to reinforcement learning training by a first reinforcement learning method to obtain a reinforced path selection, including:

[0013] The historical Q(s, a) is obtained and updated. If the detection drone has not completed the full coverage scan, the next moving position is selected according to the ε-greedy function. The historical Q(s, a) is obtained, the reward of the next moving position is evaluated according to the reward function, and the Q(s, a) value is updated according to the reward.

[0014] The selection of the next moving position according to the ε-greedy function includes: generating a random number θ1. When θ1 < ε, the detection drone selects the position with the largest Q(s, a) value as the next moving position; when θ1 ≥ ε, generating a random number θ2 and introducing the parameter when When approaching or choosing the first A-star path, When , a random position is selected;

[0015] Q(s, a) represents the value of executing action a in state s, and its value is determined by the reward function, which is expressed as:

[0016]

[0017] Among them, R explore represents the reward function that encourages the detection drone to explore the unscanned area, n represents the number of new grids scanned by the detection drone in the next move, l represents the risk level in the risk zoning established by the shore base at the next location, and α1 represents the incentive factor for exploring the unscanned area; R repeat represents the reward function that penalizes the detection drone for repeatedly covering the grid; n0 represents the number of grids that the detection drone can scan each time it moves the detector, so n0-n represents the number of grids that are repeatedly covered; b1 represents the penalty factor for repeated scanning, b1>1, and b1 increases with the increase of the number of iterations; R obstacle It represents the reward function that penalizes the detection drone when it encounters obstacles or boundaries, and c1 represents the specific penalty value given;

[0018] The reward obtained by the detection drone each time it moves is expressed as: R1 = R explore +R explore +R obstacle .

[0019] Furthermore, the dynamically adjusting the enhanced path selection to obtain the final path planning includes:

[0020] The coverage rate of the detection drone during the full coverage scanning process is calculated, and after every m iterations, η scanned After one update, the coverage is expressed as:

[0021]

[0022] Among them, A scanned Indicates the area of ​​the current scanned area; A indicates the total area of ​​the area;

[0023] Set the threshold δ limit , expressed as: limit =δ limit ·w; where w represents δ limit Attenuation coefficient, w≤1;

[0024] If η scanned The increase is less than the threshold δ limit, then a star A path is generated with the current position as the starting point and the center of the nearest uncovered area as the end point, and the detection drone flies according to the star A path. When the detection drone flies to the end point of the star A path, action a is selected according to the reward function and ε-greedy function of the first reinforcement learning method.

[0025] Furthermore, the mission-type vessel plans a navigation path to the marked position according to the A-star algorithm and the second reinforcement learning method, including:

[0026] The mission-type vessel calculates the shortest path from the current position to the marked position through the A-star algorithm to obtain the third A-star path, and then performs reinforcement learning training on the third A-star path according to the second reinforcement learning method;

[0027] Among them, the reward function of the second reinforcement learning method is expressed as:

[0028]

[0029] Among them, R deviation represents the reward function used to prevent the current path from deviating too much from the path of the third star A, s1 represents the current position of the mission-type ship, d(s1,p astar ) represents the distance from the current position to the third A-star path, α2 and b2 represent adjustment coefficients, and l represents the risk level in the risk zoning established by the shore-based corresponding position at the next position; R risk represents the reward function that encourages the ship to tend to sail along the path with lower risk, c2 represents the risk control coefficient, and c2>1;

[0030] The reward for each functional ship's movement is: R2 = R deviation +R risk ;

[0031] The ε-greedy function according to the second reinforcement learning method includes: the functional ship selects the position with the largest R2 value as the next moving position under the probability of ε; and selects a random position as the next moving position under the probability of 1-ε.

[0032] Furthermore, the comprehensive detection probability of the sonar array is expressed as:

[0033]

[0034] Where N is the number of sensors actually used; i is the i-th sensor, DE i represents the detection efficiency of the i-th sensor; P d It represents the probability of a single sonar detecting a target; F represents the effect of the arrangement on the detection efficiency;

[0035] Pd =S·e -αd (1-E), S represents the acoustic reflection coefficient of the detection target, E represents the environmental noise level, α represents the attenuation coefficient of the sound wave in water, and d represents the distance of the detection target from the sonar sensor;

[0036]

[0037] Wherein, A represents the influence coefficient brought by sensor performance and operating frequency; N represents the number of sensors actually used, N0 represents the number of sensors used theoretically; β represents the weight coefficient of the influence of the number of sensors on the detection efficiency, β>0; γ represents the influence coefficient of the arrangement form, F·γ represents the influence of the arrangement form on the detection efficiency.

[0038] Furthermore, the task-type vessel makes a decision on whether to stop the action or contact the shore to adjust resource input, including: when the marginal benefit is not less than the marginal benefit threshold, contact the shore to adjust resource input; when the marginal benefit is less than the marginal benefit threshold, stop the action;

[0039] Among them, the marginal benefit is expressed as:

[0040]

[0041] R represents the resource consumption used, ΔR represents the increment of time, cost, drone and other resource units; ΔP(S) represents the change in the probability of success after adding ΔR resources.

[0042] The beneficial effects of this application are:

[0043] (1) By conducting raster modeling of abnormal areas on the shore and combining the information from meteorological stations to conduct a comprehensive analysis of the sea and air conditions, risk zoning can be carried out, effectively preventing high-risk accidents during the navigation of drones and ships, making their deployment more targeted and fully improving the efficiency and safety of emergency response.

[0044] (2) The traditional full-coverage path planning algorithm cannot handle the full-coverage scanning of complex marine environments well. The risk zoning of the marine environment is introduced. The A-star algorithm is introduced to improve it based on the reinforcement learning algorithm. The use of dynamically adjusted path planning scheme can better ensure that the UAV can quickly achieve full-coverage scanning in a short time, and it is more conducive to the UAV to deal with complex actual situations during the scanning process.

[0045] (3) Based on the comprehensive application of drone sea surface scanning and sonar underwater scanning, combined with the complementary characteristics of drone wide-area search and sonar depth survey, it can achieve detailed coverage of vast sea areas and in-depth detection of difficult-to-detect underwater targets, which has a significant effect of effectively preventing the omission of search and rescue targets, thereby greatly improving the comprehensiveness and accuracy of maritime target monitoring.

[0046] (4) Based on comprehensive evaluation and dynamic adjustment mechanisms, the comprehensive calculation of resource consumption and marginal benefits in actual actions, combined with consideration of resource limitations and changes in the probability of success, has the effect of improving the cost-effectiveness of actions. By scientifically evaluating the relationship between resource input and actual results, it ensures that the action achieves the best balance between cost and benefit and avoids ineffective waste of resources.

[0047] (5) The present invention realizes highly coordinated operation between drone clusters and ships, and achieves optimal allocation and utilization of resources through precise scheduling and dynamic path planning. This collaborative operation mode not only increases the flexibility and adaptability of task units, but also greatly improves the timeliness and accuracy of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is an overview of the air-sea collaborative monitoring method in the embodiment of the present application;

[0049] Figure 2 This is a flowchart of multi-UAV air-sea collaborative monitoring in an embodiment of the present application;

[0050] Figure 3 This is a flow chart of path planning that integrates the A-star algorithm and reinforcement learning in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0052] like Figure 1 As shown, the drone air-sea collaborative target monitoring method described in this application includes five entities, as follows:

[0053] Shore-based: refers to a command and control center onshore that integrates advanced communication and data processing capabilities. The shore-based center works closely with the global positioning satellite system to collect real-time data and meteorological information in the sea area, and conducts detailed risk assessment and zoning of the target area through in-depth analysis. The command decision-making of the shore-based center is based on a comprehensive grasp of sea and air intelligence, aiming to guide the precise and effective deployment of marine resources to respond to various marine emergencies.

[0054] Sonar array: refers to a system of multiple sonar sensors arranged in a certain geometric pattern, including buoys and submerged buoys, carried by functional vessels and deployed in the target sea area to detect and analyze abnormal conditions in the underwater area. This system improves the signal reception quality and target positioning accuracy by comprehensively analyzing the data collected by multiple sonar sensors.

[0055] Drone swarm: refers to a group of drones deployed according to the type of emergency, including detection drones and functional drones. Detection drones are equipped with high-resolution and thermal imaging cameras, focusing on performing wide-area search, precision scanning and target identification tasks. Functional drones are equipped with special equipment to perform specialized operations according to the specific needs of abnormal situations, while providing air support and assistance for surface operations, enhancing the flexibility and efficiency of mission execution.

[0056] Mission-type vessels: refers to vessels equipped with advanced computing centers and communication equipment. Their main responsibilities include real-time processing of detection data from aerial drones and planning optimal paths for drone flights and functional ship operations. Such vessels are also responsible for clarifying specific task allocations and ensuring that maritime and air resources work together to meet challenges in complex maritime environments.

[0057] Functional vessels: refers to vessels designed specifically for sea surface monitoring and emergency response tasks, equipped with various types of drones and their necessary charging and maintenance equipment to ensure that drones can continue to patrol the sea. In addition, functional vessels also carry specially trained professional technicians for on-site analysis and emergency disposal. According to mission requirements, buoys and submerged buoys are deployed in specific areas to provide key support for maritime operations.

[0058] This application can fully demonstrate its unique advantages and efficient execution capabilities in responding to emergency situations such as fires on offshore oil platforms and rescuing people who fall into the water in the surrounding sea areas. The following are the operating steps and technical applications of this application in this scenario.

[0059] like Figure 2 and Figure 3 As shown, the multi-UAV air-sea collaborative target monitoring method integrating the A-star algorithm and reinforcement learning described in this application includes:

[0060] Step 1: During the signal monitoring process, the shore-based station receives a distress signal from an offshore oil platform fire. The shore-based station locates the area where the signal is sent and performs grid modeling. The weather station collects specific information about the sea area and performs risk zoning for the sea and air conditions. Allocate the configuration and number of drone clusters, the number of ships, and the number of people they carry. According to expert advice, an area A is determined with the abnormal situation as the center, and drones are prepared to be dispatched to conduct a full coverage scan of this area. Based on the established risk zoning, a path is planned for the drone to conduct a full coverage scan.

[0061] Step 1 The specific steps are as follows:

[0062] Step 1.1: Deploy a multi-sensor fire detection system on the offshore oil platform to monitor the ambient temperature, smoke concentration and other key parameters in real time. When a fire sign is detected (such as abnormal temperature rise, rapid increase in smoke concentration), the system automatically generates and sends a distress signal to the shore, which includes the fire location, preliminary estimated fire scale and urgency.

[0063] Step 1.2: The shore base receives the distress signal and, based on the expert's advice, determines a large area A with the offshore oil platform as the center and performs grid modeling. The shore base collects information such as sea currents, swells, wave heights, and air currents, air pressure, and humidity in the corresponding sea area through the meteorological center. The risks faced by drones and ships passing through the area are measured by calculation, and the risks are divided into levels and mapped to the corresponding specific grids to achieve risk zoning of area A.

[0064] Step 1.3: The shore-based base will first dispatch detection drones equipped with high-resolution cameras, thermal imaging cameras and smoke collection probes to conduct a full-coverage scan of area A. Considering the complexity of the ocean conditions and the difficulty of the full-coverage scan task, the shore-based base will first use the path planning algorithm that integrates the A-star algorithm and Q-learning (reinforcement learning) to plan the navigation path of the detection drone, and then dispatch the detection drone to conduct a full-coverage scan according to the planned path.

[0065] Step 1.4: Based on the risk zoning map, divide area A into basically uniform sub-areas A with the high-risk location as the center. i The shortest path between the center points of adjacent regions is calculated using the A-star algorithm. On this basis, the minimum spanning tree is calculated using the Kruskal algorithm to determine the optimal order of region traversal. According to the optimal order of traversal, the corresponding first A-star path is saved as a reference for subsequent Q-learning algorithm execution.

[0066] Step 1.5: The core of the Q-learning algorithm is to let the agent select behavior a according to a certain probability based on the current state s, and then obtain the corresponding reward r. At the same time, the environment updates the environment state to s according to the current state s and the action selected by the agent. i+1 , then based on s i+1Select the next action until the task is completed. Therefore, Q(s, a) is used to represent the value of performing behavior a in state s. The update process of Q(s, a) reflects the relationship between s, a, and r, as well as how the agent finds the optimal solution by learning action selection. In Q-learning, a two-dimensional table (Q table) is established to store the Q(s, a) corresponding to each position.

[0067] Based on the above premise, in order to ensure the comprehensiveness of full coverage scanning and reduce the repetition of the scanning process, the following three reward functions are designed to constrain the actions of the detection drone.

[0068] The first reward function R explore To encourage the detection drone to explore uncovered areas, the calculation is as follows:

[0069] R explore =α1·nl

[0070] Among them, n represents the number of new grids scanned by the detection drone in the next movement, l represents the risk level in the risk zoning established by the shore base corresponding to the next position, and α1 represents the incentive factor for exploring the unscanned area.

[0071] The second reward function R repeat Penalize the repeated coverage of the detection drone, calculated as follows:

[0072] R repeat =-b1(n0-n)-l

[0073] Among them, n0 represents the number of grids that the detection drone can scan each time it moves the detector, so n0-n represents the number of grids that are repeatedly covered; b1 represents the penalty factor for repeated scanning, b1>1, and b1 increases with the increase in the number of iterations.

[0074] The third reward function R obstacle A penalty is imposed on detection drones when they encounter obstacles or boundaries, calculated as follows:

[0075] R obstacle =-c1

[0076] Among them, c1 represents the specific penalty value given, which is adjusted according to the specific environmental conditions.

[0077] In summary, the reward for each movement of the detection drone is calculated as follows:

[0078] R1=R explore +R explore +R obstacle

[0079] Step 1.6: To avoid Q-learning only finding the local optimal solution, an ε-greedy function is established to greedily select action a. Specifically, the parameter ε and the random number θ1 are introduced so that when θ1 < ε, the detection drone greedily selects the position with the largest Q value in the Q table as the next moving position; and when θ1 ≥ ε, it randomly selects a position.

[0080] In order to allow the A-star algorithm to effectively guide Q-learning, the A-star path is integrated into the action selection process of Q-learning through the ε-greedy function, so that the detection drone can choose the A-star path or the direction close to the A-star path with a certain probability when selecting an action. Based on Step 1.4, when the detection drone selects the next moving position, it is assumed that the current position of the detection drone belongs to area A. i , Region A i From the center of the next uncovered area A j The shortest path to star A is p i .

[0081] The original ε-greedy function is modified as follows: Generate a random number θ1: When θ1 < ε, the detection drone selects the position with the largest Q(s, a) value as the next moving position; when θ1 ≥ ε, generate a random number θ2 and introduce the parameter when When approaching or choosing the first A star path p i ,when , select a random position.

[0082] Step 1.7: Introduce variable η scanned Indicates the coverage rate during full coverage scanning, calculated as follows:

[0083]

[0084] Among them, A scanned Indicates the area of ​​the current scanned area; A indicates the total area of ​​the area.

[0085] After every 20 iterations, η scanned Perform a check. Set a threshold δ limit , calculated as follows:

[0086] δ limit δ limit ·w

[0087] Where w represents δ limit The attenuation coefficient, w≤1, controls δ limit It decreases as the number of iterations increases.

[0088] If η scanneThe increase of d is less than the threshold δ limit , a second A-star path is generated with the current position as the starting point and the center of the nearest uncovered area as the end point, and the path planned by Q-learning in Step 1.5 and Step 1.6 is changed to the second A-star path. When the flight reaches the end point of the second A-star path, the path is then planned according to the Q-learning algorithm in Step 1.5 and Step 1.6.

[0089] Step 1.8: After the path planning is completed, the shore base selects detection drones, firefighting drones equipped with fire extinguishing devices and sprinkler devices to form a drone cluster, and allocates firefighting resources such as high-pressure water guns, carbon dioxide fire extinguishers, and professional firefighting personnel to functional ships. The shore base will give priority to dispatching detection drones to carry out full coverage scanning.

[0090] Step 2: The detection drone goes to area A according to the path planned by the shore base to conduct a full coverage scan, identify and mark the location of the people who jumped into the sea due to the fire and are waiting to be rescued, and transmit the specific information back to the shore base. The shore base synchronizes the information to the mission-oriented ship, which plans the path for the ship's navigation based on the path planning algorithm that integrates the A* algorithm and reinforcement learning, and goes to the abnormal area to carry out specific work.

[0091] Step 2 The specific execution steps are as follows:

[0092] Step 2.1: The detection drone detects and identifies people waiting to be rescued on the sea surface according to the full coverage scanning path planned in Step 1. Once the people waiting to be rescued on the sea surface are detected, their specific location is immediately marked and the location information is transmitted back to the shore. During the flight, the detection drone collects environmental information of the sea area through cameras and sensors at the same time to ensure that no key information is missed.

[0093] Step 2.2: The shore-based detection unit synchronizes the location marked by the drone to the mission-type vessel, which dispatches some functional vessels to the marked location for rescue and sends the remaining functional vessels equipped with firefighting equipment, drones, and sonar buoys to the abnormal location.

[0094] Step 2.3: Considering the complexity of the maritime environment, the mission-oriented vessel uses a path planning algorithm that integrates the A-star algorithm and Q-learning to plan the navigation path for each ship.

[0095] The mission-oriented ship first uses the A-star algorithm to calculate the shortest distance p from the current position to the target position astar In the Q-learning planning process, in order to consider the navigation risk and prevent astar The deviation is too large and the path is too long. The following two reward functions are designed to constrain the navigation path of the ship:

[0096] The first reward function R deviation It is used to prevent the current path from deviating too much from the path of star A. The calculation is as follows:

[0097] R deviation =-α2·exp(b2·d(s1,p astar ))

[0098] Among them, s1 represents the current position of the mission-type vessel, d(s1,p astar ) represents the distance from the current position to the shortest path determined by the A-star algorithm; α2 and b2 represent adjustment coefficients, which are used to control the penalty intensity for deviation from the path, thereby adjusting the sensitivity to the shortest path deviation; l represents the risk level in the risk zoning established on the shore corresponding to the next position.

[0099] The second reward R risk Incentivize ships to tend to follow the less risky path, calculated as follows:

[0100] R risk =-c2·l

[0101] Among them, c2 represents the risk control coefficient, and c2>1.

[0102] In summary, the reward for each ship move is calculated as follows:

[0103] R2=R deviation +R risk

[0104] Similarly to Step 1.4, in the reinforcement learning path planning process, according to the ε-greedy function, the ship will choose the position with the largest R2 value as the next moving position under the probability of ε; and choose a random position as the next moving position under the probability of 1-ε.

[0105] Step 2.4: After the initial full coverage scan is completed, the detection drone maintains real-time communication with the mission-type vessel, conducts periodic patrols of the sea area according to the rescue situation of the functional vessel, and continues to track and update the position of the target on the sea surface. Set a threshold φ. When the battery power of the detection drone is less than φ, the drone flies to the functional vessel for charging, and works alternately with the detection drone carried by the functional vessel to achieve continuous monitoring of the oil well platform and surrounding locations.

[0106] Step 3: The mission-type vessel and other functional vessels arrive at the oil well platform and start rescue operations. According to the risk zoning and on-site conditions, the functional vessel deploys a sonar array to the selected location to monitor underwater abnormalities.

[0107] Step 3 The specific execution steps are as follows:

[0108] Step 3.1: The mission-type vessel and other functional vessels arrive at the abnormal area. The mission-type vessel dispatches some functional vessels to remotely put out the fire with high-pressure water guns; the remaining functional vessels carry firefighters to the oil platform to rescue the trapped personnel. At this time, the detection drone patrols the oil platform and surrounding locations, using infrared thermal imagers to identify possible fire sources and potential personnel hotspots, and at the same time determines the placement of the sonar array through the onboard infrared equipment and radar detection equipment.

[0109] Step 3.2: The functional ship deploys a sonar array at the selected location to monitor whether there are people waiting to be rescued underwater. For the detection process of the sonar array, a probability model is established as follows:

[0110] The probability of detecting a target for a single sonar is:

[0111] P d =S·e -αd (1-E)

[0112] Among them, S represents the acoustic reflection coefficient of the detection target, which depends on the size, shape and material of the target; E represents the ambient noise level; α represents the attenuation coefficient of sound waves in water; d represents the distance of the detection target from the sonar sensor.

[0113] When the sonar array detects a target, the arrangement of the sonar array will have a great impact on the detection effect. By introducing the variable F to represent the effect of the arrangement on the detection efficiency, the detection efficiency DE can be calculated:

[0114]

[0115] Among them, A represents the influence coefficient brought by sensor performance, working frequency, etc.; N represents the number of sensors actually used, and N0 represents the number of sensors used theoretically; β represents the weight coefficient of the influence of the number of sensors on the detection efficiency, reflecting the effect of increasing the number of sensors on the improvement of detection efficiency. β>0 indicates that the increase in the number of sensors will improve the detection efficiency, but the increase decreases with the increase of N; γ represents the influence coefficient of the arrangement form, and F·γ represents the influence of the arrangement form on the detection efficiency. By adjusting γ, the difference in the influence of different arrangements of the sonar array on the detection efficiency can be reflected.

[0116] Taking all the above factors into consideration, the comprehensive detection probability of the entire sonar array is constructed:

[0117]

[0118] Where N is the number of sensors actually used; i is the i-th sensor, DE i represents the detection efficiency of the i-th sensor; Pd It represents the probability of a single sonar detecting a target; F represents the influence of the arrangement on the detection efficiency.

[0119] Set a threshold ε, when P Total ≥ε, indicating that the sonar array is very likely to detect the underwater rescue personnel, and quickly feed back the detected situation information and location coordinates to the mission-type ship, which will dispatch the functional ship with divers to carry out underwater rescue. Total <ε, indicating that there is no abnormal situation underwater. The mission-type vessel contacts the base station to consider whether to adjust the plan.

[0120] Step 3.3: While underwater sonar detection is in progress, the firefighting drone provides a high-altitude, comprehensive view for the functional ship operations, identifies the location of trapped personnel through thermal imaging cameras, and assists the functional ship in successful rescue. The detection drone continuously monitors the fire situation on the oil well platform to identify whether there is an increase in fire points. Once an abnormal increase is found, the mission-type ship is immediately notified to handle it.

[0121] Step 4: When the drone, sonar array, and ship no longer detect abnormal conditions, the mission-type ship will calculate based on the following two rules to decide whether to stop the action or contact the shore to adjust resource investment.

[0122] The specific implementation steps of Step 4 are as follows:

[0123] Step 4.1: Consider resource constraints and introduce the variable R to represent the resource consumption (including time, cost, and equipment usage). limit Indicates resource usage limit. When R>R limit , that is, if the resources required to continue additional detection and rescue missions exceed the limit, the marginal benefits of the action should be analyzed and then it should be determined whether the strategy should be adjusted or the action should be terminated.

[0124] Step 4.2: Analyze the marginal benefit impact of the action as the plan is adjusted. Based on the assumptions in Step 4.1, ΔR is used to represent the incremental increase in time, cost, drones, and other resource units; ΔP(S) represents the change in the probability of success after adding ΔR resources. The marginal benefit (ME) can be calculated as follows:

[0125]

[0126] The above formula can be used to measure the contribution of resource input to the increase in success probability, that is, how much the success probability can be increased by each additional unit of resources. During the search process, the marginal benefit threshold ME should be set according to cost-effectiveness and task objectives. threshold, the mission-type ship estimates the marginal benefit changes brought about by changing resource inputs based on the current UAV and underwater sonar search conditions. If the calculated marginal benefit is less than ME threshold , indicating that the rescue mission has been basically completed and no more resources are needed, so the operation should be stopped.

[0127] The above are exemplary embodiments of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-UAV air-sea collaborative target monitoring method integrating A-star algorithm and reinforcement learning, characterized in that: include: When the shore-based system detects an abnormal situation during the signal monitoring process, it determines area A with the abnormal situation as the center, performs grid modeling on area A, and performs risk zoning for the sea and air conditions in area A based on the grid modeling; Taking the high-risk location in the risk zoning as the center, divide area A into multiple uniform sub-areas A i ; Obtain the optimal order of traversing all sub-areas through the A-star algorithm and save the corresponding first A-star path; Perform reinforcement learning training on the first A-star path through the first reinforcement learning method to obtain the reinforced path selection; Then dynamically adjust the reinforced path selection to obtain the final path planning, and the detection drone performs full coverage scanning according to the final path planning; After the detection drone detects an abnormal target on the sea surface, it marks its specific location and transmits the marked location back to the shore base. The shore base then synchronizes the marked location to the mission-type vessel, and the mission-type vessel and functional vessel go to the marked location to start work; among them, the mission-type vessel plans the navigation path to the marked location according to the A-star algorithm and the second reinforcement learning method; After arriving at the marked location, the mission-type vessel assigns tasks to the functional vessel, and the functional vessel deploys a sonar array at the selected location to monitor underwater abnormalities through the sonar array. At the same time, the functional drone continuously monitors the abnormal area and surrounding conditions. When the sonar array, functional drones and ships no longer identify abnormal situations, the mission-type ship decides whether to stop the action or contact the shore to adjust resource input to complete the target detection; Among them, the first A-star path is subjected to reinforcement learning training by the first reinforcement learning method to obtain the reinforced path selection, including: The historical Q(s, a) is obtained and updated. If the detection drone has not completed the full coverage scan, the next moving position is selected according to the ε-greedy function. The historical Q(s, a) is obtained, the reward of the next moving position is evaluated according to the reward function, and the Q(s, a) value is updated according to the reward. The selection of the next moving position according to the ε-greedy function includes: generating a random number θ1 and introducing the parameter ε. When θ1 < ε, the detection drone selects the position with the largest Q(s, a) value as the next moving position; when θ1 ≥ ε, generating a random number θ2 and introducing the parameter when When approaching or choosing the first A-star path, When , a random position is selected; Q(s, a) represents the value of executing action a in state s, and its value is determined by the reward function, which is expressed as: Among them, R explore represents the reward function that encourages the detection drone to explore the unscanned area, n represents the number of new grids scanned by the detection drone in the next move, l represents the risk level in the risk zoning established by the shore base at the next location, and α1 represents the incentive factor for exploring the unscanned area; R repeat represents the reward function that penalizes the detection drone for repeatedly covering the grid; n0 represents the number of grids that the detection drone can scan each time it moves the detector, so n0-n represents the number of grids that are repeatedly covered; b1 represents the penalty factor for repeated scanning, b1>1, and b1 increases with the increase of the number of iterations; R obstacle It represents the reward function that penalizes the detection drone when it encounters obstacles or boundaries, and c1 represents the specific penalty value given; The reward obtained by the detection drone each time it moves is expressed as: R1 = R explore +R explore +R obstacle .

2. The multi-UAV air-sea collaborative target monitoring method according to claim 1, characterized in that: The method of obtaining the optimal order of traversing all sub-areas through the A-star algorithm and saving the corresponding A-star path includes: calculating the shortest path between the center points of adjacent sub-areas through the A-star algorithm, obtaining a minimum spanning tree based on the shortest path through the minimum cost spanning tree algorithm, determining the optimal order of traversing all sub-areas according to the minimum spanning tree, and saving the corresponding A-star path according to the optimal order of traversal.

3. The multi-UAV air-sea collaborative target monitoring method according to claim 2, characterized in that: The dynamically adjusting the enhanced path selection to obtain the final path planning includes: The coverage rate of the detection drone during the full coverage scanning process is calculated, and after every m iterations, η scanned After one update, the coverage is expressed as: Among them, A scanned Indicates the area currently scanned; S A Indicates the total area of ​​the region; Set the threshold δ limit , expressed as: limit =δ limit ·w; where w represents δ limit Attenuation coefficient, w≤1; If η scanned The increase is less than the threshold δ limit , then a second A-star path is generated with the current position as the starting point and the center of the nearest uncovered area as the end point according to the A-star algorithm, and the detection drone flies according to the second A-star path. When the detection drone flies to the end point of the second A-star path, action a is selected according to the reward function and ε-greedy function of the first reinforcement learning method.

4. The multi-UAV air-sea collaborative target monitoring method according to claim 3, characterized in that: The mission-type vessel plans a navigation path to the marked position according to the A-star algorithm and the second reinforcement learning method, including: The mission-type vessel calculates the shortest path from the current position to the marked position through the A-star algorithm to obtain the third A-star path, and then performs reinforcement learning training on the third A-star path according to the second reinforcement learning method; Among them, the reward function of the second reinforcement learning method is expressed as: Among them, R deviation represents the reward function used to prevent the current path from deviating too much from the path of the third star A, s1 represents the current position of the mission-type ship, d(s1,p astar ) represents the distance from the current position to the third A-star path, α2 and b2 represent adjustment coefficients, and l represents the risk level in the risk zoning established by the shore-based corresponding location at the next position; R risk represents the reward function that encourages the ship to sail along the path with low risk, c2 represents the risk control coefficient, and c2>1; The reward for each functional ship's movement is: R2 = R deviation +R risk ; The ε-greedy function according to the second reinforcement learning method includes: the functional ship selects the position with the largest R2 value as the next moving position under the probability of ε; and selects a random position as the next moving position under the probability of 1-ε.

5. The multi-UAV air-sea collaborative target monitoring method according to claim 1, characterized in that: The comprehensive detection probability of the sonar array is expressed as: Where N is the number of sensors actually used; i is the i-th sensor, DE i represents the detection efficiency of the i-th sensor; P d It represents the probability of a single sonar detecting a target; F represents the effect of the arrangement on the detection efficiency; P d =S·e -αd (1-E), S represents the acoustic reflection coefficient of the detection target, E represents the environmental noise level, α represents the attenuation coefficient of the sound wave in water, and d represents the distance of the detection target from the sonar sensor; Wherein, A′ represents the influence coefficient brought by sensor performance and operating frequency; N represents the number of sensors actually used, N0 represents the number of sensors used theoretically; β represents the weight coefficient of the influence of the number of sensors on the detection efficiency, β>0; γ represents the influence coefficient of the arrangement form, F·γ represents the influence of the arrangement form on the detection efficiency.

6. The multi-UAV air-sea collaborative target monitoring method according to claim 5, characterized in that: The task-type vessel makes a decision on whether to stop the action or contact the shore to adjust the resource input, including: when the marginal benefit is not less than the marginal benefit threshold, contact the shore to adjust the resource input; when the marginal benefit is less than the marginal benefit threshold, stop the action; Among them, the marginal benefit is expressed as: R represents the resource consumption used, ΔR represents the increment of time, cost, drone and other resource units; ΔP(S) represents the change in the probability of success after adding ΔR resources.

Images