A multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning

Abstract

The present invention discloses a multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning, which belongs to the field of marine environment collaborative search technology. Multiple AUVs are deployed in a mission area, and the mission area is discretized into multiple grids; the AUV plans a search path according to the current position and the probability of target existence, and performs the search task, and updates the joint probability graph model after each movement of the AUV; when the AUV initiates an optical communication request, the target AUV receiving the optical communication data will fuse the information from the AUV, update its own joint probability graph model, and the target AUV re-plans the search path based on the updated joint probability graph model; repeat the above steps until the target search in the mission area is completed. The multi-AUV acoustic-optical multi-mode collaborative search based on multi-agent reinforcement learning of the present invention can effectively improve the collaborative search efficiency of multiple AUVs under limited communication, and can be applied to AUV cluster collaborative search in complex marine scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of marine environment collaborative search, and specifically relates to a multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning. Background Art

[0002] Underwater acoustic communication is widely used in ocean exploration and multi-robot collaboration, but its inherent low data rate and high latency significantly limit the ability to transmit real-time information. This limitation not only affects the collaborative efficiency of autonomous underwater vehicles (AUVs), but also makes it difficult to respond quickly and adapt to changing environments in emergency situations. Especially in complex underwater environments, acoustic signals are susceptible to noise and propagation delays, resulting in unreliable information transmission, which in turn affects the safety and effectiveness of mission execution.

[0003] In contrast, underwater wireless optical communication (UWOC) has become an ideal choice for improving AUV collaborative work with its advantages of high data transmission rate and low latency. UWOC uses optical signals to transmit data, which can achieve faster and more efficient information exchange, which is particularly important for real-time data transmission. However, UWOC still faces challenges in practical applications such as short communication distance and changes in underwater environment (such as water quality, temperature, impurities, etc.). In addition, the directionality of optical communication requires that AUVs must maintain precise alignment, which places higher demands on their flexibility and intelligent capabilities.

[0004] To address these challenges, the advantages of acoustic and optical transmission can be combined. Multi-mode communication can not only improve the reliability and efficiency of information transmission, but also promote information sharing between AUVs, thereby more effectively utilizing the data provided by probability maps. This improves the collaborative ability of AUVs in complex underwater tasks, enabling them to better evaluate and select action paths in dynamic environments. Deep reinforcement learning (DRL) is an effective technology for dealing with complex nonlinear problems by interacting with the environment. Combining probability maps with DRL is an effective way to solve the problem of multi-AUV acoustic and optical multi-mode collaborative search. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention proposes a multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning, which has a reasonable design, solves the shortcomings of the prior art, and has good effects.

[0006] A multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning includes the following steps:

[0007] Step 1: Deploy multiple AUVs in the mission area to form a collaborative search system and initialize it, discretize the mission area into multiple grids, and initialize the joint probability graph model;

[0008] Step 2: The AUV plans the search path according to the current position and the target existence probability, and performs the search task. After each movement of the AUV, the joint probability graph model is corrected using pheromones, thereby updating the joint probability graph model.

[0009] Step 3: During the search process, when the AUV needs to share information with other AUVs, it will initiate an optical communication request to find the target AUV as the optical communication object, and gradually approach the target AUV for optical communication data transmission;

[0010] Step 4: The target AUV receiving the optical communication data will fuse the information from the AUV in step 3 and update its own joint probability graph model. The target AUV will re-plan the search path based on the updated joint probability graph model. If the target is not found, steps 2 to 4 will be repeated until the target in the mission area is searched.

[0011] Furthermore, the specific process of step 1 is as follows:

[0012] Set the task area to a The rectangular area is used as the initial working area for multi-AUV collaborative search at a depth of Deployment in the target area AUVs are equipped with acoustic communication sensors, optical communication sensors, and sonar equipment. AUVs use sonar as a sensor to capture targets. During the search process, AUVs transmit information through acoustic and optical communication sensors. AUVs form a collaborative search system to perform the search task; the task area is discretized into several grids.

[0013] Furthermore, in step 2, the AUV uses a joint probability graph combined with a multi-agent deep deterministic policy gradient reinforcement learning algorithm to make action decisions during the search process. The specific decision-making process is:

[0014] The AUV collaborative search control system uses the sonar measurements and the information received by acoustic and optical sensors as observation values. The state of the AUV is defined as ,in express When AUV receives messages from other agents, Represents the joint probability graph information of AUV maintenance, The state function in the discrete time domain is expressed as:

[0015] ; (1)

[0016] in, and For AUV The horizontal and vertical coordinates of time, , , is process noise with zero mean and covariance Gaussian distribution of and For AUV Impact velocity and yaw rate at the moment; For AUV The heading angle at the moment; is the sampling time;

[0017] For each AUV, during the search process, according to the current strategy and exploration noise Select an action , at this time the strategy is a search strategy under communication constraints, where Indicates indivual The observed value of Representing an Agent The action set is , the AUV search system selects from the action set according to the current strategy Select an action in One of all possible changes of the AUV under the communication constraints at that moment.

[0018] Furthermore, in step 2, the AUV obtains the target existence probability of the grid corresponding to the current position through sonar, and the target existence probability is represented by the pheromone corrected joint probability graph model, and the expression is:

[0019] ; (2)

[0020] in, represents the joint probability, express Due to the pheromone detection results and feedback, the time interval The amount of change, The value indicates exist The probability of the target existing at time instant;

[0021] The update formula is:

[0022] ; (3)

[0023] in, and Respectively indicate the presence or absence of a target in the grid; and They are respectively expressed as detection probability and false alarm probability, express exist Time location The probability of the target existing.

[0024] Furthermore, in step 2, after the AUV collaborative search control system performs one action, the reward function is set as follows:

[0025] ; (4)

[0026] in, represents the reward for moving to a high probability grid cell, represents the reward for finding the target, indicates the penalty for collision, represents the reward for encouraging the AUV to quickly approach the optical communication target after selecting the optical communication target, and Represents the penalty for AUVs being too close to each other after the optical communication ends. By setting the reward, the AUV can quickly approach the optical communication target under the search constraints after sending the optical request, and quickly move away from the communication target after the communication ends, so as to ensure the efficiency of the search.

[0027] Furthermore, in step 4, the target AUV receiving the optical communication data will fuse the information from the AUV in step 3 and update its own joint probability graph model. The fused probability is calculated by the following formula:

[0028] ; (5)

[0029] in, is the joint probability after fusion, Indicates The weight of the AUVs participating in the fusion, Indicates the number of AUVs involved in the fusion;

[0030] The AUV swarm uses a multi-agent deep deterministic policy gradient reinforcement learning algorithm to make action decisions;

[0031] In the AUV cluster, each AUV maintains its own Actor network and Critic network, and its network parameters are and , using the CTDE paradigm based on the Actor-Critic method for centralized training and distributed execution. During training, the Critic part of each AUV can obtain the strategy information of other AUVs;

[0032] No. Critic network parameters for each AUV By minimizing the loss function To update, the specific formula is as follows:

[0033] ; (6)

[0034] in, for The expected value of the sample in represents the experience replay pool for storing data, Indicates the target value at the current moment;

[0035] The goal of the Actor Network is to maximize Discounted expected return , No. Actor network parameters for each AUV The update is performed by performing the gradient descent method, the specific formula is:

[0036] ; (7)

[0037] in, Express Seeking guidance, Represents the optimization target, experience replay pool By tuple express, Indicates expected return, is the status information of the AUV, It is about performing the corresponding operation The next state information, Is the output of the Critic network, indicating the current state and Action The function value obtained;

[0038] Soft update target critic network parameters and target actor network parameters , the specific formula is as follows:

[0039] ; (8)

[0040] in, represents the target Critic network parameters, Represents the target Actor network parameters, Represents the soft update coefficient.

[0041] Furthermore, in step 4, since the AUV is also equipped with an inertial navigation system, the speed of the AUV is controlled according to the inertial navigation system so that the AUV moves. The moving position of the AUV at the next moment is expressed as:

[0042] ; (9)

[0043] ; (10)

[0044] ; (11)

[0045] in, represents the cosine in trigonometric function, , Respectively The horizontal and vertical coordinates of time, , and Respectively The surge velocity, sway velocity and yaw rate at the moment;

[0046] After selecting a specific action, the AUV adjusts the corresponding speed and updates the status information. Through the updated joint probability graph information and the continuous updating of network parameters, the AUV can gradually learn a better search strategy, so as to find the target more efficiently in the subsequent search process.

[0047] Beneficial technical effects brought by the present invention:

[0048] Aiming at the collaborative search scenario in the marine environment, the present invention combines the complementary advantages of acoustic and optical communications and proposes a multi-AUV acoustic-optical multi-mode search method based on deep reinforcement learning. The pheromone-corrected probability map is used to achieve rapid search for marine targets. Specifically, the pheromone-corrected probability map aims to improve the search efficiency of AUVs in complex marine environments, and the acoustic-optical multi-mode communication method is used to enhance the information sharing between AUVs and the utilization of probability maps under communication-restricted conditions.

[0049] The present invention simultaneously considers key factors such as optical alignment, mobile optical communication, and communication delay. On this basis, a multi-AUV acoustic-optical multi-mode collaborative search algorithm based on multi-agent reinforcement learning is proposed, which is the first time to realize the combination of acoustic-optical multi-mode in a search scenario. The algorithm optimizes the efficiency of information sharing and target search between AUVs by dynamically adjusting the search strategy of AUVs. When applied to complex marine environments, it can significantly improve the search success rate and response speed, especially when communication is limited, showing stronger robustness and adaptability. This method provides new ideas for future ocean exploration and monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning of the present invention.

[0051] Figure 2This is a schematic diagram of updating the joint probability graph model of each AUV after the search is completed in the simulation experiment of the present invention.

[0052] Among them, (a) is AUV 1 The joint probability graph model of (b) AUV 2 The joint probability graph model of (c) AUV 3 The joint probability graphical model of .

[0053] Figure 3 It is a schematic diagram of the change of search success rate during the training process in the simulation experiment of the present invention.

[0054] Figure 4 This is a comparison chart of the average search success rates of the acoustic-optical multi-mode multi-agent deep reinforcement learning algorithm, the acoustic communication multi-agent deep reinforcement learning algorithm, the autonomous collaborative search learning algorithm and the genetic algorithm under different numbers of AUVs in the simulation experiment of the present invention.

[0055] Figure 5 This is a comparison chart of the average search time of the acoustic-optical multi-mode multi-agent deep reinforcement learning algorithm, the acoustic communication multi-agent deep reinforcement learning algorithm, the autonomous collaborative search learning algorithm and the genetic algorithm under different numbers of AUVs in the simulation experiment of the present invention.

[0056] Figure 6 This is a comparison chart of the effective search area per unit time of the acoustic-optical multi-mode multi-agent deep reinforcement learning algorithm, the acoustic communication multi-agent deep reinforcement learning algorithm, the autonomous collaborative search learning algorithm and the genetic algorithm under different numbers of AUVs in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0057] The specific implementation of the present invention is further described below in conjunction with specific embodiments:

[0058] In the present invention, each AUV is equipped with a sonar sensor, an acoustic communication sensor and an optical communication sensor, and the complementary advantages of acoustic and optical communications are utilized to improve information sharing between AUVs, and a probability graph is used to realize a rapid search for marine targets in communication-restricted scenarios. First, the AUV search problem under communication constraints is analyzed, aiming to maximize the search success rate, minimize the search time, and maximize the effective search area per unit time. Secondly, the problem is modeled as a distributed partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP), and a multi-AUV acoustic-optical multimodal search algorithm based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is proposed. Information sharing between AUVs is enhanced through acoustic-optical multimodal communication, which promotes the accurate update of probability graphs. In addition, by incorporating pheromones into the correction probability graph, AUVs can effectively adjust their search paths, thereby improving the efficiency of collaborative search.

[0059] The present invention uses a method based on multi-agent deep reinforcement learning, which optimizes the AUV search strategy by integrating acoustic and optical communication technologies and taking advantage of the multi-agent system. At the same time, the AUV's own position, the state information of adjacent AUVs and the probability of target existence are considered to improve the AUV search efficiency under communication constraints.

[0060] A multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning includes the following steps:

[0061] Step 1: Deploy multiple AUVs in the mission area to form a collaborative search system and initialize it, discretize the mission area into multiple grids, and initialize the joint probability graph model;

[0062] like Figure 1 As shown, set the task area to a The rectangular area is used as the initial working area for multi-AUV collaborative search at a depth of Deployment in the target area AUVs are equipped with acoustic communication sensors, optical communication sensors, and sonar equipment. AUVs use sonar as a sensor to capture targets. During the search process, AUVs transmit information through acoustic and optical communication sensors. AUVs form a collaborative search system to perform the search task; the task area is discretized into several grids.

[0063] Step 2: The AUV plans the search path according to the current position and the target existence probability, and performs the search task. After each movement of the AUV, the joint probability graph model is corrected using pheromones, thereby updating the joint probability graph model.

[0064] During the search process, the AUV uses a joint probability graph combined with a multi-agent deep deterministic policy gradient reinforcement learning algorithm to make action decisions. The specific decision-making process is as follows:

[0065] The AUV collaborative search control system uses the sonar measurements and the information received by acoustic and optical sensors as observation values. The state of the AUV is defined as ,in express When AUV receives messages from other agents, Represents the joint probability graph information of AUV maintenance, The state function in the discrete time domain is expressed as:

[0066] ; (1)

[0067] in, and For AUV The horizontal and vertical coordinates of time, , , is process noise with zero mean and covariance Gaussian distribution of and For AUV Impact velocity and yaw rate at the moment; For AUV The heading angle at the moment; is the sampling time;

[0068] For each AUV, during the search process, according to the current strategy and exploration noise Select an action , at this time the strategy is a search strategy under communication constraints, where Indicates indivual The observed value of Representing an Agent The action set is , the AUV search system selects from the action set according to the current strategy Select an action in One of all possible changes of the AUV under the communication constraints at that moment.

[0069] AUV uses the pheromone-corrected joint probability graph model to record the probability of target existence in each area cell, specifically:

[0070] The AUV obtains the target existence probability of the grid corresponding to the current position through sonar. The target existence probability is expressed by the pheromone-corrected joint probability graph model, and the expression is:

[0071] ; (2)

[0072] in, represents the joint probability, express Due to the pheromone detection results and feedback, the time interval The amount of change, The value indicates exist The probability of the target existing at time instant;

[0073] The update formula is:

[0074] ; (3)

[0075] in, and Respectively indicate the presence or absence of a target in the grid; and They are respectively expressed as detection probability and false alarm probability, express exist Time location The probability of the target existing.

[0076] After the AUV collaborative search control system performs one step of action, the reward function is set as follows:

[0077] ; (4)

[0078] in, represents the reward for moving to a high probability grid cell, represents the reward for finding the target, indicates the penalty for collision, represents the reward for encouraging the AUV to quickly approach the optical communication target after selecting the optical communication target, and Represents the penalty for AUVs being too close to each other after the optical communication ends. By setting the reward, the AUV can quickly approach the optical communication target under the search constraints after sending the optical request, and quickly move away from the communication target after the communication ends, so as to ensure the efficiency of the search.

[0079] Step 3: In order to enhance information sharing between AUVs in a communication-restricted environment, the acoustic-optical multimodal technology is used. During the search process, AUVs continuously exchange information through acoustic communication, and enhance information sharing between AUVs through optical communication. The activation of optical communication is subject to the constraints of the search task. Only when the search constraints are met and the distance between each other is appropriate, will AUVs actively approach and establish optical communication links, initiate optical communication requests, search for the target AUV as the optical communication object, and gradually approach the target AUV for optical communication data transmission.

[0080] Step 4: The target AUV receiving the optical communication data will fuse the information from the AUV in step 3 and update its own joint probability graph model. The target AUV will re-plan the search path based on the updated joint probability graph model. If the target is not found, steps 2 to 4 will be repeated until the target in the mission area is searched.

[0081] In a collaborative search system, when multiple AUVs share information through optical communication, they each have different estimates of the probability of the target existing in the mission area. In order to integrate these estimates and form a more comprehensive probability map, a weighted average method is used for fusion, and the fused probability is calculated by the following formula:

[0082] ; (5)

[0083] in, is the joint probability after fusion, Indicates The weight of the AUVs participating in the fusion, Indicates the number of AUVs involved in the fusion;

[0084] The AUV cluster uses a collaborative search algorithm based on a multi-agent deep deterministic policy gradient method to make action decisions, specifically:

[0085] In the AUV cluster, each Maintain your own Actor network and Critic network, whose network parameters are and , using the CTDE paradigm based on the Actor-Critic method for centralized training and distributed execution. During training, the Critic part of each AUV can obtain the strategy information of other AUVs;

[0086] No. Critic network parameters for each AUV By minimizing the loss function To update, the specific formula is as follows:

[0087] ; (6)

[0088] in, for The expected value of the sample in represents the experience replay pool for storing data, Indicates the target value at the current moment, , is the discount factor, Represents immediate reward;

[0089] The goal of the Actor Network is to maximize Discounted expected return , No. Actor network parameters for each AUV The update is performed by performing the gradient descent method, the specific formula is:

[0090] ; (7)

[0091] in, Express Seeking guidance, Represents the optimization target, experience replay pool By tuple express, Indicates expected return, is the status information of the AUV, It is about performing the corresponding operation The next state information, Is the output of the Critic network, indicating the current state and Action The function value obtained;

[0092] Soft update target critic network parameters and target actor network parameters , the specific formula is as follows:

[0093] ; (8)

[0094] in, represents the target Critic network parameters, Represents the target Actor network parameters, Represents the soft update coefficient.

[0095] Since the AUV is also equipped with an inertial navigation system, the speed of the AUV is controlled by the inertial navigation system so that the AUV moves. The moving position of the AUV at the next moment is expressed as:

[0096] ; (9)

[0097] ; (10)

[0098] ; (11)

[0099] in, represents the cosine in trigonometric function, , Respectively The horizontal and vertical coordinates of time, , and Respectively The surge velocity, sway velocity and yaw rate at the moment;

[0100] After selecting a specific action, the AUV adjusts the corresponding speed and updates the status information. Through the updated joint probability graph information and the continuous updating of network parameters, the AUV can gradually learn a better search strategy, so as to find the target more efficiently in the subsequent search process.

[0101] The specific implementation steps of the collaborative search algorithm based on the multi-agent deep deterministic policy gradient method are as follows:

[0102] (4.1) Initialization parameters, such as Actor network parameters , Critic network parameters and its target network parameters, experience replay pool , maximum round , the maximum time step wait;

[0103] (4.2) Perform an external round loop;

[0104] (4.2.1) Randomly assign the position of the AUV;

[0105] (4.2.2) Initialize a random process , for motion exploration;

[0106] (4.2.3) Obtaining initial observations for all agents ;

[0107] (4.2.4) Perform internal time step loop;

[0108] (4.2.4.1) For each , choose an action using the current policy ;

[0109] (4.2.4.2) Execution of actions , get new observations and rewards ;

[0110] (4.2.4.3) Store to experience replay pool middle;

[0111] (4.2.4.4) ;

[0112] (4.2.4.5) Perform an internal loop;

[0113] (4.2.4.5.1) From the Experience Replay Pool Randomly draw small batches of samples from ;

[0114] (4.2.4.5.2) By minimizing the loss function To update the critic network parameters , the specific calculation formula is (5),

[0115] (4.2.4.5.3) Perform gradient descent to update Actor network parameters , the specific calculation formula is (6);

[0116] (4.2.4.6) If each agent has completed the update, the inner loop ends;

[0117] (4.2.4.7) Update each The target Critic network parameters and target Actor network parameters are calculated as follows: (7);

[0118] (4.2.5) If the search time times out, end the internal time step loop;

[0119] (4.3) If the number of steps in the external loop is reached, end the external round loop.

[0120] The implementation effect of the present invention can be further illustrated by simulation experiments:

[0121] (1) Simulation experiment conditions:

[0122] The simulation experiments were conducted using Python 3.7 and MATLAB 2022a. Four fully connected neural networks were deployed using Pytorch, each with two hidden layers. They contained 64 units and a learning rate of , the batch size is In the simulation, AUVs are randomly distributed in the target area of ​​2km×2km. For each AUV, the values ​​of the relevant parameters are set as follows:

[0123] AUV search range Set to 100m; maximum optical communication distance Set to 200m; Ocean temperature set to 1℃; Salinity coefficient set to 32; Safety distance between adjacent AUVs set to 5m; Discount factor Set to 0.95; learning rate Set to 0.01; the update rate of the target network Set to 0.01; Experience replay pool The length is set to ; Batch size Set to 1024; number of rounds Set to 20000.

[0124] (2) Simulation experiment results:

[0125] First, verify the effectiveness of the AUV in searching for targets in the mission area. Figure 2 a, b and c represent , and The probability map generated after completing the search task, where b represents Probability map of successfully finding a target. Initially, each cell has a prior probability of 0.5. The color intensity of each grid represents the probability of the target being present. During the search process, the AUVs fuse their probability maps with each other. Grids with a higher probability of the target being present attract the AUV to search further, increasing the probability of detecting the target. Figure 4 This is a diagram of the training process of the multi-agent deep deterministic policy gradient reinforcement learning algorithm in the simulation experiment of the present invention. It can be seen that the training results are convergent.

[0126] Figure 3 The comparison of the average search success rate of the proposed search algorithm with training iterations is shown. The results show that the search success rate increases rapidly with the increase in the number of training iterations, and the success rate also increases with the increase in the number of AUVs, which proves its robustness and effectiveness.

[0127] In order to verify the performance of the acoustic-optical multi-modal multi-agent deep reinforcement learning algorithm, Figure 4It shows its comparison with three other search algorithms under different numbers of AUVs, namely, acoustic communication multi-agent deep reinforcement learning algorithm, autonomous collaborative search learning algorithm and genetic algorithm. Compared with the traditional genetic algorithm, the proposed multi-agent deep reinforcement learning algorithm based on acoustic and optical multi-mode shows better performance, especially in high-dimensional complex environments. Due to communication limitations and insufficient global search capabilities, traditional genetic algorithms are difficult to adapt to complex ocean conditions. The multi-agent deep reinforcement learning algorithm improves information sharing between AUVs through acoustic and optical multi-mode communication, enabling AUVs to conduct collaborative search more effectively. In addition, the ablation experiment of traditional underwater acoustic communication further proves the necessity of optical communication in the multi-agent deep reinforcement learning algorithm. The autonomous collaborative search learning algorithm relies on information fusion and timestamp mechanism to achieve a certain degree of information optimization, but it does not fully consider the advantages of multi-mode communication, so its search success rate is still not as good as the multi-agent deep reinforcement learning algorithm.

[0128] In order to further verify the efficiency of the multi-agent deep reinforcement learning algorithm in different search schemes, the average search time of each algorithm under different numbers of AUVs was compared. The results are shown in Figure 2. Figure 5 The multi-agent deep reinforcement learning algorithm significantly shortens the average search time, mainly due to its acoustic and optical multimodal communication mechanism, which enables the AUV to obtain more comprehensive environmental information. Through precise pheromone updates and dynamic construction of the global probability map, unnecessary repeated searches are reduced, thus achieving more efficient searches in complex marine environments.

[0129] In addition, the average search efficiency of different algorithms is evaluated based on the effective search area per unit time. Figure 6 This indicator reflects the overall efficiency of each algorithm in performing search tasks. The multi-agent deep reinforcement learning algorithm shows significant advantages here, especially in complex marine environments. Due to its acoustic and optical multimodal communication strategy, it not only improves the accuracy of environmental information collection, but also effectively reduces redundant search behaviors between AUVs. Even in sparse targets or complex environments, the multi-agent deep reinforcement learning algorithm can still maintain high search efficiency, proving its robustness.

[0130] The above numerical results show that the scheme of the present invention has good performance, can effectively enhance the information sharing of AUVs in search tasks, improve the efficiency of collaborative search, and realize rapid search of targets in unknown environments.

[0131] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Deploy multiple AUVs in the mission area to form a collaborative search system and initialize it, discretize the mission area into multiple grids, and initialize the joint probability graph model; Step 2: The AUV plans the search path according to the current position and the target existence probability, and performs the search task. After each movement of the AUV, the joint probability graph model is corrected using pheromones, thereby updating the joint probability graph model. Step 3: During the search process, when the AUV needs to share information with other AUVs, it will initiate an optical communication request to find the target AUV as the optical communication object, and gradually approach the target AUV for optical communication data transmission; Step 4: The target AUV receiving the optical communication data will integrate the information from the AUV in step 3, update its own joint probability graph model, and re-plan the search path based on the updated joint probability graph model; if the target is not found, repeat steps 2 to 4 until the target in the mission area is searched; In step 2, the AUV uses a joint probability graph combined with a multi-agent deep deterministic policy gradient reinforcement learning algorithm to make action decisions during the search process. The specific decision-making process is: The AUV collaborative search control system uses the sonar measurements and the information received by acoustic and optical sensors as observation values. The state of the AUV is defined as ,in express When AUV receives messages from other agents, Represents the joint probability graph information of AUV maintenance, The state function in the discrete time domain is expressed as: ；(1) in, and For AUV The horizontal and vertical coordinates of time, , , is process noise with zero mean and covariance Gaussian distribution of and For AUV Impact velocity and yaw rate at the moment; For AUV The heading angle at the moment; is the sampling time; For each AUV, during the search process, according to the current strategy and exploration noise Select an action , at this time the strategy is a search strategy under communication constraints, where Indicates indivual The observed value of Representing an Agent The action set is , the AUV search system selects from the action set according to the current strategy Select an action in One of all possible changes of the AUV under the communication constraints at the moment; In step 2, the AUV obtains the target existence probability of the grid corresponding to the current position through sonar. The target existence probability is represented by the pheromone corrected joint probability graph model, and the expression is: ；(2) in, represents the joint probability, express Due to the pheromone detection results and feedback, the time interval The amount of change, The value indicates exist The probability of the target existing at time instant; The update formula is: ；(3) in, and Respectively indicate the presence or absence of a target in the grid; and They are respectively expressed as detection probability and false alarm probability, express exist Time location The probability of the target existing.

2. According to the method of multi-AUV acoustic-optical multi-mode collaborative search based on deep reinforcement learning in claim 1, it is characterized in that: The specific process of step 1 is as follows: Set the task area to a The rectangular area is used as the initial working area for multi-AUV collaborative search at a depth of Deployment in the target area AUVs are equipped with acoustic communication sensors, optical communication sensors, and sonar equipment. AUVs use sonar as a sensor to capture targets. During the search process, AUVs transmit information through acoustic and optical communication sensors. AUVs form a collaborative search system to perform the search task; the task area is discretized into several grids.

3. According to a multi-AUV acoustic-optical multi-mode collaborative search method based on deep reinforcement learning according to claim 2, it is characterized in that: In step 2, after the AUV collaborative search control system performs one step of action, the reward function is set as follows: ；(4) in, represents the reward for moving to a high probability grid cell, represents the reward for finding the target, indicates the penalty for collision, represents the reward for encouraging the AUV to quickly approach the optical communication target after selecting the optical communication target, and Represents the penalty for AUVs being too close after the optical communication ends. By setting the reward, the AUV can quickly approach the optical communication target under the search constraints after sending the optical request, and quickly move away from the communication target after the communication ends, so as to ensure the efficiency of the search.

4. According to the method of multi-AUV acoustic-optical multi-mode collaborative search based on deep reinforcement learning in claim 3, it is characterized in that: In step 4, the target AUV receiving the optical communication data will fuse the information from the AUV in step 3 and update its own joint probability graph model. The fused probability is calculated by the following formula: ；(5) in, is the fused joint probability, Indicates The weight of the AUVs participating in the fusion, Indicates the number of AUVs involved in the fusion; The AUV cluster uses a multi-agent deep deterministic policy gradient reinforcement learning algorithm to make action decisions; In the AUV cluster, each AUV maintains its own Actor network and Critic network, and its network parameters are and , using the CTDE paradigm based on the Actor-Critic method for centralized training and distributed execution. During training, the Critic part of each AUV can obtain the strategy information of other AUVs; No. Critic network parameters for each AUV By minimizing the loss function To update, the specific formula is as follows: ；(6) in, for The expected value of the sample in represents the experience replay pool for storing data, Indicates the target value at the current moment; The goal of the Actor Network is to maximize Discounted expected return , No. Actor network parameters for each AUV The update is performed by performing the gradient descent method, the specific formula is: ；(7) in, Express Seeking guidance, Represents the optimization target, experience replay pool By tuple express, Indicates expected return, is the status information of the AUV, It is about performing the corresponding operation The next state information, Is the output of the Critic network, indicating the current state and Action The function value obtained; Soft update target critic network parameters and target Actor network parameters , the specific formula is as follows: ；(8) in, represents the target Critic network parameters, Represents the target Actor network parameters, Represents the soft update coefficient.

5. According to the method of multi-AUV acoustic-optical multi-mode collaborative search based on deep reinforcement learning in claim 4, it is characterized in that: In step 4, since the AUV is also equipped with an inertial navigation system, the speed of the AUV is controlled according to the inertial navigation system so that the AUV moves. The moving position of the AUV at the next moment is expressed as: ； (9) ；(10) ；(11) in, represents the cosine in trigonometric function, , Respectively The horizontal and vertical coordinates of time, , and Respectively The surge velocity, sway velocity and yaw rate at the moment; After selecting a specific action, the AUV adjusts the corresponding speed and updates the status information. Through the updated joint probability graph information and the continuous updating of network parameters, the AUV can gradually learn a better search strategy, so as to find the target more efficiently in the subsequent search process.

Images