Underwater collaborative search positioning method and system based on deep reinforcement learning

Abstract

The invention relates to the technical field of underwater collaborative search positioning, in particular to an underwater collaborative search positioning method and system based on deep reinforcement learning, and the method comprises the steps: obtaining the environment information and target information of an underwater robot; obtaining six-degree-of-freedom pose information of the underwater robot; based on the environment information and the target information, determining whether the underwater robot is in the search target area; determining the current state of the underwater robot according to the judgment result; generating a decision result based on the current state and a preset Markov decision process; and according to a decision result, a next-step action instruction of the underwater robot is output, and through an innovative deep reinforcement learning framework, the problems of uncertainty and dynamic change of the underwater environment are effectively solved. According to the method, a search strategy can be dynamically adjusted according to real-time environment information and target information, more intelligent and efficient decision making is achieved, a task allocation algorithm based on Monte Carlo sampling is introduced, and the cooperation efficiency of a multi-robot system is improved.

Description

Technical Field

[0001] The present invention relates to the field of underwater collaborative search and positioning technology, and more specifically, to an underwater collaborative search and positioning method and system based on deep reinforcement learning. Background Art

[0002] With the rapid development of marine resource exploration, underwater archaeology and marine environment monitoring, underwater collaborative search and positioning technology has attracted increasing attention. Traditional underwater search and positioning methods mainly rely on a single robot or simple multi-robot collaboration, which often cannot meet the requirements of high efficiency and high precision in complex and changeable underwater environments.

[0003] In recent years, with the advancement of artificial intelligence technology, researchers have begun to try to apply machine learning methods to underwater search and positioning tasks. Among them, reinforcement learning-based methods are favored because they can continuously optimize decision-making strategies in interaction with the environment. However, existing reinforcement learning-based underwater search and positioning methods still have many problems.

[0004] First, most methods have difficulty effectively dealing with the high uncertainty and dynamic changes in underwater environments. Underwater environments are affected by factors such as ocean currents, temperature, and pressure, and their states are often difficult to accurately predict and model. Existing methods often exhibit problems such as unstable decision-making and poor adaptability when faced with such complex environments.

[0005] Secondly, the problem of task allocation in multi-robot collaborative search has not been well solved. Most existing methods use simple fixed allocation strategies or dynamic allocation methods based on heuristic rules, which makes it difficult to fully utilize the advantages of multi-robot systems and leads to low search efficiency.

[0006] Furthermore, the limited bandwidth of underwater communication seriously restricts the collaborative effect of multi-robot systems. Existing methods often ignore this key limitation, making it difficult to achieve effective information sharing and collaborative decision-making in practical applications.

[0007] In addition, the existing methods need to be improved in terms of environment modeling and target recognition. Traditional environment modeling methods are difficult to accurately describe the complexity of underwater environments, while target recognition algorithms perform poorly when dealing with typical underwater problems such as blur and occlusion.

[0008] Finally, existing underwater collaborative search and positioning systems generally lack flexibility and scalability, and are difficult to adapt to different application scenarios and mission requirements. Summary of the invention

[0009] In response to the above problems, the present invention proposes an underwater collaborative search and positioning method and system based on deep reinforcement learning. The method aims to improve the search efficiency and positioning accuracy of the underwater multi-robot system, while enhancing the adaptability and robustness of the system in complex underwater environments.

[0010] The present invention provides an underwater collaborative search and positioning method based on deep reinforcement learning, comprising:

[0011] The acquisition steps include:

[0012] Obtain environmental information and target information of the underwater robot;

[0013] Obtain the six-degree-of-freedom position information of the underwater robot;

[0014] Processing steps include:

[0015] Based on the environmental information and the target information, determining whether the underwater robot is within the search target area;

[0016] Determining the current state of the underwater robot according to the judgment result;

[0017] Generate a decision result based on the current state and a preset Markov decision process;

[0018] Output steps include:

[0019] According to the decision result, the next action instruction of the underwater robot is output.

[0020] Preferably, the processing steps specifically include:

[0021] When the underwater robot is not in the search target area, setting the current state to a search state;

[0022] When the underwater robot is in the search target area, setting the current state to a cruising state;

[0023] Based on the current state, a next state is selected from a cruising state, a searching state, a self-positioning state, and a searching planning state.

[0024] Preferably, the method further comprises:

[0025] Build steps, including:

[0026] Based on the random walk method, a three-dimensional probability map of the underwater environment is constructed;

[0027] The environmental information is processed using a deep neural network to update the three-dimensional probability map.

[0028] Preferably, the method further comprises:

[0029] The task allocation steps include:

[0030] Based on the Monte Carlo sampling method, the exploration probability of multiple underwater robots is calculated;

[0031] Using Bayesian distribution, updating the exploration probabilities of the multiple underwater robots;

[0032] According to the updated exploration probability, exploration tasks are assigned to the multiple underwater robots.

[0033] Preferably, the processing step adopts an improved deep deterministic policy gradient algorithm, specifically comprising:

[0034] Using the Actor network, we generate action distribution based on the input observations.

[0035] Using the Critic network, the value of the action distribution is evaluated;

[0036] A policy network is optimized based on the action distribution and the value evaluation.

[0037] Preferably, the method further comprises:

[0038] The target identification steps include:

[0039] Use convolutional neural network to preprocess the acquired images;

[0040] Convert the preprocessed image features into graph structures;

[0041] The graph structure is processed using a graph convolutional neural network to obtain target confidence.

[0042] Preferably, the method further comprises:

[0043] Data transmission steps include:

[0044] Process the transmitted data in blocks;

[0045] Calculate the compression ratio of each data block;

[0046] Filter the optimal data blocks based on the preset maximum allowable compression rate;

[0047] Reorder and encode the filtered data blocks;

[0048] The re-encoded data is compressed and transmitted using a compression algorithm.

[0049] Preferably, the method further comprises:

[0050] Map fusion steps include:

[0051] Obtain three-dimensional map data of multiple underwater robots;

[0052] Based on the six-degree-of-freedom posture information, registering the multiple three-dimensional map data;

[0053] Fusion of multiple registered 3D map data to generate large-scene probability maps.

[0054] Preferably, the processing steps specifically include:

[0055] Construct a state transition graph with the current state as the root node;

[0056] Determine the probability of state transition and the corresponding action, and generate a state transition decision table;

[0057] According to the current state, the state transition decision table is queried to determine the optimal action.

[0058] An underwater collaborative search and positioning system based on deep reinforcement learning for executing the method includes:

[0059] Environmental perception module, used to obtain environmental information and target information of the underwater robot;

[0060] A posture estimation module, used to obtain the six-degree-of-freedom posture information of the underwater robot;

[0061] Decision control module, used to:

[0062] Based on the environmental information and the target information, determining whether the underwater robot is within the search target area;

[0063] Determining the current state of the underwater robot according to the judgment result;

[0064] Generate a decision result based on the current state and a preset Markov decision process;

[0065] The execution module is used to execute the next action of the underwater robot according to the decision result.

[0066] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0067] The method of the present invention effectively solves the uncertainty and dynamic change problems of the underwater environment through an innovative deep reinforcement learning framework. The method can dynamically adjust the search strategy according to real-time environmental information and target information to achieve more intelligent and efficient decision-making. At the same time, the present invention introduces a task allocation algorithm based on Monte Carlo sampling, which greatly improves the collaborative efficiency of the multi-robot system.

[0068] In terms of communication, the present invention proposes an innovative data compression and transmission strategy, which effectively solves the problem of limited underwater bandwidth. This not only improves the efficiency of information sharing, but also provides a reliable basis for collaborative decision-making of multi-robot systems.

[0069] The present invention also adopts an environment modeling and target recognition method based on deep learning, which significantly improves the system's ability to understand complex underwater environments and the accuracy of target recognition. In particular, it shows obvious advantages when dealing with typical underwater problems such as blur and occlusion.

[0070] In addition, the underwater collaborative search and positioning system proposed by the present invention has high flexibility and scalability. Through modular design, the system can easily adapt to different application scenarios and mission requirements, providing a universal and powerful technical platform for underwater operations.

[0071] In general, the method and system of the present invention have made significant progress in improving search efficiency, enhancing positioning accuracy, adapting to complex environments, optimizing resource utilization, etc. This not only provides strong technical support for underwater resource exploration, marine environmental monitoring, underwater archaeology and other fields, but also points out a new direction for the development of underwater robot technology in the future. The results of the present invention are expected to play an important role in promoting the progress of marine science and technology and promoting the development of the marine economy. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The figure is a flow chart of the method of the present invention.

[0073] Figure 2 A logic block diagram of the task allocation steps of the present invention.

[0074] Figure 3 A logic block diagram of the task allocation steps of the present invention.

[0075] Figure 4 A logic block diagram of the task allocation steps of the present invention.

[0076] Figure 5 It is a logic block diagram of the data transmission step of the present invention.

[0077] Figure 6 It is a logic block diagram of the map fusion step of the present invention. DETAILED DESCRIPTION

[0078] Please refer to Figure 1-6 The present invention provides an underwater collaborative search and positioning method and system based on deep reinforcement learning. The method can effectively solve the problem of multi-robot collaborative search and positioning in an underwater environment, and improve search efficiency and positioning accuracy.

[0079] First, the method of the present invention includes an acquisition step, a processing step, and an output step. In the acquisition step, the method acquires the environmental information and target information of the underwater robot, as well as the six-degree-of-freedom posture information of the underwater robot. This information is an important basis for subsequent decision-making. Preferably, the environmental information may include sonar data, hydrological data, etc., and the target information may include the estimated position and feature description of the target. The six-degree-of-freedom posture information includes the position and posture of the underwater robot in three-dimensional space.

[0080] Next, in the processing step, the method first determines whether the underwater robot is in the search target area based on the acquired environmental information and target information. This judgment is very critical because it determines the subsequent behavior strategy of the robot. The specific implementation of the judgment can adopt a method based on geographic coordinate comparison or feature matching. For example, a boundary threshold of the search target area can be set, and when the position coordinates of the robot are within the boundary, it is determined to be in the search target area.

[0081] According to the above judgment results, the method further determines the current state of the underwater robot. The state here refers to the working mode of the robot, which may include but is not limited to the cruising state, the searching state, the self-positioning state and the searching planning state. The choice of state directly affects the behavior strategy of the robot. For example, when the robot is determined to be in the searching target area, it may enter a more refined searching state.

[0082] After determining the current state, the method generates a decision result based on the state and a preset Markov decision process. The Markov decision process is a commonly used sequential decision model, which is applicable to the underwater collaborative search and positioning scenario in the present invention. In one embodiment of the present invention, the Markov decision process can be represented as a five-tuple group (S, A, P, R, γ), where:

[0083] S represents the state space, which contains all possible robot states;

[0084] A represents the action space, which contains all the actions that the robot can perform;

[0085] P represents the state transition probability function, which defines the probability of transitioning to a new state after taking a certain action in a certain state;

[0086] R represents the reward function, which defines the immediate reward obtained by taking a certain action in a certain state;

[0087] γ represents the discount factor, which is used to balance immediate rewards and long-term benefits.

[0088] Based on this Markov decision process, this method uses a deep reinforcement learning algorithm to learn the optimal strategy. Specifically, algorithms such as the deep Q network (DQN) or the policy gradient method can be used. For example, when using the DQN algorithm, the Q function can be defined as follows:

[0089] Q(s,a)=E[R t +γR t+1 +γ 2 R t+2 +…|s t =s,a t =a]

[0090] Among them, s represents the current state, a represents the current action, and R t represents the reward obtained at time t.

[0091] Finally, in the output step, this method outputs the next action instructions of the underwater robot based on the generated decision results. These instructions may include changing the heading, adjusting the depth, activating specific sensors, etc.

[0092] An important feature of the method of the present invention is the refinement of its processing steps. When the underwater robot is not in the search target area, the method sets the current state to the search state. This is to enable the robot to quickly reach the target area. Conversely, when the underwater robot is in the search target area, the method sets the current state to the cruise state. In the cruise state, the robot can search the target area more systematically.

[0093] Based on the determined current state, the method selects the next state from the cruise state, search state, self-positioning state, and search planning state. This state transition mechanism enables the underwater robot to flexibly respond to different situations. For example, when a suspicious target is detected in the cruise state, it may switch to a more sophisticated search state.

[0094] Another important innovation is that the method of the present invention introduces an environmental modeling step. In this step, the method constructs a three-dimensional probability map of the underwater environment based on the random walk method. Random walk is a simple and effective exploration strategy, especially suitable for unknown or semi-known environments. In one embodiment of the present invention, the random walk can be expressed as:

[0095] x t+1 =x t +Δx

[0096] y t+1 =y t +Δy

[0097] z t+1 =z t +Δz

[0098] Among them, (x t ,y t ,z t ) represents the position of the robot at time t, and (Δx, Δy, Δz) are displacements randomly sampled from some probability distribution (such as normal distribution).

[0099] The constructed three-dimensional probability map not only reflects the geometric structure of the environment, but also contains uncertainty information. For example, an occupancy grid map can be used to represent the environment, where the value of each grid represents the probability of the location being occupied.

[0100] In order to improve the accuracy and adaptability of the map, this method also uses deep neural networks to process environmental information and dynamically update the three-dimensional probability map. Here, structures such as convolutional neural networks (CNN) or recurrent neural networks (RNN) can be used to convert raw sensor data into high-level features, which are then used to update the map. For example, a map update function can be defined:

[0101] m t+1 =f(m t ,o t ,θ)

[0102] Among them, m t is a map at time t, o t is a new observation, θ is the neural network parameter, and f is the nonlinear function defined by the deep neural network.

[0103] In this way, the method of the present invention can continuously optimize the environmental model, providing a more reliable basis for subsequent search and positioning tasks. This ability of dynamic environmental modeling makes the method particularly suitable for complex and changeable underwater environments.

[0104] In summary, the underwater collaborative search and positioning method based on deep reinforcement learning provided by the present invention realizes the intelligent collaborative search and precise positioning of underwater robots by combining the steps of environmental perception, state judgment, decision generation and environmental modeling. This method can not only effectively cope with the uncertainty and complexity of the underwater environment, but also continuously optimize the decision-making strategy through deep learning technology to improve the search efficiency and positioning accuracy. This has important application value in the fields of marine resource exploration, underwater archaeology, submarine pipeline detection, etc. The method of the present invention also includes a key task allocation step, which aims to optimize the collaborative search efficiency of multiple underwater robots. In this step, the method adopts an innovative algorithm based on Monte Carlo sampling to calculate and update the exploration probability of multiple underwater robots.

[0105] Specifically, the method first calculates the exploration probability of multiple underwater robots based on the Monte Carlo sampling method. The core idea of ​​the Monte Carlo method is to approximate the solution of a complex problem through a large number of random sampling. In a preferred embodiment of the present invention, the calculation of the exploration probability can be expressed as:

[0106]

[0107] Among them, P(e i ) represents the exploration probability of robot i, N is the number of sampling times, s j is the state of the jth sampling, A i is the reachable area of ​​robot i, and I(·) is the indicator function. This method can effectively estimate the exploration effect of each robot in different areas.

[0108] Next, the method uses the Bayesian distribution to update these exploration probabilities. Bayesian updating allows us to dynamically adjust probability estimates as new information is obtained. In another embodiment of the present invention, Bayesian updating can be expressed as:

[0109]

[0110] Among them, P(e i |o) is the updated exploration probability of robot i after observing o, P(o|e i ) is the likelihood function, P(e i ) is the prior probability and P(o) is the evidence factor.

[0111] Based on these updated exploration probabilities, the method assigns exploration tasks to multiple underwater robots. Preferably, task assignment can use a combinatorial optimization method such as a greedy algorithm or a Hungarian algorithm. For example, when using a greedy algorithm, the area with the highest current exploration probability can be iteratively assigned to each robot. This dynamic task assignment mechanism can effectively balance exploration and utilization and improve the search efficiency of the entire system.

[0112] The method of the present invention adopts an improved deep deterministic policy gradient (DDPG) algorithm in the processing step. This is an advanced algorithm that combines deep learning and reinforcement learning, and is particularly suitable for problems with continuous action space, such as the control of underwater robots.

[0113] In one embodiment of the present invention, the DDPG algorithm includes two key networks: the Actor network and the Critic network. The Actor network is responsible for generating actions, and its structure can be expressed as:

[0114] a=μ(s|θ μ )

[0115] Among them, a is the action, s is the state, μ is the Actor network function, θ μ is the network parameter.

[0116] The Critic network is responsible for evaluating the value of the action, and its structure can be expressed as:

[0117] Q(s,a)=Q(s,a|θ Q )

[0118] Where Q is the state-action value function, θ Q is the Critic network parameter.

[0119] The innovation of this method is that it uses the Actor network to generate an action distribution based on the input observations, rather than a single deterministic action. This approach increases the algorithm's exploration capabilities. The action distribution can be expressed as:

[0120]

[0121] in, represents the normal distribution, σ 2 is the variance, which can be a fixed value or learned through the network. The Critic network evaluates the value of this action distribution.

[0122] The Critic network evaluates the value of this action distribution. The evaluation process can be expressed as:

[0123] V(s)=E a～P(a∣s) [Q(s,a)]

[0124] Among them, V(s) is the state value function.

[0125] Based on action distribution and value evaluation, this method optimizes the policy network. The optimization objective can be expressed as:

[0126]

[0127] Among them, ρ β Represents the state distribution.

[0128] Through this improved DDPG algorithm, this method can generate more flexible and robust control strategies in complex underwater environments and improve the search and positioning performance of underwater robots.

[0129] The method of the present invention also includes an innovative target recognition step, which utilizes advanced deep learning technology to achieve accurate recognition of underwater targets. The implementation of this step is crucial to improving the search efficiency and positioning accuracy of the entire system.

[0130] First, the method uses a convolutional neural network (CNN) to preprocess the acquired image. In a preferred embodiment of the present invention, the structure of the CNN may include multiple convolutional layers, pooling layers, and fully connected layers. The convolution operation can be expressed as:

[0131]

[0132] Among them, I is the input image, K is the convolution kernel, and f(i,j) is the value of the (i,j) position of the output feature map.

[0133] After CNN processing, the preprocessed image features are converted into a graph structure. This step is another innovation of this method, which allows us to use the structural information of the graph to improve the recognition accuracy. The graph structure can be expressed as:

[0134] G=(V,E)

[0135] Among them, V is the node set, representing the local features of the image, and E is the edge set, representing the relationship between features.

[0136] Next, this method uses a graph convolutional neural network (GCN) to process this graph structure. The operation of a layer of GCN can be expressed as:

[0137]

[0138] Among them, H (l) is the node feature matrix of the lth layer, For the adjacency matrix with self-loops added, is the corresponding degree matrix, W (l) is a learnable weight matrix and σ is a nonlinear activation function.

[0139] Through multi-layer GCN operations, this method can effectively capture the complex relationship between image features and finally obtain the confidence of the target. This graph-based target recognition method is particularly suitable for processing complex scenes in underwater environments and can improve the accuracy and robustness of recognition.

[0140] In summary, the underwater collaborative search and positioning method based on deep reinforcement learning of the present invention realizes efficient collaboration and precise positioning of underwater multi-robot systems through innovative task allocation strategies, improved deep reinforcement learning algorithms, and target recognition technology based on graph structures. The combination of these technologies not only improves the search efficiency and positioning accuracy of the system, but also enhances the adaptability and robustness of the system in complex underwater environments. This method provides strong technical support for underwater resource exploration, marine environment monitoring, underwater archaeology and other fields, and has broad application prospects. The method of the present invention also includes an innovative data transmission step, which is intended to solve the problem of bandwidth-limited communication in underwater environments. In underwater environments, effective data transmission is crucial for the collaboration of multi-robot systems, and this method provides an efficient solution.

[0141] First, the method processes the transmission data in blocks. Preferably, the block size can be dynamically adjusted according to the actual communication conditions. For example, in one embodiment, the block size can be set to between 512 bytes and 4096 bytes, which is usually able to achieve a good balance between transmission efficiency and robustness.

[0142] Next, the method calculates the compression ratio of each data block. The calculation of the compression ratio can be expressed as:

[0143]

[0144] Among them, CR represents the compression ratio, S original Indicates the original data block size, S comprcssed Indicates the size of the compressed data block.

[0145] Based on the preset maximum allowable compression rate, the method selects the optimal data block. The setting of the maximum allowable compression rate needs to consider the balance between data integrity and transmission efficiency. In a preferred embodiment of the present invention, the maximum allowable compression rate can be set to 10:1, that is, one tenth of the original data volume. This value can usually provide sufficient compression effect in most underwater environments while ensuring data reliability.

[0146] For the filtered data blocks, this method reorders and encodes them. The purpose of reordering is to optimize the transmission order of data so that important information can be transmitted first. The encoding process can use entropy coding methods such as Huffman coding or arithmetic coding to further improve the data compression efficiency.

[0147] Finally, the method uses a compression algorithm to compress and transmit the re-encoded data. In an underwater environment, compression algorithms such as ZLIB or LZMA can be selected, which achieve a good balance between compression efficiency and decompression speed.

[0148] Through this multi-step data processing and transmission method, the present invention can significantly improve the efficiency of underwater communication and provide strong support for the collaborative search and positioning of multi-robot systems.

[0149] The method of the present invention also includes an important map fusion step, which is intended to integrate the environmental perception information of multiple underwater robots to build a global, high-precision environmental map. This step is crucial to improving the positioning accuracy and environmental understanding ability of the entire system.

[0150] First, the method obtains 3D map data of multiple underwater robots. Each robot builds a local 3D map based on its own sensor data, which is usually represented in the form of point cloud or voxel grid.

[0151] Next, based on the previously acquired six-degree-of-freedom pose information, the method aligns multiple three-dimensional map data. The alignment process can use the iterative closest point (ICP) algorithm or its variants. The objective function of the ICP algorithm can be expressed as:

[0152]

[0153] Among them, R is the rotation matrix, t is the translation vector, and p i and q i are the corresponding point pairs in the two point clouds, N p is the number of point pairs.

[0154] By minimizing this objective function, the optimal rotation and translation parameters can be found, thus achieving accurate alignment between different robot maps.

[0155] After completing the registration, this method fuses these aligned 3D map data to generate a large scene probability map. The fusion process can be expressed as:

[0156] P(m|z 1:t )=ηP(z t |m)P(m|z 1:t-1 )

[0157] Where m represents the map, z1:t represents all observations up to time t, and η is the normalization factor.

[0158] This probability map not only contains the geometric information of the environment, but also reflects the uncertainty of environmental cognition, providing a more reliable basis for subsequent decision-making.

[0159] The method of the present invention also includes an innovative decision-making mechanism in the processing step, which is based on a state transition diagram and a state transition decision table and can effectively handle complex situations and uncertainties in an underwater environment.

[0160] First, this method constructs a state transition graph with the current state as the root node. The state transition graph can be represented as a directed graph:

[0161] G=(V,E)

[0162] Among them, V is the vertex set, representing all possible states, and E is the edge set, representing the possible transitions between states.

[0163] Next, this method determines the probability of state transition and the corresponding action, and generates a state transition decision table. The decision table can be represented as a matrix D, where D ij represents the probability of transitioning from state i to state j and the corresponding optimal action. This probability can be continuously updated and optimized through the reinforcement learning process.

[0164] In actual decision-making, this method queries the state transition decision table based on the current state to determine the optimal action. The selection of the optimal action can be based on the maximum expected reward principle, expressed as:

[0165]

[0166] Among them, a * is the optimal action, and R(i,j,a) is the reward for executing action a from state i to state j. This method based on state transition diagram and decision table can effectively handle various complex situations in underwater environments and improve the accuracy and efficiency of decision-making.

[0167] This method based on state transition diagram and decision table can effectively handle various complex situations in underwater environment and improve the accuracy and efficiency of decision-making.

[0168] Finally, the present invention also proposes an underwater collaborative search and positioning system based on deep reinforcement learning, which is a specific implementation of the above method. The system includes an environment perception module 1, a posture estimation module 2, a decision control module 3 and an execution module 4.

[0169] The environment perception module 1 is responsible for obtaining the environment information and target information of the underwater robot. This module can include various sensors, such as sonar, water pressure sensor, optical camera, etc. The data of these sensors provide the system with comprehensive environment perception capabilities through appropriate signal processing and fusion algorithms.

[0170] The pose estimation module 2 is used to obtain the six-degree-of-freedom pose information of the underwater robot. This module can combine technologies such as inertial measurement unit (IMU), depth sensor and visual odometer to achieve high-precision pose estimation. In underwater environments, pose estimation is particularly challenging, so this system uses multi-sensor fusion and filtering technology to improve the accuracy and robustness of the estimation.

[0171] The decision control module 3 is the core of the whole system, which implements the decision algorithm based on deep reinforcement learning. The module first determines whether the underwater robot is in the search target area, then determines the current state of the robot, and finally generates a decision result based on the current state and the preset Markov decision process. The design of this module fully considers the particularity of the underwater environment and can adapt to various complex search and positioning scenarios.

[0172] The execution module 4 is responsible for executing the next action of the underwater robot according to the decision result. This module converts the high-level decision instructions into specific control signals to drive the robot's propellers, steering gears and other actuators to achieve the expected movement and operation.

[0173] Through the collaborative work of these modules, this system can realize the intelligent collaborative search and precise positioning of underwater robots, providing strong technical support for applications such as underwater resource exploration, marine environment monitoring, and underwater archaeology.

[0174] In summary, the underwater collaborative search and positioning method and system based on deep reinforcement learning provided by the present invention effectively solves the problem of multi-robot collaborative search and positioning in underwater environments through innovative data transmission strategies, map fusion technology, graph-based decision-making mechanisms, and multi-module collaborative system design. This method and system not only improves search efficiency and positioning accuracy, but also enhances the adaptability and robustness of the entire system in complex underwater environments, and has important theoretical value and broad application prospects.

[0175] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, replacement, and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An underwater collaborative search and positioning method based on deep reinforcement learning, characterized in that: include: The acquisition steps include: Obtain environmental information and target information of the underwater robot; Obtain the six-degree-of-freedom position information of the underwater robot; Processing steps include: Based on the environmental information and the target information, determining whether the underwater robot is within the search target area; Determining the current state of the underwater robot according to the judgment result; Generate a decision result based on the current state and a preset Markov decision process; Output steps include: According to the decision result, the next action instruction of the underwater robot is output.

2. The method according to claim 1, characterized in that The processing steps specifically include: When the underwater robot is not in the search target area, setting the current state to a search state; When the underwater robot is in the search target area, setting the current state to a cruising state; Based on the current state, a next state is selected from a cruising state, a searching state, a self-positioning state, and a searching planning state.

3. The method according to claim 1, characterized in that The method further comprises: Build steps, including: Based on the random walk method, a three-dimensional probability map of the underwater environment is constructed; The environmental information is processed using a deep neural network to update the three-dimensional probability map.

4. The method according to claim 1, characterized in that The method further comprises: The task allocation steps include: Based on the Monte Carlo sampling method, the exploration probability of multiple underwater robots is calculated; Using Bayesian distribution, updating the exploration probabilities of the multiple underwater robots; According to the updated exploration probability, exploration tasks are assigned to the multiple underwater robots.

5. The method according to claim 1, characterized in that The processing steps adopt an improved deep deterministic policy gradient algorithm, specifically including: Using the Actor network, we generate action distribution based on the input observations. Using the Critic network, the value of the action distribution is evaluated; A policy network is optimized based on the action distribution and the value evaluation.

6. The method according to claim 1, characterized in that The method further comprises: The target identification steps include: Use convolutional neural network to preprocess the acquired images; Convert the preprocessed image features into graph structures; The graph structure is processed using a graph convolutional neural network to obtain target confidence.

7. The method according to claim 1, characterized in that The method further comprises: Data transmission steps include: Process the transmitted data in blocks; Calculate the compression ratio of each data block; Filter the optimal data blocks based on the preset maximum allowable compression rate; Reorder and encode the filtered data blocks; The re-encoded data is compressed and transmitted using a compression algorithm.

8. The method according to claim 1, characterized in that: The method further comprises: Map fusion steps include: Obtain three-dimensional map data of multiple underwater robots; Based on the six-degree-of-freedom posture information, registering the multiple three-dimensional map data; Fusion of multiple registered 3D map data to generate large-scene probability maps.

9. The method according to claim 1, characterized in that: The processing steps specifically include: Construct a state transition graph with the current state as the root node; Determine the probability of state transition and the corresponding action, and generate a state transition decision table; According to the current state, the state transition decision table is queried to determine the optimal action.

10. An underwater collaborative search and positioning system based on deep reinforcement learning that executes the method according to any one of claims 1 to 9, characterized in that: include: Environmental perception module, used to obtain environmental information and target information of the underwater robot; A posture estimation module, used to obtain the six-degree-of-freedom posture information of the underwater robot; Decision control module, used to: Based on the environmental information and the target information, determining whether the underwater robot is within the search target area; Determining the current state of the underwater robot according to the judgment result; Generate a decision result based on the current state and a preset Markov decision process; The execution module is used to execute the next action of the underwater robot according to the decision result.

Images