Three-dimensional covert path planning method and system for underwater vehicle based on TAM-HRL

Abstract

The present application belongs to the technical field of underwater navigation control, and particularly relates to a three-dimensional covert path planning method and system for underwater vehicles based on TAM-HRL, which acquires a three-dimensional terrain field and a three-dimensional acoustic detection probability field of a target water area, and constructs a combined state containing global navigation, local terrain, local detection probability and maneuvering state; the combined state is input into a layered time sequence decision model, in which a high-level network outputs fusion weights based on a historical state sequence, and the action probability distribution output by each sub-strategy is weighted and fused to obtain a final action. A compound reward signal and a responsibility allocation experience replay mechanism are used in training, and each sub-strategy is updated only with the samples generated by it. The present application can effectively suppress strategy shock and frequent switching of sub-tasks, improve training convergence stability and path success rate, and generate a safe, covert and smooth three-dimensional navigation path.

Description

Technical Field

[0001] This invention belongs to the field of underwater navigation control technology, specifically relating to a three-dimensional stealth path planning method and system for underwater vehicles based on TAM-HRL. Background Technology

[0002] Autonomous underwater vehicles (AUVs) can perform tasks such as detection, cruising, searching, monitoring, and target approach in complex underwater environments. Existing path planning methods for AUVs typically discretize the mission area into a two-dimensional or three-dimensional grid and employ algorithms such as A*, Dijkstra's algorithm, fast expanding random tree algorithm, ant colony algorithm, particle swarm optimization, or reinforcement learning for path search. Some methods introduce the concept of a sonar threat zone, simplifying acoustic detection risk into a fixed circular area or an empirical risk area based on distance attenuation. Regarding path optimization objectives, existing methods mainly focus on path length, obstacle avoidance safety, or navigation energy consumption, simultaneously learning multiple behavioral preferences such as terrain obstacle avoidance, detection avoidance, target approach, and maneuver smoothing through a single agent.

[0003] However, existing technologies still have the following shortcomings: First, simplifying acoustic detection risk into a fixed circular region or distance attenuation model makes it difficult to reflect the non-uniform acoustic risk distribution under the combined effects of sound speed profiles, seabed undulations, propagation losses, and multi-source detection in the real ocean environment. Second, methods based on a single reinforcement learning agent couple multiple types of targets into the same state space and reward function, leading to competition between different targets and easily causing problems such as training instability, policy oscillations, local path detours, or interference between obstacle avoidance and target avoidance. Third, if high-level decision-making relies solely on current-moment observations, it is difficult to accurately determine the timing of sub-task switching, easily leading to frequent sub-policy switching and uneven trajectories, failing to meet the requirements of underwater vehicles for path continuity and smooth maneuvering. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional stealth path planning method and system for underwater vehicles based on TAM-HRL, so as to solve the problems of unstable training, policy oscillation and uneven trajectory caused by frequent subtask switching in existing autonomous underwater vehicle path planning methods due to multi-target coupling.

[0005] The present invention achieves the above objectives through the following technical solutions: This invention proposes a three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL, including: Obtain the three-dimensional constraint field corresponding to the target water area. The three-dimensional constraint field includes a three-dimensional topographic field and a three-dimensional acoustic detection probability field. The current position of the underwater vehicle, the target position, local terrain information, local detection probability information, and historical maneuver information are determined based on the three-dimensional constraint field and constructed as a combined state. The combined state is input into a pre-constructed time-sharing decision model, which includes multiple low-level sub-policy networks and a high-level time-series decision network. The action probability distribution is output by multiple low-level sub-policy networks, and the fusion weight of each low-level sub-policy network is output by a high-level temporal decision network based on the combined state of multiple consecutive time points. The probability distributions of multiple actions are weighted and fused according to the fusion weights to obtain the final action probability distribution in the preset action space. The autonomous underwater vehicle's motion is determined based on the final motion probability distribution, and the hierarchical temporal decision model is trained based on the composite feedback signal until a three-dimensional acoustic concealment path from the starting point to the target area is output. The preset action space includes a discrete combination of horizontal heading change and vertical depth change; wherein the horizontal heading change includes at least one of left turn, maintaining heading, and right turn, and the vertical depth change includes at least one of surfacing, maintaining depth, and diving.

[0006] Preferably, the combined states include global navigation state, local terrain state, local detection probability state, and maneuver state. Specifically, The global navigation status includes the underwater vehicle's current position, relative target position, and heading information, including the following formula: in, This indicates the current location of the underwater vehicle. These represent the axis-by-axis normalized offsets of the underwater vehicle relative to the target point. The horizontal heading angle of the underwater vehicle; The local terrain state and the local detection probability state are respectively extracted through multi-scale windows: taking the current position of the autonomous underwater vehicle as the center, at least one scale of two-dimensional local window is extracted from the three-dimensional terrain field to form the local terrain state, and at least one scale of two-dimensional local window is extracted from the three-dimensional acoustic detection probability field to form the local detection probability state. The maneuver status includes the actual changes in heading and depth performed by the underwater vehicle at the previous moment.

[0007] Preferably, the plurality of low-level sub-strategy networks include an obstacle avoidance sub-strategy network, a detection and avoidance sub-strategy network, and a maneuver control sub-strategy network; The obstacle avoidance sub-strategy network is used to output an obstacle avoidance action probability distribution based on the global navigation state and the local terrain state; The detection avoidance sub-strategy network is used to output a probability distribution of detection avoidance actions based on the global navigation state and the local detection probability state. The maneuver control sub-policy network is used to output a smooth maneuver probability distribution based on the global navigation state and the maneuver state.

[0008] Preferably, the obstacle avoidance sub-strategy network and the detection avoidance sub-strategy network include a multi-scale convolutional feature extraction module; The multi-scale convolutional feature extraction module includes large-scale convolutional branches and small-scale convolutional branches; The large-scale convolutional branch is used to receive environmental data within a large window range, extract far-field terrain undulation trends and / or detect risk distribution trends; The small-scale convolutional branch is used to receive environmental data within a small window range and extract detailed features of near-field obstacles, local boundaries and / or local high-probability detection areas. The global context features output by the large-scale convolutional branch are fused with the local detail features output by the small-scale convolutional branch to obtain environment-aware features for action decision-making.

[0009] Preferably, the high-level decision-making network includes a gated recurrent unit (GRU). The high-level decision network receives a state sequence consisting of combined states at multiple consecutive time points, extracts temporal features through the gated recurrent unit (GRU), and outputs the fusion weights of the multiple low-level sub-policy networks based on the temporal features. The final action probability distribution is weighted and fused from the action probability distributions output by each low-level policy network according to the following formula: Where N is the number of low-level sub-policy networks. Let be the combination state at time t. Let be the input state of the i-th lower-level sub-policy network. The fusion weights of the i-th low-level sub-policy network output by the high-level temporal decision network are: Let i be the action probability distribution output by the i-th low-level sub-policy network. This represents the final action probability distribution after fusion.

[0010] Preferably, the preset action space includes a discrete combination of horizontal heading change and vertical depth change; wherein the horizontal heading change includes at least one of left turn, maintaining heading, and right turn, and the vertical depth change includes at least one of surfacing, maintaining depth, and diving.

[0011] Preferably, the composite feedback signal includes: Proximity feedback is used to guide underwater vehicles toward a target area; Collision constraint feedback is used to constrain the distance between underwater vehicles and the seabed, sea surface, and horizontal boundaries; Detection avoidance feedback is used to penalize behaviors that enter areas with a high probability of detection. Maneuvering smoothing feedback is used to suppress drastic changes in course and depth.

[0012] Preferably, during the training of the hierarchical time-series decision model, a responsibility allocation operation is also performed: Based on the fusion weights output by the high-level temporal decision network and the output probabilities of each low-level policy network for the executed actions, the responsibility allocation coefficient of each low-level policy network for this decision is calculated as follows: in, For the first The responsibility allocation coefficients of the low-level sub-policy network at time t. For high-level time-series decision networks at time... The output of the first Each fusion weight, For the first A low-level sub-policy network for the selected action The output probability, ε is a constant to prevent the denominator from being zero; when the k-th low-level sub-policy network satisfies The current environment interaction samples are used as training samples for the k-th low-level sub-policy network. The current environmental interaction samples are stored in the experience replay buffer corresponding to the low-level policy network with the largest responsibility allocation coefficient, so that each low-level policy network can update its parameters based on the samples it generates.

[0013] This invention also proposes a three-dimensional stealth path planning system for underwater vehicles based on TAM-HRL, used to implement the three-dimensional stealth path planning method for underwater vehicles as described above. The system includes: An environmental data acquisition module is used to acquire a three-dimensional constraint field corresponding to the target water area. The three-dimensional constraint field includes a three-dimensional topographic field and a three-dimensional acoustic detection probability field. The state construction module is used to construct a combined state based on the current position of the underwater vehicle, the target position, and local terrain information and local detection probability information extracted from the three-dimensional constraint field, combined with historical maneuver information. The hierarchical temporal decision model includes multiple low-level policy networks and one high-level temporal decision network. The multiple low-level policy networks are used to output action probability distributions respectively. The high-level temporal decision network is used to output the fusion weights of each low-level policy network based on the combined state of multiple consecutive time points, and to perform weighted fusion of multiple action probability distributions according to the fusion weights to obtain the final action probability distribution in the preset action space. The training optimization module is used to train the hierarchical time-series decision model based on the composite feedback signal. The path output module is used to output a 3D hidden path from the starting point to the target region using the trained hierarchical temporal decision model.

[0014] Preferably, the state construction module includes: The navigation status acquisition unit is used to acquire the current position, target position, and heading angle information of the underwater vehicle in real time; The multi-scale window extraction unit is used to extract at least two two-dimensional local windows of different scales from the three-dimensional topographic field and the three-dimensional acoustic detection probability field, respectively, with the current position of the underwater vehicle as the center, and then flatten and stitch the windows of each scale to form a local state vector. The maneuvering status recording unit is used to store the actual changes in heading and depth performed by the underwater vehicle at the previous moment.

[0015] Preferably, the training optimization module includes: The composite reward calculation unit is used to calculate the instantaneous reward value of the current step based on distance approach feedback, collision constraint feedback, detection avoidance feedback and maneuver smoothing feedback; The experience replay buffer group contains multiple independent experience replay buffers, and each experience replay buffer corresponds one-to-one with the multiple low-level policy networks. The parameter update unit is used to randomly sample training samples from the experience replay buffer group and update the network parameters of each low-level policy network and the high-level temporal decision network using gradient descent.

[0016] The beneficial effects of this invention are as follows: First, this invention proposes a TAM-HRL temporal-aware multi-scale hierarchical reinforcement learning architecture, which decomposes the 3D path planning task of autonomous underwater vehicles into multiple sub-tasks such as terrain obstacle avoidance, detection and avoidance, and maneuver smoothing. A high-level temporal decision network then coordinates the control of multiple low-level sub-policy networks. Compared to a single reinforcement learning agent, this invention reduces the learning difficulty of multi-objective, strongly coupled path planning tasks and improves the stability of policy learning in complex environments.

[0017] Second, this invention uses the three-dimensional topographic field and the three-dimensional acoustic detection probability field as the path planning constraint input, so that the path planning model can not only perceive the topographic safety constraints such as the seabed, sea surface and horizontal boundaries, but also perceive the non-uniform acoustic high-risk area formed by the acoustic propagation environment, sonar parameters, seabed undulations and multi-source detection, thereby improving the safety and stealth of the autonomous underwater vehicle's three-dimensional path.

[0018] Third, this invention introduces a multi-scale environmental perception structure into the low-level sub-policy network. The large-scale perception branch extracts the terrain change trend and acoustic risk distribution trend over a large area, while the small-scale perception branch extracts local terrain details and local high-risk area features within a short distance. This enables autonomous underwater vehicles to simultaneously possess far-field risk prediction capabilities and near-field fine-grained risk avoidance capabilities, thereby improving the robustness of path planning in complex underwater environments.

[0019] Fourth, this invention introduces a gated recurrent unit (GRU) into the high-level temporal decision network, enabling the high-level decision network to receive combined state sequences from multiple consecutive time points and dynamically output the fusion weights of each low-level sub-policy network based on historical navigation status, local terrain changes, acoustic risk changes, and maneuver trends. Compared to high-level decision-making methods that rely solely on observations at the current time, this invention reduces policy oscillations caused by frequent sub-policy switching and improves the temporal continuity and motion smoothness of path decisions.

[0020] Fifth, this invention fuses the action probability distributions output by multiple low-level sub-policy networks through an action probability weighted fusion mechanism, rather than fixedly selecting the action output of a single low-level sub-policy network. This approach enables flexible coordination among multiple task objectives such as terrain obstacle avoidance, detection avoidance, and maneuver smoothing, ensuring that the final action decision simultaneously considers safety, stealth, and movement rationality.

[0021] Sixth, this invention proposes a responsibility allocation experience replay mechanism. Based on the fusion weights output by the high-level temporal decision network and the output probabilities of each low-level sub-policy network for the selected action, the responsibility allocation coefficient corresponding to the current interaction sample is calculated, and the sample is allocated to the experience replay buffer of the dominant low-level sub-policy network. This mechanism can reduce interference from samples irrelevant to the sub-task objective, improve the training targeting of the low-level sub-policy network and the overall convergence stability of the hierarchical reinforcement learning model. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a three-dimensional path planning method for an autonomous underwater vehicle in an embodiment of the present invention. Figure 2 This is a schematic diagram of a three-dimensional path planning scenario for an autonomous underwater vehicle in an embodiment of the present invention; Figure 3 This is a schematic diagram of the TAM-HRL path planning model structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-scale convolutional feature extraction module structure in an embodiment of the present invention; Figure 5 This is a schematic diagram of the high-level GRU decision network structure in an embodiment of the present invention; Figure 6This is a comparison scenario diagram of the methods in the embodiments of the present invention; Figure 7 These are comparative effect diagrams of the methods in the embodiments of the present invention; Figure 8 This is a two-dimensional path and enemy detection probability map at time t=0.8h in an embodiment of the present invention; Figure 9 This is a two-dimensional path and enemy detection probability map at time t=1.6h in an embodiment of the present invention; Figure 10 This is a two-dimensional path and enemy detection probability map at time t=2.4h in an embodiment of the present invention; Figure 11 This is a two-dimensional path and enemy detection probability map at time t=3.1h in an embodiment of the present invention; Figure 12 This is a two-dimensional path and enemy detection probability diagram at time t=3.9h in an embodiment of the present invention; Figure 13 This is a two-dimensional path and enemy detection probability map at time t=4.7h in an embodiment of the present invention; Figure 14 This is a three-dimensional path and enemy detection probability map at time t=0.8h in an embodiment of the present invention; Figure 15 This is a three-dimensional path and enemy detection probability map at time t=1.6h in an embodiment of the present invention; Figure 16 This is a three-dimensional path and enemy detection probability map at time t=2.4h in an embodiment of the present invention; Figure 17 This is a three-dimensional path and enemy detection probability map at time t=3.1h in an embodiment of the present invention; Figure 18 This is a diagram showing the three-dimensional path and enemy detection probability at time t=3.9h in an embodiment of the present invention. Figure 19 This is a diagram showing the three-dimensional path and enemy detection probability at time t=4.7h in an embodiment of the present invention. Detailed Implementation

[0023] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0024] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0025] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0026] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0027] This invention provides a method and system for acoustic stealth 3D path planning for autonomous underwater vehicles based on the TAM-HRL architecture. TAM-HRL stands for Temporal-Aware Multi-scale Hierarchical Reinforcement, which refers to temporal-aware multi-scale hierarchical reinforcement learning. This method addresses stealth navigation tasks constrained by complex seabed terrain and non-uniform acoustic detection risks. By constructing a temporal-aware multi-scale hierarchical reinforcement learning model, the path planning process is decomposed into multiple sub-tasks, including terrain obstacle avoidance, detection avoidance, and maneuver smoothing. A high-level temporal decision network dynamically fuses multiple low-level sub-strategies to generate a 3D navigation path that balances safety, stealth, and smooth motion.

[0028] Another objective of this invention is to provide a three-dimensional path planning system for autonomous underwater vehicles. This system can use three-dimensional terrain fields and three-dimensional acoustic detection probability fields as environmental constraints, learn obstacle avoidance, detection avoidance, and maneuver control behaviors through multiple low-level sub-policy networks, and perform fusion decision-making on multiple sub-policies through a high-level decision network.

[0029] First Embodiment like Figure 2As shown, this embodiment provides a 3D path planning method for autonomous underwater vehicles (AUVs) based on TAM-HRL. This method is designed for underwater 3D environments with complex terrain constraints and acoustic detection risks. The AUV needs to start from a given point and reach the target area while satisfying constraints related to navigation boundaries, seabed safety clearance, and detection risks.

[0030] In this embodiment, the three-dimensional topographic field can be constructed from a seabed digital elevation model, bathymetry data, nautical chart data, or other topographic data. The three-dimensional acoustic detection probability field can be constructed from sound propagation simulation, sonar detection model, historical observation data, measured sound field data, or multi-source data fusion methods. The core of this invention is not to limit the unique generation method of the three-dimensional acoustic detection probability field, but to use the three-dimensional acoustic detection probability field as an acoustic risk constraint input to the TAM-HRL path planning model, and to use a hierarchical reinforcement learning model for path decision-making.

[0031] like Figure 2 As shown, the method includes the following steps.

[0032] S1. Obtain the three-dimensional constraint field corresponding to the target water area. The three-dimensional constraint field includes a three-dimensional topographic field and a three-dimensional acoustic detection probability field. The three-dimensional topographic field is used to describe the seabed undulations, sea surface boundaries, maximum permissible depth, and non-navigable areas in the target water area. The three-dimensional acoustic detection probability field is used to describe the probability that the autonomous underwater vehicle will be detected by an acoustic detection source at different locations and / or at different times.

[0033] S2, based on the three-dimensional constraint field, determine the current position of the underwater vehicle, the target position, local terrain information, local detection probability information, and historical maneuver information, and construct a combined state. The combined state consists of a global navigation state, a local terrain state, a local detection probability state, and a maneuver state.

[0034] S3, input the combined state into a pre-constructed time-sharing decision model (TAM-HRL path planning model). The time-sharing decision model includes multiple low-level sub-policy networks and one high-level time-series decision network; the TAM-HRL path planning model includes low-level sub-policy networks and a high-level decision network. The low-level sub-policy networks include obstacle avoidance sub-policy networks, detection and avoidance sub-policy networks, and maneuver control sub-policy networks.

[0035] S4, the obstacle avoidance sub-strategy network, the detection and avoidance sub-strategy network, and the maneuver control sub-strategy network output the action probability distribution according to their respective input states.

[0036] S5, the high-level decision network outputs the fusion weights of multiple low-level sub-policy networks based on the combined state.

[0037] S6. Multiple low-level action probability distributions are weighted and fused according to fusion weights to obtain the final action probability distribution in the preset action space. The next action of the autonomous underwater vehicle is then determined based on the final action probability distribution. The preset action space includes discrete combinations of horizontal heading changes and vertical depth changes; wherein the horizontal heading changes include at least one of left turn, maintaining heading, and right turn, and the vertical depth changes include at least one of surfacing, maintaining depth, and diving.

[0038] S7, execute the next action, update the autonomous underwater vehicle's position, and calculate the distance reward, collision constraint reward, detection avoidance reward, maneuvering smoothing reward, and termination penalty. The above update operation aims to train the TAM-HRL path planning model using composite feedback signals. These composite feedback signals include distance approach feedback to guide the underwater vehicle towards the target area, collision constraint feedback to constrain the distance between the underwater vehicle and the seabed, sea surface, and horizontal boundaries, detection avoidance feedback to penalize behavior entering high-probability detection areas, and maneuvering smoothing feedback to suppress drastic changes in course and depth.

[0039] S8 stores the interaction samples into the corresponding experience replay buffer and trains and updates the TAM-HRL path planning model.

[0040] S9, repeat S2-S8 until the model reaches the training termination condition. Use the trained model to output a 3D path from the starting point to the target region during the testing phase.

[0041] In one embodiment, the three-dimensional terrain field can be represented as ,in and The horizontal position coordinates are... This indicates the seabed depth at the corresponding horizontal position. The position of an autonomous underwater vehicle (AUV) is indicated as... ,in This indicates the current depth of the AUV, with the depth direction represented as positive downwards.

[0042] In one embodiment, the three-dimensional acoustic detection probability field can be expressed as: Its value range is .when The closer the value is to 1, the higher the risk of acoustic detection at the current location and / or time. The closer it is to 0, the stronger the concealment of the current position and / or the current time.

[0043] In one embodiment, when multiple acoustic detection sources exist, the detection probability field corresponding to each acoustic detection source can be obtained separately. Furthermore, a joint detection probability field is obtained through multi-source fusion. For example, under the condition that each detection event is independent, the joint detection probability can be expressed as: Where M represents the number of acoustic detection sources. Let be the detection probability field corresponding to the m-th acoustic detection source.

[0044] In this embodiment, the three-dimensional acoustic detection probability field can be a pre-constructed static detection probability field or a time-varying detection probability field that is updated according to the motion state of the acoustic detection source. When the three-dimensional acoustic detection probability field is a time-varying detection probability field, the autonomous underwater vehicle reads the detection probability value at the corresponding time for both state construction and reward calculation at time t.

[0045] In one embodiment, the autonomous underwater vehicle is simplified as a point mass model with a fixed step size, ignoring roll, pitch, sideslip and complex dynamic processes, while retaining the horizontal heading adjustment and vertical depth adjustment capabilities required for path planning.

[0046] The position state of an autonomous underwater vehicle at time t is defined as follows: in and These represent the position coordinates within the horizontal plane. This indicates the current depth of the AUV.

[0047] The maneuver consists of a combination of horizontal heading changes and vertical depth changes: in here, This indicates the amount of heading adjustment for the current step. Indicates rising. This indicates that the depth remains constant. This indicates a descent. This can be used to construct a set of discrete actions: In one embodiment, the autonomous underwater vehicle updates its position based on the current heading angle and the selected maneuver: in, and For horizontal and vertical step sizes.

[0048] To enable the autonomous underwater vehicle to simultaneously perform target approach, terrain obstacle avoidance, detection avoidance, and smooth maneuvering, this embodiment defines the combined state as follows: in, Indicates global navigation information. Indicates local topographic observation. Indicates local detection probability observation. This represents the agent's maneuvering information.

[0049] The global navigation state describes the current pose of the autonomous underwater vehicle and its position relative to the target, and can be represented as: in, These represent the axis-by-axis normalized offset of the AUV relative to the target point. Using a normalization method can improve the stability of the state input under different task scales.

[0050] The local terrain condition is obtained by extracting multi-scale local terrain windows centered on the current position of the autonomous underwater vehicle. Let the first... The radius of each scale window is Then the window side length is The local terrain condition can be represented as: The local detection probability state is obtained by extracting a multi-scale local detection probability window centered on the current position of the autonomous underwater vehicle: The maneuver state describes the maneuver information actually performed in the previous moment, and can be represented as: Where represents the normalized value of the actual steering angle increment at the previous moment, and represents the normalized value of the actual depth change at the previous moment.

[0051] like Figure 3 As shown, the TAM-HRL path planning model in this embodiment includes a low-level control network and a high-level decision network. The low-level control network is used to propose action preferences from different task perspectives, while the high-level decision network is used to coordinate the contributions of different low-level sub-policy networks based on the current state or historical state sequence.

[0052] The lower-level control network includes the collision avoidance network CA, the detection and avoidance network DA, and the maneuver control network CU.

[0053] The input to the collision avoidance network (CA) is the global navigation state and the local terrain state, namely: The input to the Detection Avoidance Network (DA) is the global navigation state and the local detection probability state, i.e.: The inputs to the maneuver control network (CU) are the global navigation state and the maneuver state, namely: The input state of the high-level decision network DM uses the complete state, i.e. The low-level control network outputs the action probability distributions respectively: The high-level decision network (DM) outputs sub-policy weight vectors based on the global states generated at multiple time points by the sequence generator: in, And each weight is not less than 0.

[0054] The final action probability distribution is obtained by weighted fusion using a fusion unit: During the training phase, random sampling can be performed based on the probability distribution of the final action before interacting with the environment to maintain exploratory nature; during the testing phase, the action with the highest probability can be selected as the final action to be executed.

[0055] like Figure 4 As shown, to enhance the low-level sub-policy network's ability to perceive the local environment, this embodiment incorporates a multi-scale convolutional feature extraction module in the collision avoidance network (CA) and the detection avoidance network (DA). This module includes a large-scale convolutional branch 1 and a small-scale convolutional branch 2.

[0056] Large-scale convolution branch 1 receives terrain patches or detection probability patches within a larger window range to extract contextual information such as far-field terrain undulation trends, threat coverage, and global detour directions. Small-scale convolution branch 2 receives terrain patches or detection probability patches within a smaller window range to extract detailed information such as near-field obstacles, local boundaries, local high detection probability regions, and fine risk gradients. Furthermore, the global contextual information extracted by large-scale branch 1 is injected into small-scale branch 2 through a connection, allowing the local branch to receive guidance on broader environmental changes while focusing on neighborhood details.

[0057] In one embodiment, large-scale convolutional branch 1 includes multiple convolutional layers, activation functions, and pooling layers, while small-scale convolutional branch 2 includes multiple convolutional layers, activation functions, and pooling layers. The feature vectors FC-1 and FC-2 output from the two branches after preprocessing are concatenated and then combined with the global navigation state. The input is a fully connected layer FC-3, and the final output is an optional policy in the corresponding action space.

[0058] With the above structure, autonomous underwater vehicles can both use large-scale environmental information to make advance detours and use small-scale environmental information to complete near-field fine-grained hazard avoidance, thereby improving the safety and robustness of path planning in complex environments.

[0059] like Figure 3 As shown, in one embodiment, the high-level decision network DM includes a GRU time series generator and a weight output layer. Since path planning is a continuous sequence decision-making process, relying solely on the current state may lead to frequent changes in the high-level weights. Therefore, this embodiment constructs a state sequence from the combined states of the most recent k time steps: Where a single-step observation can be represented as like Figure 5 As shown, the state update process of GRU is as follows: in, and These represent the update gate and reset gate in the GRU structure, respectively. The current hidden state. Based on the hidden state. The high-level temporal decision network outputs the fusion weight vector of each low-level sub-policy network: in, Indicates the first The fusion weights of the low-level sub-policy networks at time t.

[0060] By introducing GRU, the high-level decision network can integrate recent terrain changes, detection risk changes, and navigation maneuver trends, thereby more stably determining the contribution ratio of each low-level sub-policy network and reducing path oscillations caused by frequent sub-policy switching.

[0061] To guide the autonomous underwater vehicle in generating a safe, stealthy, and smooth 3D path, this embodiment constructs a composite reward function: in, As a distance reward, For collision constraint rewards, To detect and avoid rewards, For smooth reward of maneuver, As a reward for success, As a penalty for failure, Penalty for each step of time, , , , The weights for each reward item.

[0062] Distance rewards are used to encourage autonomous underwater vehicles to approach a target area. In one embodiment, the distance reward can be constructed based on the reduction in distance to the target area between two adjacent steps.

[0063] Collision constraint rewards are used to constrain autonomous underwater vehicles (AUVs) away from the seabed, sea surface, and horizontal boundaries. In one embodiment, a safety buffer zone can be set up to apply continuous penalties when the AUV approaches the seabed, sea surface, or horizontal boundary; and a failure termination penalty can be triggered when the AUV crosses the boundary, hits the bottom, or enters an unnavigable area.

[0064] Detection avoidance rewards are used to constrain autonomous underwater vehicles to avoid areas with high detection probability. In one embodiment, the detection probability can be based on the current position. and detection safety threshold Construct a reward system for detection and evasion. When... When the detection safety threshold is exceeded, the risk penalty is increased.

[0065] Maneuvering smoothing rewards are used to suppress sharp turns, frequent large depth changes, and reverse swaying. In one embodiment, maneuvering smoothing rewards can be constructed based on the current change in heading, the change in depth, and whether the directions of two adjacent heading changes are opposite.

[0066] Termination rewards and penalties are used to provide feedback on mission success or failure. When the autonomous underwater vehicle reaches the target area, a success reward is given; when it crosses the boundary, hits the bottom, exceeds the maximum number of steps, or the detection risk exceeds the termination conditions, a failure penalty is given.

[0067] In a hierarchical reinforcement learning architecture, the final action is obtained by weighted fusion of the action probability distributions of multiple low-level sub-policy networks. Since the final action after fusion may be influenced by multiple low-level sub-policy networks simultaneously, evenly distributing all interaction samples among them can easily lead to mixed learning objectives for the low-level sub-policy networks. This embodiment determines the dominant source of samples through a responsibility allocation coefficient, thereby reducing the interference of irrelevant samples on the training of low-level sub-policy networks.

[0068] To address the aforementioned issues, this embodiment proposes a responsibility allocation experience replay mechanism. For the selected actions at time t... Based on the fusion weights output by the high-level decision network and the probability outputs of each low-level sub-policy network for the action, the responsibility allocation coefficients of each low-level sub-policy network are calculated: in, The larger the value, the higher the value. The greater the contribution of each lower-level sub-policy network to the final action.

[0069] In one embodiment, the low-level sub-policy network with the largest responsibility allocation coefficient is selected as the dominant low-level sub-policy network, and the current interaction sample is stored in the experience replay buffer corresponding to that network. The experience replay buffer of the high-level decision network stores the complete combinatorial state, fusion weights, final action, total reward, next combinatorial state, and termination flag.

[0070] In another embodiment, the interaction samples can also be stored or updated in a weighted manner according to the responsibility allocation coefficient, so that the low-level sub-policy network with a larger contribution receives a higher training weight, and the low-level sub-policy network with a smaller contribution receives a lower training weight.

[0071] In one embodiment, both the low-level sub-policy network and the high-level decision network can be trained using a soft actor-critic reinforcement learning algorithm in the form of discrete actions. Each sub-policy network includes a policy network, a double-Q network, a target Q network, and a temperature parameter. The policy network outputs the discrete action probability distribution, the double-Q network estimates the action value, the target Q network constructs a stable target value, and the temperature parameter balances reward maximization and policy entropy maximization.

[0072] In each training round, the autonomous underwater vehicle (AUV) initializes from the starting point or starting area. The environment extracts a combined state based on its current position and inputs it into the TAM-HRL path planning model. The low-level sub-policy network outputs the action probability distribution, the high-level decision network outputs the fusion weights, and the action fusion module generates the final action probability distribution and samples the actions. After executing an action, the environment returns the next state, reward, and termination flag. The training optimization module updates the network parameters based on samples in the experience replay buffer.

[0073] Once training is complete, during the testing or practical application phase, the system reads the current combined state, the TAM-HRL path planning model outputs the final action probability distribution, and selects the action with the highest probability as the action to be executed. This process is repeated until the autonomous underwater vehicle reaches the target area or triggers the termination condition. The final output is a three-dimensional path composed of multiple waypoints.

[0074] Second Embodiment This embodiment proposes a three-dimensional covert path planning system for underwater vehicles based on TAM-HRL, which is used to implement the three-dimensional covert path planning method for underwater vehicles as in Embodiment 1. The system includes an environmental data acquisition module, a state construction module, a hierarchical time-series decision model, a training and optimization module, and a path output module.

[0075] The environmental data acquisition module is used to acquire the three-dimensional topographic field and the three-dimensional acoustic detection probability field of the target water area. The three-dimensional topographic field can come from topographic survey data, nautical chart data, or simulation data; the three-dimensional acoustic detection probability field can come from acoustic propagation simulation, detection probability calculation, sensor observation, or external input.

[0076] The state construction module is used to construct a combined state based on the current position of the autonomous underwater vehicle, the target position, local terrain information, local detection probability information, and historical maneuver information.

[0077] The hierarchical temporal decision-making model includes a low-level sub-policy decision-making module, a high-level decision-making module, and an action fusion module. The low-level sub-policy decision-making module includes an obstacle avoidance sub-policy unit, a detection and avoidance sub-policy unit, and a maneuver control sub-policy unit. The obstacle avoidance sub-policy unit outputs the probability distribution of obstacle avoidance actions, the detection and avoidance sub-policy unit outputs the probability distribution of detection and avoidance actions, and the maneuver control sub-policy unit outputs the probability distribution of smooth maneuvering actions. The high-level decision-making module outputs the fusion weights of each low-level sub-policy unit based on the combined state or state sequence. In one embodiment, the high-level decision-making module includes a GRU temporal encoder and a weight output layer. The action fusion module performs weighted fusion of multiple action probability distributions based on the fusion weights to obtain the final action probability distribution and determine the motion actions of the autonomous underwater vehicle.

[0078] The training and optimization module is used to train the low-level sub-policy decision module and the high-level decision module based on the reward function and samples in the experience replay buffer.

[0079] The path output module is used to output the three-dimensional path of the autonomous underwater vehicle from the starting point to the target area, and can store path points, detection probabilities, terrain safety distances and maneuver sequences.

[0080] Preferably, the environmental data acquisition module includes: A terrain data interface is used to connect to external terrain surveying equipment or read locally stored three-dimensional digital elevation model data to form the three-dimensional terrain field; An acoustic field data interface is used to receive three-dimensional acoustic detection probability data output from an acoustic propagation simulation model or fed back from a sonar detection system, in order to construct the three-dimensional acoustic detection probability field.

[0081] Preferably, the state construction module includes: The navigation status acquisition unit is used to acquire the current position, target position, and heading angle information of the underwater vehicle in real time; The multi-scale window extraction unit is used to extract at least two two-dimensional local windows of different scales from the three-dimensional topographic field and the three-dimensional acoustic detection probability field, respectively, with the current position of the underwater vehicle as the center, and then flatten and stitch the windows of each scale to form a local state vector. The maneuvering status recording unit is used to store the actual changes in heading and depth performed by the underwater vehicle at the previous moment.

[0082] Preferably, the training optimization module includes: The composite reward calculation unit is used to calculate the instantaneous reward value of the current step based on distance approach feedback, collision constraint feedback, detection avoidance feedback and maneuver smoothing feedback; The experience replay buffer group contains multiple independent experience replay buffers, and each experience replay buffer corresponds one-to-one with the multiple low-level policy networks. The parameter update unit is used to randomly sample training samples from the experience replay buffer group and update the network parameters of each low-level policy network and the high-level temporal decision network using gradient descent.

[0083] Experimental verification To verify the effectiveness of the method of the present invention, path planning tests of the single SAC method and the TAM-HRL-SAC method proposed in this invention were conducted in a three-dimensional simulation environment containing complex seabed topography and a three-dimensional acoustic detection probability field.

[0084] (1) Construction of the experimental platform The experimental site adopts Figure 6 The three-dimensional terrain and detection scenario shown (geographic location n30.0_s25.0_w125.0_e130.0) are illustrated in Figure (a), which shows the starting area, target area, and fixed sonar detection range of the test scenario. Figure (b) shows the three-dimensional model of the seabed terrain, which can realistically reflect the seabed undulations.

[0085] (2) Comparison Algorithm The following path planning algorithms were selected for comparison in the experiment: SAC (Soft Actor-Critic): A single reinforcement learning agent trained by combining distance reward, collision constraint, detection avoidance and smooth maneuver reward. TAM-HRL-SAC: The temporal-aware multi-scale hierarchical reinforcement learning method proposed in this invention includes an obstacle avoidance sub-policy network, a detection and avoidance sub-policy network, and a maneuver control sub-policy network, with a high-level GRU network performing dynamic fusion weight decision-making.

[0086] (3) Experimental setup and evaluation indicators The experiment was set up in two phases: a training phase and a testing phase.

[0087] Training phase: Each algorithm undergoes 3000 training rounds; in each round, the agent is randomly initialized from the starting region.

[0088] Testing phase: Using the trained model, output a 3D path from the starting point to the target area, and record the detection probability, depth, and heading information of the path points.

[0089] To facilitate a quantitative comparison of the training results, the following key performance indicators (KPIs) are used to conduct a detailed comparison of the algorithms: (a) Convergence Time (CT): The number of training epochs at which the success rate first reaches 70%.

[0090] (b) Convergence stability (SC): Standardized deviation of the success rate in the last 1000 rounds of training.

[0091] (c) Success Rate After Convergence (SRC): The average success rate over the last 1000 rounds of training.

[0092] (d) Average reward after convergence (ARC): The average reward of the last 1000 rounds of the training process.

[0093] (e) Normalized Cumulative Detection Risk (NCDR): The undetected probability of the entire path is converted into the equivalent single-step undetected probability, resulting in the equivalent single-step detection risk: in, Indicates the path where AUVs are generated. This represents the actual number of moves in the current round. Indicates that AUV is in the first The detection probability at each step position. Compared to directly accumulating detection probabilities, normalized cumulative detection probabilities can more fairly compare the concealment performance of different algorithms under different path lengths.

[0094] (4) Comparative analysis of experimental results Under the simulation conditions of this example, such as Figure 7 As shown in the left figure, the average reward curve of the TAM-HRL-SAC algorithm exhibits faster convergence and lower volatility compared to the single SAC algorithm. Figure 7 As shown in the figure on the right, the normalized cumulative detect risk (NCDR) curve of the TAM-HRL-SAC algorithm decreases faster and more thoroughly, demonstrating superior concealment performance compared to the single SAC algorithm.

[0095] Figures 8-10 and Figures 14-16 The early and mid-stages of two-dimensional and three-dimensional path planning are shown respectively. The TAM-HRL-SAC algorithm chooses to avoid the main acoustic threat area and take the right-hand channel with a large detection area gap, while the SAC algorithm chooses to take the left-hand channel with a small detection area gap. Figures 11-13 and Figures 17-19The later stages of two-dimensional and three-dimensional path planning were demonstrated respectively. The TAM-HRL-SAC algorithm showed a smooth overall navigation path with moderate vertical fluctuations, taking into account both depth changes and heading adjustments. In contrast, the SAC algorithm not only made significant turns but also showed a tendency to cross certain high-risk areas.

[0096] Table 1

[0097] Table 1 shows that the experimental results indicate that TAM-HRL-SAC outperforms the single SAC algorithm in terms of convergence speed, strategy stability, and success rate, and can generate safer, more concealed, and smoother 3D paths in complex terrain and non-uniform acoustic risk environments.

[0098] In summary, the TAM-HRL-based 3D path planning method for autonomous underwater vehicles proposed in this invention can: improve training convergence speed and shorten CT; enhance strategy stability and reduce SC; significantly improve success rate SRC and average round reward ARC; in terms of stealth performance, it demonstrates effective avoidance of acoustic threats through the normalized cumulative detection risk index NCDR; and generate smooth and reasonable 3D navigation paths, taking into account the comprehensive performance of terrain obstacle avoidance, detection avoidance, and smooth maneuvering.

[0099] Without departing from the concept of the present invention, the present invention may have a variety of alternative implementation methods.

[0100] In an alternative implementation, the number of low-level sub-policy networks is not limited to three, and can be expanded to include energy consumption control sub-policy networks, communication maintenance sub-policy networks, trajectory tracking sub-policy networks, or other task sub-policy networks, depending on mission requirements.

[0101] In an alternative implementation, the three-dimensional acoustic detection probability field can be a static field or a dynamic field; as the position of the detection source changes over time, the three-dimensional acoustic detection probability field can be read according to a time index or updated in real time.

[0102] In an alternative implementation, the high-level decision network can employ GRU, LSTM, Transformer encoder, or temporal convolutional network to extract temporal features.

[0103] In an alternative implementation, the action space can be a discrete action space or extended to a continuous action space; when a continuous action space is used, each low-level sub-policy network outputs a continuous action distribution, and the high-level decision network fuses the continuous action distribution or the action mean.

[0104] In an alternative implementation, the experience replay mechanism may employ a maximum responsibility allocation method, a weighted responsibility allocation method, or a threshold-screened responsibility allocation method.

[0105] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0106] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0107] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL, characterized in that, include: Obtain the three-dimensional constraint field corresponding to the target water area. The three-dimensional constraint field includes a three-dimensional topographic field and a three-dimensional acoustic detection probability field. The current position of the underwater vehicle, the target position, local terrain information, local detection probability information, and historical maneuver information are determined based on the three-dimensional constraint field and constructed as a combined state. The combined state is input into a pre-constructed time-sharing decision model, which includes multiple low-level sub-policy networks and a high-level time-series decision network. The action probability distribution is output by multiple low-level sub-policy networks, and the fusion weight of each low-level sub-policy network is output by a high-level temporal decision network based on the combined state of multiple consecutive time points. The probability distributions of multiple actions are weighted and fused according to the fusion weights to obtain the final action probability distribution in the preset action space. The autonomous underwater vehicle's motion is determined based on the final motion probability distribution, and the hierarchical temporal decision model is trained based on the composite feedback signal until a three-dimensional acoustic concealment path from the starting point to the target area is output. The preset action space includes a discrete combination of horizontal heading change and vertical depth change; wherein the horizontal heading change includes at least one of left turn, maintaining heading, and right turn, and the vertical depth change includes at least one of surfacing, maintaining depth, and diving.

2. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 1, characterized in that, The combined states include global navigation state, local terrain state, local detection probability state, and maneuver state. Specifically, The global navigation status includes the underwater vehicle's current position, relative target position, and heading information, including the following formula: in, This indicates the current location of the underwater vehicle. These represent the axis-by-axis normalized offsets of the underwater vehicle relative to the target point. The horizontal heading angle of the underwater vehicle; The local terrain state and the local detection probability state are respectively extracted through multi-scale windows: taking the current position of the autonomous underwater vehicle as the center, at least one scale of two-dimensional local window is extracted from the three-dimensional terrain field to form the local terrain state, and at least one scale of two-dimensional local window is extracted from the three-dimensional acoustic detection probability field to form the local detection probability state. The maneuver status includes the actual changes in heading and depth performed by the underwater vehicle at the previous moment.

3. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 1, characterized in that, The multiple low-level sub-strategy networks include an obstacle avoidance sub-strategy network, a detection and avoidance sub-strategy network, and a maneuver control sub-strategy network; The obstacle avoidance sub-strategy network is used to output an obstacle avoidance action probability distribution based on the global navigation state and the local terrain state; The detection avoidance sub-strategy network is used to output a probability distribution of detection avoidance actions based on the global navigation state and the local detection probability state. The maneuver control sub-policy network is used to output a smooth maneuver probability distribution based on the global navigation state and the maneuver state.

4. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 3, characterized in that, The obstacle avoidance sub-strategy network and the detection avoidance sub-strategy network include a multi-scale convolutional feature extraction module; The multi-scale convolutional feature extraction module includes large-scale convolutional branches and small-scale convolutional branches; The large-scale convolutional branch is used to receive environmental data within a large window range, extract far-field terrain undulation trends and / or detect risk distribution trends; The small-scale convolutional branch is used to receive environmental data within a small window range and extract detailed features of near-field obstacles, local boundaries and / or local high-probability detection areas. The global context features output by the large-scale convolutional branch are fused with the local detail features output by the small-scale convolutional branch to obtain environment-aware features for action decision-making.

5. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 1, characterized in that, The high-level decision-making network includes a gated recurrent unit (GRU). The high-level decision network receives a state sequence consisting of combined states at multiple consecutive time points, extracts temporal features through the gated recurrent unit (GRU), and outputs the fusion weights of the multiple low-level sub-policy networks based on the temporal features. The final action probability distribution is weighted and fused from the action probability distributions output by each low-level policy network according to the following formula: Where N is the number of low-level sub-policy networks. Let be the combination state at time t. Let be the input state of the i-th lower-level sub-policy network. The fusion weights of the i-th low-level sub-policy network output by the high-level temporal decision network are: Let i be the action probability distribution output by the i-th low-level sub-policy network. This represents the final action probability distribution after fusion.

6. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 1, characterized in that, The composite feedback signal includes: Proximity feedback is used to guide underwater vehicles toward a target area; Collision constraint feedback is used to constrain the distance between underwater vehicles and the seabed, sea surface, and horizontal boundaries; Detection avoidance feedback is used to penalize behaviors that enter areas with a high probability of detection. Maneuvering smoothing feedback is used to suppress drastic changes in course and depth.

7. The three-dimensional stealth path planning method for underwater vehicles based on TAM-HRL according to claim 1, characterized in that, During the training of the hierarchical time-series decision model, a responsibility allocation operation is also performed: Based on the fusion weights output by the high-level temporal decision network and the output probabilities of each low-level policy network for the executed actions, the responsibility allocation coefficient of each low-level policy network for this decision is calculated as follows: in, For the first The responsibility allocation coefficients of the low-level sub-policy network at time t. For high-level time-series decision networks at time... The output of the first Each fusion weight, For the first A low-level sub-policy network for the selected action The output probability, ε is a constant to prevent the denominator from being zero; when the k-th low-level sub-policy network satisfies The current environment interaction samples are used as training samples for the k-th low-level sub-policy network. The current environmental interaction samples are stored in the experience replay buffer corresponding to the low-level policy network with the largest responsibility allocation coefficient, so that each low-level policy network can update its parameters based on the samples it generates.

8. A three-dimensional stealth path planning system for underwater vehicles based on TAM-HRL, used to implement the three-dimensional stealth path planning method for underwater vehicles as described in any one of claims 1-7, characterized in that, The system includes: An environmental data acquisition module is used to acquire a three-dimensional constraint field corresponding to the target water area. The three-dimensional constraint field includes a three-dimensional topographic field and a three-dimensional acoustic detection probability field. The state construction module is used to construct a combined state based on the current position of the underwater vehicle, the target position, and local terrain information and local detection probability information extracted from the three-dimensional constraint field, combined with historical maneuver information. The hierarchical temporal decision model includes multiple low-level policy networks and one high-level temporal decision network. The multiple low-level policy networks are used to output action probability distributions respectively. The high-level temporal decision network is used to output the fusion weights of each low-level policy network based on the combined state of multiple consecutive time points, and to perform weighted fusion of multiple action probability distributions according to the fusion weights to obtain the final action probability distribution in the preset action space. The training optimization module is used to train the hierarchical time-series decision model based on the composite feedback signal. The path output module is used to output a 3D hidden path from the starting point to the target region using the trained hierarchical temporal decision model.

9. The three-dimensional stealth path planning system for underwater vehicles based on TAM-HRL according to claim 8, characterized in that, The state construction module includes: The navigation status acquisition unit is used to acquire the current position, target position, and heading angle information of the underwater vehicle in real time; The multi-scale window extraction unit is used to extract at least two two-dimensional local windows of different scales from the three-dimensional topographic field and the three-dimensional acoustic detection probability field, respectively, with the current position of the underwater vehicle as the center, and then flatten and stitch the windows of each scale to form a local state vector. The maneuvering status recording unit is used to store the actual changes in heading and depth performed by the underwater vehicle at the previous moment.

10. The three-dimensional stealth path planning system for underwater vehicles based on TAM-HRL according to claim 8, characterized in that, The training optimization module includes: The composite reward calculation unit is used to calculate the instantaneous reward value of the current step based on distance approach feedback, collision constraint feedback, detection avoidance feedback and maneuver smoothing feedback; The experience replay buffer group contains multiple independent experience replay buffers, and each experience replay buffer corresponds one-to-one with the multiple low-level policy networks. The parameter update unit is used to randomly sample training samples from the experience replay buffer group and update the network parameters of each low-level policy network and the high-level temporal decision network using gradient descent.

Images