A Deep Reinforcement Learning-Based Adaptive Path Planning Method for Unmanned Underwater Vehicles in Multi-Constraint Environments

Abstract

This invention relates to the field of intelligent path planning technology, specifically to an adaptive path planning method for unmanned underwater vehicles (UUVs) in multi-constraint environments based on deep reinforcement learning. The method includes the following steps: S1: Encoding the augmented state vector at the current moment; S2: Calculating the immediate reward based on the augmented state vector and the actions performed by the UUV; S3: Initializing a dual-deep Q-network based on a Dueling architecture as the policy network; S4: Obtaining a new augmented state vector and immediate reward; S5: Calculating the target Q-value using the target network and policy network, and updating the policy network parameters through a loss function; S6: The action sequence output by the policy network constitutes the final adaptively planned path. This invention, through the synergistic guidance of augmented state modeling and multi-objective composite rewards, combined with a stable and efficient deep reinforcement learning update mechanism, achieves an overall improvement in the safety and path planning quality of UUVs in multi-constraint dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent path planning technology, and in particular to an adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning. Background Technology

[0002] In complex marine environments, when unmanned underwater vehicles (UUVs) perform autonomous missions, their path planning not only needs to avoid static and dynamic obstacles, but also needs to adapt in real time to changing ocean currents, energy constraints, and mission objective requirements. Path planning tasks exhibit characteristics such as multiple constraints, high dynamics, and strong nonlinearity. In recent years, with the development of deep reinforcement learning technology, its application in UUV path planning has gradually emerged. Learning state-action mapping relationships to achieve autonomous decision-making and control has become an important direction for intelligent path generation.

[0003] However, most existing reinforcement learning methods generally employ simple state modeling and a single reward-driven approach, lacking detailed modeling and feedback on key task elements such as environmental evolution trends and energy sufficiency. This leads to significant problems of policy instability and poor path quality in highly dynamic environments. Specifically, current path planning reinforcement learning schemes suffer from the following technical bottlenecks: First, the state space representation has limited dimensions, failing to reflect obstacle approach trends and dynamic changes in energy consumption constraints, making it difficult for the agent to identify risk evolution and energy shortages in advance. Second, reward mechanisms are generally sparse and singular, lacking multi-objective guidance encompassing safety, energy efficiency, and path progress, resulting in unclear policy optimization directions and low training efficiency. Third, the policy network update mechanism does not adequately utilize key experiences, easily getting trapped in local optima and exhibiting slow training convergence. Therefore, there is an urgent need for an adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning to address these issues. Summary of the Invention

[0004] To achieve the above objectives, this invention provides an adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning.

[0005] A deep reinforcement learning-based adaptive path planning method for unmanned underwater vehicles in multi-constraint environments includes the following steps: S1: Acquire the real-time status and environmental information of the unmanned underwater vehicle, and calculate the obstacle proximity trend characteristics and energy-range ratio characteristics, which are jointly encoded to form the enhanced state vector at the current moment; S2: Based on the enhanced state vector and the actions performed by the unmanned underwater vehicle, calculate the immediate reward obtained by weighted summation of safety reward, energy efficiency reward, progress reward and goal achievement reward; S3: Initialize a dual-depth Q-network based on the Dueling architecture as the policy network, and construct the corresponding target network and priority experience replay pool; S4: Input the enhanced state vector into the policy network to select an action. After execution, a new enhanced state vector and an immediate reward are obtained, forming a transition sample. After calculating its priority, it is stored in the priority experience replay pool. S5: Sample samples from the priority experience replay pool according to priority, calculate the target Q value using the target network and policy network, update the policy network parameters through the loss function, and recalculate the sample priority; S6: Periodically synchronize the parameters of the policy network and the target network, and use the action sequence output by the policy network to form the final adaptive planning path.

[0006] Optionally, S1 specifically includes: S11: Acquire real-time status information and environmental perception information of the unmanned underwater vehicle; real-time status information includes current position coordinates, heading angle, speed and remaining energy; environmental perception information includes static obstacle positions, dynamic obstacle positions and velocities, current ocean current velocity vector field and target point position, all information is synchronized with the current time t as the reference; S12: Based on the obstacle distance difference between the current and previous time, calculate the obstacle proximity trend characteristics that reflect the changing trend of obstacle avoidance pressure; S13: Calculate the energy-range ratio characteristic based on the ratio between the current remaining energy and the remaining range; S14: The real-time state information and environmental perception information obtained in S11, together with the two feature values ​​calculated in S12 and S13, are structured and encoded to form the enhanced state vector at the current moment.

[0007] Optionally, S2 specifically includes: S21: After the unmanned underwater vehicle performs the current action, call the current time-based enhanced state vector output by S1; S22: Calculate the safety reward based on the obstacle proximity trend features contained in the enhanced state vector; S23: Calculate the energy efficiency bonus based on the energy consumption of the unmanned underwater vehicle under the current action and the energy-range ratio feature in the enhanced state vector; S24: Calculate the progress reward based on the displacement change of the unmanned underwater vehicle relative to the target point before and after the current action is performed; S25: Based on the distance relationship between the current position of the unmanned underwater vehicle and the target point, determine whether the target arrival condition is met; when the unmanned underwater vehicle reaches the target point, generate a preset maximum positive target achievement reward item, otherwise the target achievement reward item is zero; S26: According to the preset weighting coefficients, the safety reward items, energy efficiency reward items, progress reward items and target achievement reward items obtained from S22 to S25 are weighted and summed to obtain the instant reward value at the current moment.

[0008] Optionally, S26 specifically includes: S261: Pre-set the weight coefficients for safety rewards, energy efficiency rewards, progress rewards, and target achievement rewards, denoted as... , and The weighting coefficients are all non-negative real numbers and satisfy the normalization constraint conditions: ; S262: Call the safety reward items, energy efficiency reward items, progress reward items, and goal achievement reward items from S22 to S25 as reward component inputs at the same decision moment; S263: By linearly weighting and summing each reward item according to the one-to-one correspondence between each reward component and its corresponding weight coefficient, the instantaneous reward value at the current moment is obtained. .

[0009] Optionally, S3 specifically includes: S31: Construct a policy network based on the Dueling architecture, which includes an input layer, a feature extraction layer, a state value branch, and an action advantage branch; S32: Copy the structure of the policy network and parameterize it into the target network, maintaining the same hierarchical structure and initialization method as the policy network, and use it to output the target Q value during training; S33: Initialize the priority experience replay pool, set a fixed capacity, and assign an initial priority value to each sample; the experience replay pool supports sampling and updating operations according to sample priority.

[0010] Optionally, S33 specifically includes: S331: When initializing the priority experience replay pool, a fixed capacity limit is set. When a new sample arrives and the capacity is full, the lowest priority sample is replaced according to the set replacement strategy. S332: When writing each newly generated state transition sample into the replay pool, an initial priority value is assigned to the sample. The initial priority is calculated based on the absolute value of the temporal difference error corresponding to the sample in the current policy network. The larger the error, the higher the corresponding priority. S333: When drawing training samples from the replay pool, calculate the sampling probability of each sample based on its current priority value; S334: After each round of policy network parameter updates, recalculate the temporal difference error of the samples used in this round, and update their priority value in the replay pool based on the error value.

[0011] Optionally, S4 specifically includes: S41: Call the current time-increased state vector output by S1, input the enhanced state vector into the policy network initialized by S3, and obtain the action value evaluation results of each candidate action; S42: Determine the action at the current moment based on the action value assessment result and the preset ε-greedy action selection rule. Specifically, when the exploration selection conditions are met, select the exploration action from the preset action set; when the exploration selection conditions are not met, select the action with the highest action value assessment result as the current action. S43: Control the unmanned underwater vehicle to perform actions, obtain the environmental feedback information corresponding to the execution result after the execution is completed, and execute S1 again to generate the enhanced state vector for the next moment. At the same time, calculate the instant reward corresponding to this execution based on S2. S44: Encapsulate the current augmented state vector, current action, immediate reward, and next augmented state vector in a structured manner to form a state transition sample; S45: Calculate the absolute value of the time difference error based on the state transition sample, and use this absolute value as the basis for calculating the sample priority to obtain the sample priority value; S46: Write the state transition sample and its corresponding priority value into the priority experience replay pool.

[0012] Optionally, S5 specifically includes: S51: Calculate the sampling probability from the priority experience replay pool based on the priority value corresponding to each state transition sample, and extract a fixed number of state transition samples according to the sampling probability. S52: For each state transition sample in the mini-batch sample set, call the policy network to evaluate the action value of its next time step augmented state vector, and select the action with the highest action value as the target action corresponding to the target Q value calculation; S53: Call the target network, calculate the corresponding target Q value based on the target action determined in S52, and combine the immediate reward with the target Q value to obtain the target Q value corresponding to each state transition sample; S54: Call the policy network, calculate the predicted Q value of the state transition sample under the current policy network parameters, and compare the predicted Q value with the target Q value obtained in S53 to construct a loss function to measure the difference between the two. S55: Based on the loss function, the gradient of the parameters of each layer of the policy network is calculated through the backpropagation mechanism, and the policy network parameters are updated by gradient descent to reduce the deviation between the predicted Q value and the target Q value. S56: After completing the policy network parameter update, recalculate the temporal difference error corresponding to the state transition sample participating in the training in this round, and update its priority value in the priority experience replay pool based on the temporal difference error.

[0013] Optionally, S55 specifically includes: S551: For the small batch sample set determined by S51, call the loss function constructed by S54, calculate the deviation between the predicted Q value and the target Q value corresponding to each state transition sample, and use the deviation as the error signal input for backpropagation calculation; S552: Using the loss function as the optimization objective, the error signal is propagated backward along the network layers starting from the output layer of the policy network. The partial derivatives of the loss function with respect to the weight parameters and bias parameters of each layer are calculated layer by layer to obtain the gradient set of the parameters of each layer of the policy network. S553: ​​Summarize the gradient sets corresponding to each state transition sample in the mini-batch sample set to generate the batch gradient for this round of parameter update; S554: Based on the preset learning rate, the gradient descent method is used to update the parameters of the policy network. The parameters of each layer are corrected in the opposite direction of their gradient, so that the predicted Q value output by the policy network converges to the target Q value, thereby reducing the deviation between the predicted Q value and the target Q value.

[0014] Optionally, S6 specifically includes: S61: Set a fixed parameter synchronization period threshold. Whenever the policy network completes a continuous After this parameter update operation, a target network parameter synchronization operation is immediately triggered; S62: When a synchronization operation is triggered, the parameter values ​​of each layer in the policy network are copied and overwritten to the corresponding layer in the target network, so that the structural parameters between the policy network and the target network are consistent. S63: During the continuous execution of actions and interaction with the environment by the unmanned underwater vehicle, the optimal action decision output by the policy network at each state moment is recorded in real time and stored sequentially according to the state transition order. S64: When the unmanned underwater vehicle reaches the target point or the mission termination condition is met, extract the complete action sequence and use it as the final adaptive path planning result.

[0015] The beneficial effects of this invention are: This invention improves the ability to express key features in multi-constrained dynamic environments by introducing an enhanced state vector that includes obstacle proximity trends and energy-range ratios. This enables the agent to perceive changes in environmental risks and energy usage stress in advance, enhancing the foresight and safety of the path planning process. Combined with a phased weighted composite reward mechanism, it achieves multi-objective collaborative guidance, effectively solving the learning dilemma under sparse rewards and improving policy learning efficiency and path quality.

[0016] This invention enhances the learning and utilization of high-value samples by using a dual-depth Q-network based on the Dueling architecture and a priority experience replay mechanism. By separating the calculation of the target Q-value and updating the parameters synchronously, the stability and convergence speed of the training process are improved, ensuring that the unmanned underwater vehicle generates high-quality path planning results with safety, energy efficiency and mission completion capabilities in complex environments with multiple constraints. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the adaptive path planning method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the network parameter update strategy process according to an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0020] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0021] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0022] like Figures 1-2As shown, the adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning includes the following steps: S1: Acquire the real-time status and environmental information of the unmanned underwater vehicle, and calculate the obstacle proximity trend characteristics and energy-range ratio characteristics, which are jointly encoded to form the enhanced state vector at the current moment; S2: Based on the enhanced state vector and the actions performed by the unmanned underwater vehicle, calculate the immediate reward obtained by weighted summation of safety reward, energy efficiency reward, progress reward and goal achievement reward; S3: Initialize a dual-depth Q-network based on the Dueling architecture as the policy network, and construct the corresponding target network and priority experience replay pool; S4: Input the enhanced state vector into the policy network to select an action. After execution, a new enhanced state vector and an immediate reward are obtained, forming a transition sample. After calculating its priority, it is stored in the priority experience replay pool. S5: Sample samples from the priority experience replay pool according to priority, calculate the target Q value using the target network and policy network, update the policy network parameters through the loss function, and recalculate the sample priority; S6: Periodically synchronize the parameters of the policy network and the target network, and use the action sequence output by the policy network to form the final adaptive planning path.

[0023] S1 specifically includes: S11: Acquire real-time status information and environmental perception information of the unmanned underwater vehicle to support subsequent augmented state construction; real-time status information includes current position coordinates, heading angle, speed and remaining energy; environmental perception information includes static obstacle positions, dynamic obstacle positions and velocities, current ocean current velocity vector field and target point position, all information is synchronized with the current time t as the reference; S12: Based on the obstacle distance difference between the current and previous time steps, calculate the obstacle proximity trend feature, which reflects the changing trend of obstacle avoidance pressure, to represent the evolution of obstacle risk; this feature is defined as: ,in, Obstacle proximity trend characteristics; The Euclidean distance between the unmanned underwater vehicle and the nearest obstacle at the current moment; This is the Euclidean distance corresponding to the previous time step; S13: Based on the ratio between current remaining energy and remaining range, calculate the energy-range ratio characteristic to measure energy sufficiency; this characteristic is defined as: ,in, This is a characteristic of the energy-to-range ratio; The remaining energy at the current moment; This represents the Euclidean distance from the current location to the target point. S14: The real-time state information and environmental perception information obtained in S11, together with the two feature values ​​calculated in S12 and S13, are structured and encoded to form the enhanced state vector at the current moment, which serves as the input state for subsequent action decisions and Q-value calculations. The above steps reflect the speed of risk change by using the obstacle proximity trend and the energy-range ratio to reflect the task coverage capability of remaining energy. This enables the enhanced state vector to not only have the ability to express information at the current moment, but also incorporates predictive cognition of future obstacle avoidance pressure and energy efficiency regulation, thereby improving the agent's foresight and policy robustness in dynamic multi-constraint environments.

[0024] S2 specifically includes: S21: After the unmanned underwater vehicle performs the current action, call the current enhanced state vector output by S1 and obtain the state transition result corresponding to the action to support the item-by-item calculation of the immediate reward; S22: Calculate the safety reward based on the obstacle proximity trend features contained in the enhanced state vector; S23: Calculate the energy efficiency bonus based on the energy consumption of the unmanned underwater vehicle under the current action and the energy-range ratio feature in the enhanced state vector; S24: Calculate the progress reward based on the displacement change of the unmanned underwater vehicle relative to the target point before and after the current action is performed; S25: Based on the distance relationship between the current position of the unmanned underwater vehicle and the target point, determine whether the target arrival condition is met; when the unmanned underwater vehicle reaches the target point, generate a preset maximum positive target achievement reward item, otherwise the target achievement reward item is zero; S26: According to the preset weight coefficients, the safety reward, energy efficiency reward, progress reward, and goal achievement reward obtained from S22 to S25 are weighted and summed to obtain the instant reward value at the current moment, and the instant reward value is output for subsequent training and updating of the policy network. The above steps decompose safety, energy efficiency, path progress, and goal achievement state into mutually independent and controllable reward items, and perform unified calculation and weighted fusion based on the enhanced state vector, so that the instant reward can comprehensively reflect the decision quality of the unmanned underwater vehicle in a multi-constraint environment, thereby providing a stable, continuous, and clearly directional learning signal for the reinforcement learning process.

[0025] The calculation principle for safety bonus items is as follows: If the proximity trend between the unmanned underwater vehicle and the nearest obstacle is negative, that is... This indicates that the device is approaching an obstacle and is penalized with a negative value. The expression is: ,in , which is a negative weighting coefficient; if a collision occurs, i.e., the current distance If the value is below the threshold, it is directly assigned a maximum negative constant, expressed as: ; Example as follows: ,like ,but If a collision occurs, .

[0026] The calculation principle for energy efficiency bonuses is as follows: Energy efficiency bonuses consider two factors: energy consumption per unit of motion and energy-to-range ratio. The higher the energy consumption value, the greater the penalty; conversely, the higher the energy-range ratio, the greater the reward. The expression is: ,in, The estimated energy consumed by this action; ; ; Example as follows: Let , If the current action consumes energy ,but .

[0027] The principle behind assigning values ​​to progress reward items is as follows: The reward is positively correlated with the forward displacement in the target direction. A positive reward is given if the Euclidean distance to the target point decreases after the movement, and a negative reward is given if it increases. The expression is: ,in, These represent the distances from the unmanned underwater vehicle to the target point at the previous and current moments, respectively. ; Example as follows: Let If the distance to the target point decreases by 0.6m, then ; The principle behind assigning values ​​to reward items for achieving goals is as follows: If the current distance to the target point is less than the set distance threshold If the goal is deemed to have been reached, a fixed maximum positive reward is given, expressed as follows: ; Example as follows: If =0.5m, then If the current distance to the target is 0.3m, then .

[0028] S26 specifically includes: S261: Pre-set the weight coefficients for safety rewards, energy efficiency rewards, progress rewards, and target achievement rewards, denoted as... , and The weighting coefficients are all non-negative real numbers and satisfy the normalization constraint: ; S262: Call the safety reward items, energy efficiency reward items, progress reward items, and goal achievement reward items from S22 to S25 as reward component inputs at the same decision moment; S263: By linearly weighting and summing each reward item according to the one-to-one correspondence between each reward component and its corresponding weight coefficient, the instantaneous reward value at the current moment is obtained. Its calculation expression is: The above steps introduce normalized weights and perform linear weighted fusion on four reward categories: safety, energy efficiency, progress, and goal achievement. This ensures that the instantaneous reward value remains stable and controllable on a numerical scale, and can comprehensively reflect the overall quality of a single decision made by an unmanned underwater vehicle under multiple constraints.

[0029] S3 specifically includes: S31: Construct a policy network based on the Dueling architecture, which includes an input layer, a feature extraction layer, a state value branch, and an action advantage branch. The input layer is used to receive the enhanced state vector at the current time step. The feature extraction layer consists of a multi-layer fully connected neural network and is used to extract high-order feature representations of the state. The state value branch and the action advantage branch are constructed with independent fully connected network structures to output the state value and the advantage value of each possible action in parallel. S32: Copy the structure of the policy network and parameterize it into the target network, maintaining the same hierarchical structure and initialization method as the policy network. This is used to output the target Q value during training, avoiding training instability caused by policy updates. S33: Initialize the priority experience replay pool, set a fixed capacity to store the state transition samples generated by S4, and assign an initial priority value to each sample; the experience replay pool supports sampling and updating operations according to sample priority to support the key learning of key experiences in subsequent training; the above steps, by adopting a Dueling architecture to divide state value and action advantage flow, can effectively improve the expressive power of Q-value estimation and training stability; replicating the policy network structure to generate the target network helps to stabilize the calculation of target values ​​during training; introducing the priority experience replay pool can enhance the utilization efficiency of high-value learning samples and accelerate the policy convergence speed of the unmanned underwater vehicle in a multi-constraint environment.

[0030] The core function of the policy network is to evaluate the merits of different actions chosen by the unmanned underwater vehicle in the current state based on the enhanced state vector at the current moment, and to guide the action selection accordingly. Specifically, the policy network receives the enhanced state vector constructed by S1 as input. This enhanced state vector already contains the unmanned underwater vehicle's own state, environmental information, obstacle proximity trend features, and energy-range ratio features. The policy network first processes the state vector through a feature extraction layer, converting the original numerical information into a high-level feature representation that can reflect the relationship between environmental risk, energy status, and target direction. After feature extraction, the policy network decomposes the decision-making process into two parallel parts using the Dueling architecture: one part evaluates the overall value of the current state itself, and the other part evaluates the advantage of different actions relative to the average level in the current state. Through this decomposition, the policy network can distinguish between two different questions: whether the current state is good or bad, and which action is more appropriate to choose in the current state. Based on the above two parts of information, the strategy network finally obtains the evaluation result of each optional action in the current state. In actual decision-making, the unmanned underwater vehicle selects an action and executes it according to the evaluation result given by the strategy network and the preset action selection rules, thereby realizing real-time control of heading and motion behavior. During training, the policy network continuously adjusts its internal parameters based on the immediate reward feedback obtained in subsequent steps, enabling it to gradually learn to prioritize actions that are safe, energy-efficient, and move towards the goal in complex and multi-constrained environments.

[0031] S33 specifically includes: S331: When initializing the priority experience replay pool, a fixed capacity limit is set. The capacity is expressed in units of the number of entries and is used to limit the number of state transition samples that can be stored. When a new sample arrives and the capacity is full, the sample with the lowest priority is replaced according to the set replacement strategy. S332: When writing each newly generated state transition sample into the replay pool, an initial priority value is assigned to the sample. The initial priority is calculated based on the absolute value of the temporal difference error corresponding to the sample in the current policy network; the larger the error, the higher the priority. When the sample has not yet participated in training and updates, the default initialization constant is used as the initial priority. The expression for the absolute value of the temporal difference error is: , in, The time difference error of the i-th sample; The immediate reward for the current sample; Indicates the discount factor; The estimated Q value for the target network; Q is the current Q value output by the policy network; These are the parameters for the policy network and the target network, respectively; The current state and the action to be performed; This is the new state achieved after execution; S333: When drawing training samples from the replay pool, the sampling probability of each sample is calculated based on its current priority value. The higher the priority value, the higher the corresponding sampling probability, in order to achieve priority learning of key experiences; the calculation formula is: ,in, Let i be the probability that sample i is sampled. This represents the current priority value of the sample. A moderating factor to control the degree of influence of priority importance; This is the normalized denominator for the priority values ​​of all samples in the replay pool; S334: After each round of policy network parameter update, the temporal difference error of the samples used in this round is recalculated, and their priority values ​​in the replay pool are updated based on the error values ​​to reflect their current changes in importance to learning. The above steps introduce a priority calculation and sampling mechanism based on temporal difference error, which makes high-value samples receive more attention during training, while low-value samples are used less, further improving the utilization efficiency of training samples, strengthening key decision learning, and accelerating the convergence and generalization ability of the policy network in complex environments.

[0032] S4 specifically includes: S41: Call the current time-increased state vector output by S1, input the enhanced state vector into the policy network initialized by S3, and obtain the action value evaluation results of each candidate action; The steps are as follows: The enhanced state vector obtained in S1 is input into the input layer of the policy network initialized in step S3. This input layer performs dimensional processing on the state vector. The feature extraction layer performs feature transformation on the input state vector to extract key decision features such as environmental risk, target direction and energy adequacy contained in the state vector, and outputs a high-order state representation vector. The higher-order state representation vector is passed to the state value branch and action advantage branch in the policy network, respectively. The state value branch is used to evaluate the overall value of the current state itself, and the action advantage branch is used to evaluate the degree of advantage of each candidate action relative to the state. The state value output and the advantage output of each action are structurally combined to form the action value evaluation result of all candidate actions in the current state.

[0033] S42: Determine the action at the current moment based on the action value assessment result and the preset ε-greedy action selection rule. Specifically, when the exploration selection conditions are met, select the exploration action from the preset action set; when the exploration selection conditions are not met, select the action with the highest action value assessment result as the current action. S43: Control the unmanned underwater vehicle to perform actions, obtain the environmental feedback information corresponding to the execution result after the execution is completed, and execute S1 again to generate the enhanced state vector for the next moment. At the same time, calculate the instant reward corresponding to this execution based on S2. S44: Encapsulate the current augmented state vector, current action, immediate reward, and next augmented state vector in a structured manner to form a state transition sample; S45: Calculate the absolute value of the time difference error based on the state transition sample, and use this absolute value as the basis for calculating the sample priority to obtain the sample priority value; S46: Write the state transition samples and their corresponding priority values ​​into the priority experience replay pool as the training sample source for the subsequent S5 priority-based sampling update of the policy network parameters. The above steps complete the action value evaluation by inputting the enhanced state vector into the policy network and generating the enhanced state vector and immediate reward at the next moment after execution. Then, the state transition samples are assigned priority according to the time difference error and stored in the replay pool. This makes the training sample formation process and the learning update process have a consistent data structure and clear priority label, which is conducive to enhancing the utilization efficiency of key decision experience, thereby improving the learning stability and convergence speed of the path planning strategy of the unmanned underwater vehicle in a multi-constraint environment.

[0034] The above ε-greedy action selection rule is a probability-based policy decision-making mechanism. Its core function is to dynamically balance the optimal action selection given by the policy network and the proportion of behavior exploring the unknown action space during the training process, so as to take into account both the current policy effect and the long-term policy optimization goal. At each decision time t, the specific execution process is as follows: Set the exploration probability value at the current moment. This value typically decreases gradually as the training process progresses. The initial value is relatively large to encourage early action exploration; it gradually decreases as training progresses. To enhance strategy convergence; Generate an action by selecting a random value: Generate an action that obeys the action at the current moment. A random number r in the interval; compare r with... Compare; Determine the current action :like Then, an exploration strategy is executed, starting from the action set. Randomly select an action ;like Then, the action value vector output from the policy network is executed using the policy. The rule selects the action with the highest value; this rule ensures that the model can fully explore the action space in the early stages of training.

[0035] S5 specifically includes: S51: Calculate the sampling probability from the priority experience replay pool based on the priority value corresponding to each state transition sample, and extract a fixed number of state transition samples according to the sampling probability to form a small batch sample set for this round of training updates. S52: For each state transition sample in the mini-batch sample set, call the policy network to evaluate the action value of its next time step augmented state vector, and select the action with the highest action value as the target action corresponding to the target Q value calculation; S53: Call the target network, calculate the corresponding target Q value based on the target action determined in S52, and combine the immediate reward with the target Q value to obtain the target Q value corresponding to each state transition sample; S54: Call the policy network, calculate the predicted Q value of the state transition sample under the current policy network parameters, and compare the predicted Q value with the target Q value obtained in S53 to construct a loss function to measure the difference between the two. The expression for calculating the target Q value is as follows: ; The expression for calculating the loss function is: ,in, This represents the target Q value corresponding to the i-th sample; The immediate reward for the current sample; Represents the discount factor; Q and These represent the Q values ​​output by the policy network and the target network, respectively. represents the parameters of the policy network and the target network, respectively; N represents the number of mini-batch samples; S55: Based on the loss function, the gradient of the parameters of each layer of the policy network is calculated through the backpropagation mechanism, and the policy network parameters are updated by gradient descent to reduce the deviation between the predicted Q value and the target Q value. S56: After updating the policy network parameters, recalculate the temporal difference error corresponding to the state transition samples participating in the training round, and update their priority values ​​in the priority experience replay pool based on the temporal difference error. The above steps, by introducing an update mechanism based on priority sampling, which selects target actions by the policy network and calculates target Q values ​​by the target network, can effectively reduce the bias in the action value estimation process. Combined with the loss function, the policy network parameters are updated stably, so that the reinforcement learning training process maintains convergence stability and decision consistency in a multi-constraint environment.

[0036] S55 specifically includes: S551: For the small batch sample set determined by S51, call the loss function constructed by S54, calculate the deviation between the predicted Q value and the target Q value corresponding to each state transition sample, and use the deviation as the error signal input for backpropagation calculation; S552: Using the loss function as the optimization objective, the error signal is propagated backward along the network layers starting from the output layer of the policy network. The partial derivatives of the loss function with respect to the weight parameters and bias parameters of each layer are calculated layer by layer to obtain the gradient set of the parameters of each layer of the policy network. S553: ​​Summarize the gradient sets corresponding to each state transition sample in the mini-batch sample set to generate the batch gradient for this round of parameter update, and perform gradient magnitude constraint processing on the batch gradient to control the parameter update magnitude to remain stable. S554: Based on a preset learning rate, gradient descent is used to update the parameters of the policy network. The parameters of each layer are corrected in the opposite direction of their gradients, so that the predicted Q value output by the policy network converges to the target Q value, thereby reducing the deviation between the predicted Q value and the target Q value. The above steps use the loss function deviation as an error signal to perform backpropagation and calculate the gradient layer by layer. Then, gradient descent is used to iteratively correct the policy network parameters in a consistent direction, so that the policy network can stably reduce the difference between the predicted Q value and the target Q value, and improve the accuracy of action value estimation.

[0037] S6 specifically includes: S61: Set a fixed parameter synchronization period threshold. Whenever the policy network completes a continuous After this parameter update operation, a target network parameter synchronization operation is immediately triggered; S62: When a synchronization operation is triggered, the parameter values ​​of each layer in the policy network are copied and overwritten to the corresponding layer in the target network, so that the structural parameters between the policy network and the target network are consistent. S63: During the continuous execution of actions and interaction with the environment by the unmanned underwater vehicle, the optimal action decision output by the policy network at each state moment is recorded in real time and stored sequentially according to the state transition order. S64: When the unmanned underwater vehicle reaches the target point or the mission termination condition is met, extract the complete action sequence and use it as the final adaptive path planning result. The above steps can effectively improve the stability of the target Q value calculation during the training process by setting a fixed parameter synchronization period and periodically copying the policy network parameters to the target network, and avoid over-reliance on the instantaneous fluctuations of the policy network. At the same time, the path is constructed by the action sequence continuously output by the policy network, enabling the unmanned underwater vehicle to form an overall path decision with safety, energy efficiency and target orientation in a multi-constraint environment.

[0038] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0039] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for adaptive path planning in multi-constraint environments for unmanned underwater vehicles based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Acquire the real-time status and environmental information of the unmanned underwater vehicle, and calculate the obstacle proximity trend characteristics and energy-range ratio characteristics, which are jointly encoded to form the enhanced state vector at the current moment; S2: Based on the enhanced state vector and the actions performed by the unmanned underwater vehicle, calculate the immediate reward obtained by weighted summation of safety reward, energy efficiency reward, progress reward and goal achievement reward; S3: Initialize a dual-depth Q-network based on the Dueling architecture as the policy network, and construct the corresponding target network and priority experience replay pool; S4: Input the enhanced state vector into the policy network to select an action. After execution, a new enhanced state vector and an immediate reward are obtained, forming a transition sample. After calculating its priority, it is stored in the priority experience replay pool. S5: Sample samples from the priority experience replay pool according to priority, calculate the target Q value using the target network and policy network, update the policy network parameters through the loss function, and recalculate the sample priority; S6: Periodically synchronize the parameters of the policy network and the target network, and use the action sequence output by the policy network to form the final adaptive planning path.

2. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S1 specifically includes: S11: Acquire real-time status information and environmental perception information of the unmanned underwater vehicle; real-time status information includes current position coordinates, heading angle, speed and remaining energy; environmental perception information includes static obstacle positions, dynamic obstacle positions and velocities, current ocean current velocity vector field and target point position, all information is synchronized with the current time t as the reference; S12: Based on the obstacle distance difference between the current and previous moments, calculate the obstacle proximity trend characteristics that reflect the changing trend of obstacle avoidance pressure; S13: Calculate the energy-range ratio characteristic based on the ratio between the current remaining energy and the remaining range; S14: The real-time state information and environmental perception information obtained in S11, together with the two feature values ​​calculated in S12 and S13, are structured and encoded to form the enhanced state vector at the current moment.

3. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21: After the unmanned underwater vehicle performs the current action, call the current time-based enhanced state vector output by S1; S22: Calculate the safety reward based on the obstacle proximity trend features contained in the enhanced state vector; S23: Calculate the energy efficiency bonus based on the energy consumption of the unmanned underwater vehicle under the current action and the energy-range ratio feature in the enhanced state vector; S24: Calculate the progress reward based on the displacement change of the unmanned underwater vehicle relative to the target point before and after the current action is performed; S25: Based on the distance relationship between the current position of the unmanned underwater vehicle and the target point, determine whether the target arrival condition is met; when the unmanned underwater vehicle reaches the target point, generate a preset maximum positive target achievement reward item, otherwise the target achievement reward item is zero; S26: According to the preset weighting coefficients, the safety reward items, energy efficiency reward items, progress reward items and target achievement reward items obtained from S22 to S25 are weighted and summed to obtain the instant reward value at the current moment.

4. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 3, characterized in that, S26 specifically includes: S261: Pre-set the weight coefficients for safety rewards, energy efficiency rewards, progress rewards, and target achievement rewards, denoted as... , and The weighting coefficients are all non-negative real numbers and satisfy the normalization constraint conditions: ; S262: Call the safety reward items, energy efficiency reward items, progress reward items, and target achievement reward items from S22 to S25 as reward component inputs at the same decision moment; S263: By linearly weighting and summing each reward item according to the one-to-one correspondence between each reward component and its corresponding weight coefficient, the instantaneous reward value at the current moment is obtained. .

5. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31: Construct a policy network based on the Dueling architecture, which includes an input layer, a feature extraction layer, a state value branch, and an action advantage branch; S32: Copy the structure of the policy network and parameterize it into the target network, maintaining the same hierarchical structure and initialization method as the policy network, and use it to output the target Q value during training; S33: Initialize the priority experience replay pool, set a fixed capacity, and assign an initial priority value to each sample; the experience replay pool supports sampling and updating operations according to sample priority.

6. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 5, characterized in that, S33 specifically includes: S331: When initializing the priority experience replay pool, a fixed capacity limit is set. When a new sample arrives and the capacity is full, the lowest priority sample is replaced according to the set replacement strategy. S332: When writing each newly generated state transition sample into the replay pool, an initial priority value is assigned to the sample. The initial priority is calculated based on the absolute value of the temporal difference error corresponding to the sample in the current policy network. The larger the error, the higher the corresponding priority. S333: When drawing training samples from the replay pool, calculate the sampling probability of each sample based on its current priority value; S334: After each round of policy network parameter updates, recalculate the temporal difference error of the samples used in this round, and update their priority value in the replay pool based on the error value.

7. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41: Call the current time-increased state vector output by S1, input the enhanced state vector into the policy network initialized by S3, and obtain the action value evaluation results of each candidate action; S42: Determine the action at the current moment based on the action value assessment result and the preset ε-greedy action selection rule. Specifically, when the exploration selection conditions are met, select the exploration action from the preset action set; when the exploration selection conditions are not met, select the action with the highest action value assessment result as the current action. S43: Control the unmanned underwater vehicle to perform actions, obtain the environmental feedback information corresponding to the execution result after the execution is completed, and execute S1 again to generate the enhanced state vector for the next moment. At the same time, calculate the instant reward corresponding to this execution based on S2. S44: Encapsulate the current augmented state vector, current action, immediate reward, and next augmented state vector in a structured manner to form a state transition sample; S45: Calculate the absolute value of the time difference error based on the state transition sample, and use this absolute value as the basis for calculating the sample priority to obtain the sample priority value; S46: Write the state transition sample and its corresponding priority value into the priority experience replay pool.

8. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51: Calculate the sampling probability from the priority experience replay pool based on the priority value corresponding to each state transition sample, and extract a fixed number of state transition samples according to the sampling probability. S52: For each state transition sample in the mini-batch sample set, call the policy network to evaluate the action value of its next time step augmented state vector, and select the action with the highest action value as the target action corresponding to the target Q value calculation; S53: Call the target network, calculate the corresponding target Q value based on the target action determined in S52, and combine the immediate reward with the target Q value to obtain the target Q value corresponding to each state transition sample; S54: Call the policy network, calculate the predicted Q value of the state transition sample under the current policy network parameters, and compare the predicted Q value with the target Q value obtained in S53 to construct a loss function to measure the difference between the two. S55: Based on the loss function, the gradient of the parameters of each layer of the policy network is calculated through the backpropagation mechanism, and the policy network parameters are updated by gradient descent to reduce the deviation between the predicted Q value and the target Q value. S56: After completing the policy network parameter update, recalculate the temporal difference error corresponding to the state transition sample participating in the training in this round, and update its priority value in the priority experience replay pool based on the temporal difference error.

9. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 8, characterized in that, Specifically, S55 includes: S551: For the small batch sample set determined by S51, call the loss function constructed by S54, calculate the deviation between the predicted Q value and the target Q value corresponding to each state transition sample, and use the deviation as the error signal input for backpropagation calculation; S552: Using the loss function as the optimization objective, the error signal is propagated backward along the network layers starting from the output layer of the policy network. The partial derivatives of the loss function with respect to the weight parameters and bias parameters of each layer are calculated layer by layer to obtain the gradient set of the parameters of each layer of the policy network. S553: ​​Summarize the gradient sets corresponding to each state transition sample in the mini-batch sample set to generate the batch gradient for this round of parameter update; S554: Based on the preset learning rate, the gradient descent method is used to update the parameters of the policy network. The parameters of each layer are corrected in the opposite direction of their gradient, so that the predicted Q value output by the policy network converges to the target Q value, thereby reducing the deviation between the predicted Q value and the target Q value.

10. The adaptive path planning method for unmanned underwater vehicles in multi-constraint environments based on deep reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61: Set a fixed parameter synchronization period threshold. Whenever the policy network completes a continuous After this parameter update operation, a target network parameter synchronization operation is immediately triggered; S62: When a synchronization operation is triggered, the parameter values ​​of each layer in the policy network are copied and overwritten to the corresponding layer in the target network, so that the structural parameters between the policy network and the target network are consistent. S63: During the continuous execution of actions and interaction with the environment by the unmanned underwater vehicle, the optimal action decision output by the policy network at each state moment is recorded in real time and stored sequentially according to the state transition order. S64: When the unmanned underwater vehicle reaches the target point or the mission termination condition is met, extract the complete action sequence and use it as the final adaptive path planning result.

Images