Underwater vehicle autonomous navigation method based on security diffusion reinforcement learning and related equipment

Abstract

The application discloses an underwater vehicle autonomous navigation method based on security diffusion reinforcement learning and related equipment, and the method comprises the following steps: inputting a to-be-trained state vector corresponding to an underwater vehicle into an initial condition diffusion strategy network to generate a to-be-trained smooth action; performing action post-processing on the to-be-trained smooth action to obtain a to-be-trained target action; constructing a composite loss function based on the to-be-trained target action and the to-be-trained state vector; the composite loss function comprises an imitation learning loss, a value promotion loss and a safety constraint loss; training the initial condition diffusion strategy network according to the composite loss function to obtain a target condition diffusion strategy network; inputting a current state vector corresponding to a current underwater vehicle into the target condition diffusion strategy network to generate a current smooth action; and performing action post-processing on the current smooth action to obtain a current target action. The application can improve the efficiency and safety of underwater vehicle autonomous navigation and can be widely applied in the field of artificial intelligence technology.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an autonomous navigation method and related equipment for underwater vehicles based on security diffusion reinforcement learning. Background Technology

[0002] Autonomous underwater vehicles (AUVs) are widely used in missions such as marine resource exploration, environmental monitoring, and underwater search and rescue. As mission scenarios become increasingly complex, AUVs often need to perform autonomous operations in unknown, dynamic, and uncertain underwater environments. Therefore, efficient, safe, and robust autonomous navigation and obstacle avoidance capabilities have become the core technologies for the successful execution of AUV operations.

[0003] Currently, AUV navigation methods mainly include traditional path planning methods, online reinforcement learning algorithms, and traditional offline reinforcement learning methods. However, traditional path planning methods heavily rely on prior environmental information, have poor adaptability to dynamic obstacles and model uncertainties, and produce non-smooth and suboptimal paths. Online reinforcement learning algorithms rely entirely on continuous interaction with the simulation environment; differences between the simulation and the real marine environment can lead to policy transfer failures, and require numerous online interaction steps, making them difficult to apply directly to real AUVs. Traditional offline reinforcement learning methods overestimate out-of-distribution (OOD) actions, have conservative policies, struggle to learn multimodal behaviors from data, and have low navigation efficiency.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of this application aim to at least partially address one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose an autonomous navigation method and related equipment for underwater vehicles based on safety diffusion reinforcement learning, which can overcome the representational limitations of traditional reinforcement learning strategies, improve the efficiency and safety of autonomous navigation of underwater vehicles, and reduce the risks of online exploration and collisions.

[0006] To achieve the above objectives, one aspect of this application proposes an autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning, the method comprising the following steps: Obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information; The state vector to be trained is input into the initial conditional diffusion policy network to generate a smooth action to be trained, and the smooth action to be trained is post-processed to obtain the target action to be trained. Based on the target action to be trained and the state vector to be trained, a composite loss function is constructed; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; The initial conditional diffusion policy network is jointly optimized and trained according to the composite loss function to obtain the target conditional diffusion policy network. The current state vector corresponding to the current underwater vehicle is input into the target conditional diffusion strategy network to generate the current smooth action; The current smooth action is post-processed to obtain the current target action, and the current target action is output to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

[0007] In some embodiments, the method further includes: During the current underwater vehicle's execution of the current target action, environmental interaction data is collected; When the preset offline training conditions are met, the environmental perception state information to be trained is updated according to the environmental interaction data. Based on the updated environment-aware state information to be trained, return to the step of constructing the state vector to be trained based on the environment-aware state information to be trained, until the updated target conditional diffusion policy network is obtained.

[0008] In some embodiments, constructing a composite loss function based on the target action to be trained and the state vector to be trained includes: The target action to be trained and the state vector to be trained are concatenated to obtain the state-target action vector; The state-target action vector is input into the value critic network to generate state-action value; The state-target action vector is input into the cost critic network to generate the expected cumulative cost mean and cumulative cost variance. The imitation learning loss is determined based on the denoising loss of the initial conditional diffusion strategy network. Based on the state-action value, determine the value enhancement loss; The safety constraint loss is determined based on the expected average cumulative cost and the cumulative cost variance. The imitation learning loss, the value enhancement loss, and the security constraint loss are weighted and fused to obtain the composite loss function.

[0009] In some embodiments, the method further includes: In constructing the composite loss function, the Lagrange multiplier method is used to dynamically update the safety weights corresponding to the safety constraint loss.

[0010] In some embodiments, the step of inputting the current state vector corresponding to the current underwater vehicle into the target conditional diffusion policy network to generate the current smoothing action includes: Obtain the current environmental perception state information corresponding to the current underwater vehicle, and construct the current state vector based on the current environmental perception state information; In the target conditional diffusion strategy network, the current state vector and the current noise action are concatenated to obtain the current state-noise action vector; The current state-noise action vector is input into the noise prediction network in the target conditional diffusion strategy network to generate the current estimated noise. The current estimated noise is denoised according to the denoising formula by a preset number of diffusion steps to obtain the current smoothing action.

[0011] In some embodiments, the step of performing post-processing on the current smoothing motion to obtain the current target motion, and outputting the current target motion to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target motion, includes: The current smooth action is processed by instruction mapping to obtain the current control command corresponding to the current underwater vehicle; wherein, the current control command includes thrust command and rudder angle command; Based on the dynamic constraints of the current underwater vehicle, the current control command is trimmed to obtain the target control command; The target control command is output as the current target action to the actuator of the current underwater vehicle, so that the current underwater vehicle performs the current target action.

[0012] To achieve the above objectives, another aspect of this application proposes an autonomous navigation device for underwater vehicles based on security diffusion reinforcement learning, the device comprising the following modules: The training state vector construction module is used to obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information. The training target action generation module is used to input the training state vector into the initial conditional diffusion policy network, generate the training smooth action, and perform post-processing on the training smooth action to obtain the training target action. The composite loss function construction module is used to construct a composite loss function based on the target action to be trained and the state vector to be trained; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; The policy network joint optimization training module is used to perform joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain the target conditional diffusion policy network. The current smooth action generation module is used to input the current state vector corresponding to the current underwater vehicle into the target conditional diffusion strategy network to generate the current smooth action; The current target action generation module is used to perform post-processing on the current smooth action to obtain the current target action, and output the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0015] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides an autonomous navigation method and related equipment for underwater vehicles based on safety diffusion reinforcement learning. This scheme obtains the environmental perception state information corresponding to the underwater vehicle to be trained, and constructs a training state vector based on the environmental perception state information; inputs the training state vector into an initial conditional diffusion policy network to generate a training smooth action, and performs post-processing on the training smooth action to obtain a training target action; constructs a composite loss function based on the training target action and the training state vector; wherein the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the safety constraint loss corresponding to the cost critic network; performs joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain a target conditional diffusion policy network; inputs the current state vector corresponding to the current underwater vehicle into the target conditional diffusion policy network to generate a current smooth action; performs post-processing on the current smooth action to obtain the current target action, and outputs the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target action. This application embodiment uses a conditional diffusion model as the policy network, which can effectively model multimodal and asymmetric action distributions. It accurately generates navigation actions that conform to dynamic constraints through a multi-step denoising process, which can overcome the representation limitations of traditional reinforcement learning strategies. By constructing a composite loss function that includes imitation generation terms, value enhancement terms, and safety cost control terms, it can achieve policy optimization and risk constraints on the policy network, thereby improving the efficiency and safety of autonomous navigation of underwater vehicles and reducing the risks of online exploration and collisions. Attached Figure Description

[0017] Figure 1 This is a flowchart of the steps of the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of the underwater vehicle autonomous navigation device based on security diffusion reinforcement learning provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0023] (1) AUV (Autonomous Underwater Vehicle) is an unmanned equipment that can perform tasks such as exploration, navigation and data collection underwater without human intervention.

[0024] (2) Diffusion Model: A generative model that generates structured data (such as images and action sequences) from random noise by progressively denoising. It is suitable for modeling complex continuous action spaces.

[0025] (3) Deep Reinforcement Learning (DRL) is a technique that combines deep learning and reinforcement learning. It aims to enable agents to learn optimal behavioral strategies by interacting with the environment and receiving reward signals from the environment.

[0026] (4) Safe Diffusion Reinforcement Learning (SDQ) is the method proposed in this application. It combines a diffusion model with a safety constraint mechanism to achieve efficient and safe AUV navigation strategy learning.

[0027] (5) Conditional Value at Risk (CVaR) is a risk metric used to assess the tail risk of a strategy under extreme conditions and to achieve safety control.

[0028] (6) Wasserstein distance, used to measure the difference between two probability distributions, is used in this application for fitting the cost distribution and quantifying risk.

[0029] (7) Lagrangian Multiplier Method, a constrained optimization method, is used to dynamically adjust safety weights and balance policy performance with safety constraints.

[0030] (8) Optimal Reciprocal Collision Avoidance (ORCA) is an algorithm for solving the collision avoidance problem in multi-agent systems.

[0031] As an example, autonomous underwater vehicles (AUVs) are widely used in missions such as marine resource exploration, environmental monitoring, and underwater search and rescue. With increasingly complex mission scenarios, AUVs often need to perform autonomous operations in unknown, dynamic, and uncertain underwater environments. Therefore, efficient, safe, and robust autonomous navigation and obstacle avoidance capabilities have become core technologies for the successful execution of AUV operations.

[0032] Currently, AUV navigation methods mainly include traditional path planning methods, online reinforcement learning algorithms, and traditional offline reinforcement learning methods. Traditional path planning methods, such as artificial potential field methods, rapid exploration random trees (RRT), and model predictive control (MPC), rely on accurate environmental models and struggle to adapt to dynamic obstacles and unknown disturbances. Online reinforcement learning algorithms, such as soft actor critic (SAC), double-delay deterministic policy gradient (TD3), and proximal policy optimization (PPO), require continuous interaction with the simulation environment for training. If the simulation differs from the real ocean environment (flow field, hydrodynamics), policy transfer will fail; furthermore, training relies heavily on online interaction, resulting in high costs and poor security. Traditional offline reinforcement learning methods, such as conservative Q-learning (CQL) and decision transformers (DT), learn policies directly from data, but the policies are conservative and lack the ability to model multimodal behavior.

[0033] Therefore, the relevant technology has the following main drawbacks: (1) Traditional path planning methods: heavily rely on prior environmental information, have poor adaptability to dynamic obstacles and model uncertainties, and produce non-smooth and suboptimal paths; (2) "Environment dependence and transfer failure" of online reinforcement learning algorithms: They rely entirely on continuous interaction with the simulation environment. The difference between the simulation and the real marine environment will lead to policy transfer failure. In addition, a large number of online interaction steps are required, making it difficult to apply directly to real AUVs.

[0034] (3) Traditional offline reinforcement learning methods: overestimation of out-of-distribution (OOD) actions, conservative policies, difficulty in learning multimodal behaviors from data, and low navigation efficiency.

[0035] In view of this, embodiments of this application provide an autonomous navigation method and related equipment for underwater vehicles based on safety diffusion reinforcement learning. This scheme obtains the environmental perception state information corresponding to the underwater vehicle to be trained, and constructs a training state vector based on this information. The training state vector is input into an initial conditional diffusion policy network to generate a smooth action to be trained, and post-processing is performed on the smooth action to obtain the target action to be trained. A composite loss function is constructed based on the target action and the training state vector. This composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the safety constraint loss corresponding to the cost critic network. The initial conditional diffusion policy network is jointly optimized and trained according to the composite loss function to obtain a target conditional diffusion policy network. The current state vector corresponding to the current underwater vehicle is input into the target conditional diffusion policy network to generate a current smooth action. Post-processing is performed on the current smooth action to obtain the current target action, and the current target action is output to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target action. This application embodiment uses a conditional diffusion model as the policy network, which can effectively model multimodal and asymmetric action distributions. It accurately generates navigation actions that conform to dynamic constraints through a multi-step denoising process, which can overcome the representation limitations of traditional reinforcement learning strategies. By constructing a composite loss function that includes imitation generation terms, value enhancement terms, and safety cost control terms, it can achieve policy optimization and risk constraints on the policy network, thereby improving the efficiency and safety of autonomous navigation of underwater vehicles and reducing the risks of online exploration and collisions.

[0036] The underwater vehicle autonomous navigation method based on secure diffusion reinforcement learning provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the underwater vehicle autonomous navigation method based on secure diffusion reinforcement learning, but is not limited to the above forms.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] Please see Figure 1 , Figure 1 This is an optional flowchart of the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning provided in this application embodiment. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0039] Step S101: Obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information. The "training environment perception state information" refers to the raw, unprocessed data of the interaction between the environment and the underwater vehicle (AUV) itself, collected from simulation scenarios or historical operational data during the offline training phase of reinforcement learning. This training environment perception state information serves as the training dataset for subsequent training and primarily includes the target state, obstacle perception state, AUV's own state, and ORCA guidance state. Specifically, the target state... Including but not limited to the relative position and orientation of the AUV and the target point; obstacle perception status. Including but not limited to the distance and location of the nearest obstacle obtained through sonar or multibeam sensors; the AUV's own status. Including but not limited to position, speed, heading angle, depth, etc.; ORCA guidance status This includes, but is not limited to, the suggested collision avoidance velocity direction calculated using the optimal reciprocal collision avoidance algorithm.

[0040] The state vector to be trained is a fixed-dimensional tensor formed by concatenating the environmental perception state information to be trained after normalization and structuring (such as normalization, fixed-length padding, and dimension alignment). The state vector to be trained is the input data of the reinforcement learning policy network.

[0041] Step S102: Input the state vector to be trained into the initial conditional diffusion policy network to generate a smooth action to be trained, and perform post-processing on the smooth action to be trained to obtain the target action to be trained. In the embodiments of this application, the basic model of the conditional diffusion policy network is the diffusion model. This application aims to achieve efficient and safe AUV navigation policy learning by combining the diffusion model with a safety constraint mechanism.

[0042] In step S102, the process of inputting the state vector to be trained into the initial conditional diffusion policy network to generate the smooth action to be trained, and performing post-processing on the smooth action to obtain the target action to be trained, is as follows: First, the state vector to be trained is input into the initial conditional diffusion policy network, and noisy actions are initialized: At each decision step, the conditional diffusion policy network first samples a random Gaussian noise action. This serves as the starting point for the diffusion process; then, a multi-step reverse denoising process is performed: the current noise is processed... With state splicing, input noise prediction network Then, the noise prediction network Output estimated noise Then, based on the denoising formula, the estimated noise is... Denoising is performed to update the motion, generating a smooth motion for training. Next, post-processing of the motion is performed: smoothing the motion to be trained. The actual control commands are mapped to the AUV and then trimmed according to the AUV's dynamic limits (such as maximum thrust and rudder angle range) to ensure the physical feasibility of the actions. Finally, the trimmed actual control commands are used as the target actions to be trained. .

[0043] It should be noted that in subsequent practical applications of the model, the process of inputting the real-time state vector into the trained target conditional diffusion policy network, outputting a smooth action, and then performing instruction mapping and dynamic constraint pruning on the smooth action to output the target action is basically the same as the process in step S102. It can be understood that the parameters of corresponding nodes can be represented by the same parameter; for example, the parameters of the smooth action to be trained during the training phase can be represented as follows: The real-time smooth motion obtained in practical applications can also be used This can be expressed, but is not limited to; that is, the same parameters can be used for the same nodes at different stages, but different parameters can be used for easy differentiation. For example, the parameters of the smooth action to be trained during the training stage can be expressed as follows: The parameters of the real-time smoothing motion obtained in the practical application stage are expressed as follows: The embodiments of this application do not limit this, and can be set according to the actual situation. The parameter representation of the same node in other different stages is similar, and the embodiments of this application will not be described in detail here.

[0044] Step S103: Based on the target action to be trained and the state vector to be trained, construct a composite loss function; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; In some embodiments, step S103 may include: concatenating the target action to be trained and the state vector to be trained to obtain a state-target action vector; inputting the state-target action vector into a value critic network to generate a state-action value; inputting the state-target action vector into a cost critic network to generate the expected cumulative cost mean and cumulative cost variance; determining the imitation learning loss based on the denoising loss of the initial conditional diffusion policy network; determining the value enhancement loss based on the state-action value; determining the safety constraint loss based on the expected cumulative cost mean and cumulative cost variance; and performing weighted fusion processing on the imitation learning loss, value enhancement loss, and safety constraint loss to obtain a composite loss function.

[0045] In some embodiments, the method may further include: during the construction of the composite loss function, using the Lagrange multiplier method to dynamically update the safety weights corresponding to the safety constraint loss.

[0046] The composite loss function includes the imitation learning loss for the initial condition diffusion strategy network, the value enhancement loss for the value commentator network, and the security constraint loss for the cost commentator network.

[0047] Optionally, both the value critic network and the cost critic network use a multilayer perceptron (MLP) as their basic network structure, with states as inputs. With action The concatenated vector.

[0048] Step S104: Perform joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain the target conditional diffusion policy network; Among them, the target condition diffusion strategy network is a network model that meets the autonomous control requirements of underwater vehicles after joint optimization training with composite loss functions. It has the core capability of "taking the underwater vehicle state vector as input and outputting smooth actions that conform to dynamic constraints".

[0049] Step S105: Input the current state vector corresponding to the current underwater vehicle into the target conditional diffusion strategy network to generate the current smooth action; In some embodiments, step S105 may include: acquiring the current environmental perception state information corresponding to the current underwater vehicle, and constructing a current state vector based on the current environmental perception state information; concatenating the current state vector and the current noise action in the target conditional diffusion strategy network to obtain a current state-noise action vector; inputting the current state-noise action vector into the noise prediction network in the target conditional diffusion strategy network to generate the current estimated noise; and denoising the current estimated noise according to the denoising formula by a preset diffusion step number to obtain the current smooth action.

[0050] The current state vector is a fixed-dimensional tensor formed by concatenating the current environmental perception state information of the underwater vehicle after normalization and structuring (such as normalization, fixed-length padding, and dimension alignment). The current environmental perception state information also includes the target state, obstacle perception state, AUV's own state, and ORCA guidance state. Specifically, the target state... Including but not limited to the relative position and orientation of the AUV and the target point; obstacle perception status. Including but not limited to the distance and location of the nearest obstacle obtained through sonar or multibeam sensors; the AUV's own status. Including but not limited to position, speed, heading angle, depth, etc.; ORCA guidance status This includes, but is not limited to, the suggested collision avoidance velocity direction calculated using the optimal reciprocal collision avoidance algorithm.

[0051] Step S106: Perform post-processing on the current smooth action to obtain the current target action, and output the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

[0052] In some embodiments, step S106 may include: performing instruction mapping processing on the current smooth action to obtain the current control command corresponding to the current underwater vehicle; wherein the current control command includes thrust command and rudder angle command; performing trimming processing on the current control command according to the dynamic constraints of the current underwater vehicle to obtain the target control command; and outputting the target control command as the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle performs the current target action.

[0053] The current control commands include thrust commands and rudder angle commands.

[0054] Optionally, motion post-processing refers to a series of engineering transformations and constraint processing performed on the current smooth motion, with the aim of transforming the abstract motion vector output by the network into actual motions that can be executed by the underwater vehicle actuators and conform to its own dynamic characteristics.

[0055] Among them, the target control command is the final control command obtained after trimming and conforming to the dynamic constraints of the underwater vehicle. It is the specific manifestation of the current target action and is directly output to the actuator for execution. The current target action is the action command obtained after post-action processing and finally output to the underwater vehicle actuator. It is the final decision action that takes into account both the smoothness of algorithm generation and the feasibility and safety of hardware execution.

[0056] It should be noted that the diffusion model outputs normalized commands. These normalized commands need to be mapped to actual control commands executable by the AUV based on the actual thrust or rudder angle range. Trimming can be performed before or after mapping. For example, for thrust T, it can be directly trimmed to [-1,1] before mapping, or trimmed to [...] after mapping. , ],in, This is the minimum output thrust of the AUV propulsion system. This is the maximum output thrust of the AUV propulsion system.

[0057] In some embodiments, the method may further include: collecting environmental interaction data during the current underwater vehicle's execution of the current target action; updating the environmental perception state information to be trained based on the environmental interaction data when preset offline training conditions are met; and returning to the step of constructing the training state vector based on the updated training environmental perception state information, until the updated target condition diffusion policy network is obtained.

[0058] Environmental interaction data refers to the collection of feedback data on the underwater vehicle's own motion state and the underwater environment collected by various sensors during the execution of the current target action. This data reflects the actual effect of the model's action decisions and the true characteristics of the environment, and is also the basis for updating the model's training data. Environmental interaction data may include, but is not limited to, the underwater vehicle's actual position, speed, heading, real-time changes in the water environment (flow velocity changes, etc.), relative position changes of obstacles, actual actuator parameters (actual thrust of the propeller, actual deflection angle of the servo motor), and navigation stability data after the action is performed.

[0059] Among them, the preset offline training conditions are the judgment conditions that are pre-set to trigger the offline iterative training of the target condition diffusion strategy network. They can be flexibly configured according to the actual application needs of the underwater vehicle, and the embodiments of this application do not limit them.

[0060] Steps S101 to S106 as shown in the embodiments of this application involve: acquiring the training environment perception state information corresponding to the underwater vehicle; constructing a training state vector based on the training environment perception state information; inputting the training state vector into an initial conditional diffusion policy network to generate a training smooth action; performing post-processing on the training smooth action to obtain a training target action; constructing a composite loss function based on the training target action and the training state vector; wherein the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the safety constraint loss corresponding to the cost critic network; jointly optimizing and training the initial conditional diffusion policy network according to the composite loss function to obtain a target conditional diffusion policy network; inputting the current state vector corresponding to the current underwater vehicle into the target conditional diffusion policy network to generate a current smooth action; performing post-processing on the current smooth action to obtain the current target action; and outputting the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target action. This application embodiment uses a conditional diffusion model as the policy network, which can effectively model multimodal and asymmetric action distributions. It accurately generates navigation actions that conform to dynamic constraints through a multi-step denoising process, which can overcome the representation limitations of traditional reinforcement learning strategies. By constructing a composite loss function that includes imitation generation terms, value enhancement terms, and safety cost control terms, it can achieve policy optimization and risk constraints on the policy network, thereby improving the efficiency and safety of autonomous navigation of underwater vehicles and reducing the risks of online exploration and collisions.

[0061] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0062] Please see Figure 2 , Figure 2 This is a flowchart illustrating the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning provided in an embodiment of this application, as shown below. Figure 2 As shown, Figure 2 This paper demonstrates the overall implementation flow of the autonomous navigation method for underwater vehicles based on safety diffusion reinforcement learning provided in this application embodiment. The main steps include state awareness, diffusion policy generation, critic evaluation, safety constraint optimization, and action execution. In this application embodiment, the core system architecture includes: a diffusion policy network, a value critic network, and a safety cost critic network, where the safety cost critic network is simply referred to as the cost critic network. The system workflow provided in this application embodiment is as follows: (1) Modeling of AUV dynamics and navigation problems: State space S: includes the target state Obstacle perception status AUV's own status Optimal reciprocal collision avoidance guidance term ; Action space Thrust and rudder angle , where the square brackets contain Indicates thrust. and These represent the horizontal and vertical rudder angles, respectively, indicated by the superscript square brackets. Indicates transpose; reward function Including target rewards Optimal reciprocal collision avoidance guidance reward Obstacle avoidance rewards based on artificial potential fields Energy efficiency reward ; Cost function : Defined as whether a collision has occurred (1 if yes, 0 otherwise); Constraint: Long-term cumulative costs do not exceed a preset threshold. .

[0063] In this embodiment, the AUV dynamics and navigation problem model established in this step provides a formalized problem definition and algorithm design foundation for the proposed method. The defined state space specifically indicates the information elements constituting the input vectors of the policy network and the critic network; the action space clarifies the physical control command dimensions required for mapping the policy network output layer and the post-processing module; and the detailed composition of the reward function and cost function provides the precise calculation basis for the immediate reward value r and cost value c in the data samples required for offline training, respectively used to optimize the value critic network to improve policy performance and train the cost critic network to quantify navigation risk; the set long-term cumulative cost constraint is directly related to the safety constraint loss, ensuring that the policy meets the safety threshold through mechanisms such as the Lagrange multiplier method.

[0064] Based on this AUV dynamics and navigation problem model, a high-fidelity simulation environment can be constructed. Using online reinforcement learning or model predictive control methods, trajectory data conforming to the above definition can be generated in batches, with a unified format of (s, a, s', r, c, done), where s represents the state, a represents the action, and the new state s' and the termination flag done are representations after environmental interaction data transformation. This data constitutes the expert or suboptimal sample set required for offline training, supporting the subsequent collaborative training and iterative optimization of the diffusion policy network and the critic network, thus forming a complete technical chain from problem modeling and data generation to network training and deployment.

[0065] (2) Diffusion strategy network construction and action generation process: The core execution entity of this application is the "policy network of the conditional diffusion model," or simply the conditional diffusion policy network. It receives state input from the environment perception module and generates navigation actions that conform to AUV dynamics constraints through a multi-step denoising process. The specific implementation flow is as follows: 1) Step 1: Status data acquisition and input: The environmental perception system collects the following state information in real time and constructs a state vector based on the state information. : Target state The relative position and orientation of the AUV and the target point; Obstacle perception status The distance and location of the nearest obstacle obtained through sonar or multibeam sensors; AUV's own status This includes position, speed, heading angle, depth, etc. ORCA boot status The suggested collision avoidance velocity direction is calculated using the optimal reciprocal collision avoidance algorithm.

[0066] In the specific implementation, the state vector The construction process involves standardizing and structuring the environmental perception state information through the system, and then concatenating it into a fixed-dimensional tensor, which serves as the unified input to the policy network. Specifically, the target state... Distance between AUV and target point and relative azimuth (Addition in a 3D environment) ( The target point's azimuth (pitch dimension) relative to the AUV; obstacle perception state. Extract the relative distances of the N nearest obstacles Azimuth And relative velocity (if dynamic), and fill the fixed-length vector after sorting by distance; AUV's own state Using linear velocity (u, v, w) and attitude angle (pitch angle) Yaw angle The ORCA guiding state is represented by angular velocity (q, r), etc. Then, a suggested velocity vector calculated by the optimal reciprocal collision avoidance algorithm is provided. ) and the corresponding expected course Finally, the target state after normalization and structuring is... Obstacle perception status AUV's own status and ORCA boot status The components are concatenated according to a preset dimensional order to form a fixed-dimensional tensor, which serves as the unified input to the policy network.

[0067] 2) Step two, the forward generation process of the conditional diffusion strategy network: a. Initializing the noisy action: At each decision step, the conditional diffusion policy network first samples a random Gaussian noise action. , as the starting point of the diffusion process.

[0068] b. Multi-step reverse noise reduction: such as setting the number of diffusion steps ,from arrive Perform noise reduction step by step: First, take action on the current noise. With state splicing, input noise prediction network Then, the noise is estimated by the output of the noise prediction network. Finally, the estimated noise is analyzed according to the denoising formula. Denoising is performed to update the action. The denoising formula is as follows: ; in, The first step is obtained after single-step denoising. Step movements, compared to Smoother and closer to the optimal real-world motion; For the first The current action with noise is the raw input for denoising; The noise standard deviation of the denoising step, and satisfying This is used to dynamically adjust the intensity of supplementary noise; For random noise, follow the rules of random noise. distributed, The distribution is the standard multivariate normal distribution, which can introduce exploratory features to action updates and avoid excessive noise reduction that leads to action determinism. Represents a normal distribution. It is a unit covariance matrix, whose dimensions are consistent with the action dimensions to ensure that the exploratory nature of each action dimension is independent and uniform; These are the preset diffusion scheduling parameters; This is the cumulative noise scaling factor. Finally, the final candidate actions are generated: after... After noise reduction, smooth motion is obtained. .

[0069] 3) Step three, post-processing of actions: a. Smooth the motion Mapped to the actual control commands of the AUV: thrust and rudder angle ; b. Tailor the actual control commands output according to the dynamic limits of the AUV (such as maximum thrust and rudder angle range) to ensure that the action is physically feasible; c. Finally, the actual control commands obtained after trimming are used as actions that the AUV can execute. The output is sent to the AUV actuator to achieve autonomous navigation.

[0070] It should be noted that the diffusion model outputs normalized commands. These normalized commands need to be mapped to actual control commands executable by the AUV based on the actual thrust or rudder angle range. Trimming can be performed before or after mapping. For example, for thrust T, it can be directly trimmed to [-1,1] before mapping, or trimmed to [...] after mapping. , ],in, This is the minimum output thrust of the AUV propulsion system. This is the maximum output thrust of the AUV propulsion system.

[0071] 4) Step Four: Strategy Optimization: During the training phase, the conditional diffusion policy network minimizes the composite loss function. Update.

[0072] (3) Construction and training of critic networks: Critics networks are used to evaluate and guide the learning direction of diffusion strategies. These networks include value critic networks and cost critic networks, and their construction and training process is as follows: 1) Step One: Construction and Training of the Value Critic Network (Q-network): Network structure: A multilayer perceptron (MLP) is used as the basic network structure, with the input being the state. With action The concatenated vector outputs a state-action value. .

[0073] Training objective: Minimize the temporal difference (TD) error, also known as the Bellman error, the loss function of the value critic network (Q-network). The calculation formula is as follows: ; in, These are the learnable parameters of the Q-network; For the next state The following are candidate actions; The discount factor (values ​​∈ [0,1]); These are the target network parameters, used for stable training. They are updated using gradient descent. ,make Approximating the expected cumulative rewards provides valuable guidance for strategy optimization.

[0074] 2) Step Two: Construction and Training of the Cost Critic Networks. The Cost Critic Networks include a cost mean network and a cost variance network: Network structure: A multilayer perceptron (MLP) is used as the basic network structure, with the input being the state. With action The concatenated vector; Cost Mean Network: Outputs the expected cumulative cost mean. Cost variance network: outputs the variance of cumulative cost. .

[0075] Training objective: Assume that the long-run cumulative cost follows a Gaussian distribution. By minimizing the 2-Wasserstein distance To fit the distribution: where, The target distribution parameters are obtained through temporal difference objective calculation. Finally, the loss function of the cost-means network is... Loss function of cost-variance network The calculation formula is as follows: ; ; in, The loss function of the cost mean network is used to minimize the Euclidean distance error between the network's predicted cumulative cost mean and the target distribution mean, so that the output of the mean network accurately approximates the true cumulative cost expectation. The loss function of the cost variance network is derived based on the 2-Wasserstein distance. By minimizing the distribution difference between the cost variance predicted by the network and the target distribution variance through trace operation, the variance network can accurately represent the actual uncertainty of the accumulated cost.

[0076] By fitting the cost distribution, the long-term risk of actions is assessed, providing a quantitative basis for safety constraints.

[0077] In this embodiment, the value critic network and the cost critic network are trained offline and do not interact with the environment. During training, the algorithm collects batches of offline data from the offline data pool, updates the network according to the corresponding loss function, and finally obtains a navigation strategy that surpasses the sample data, which can be directly used for actual deployment.

[0078] (4) Secure diffusion Q-learning strategy: The strategy optimization in this embodiment is achieved by minimizing a composite loss function, which integrates three objectives: imitation learning, value enhancement, and security constraints. The specific process is as follows: 1) Step 1: Construct a composite loss function : Conditional Diffusion Strategy Network The optimization objective is: ; in, These are adjustable weighting coefficients used to balance various losses.

[0079] 2) Step Two: Definition and Calculation Process of Each Loss Item: a. Mimicry generation term loss (i.e., imitation learning loss): The denoising loss of the diffusion model is directly applied using the Denoising Diffusion Probabilistic Models (DDPM) model. The calculation method is as follows: ; in, For true Gaussian noise that follows a standard multivariate normal distribution, i.e. The loss of this imitation generation term. It enables policies to learn expert behavioral patterns from offline datasets.

[0080] b. Value-added items (i.e., loss of value enhancement): By maximizing Value (i.e.) The strategy guides the generation of high-return actions, thereby improving the strategy. The calculation method is as follows: ; c. Safety cost control items (i.e., safety constraint loss): The worst-case cost is assessed based on the Conditional Value at Risk (CVaR). The formula for calculating CVaR is as follows: ; in, Indicates the level of confidence in the risk. and These represent the inverse functions of the probability density function and cumulative distribution function of the standard normal distribution, respectively.

[0081] By explicitly penalizing high-risk actions, the constraint policy operates within a safety threshold. The calculation formula is as follows: ; 3) Step three, adaptive adjustment of safety weights: Dynamically update the safety weights using the Lagrange multiplier method. The specific calculation formula is as follows: ; in, A preset cost threshold is set to ensure that the accumulated cost does not exceed the limit.

[0082] In conclusion, Figure 2 The overall implementation process of the underwater vehicle autonomous navigation method based on safety diffusion reinforcement learning is as follows: First, a state vector is constructed based on the underwater vehicle's environmental perception information. Then, this state vector is input into the initial conditional diffusion policy network to generate preliminary actions, and the preliminary actions are post-processed to obtain executable target actions. Subsequently, a composite loss function is constructed by combining the target action and the state vector, which integrates imitation learning loss, value enhancement loss, and safety constraint loss. This composite loss function is used to jointly optimize and train the initial conditional diffusion policy network to obtain the target conditional diffusion policy network. In the deployment phase, the real-time state vector is input into the trained target conditional diffusion policy network, which outputs smooth actions. After instruction mapping and dynamic constraint pruning, the smooth actions are transformed into actual control commands and issued to the actuators to drive the vehicle to perform navigation tasks. To further improve the policy performance and adaptability, the system continuously collects environmental interaction data during execution and periodically restarts the offline training process based on the accumulated new data (including environmental interaction data) to achieve iterative optimization of the policy network, thus forming an autonomous navigation closed loop of "offline training - online execution - data collection - iterative update".

[0083] The underwater vehicle autonomous navigation method based on safety diffusion reinforcement learning provided in this application adopts an offline training paradigm. The optimization and updating of the conditional diffusion policy network and the critic network are based on batch and iterative training using pre-collected or historically accumulated trajectory datasets. Specifically, during the training phase, the algorithm starts from the format (s, a, ... The static dataset (r, c, done) is sampled centrally, where r and c represent the immediate reward and cost calculated according to the preset reward function and cost function during the data collection phase, respectively. The environmental interaction data is represented by the new transition state s' and the termination flag done, which are used together to construct the training samples. Then, the joint optimization of the Conditional Diffusion Policy Network and the Critics Network is completed synchronously within the training loop. The Conditional Diffusion Policy Network is updated by minimizing a composite loss function that integrates imitation learning loss, value enhancement loss, and safety constraint loss. The value enhancement loss depends on the output Q(s,a) of the value critic network for the state-action pair (s,a), aiming to guide the policy to generate actions with high expected returns. The safety constraint loss depends on the mean and variance of the expected cumulative cost predicted by the cost critic network for the same input, using conditional risk value (CVaR) constraints to suppress high-risk behavior. Simultaneously, both critic networks are also optimized through training: the value critic is updated by minimizing the temporal difference error based on the reward r and the next state Q objective; the cost critic is updated by minimizing the 2-Wasserstein distance between its predicted cost distribution and the target cost distribution to accurately assess long-term risk. The entire training process is achieved by repeatedly sampling batches from the dataset, calculating the above losses, and performing gradient backpropagation and parameter updates.

[0084] Once the conditional diffusion policy network and the critic network converge during training, a target conditional diffusion policy network can be established, which can be directly used for online navigation decisions of AUVs. This process does not require further real-time optimization. However, to continuously improve policy performance, this application supports iterative offline training improvements: during the deployment of the trained policy execution task, newly collected interaction data (s, a, ...) are used to improve the policy performance. The data (r, c, done) can be stored in the experience pool and mixed with the old data to form a new training dataset. Then, the already trained target conditional diffusion policy network and critic network can be used as the starting point to restart the above offline training process on the new dataset, thereby achieving continuous evolution of the policy and continuous enhancement of security performance without the need for high-risk and inefficient online learning during task execution.

[0085] It should be noted that this embodiment is only a brief illustrative description of the overall process of the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that this application does not impose any limitations on this.

[0086] In addition, to verify the policy network navigation performance of the underwater vehicle autonomous navigation method based on security diffusion reinforcement learning provided in this application embodiment, this application embodiment also provides experimental procedures and experimental results: (1) Experimental setup: Based on the REMUS AUV dynamics model, to simulate complex underwater working conditions, the AUV operating space is set as a 30m×30m rectangular area, with the starting point fixed at (0m, 0m) and the target point fixed at (30m, 30m), with a straight-line distance of approximately 42m. This area includes 5-8 static obstacles (radius 0.5-2m, randomly distributed within the area), and the maximum time step per evaluation cycle is 600 steps. If the AUV fails to reach the target within 1m or collides with an obstacle within 60s (0.1s), the mission is considered a failure.

[0087] 1) Reward function settings: a. Target Rewards This feature incentivizes AUVs to continuously approach the target and complete the mission, improving navigation efficiency; target reward. The mathematical expression is as follows: ; in, Rewards for mission termination; weights ; The Euclidean distance between the AUV and the target location at the current moment is the core indicator for measuring the spatial distance between the AUV and the mission target. For the AUV at the previous time (t) Distance from the target location at time 1; Let be the distance between the AUV and the target location at the current time (time t). This is used to quantify the incremental distance an AUV moves toward a target in a single step. Positive increments result in a reward, incentivizing the AUV to continue approaching the target.

[0088] b. ORCA Guiding Rewards This feature reduces the exploration complexity of local obstacle avoidance by encouraging AUVs to match the collision avoidance velocity vector calculated by the ORCA algorithm. ORCA-guided reward. The mathematical expression is: ; in, , , To guide weights; The current heading angle of the AUV; Pi is the mathematical constant of a circle.

[0089] c. Obstacle Avoidance Rewards This design is based on the potential field method. The mathematical expression is as follows: ; in, As a penalty for collision, For a safe distance, Preventing numerical singularities and obstacle avoidance weights , These are the three obstacles closest to the AUV.

[0090] d. Energy efficiency bonus This measure optimizes AUV energy consumption, extends mission endurance, and rewards energy efficiency by penalizing high-energy-consuming behaviors. The mathematical expression is as follows: ; in, , This represents the current thrust output of the AUV's thruster. This represents the maximum output thrust of the AUV propulsion system. This represents the current rudder angle deflection of the AUV. This represents the maximum permissible deflection of the AUV's rudder angle. By applying a normalized sum of squares penalty to the thrust and rudder angle, precise constraints are achieved on the high-energy-consuming behavior of the AUV's power and steering.

[0091] 2) Training parameters: Adam optimizer is used, Mish activation function is used for policy network, ReLU activation function is used for critic network, diffusion step is 20, learning rate is 0.0001, and batch size is 256.

[0092] 3) Evaluation indicators: average trajectory length, collision avoidance success rate, and task time.

[0093] (2) Experimental results and analysis: The experimental results of the Secure Diffusion Reinforcement Learning (SDQ) method provided in this application embodiment under the above experimental settings in a 2D environment are shown in Table 1: Table 1: Comparison of Trajectory Data Indicators

[0094] As shown in Table 1, compared with unconstrained algorithms (Q-Diffusion, Diffusion), the Safety Diffusion Reinforcement Learning (SDQ) method provided in this application embodiment achieves the shortest path length and lowest task time in the 2D AUV navigation environment. Its collision cost is significantly lower than that of the unconstrained algorithms (Q-Diffusion, Diffusion), the obstacle avoidance success rate is close to 100%, and tail risk can be effectively controlled, thereby improving navigation efficiency. It should be noted that the Safety Diffusion Reinforcement Learning (SDQ) method provided in this application embodiment can be extended to the 3D AUV navigation environment.

[0095] In summary, the underwater vehicle autonomous navigation method based on security diffusion reinforcement learning provided in this application aims to solve the following technical problems: (1) How to model multimodal behavior in complex continuous action space and overcome the problem of insufficient representation ability of traditional reinforcement learning strategies; (2) How to improve the performance of the policy in the offline learning framework and solve the problem of overestimation of out-of-distribution actions by the policy; (3) How to combine diffusion models to learn multimodal strategies from data and avoid pure imitation learning being limited by the quality of the dataset; (4) How to explicitly embed safety constraints, control tail risk behaviors, and ensure the physical safety of AUVs in unknown environments.

[0096] To address the aforementioned technical problems, the autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning provided in this application offers the following key technologies: (1) AUV policy representation method based on diffusion model: By using the conditional diffusion model as the policy network, it is possible to effectively model multimodal and asymmetric action distribution. The conditional diffusion model generates navigation actions that conform to dynamic constraints through a multi-step denoising process, overcoming the limitations of traditional reinforcement learning policy representation.

[0097] (2) Safety diffusion reinforcement learning loss function: The loss function includes imitation generation term, value enhancement term and safety cost control term. The strategy optimization and risk constraint are achieved through weighted combination. Among them, the tail risk is explicitly controlled through conditional risk constraint, and the collision cost is significantly lower than that of the unconstrained algorithm.

[0098] (3) An offline safety reinforcement learning framework for AUV navigation: It integrates diffusion strategy, value and cost commentator, and dynamic safety weight adjustment mechanism to achieve efficient and safe autonomous navigation; moreover, the offline safety reinforcement learning framework is trained entirely on static datasets, which can avoid the risks of online exploration and is suitable for actual AUV deployment.

[0099] Please see Figure 3 This application also provides an autonomous navigation device 300 for underwater vehicles based on security diffusion reinforcement learning, which can implement the above-described method. The device includes the following modules: The training state vector construction module 301 is used to obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information. The target action generation module 302 is used to input the state vector to be trained into the initial conditional diffusion policy network, generate the smooth action to be trained, and perform post-processing on the smooth action to be trained to obtain the target action to be trained. The composite loss function construction module 303 is used to construct a composite loss function based on the target action to be trained and the state vector to be trained; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; The joint optimization training module 304 for policy network is used to perform joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain the target conditional diffusion policy network. The current smooth action generation module 305 is used to input the current state vector corresponding to the current underwater vehicle into the target condition diffusion strategy network to generate the current smooth action; The current target action generation module 306 is used to perform post-processing on the current smooth action to obtain the current target action, and output the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

[0100] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0101] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0102] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0103] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 using the methods described above in the embodiments of this application. Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0104] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0105] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0107] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The autonomous navigation method and related equipment for underwater vehicles based on safety diffusion reinforcement learning provided in this application embodiment acquire the environmental perception state information of the underwater vehicle to be trained, and construct a training state vector based on the environmental perception state information; input the training state vector into an initial conditional diffusion policy network to generate a training smooth action, and perform post-processing on the training smooth action to obtain a training target action; construct a composite loss function based on the training target action and the training state vector; wherein the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the safety constraint loss corresponding to the cost critic network; perform joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain a target conditional diffusion policy network; input the current state vector corresponding to the current underwater vehicle into the target conditional diffusion policy network to generate a current smooth action; perform post-processing on the current smooth action to obtain the current target action, and output the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target action. This application embodiment uses a conditional diffusion model as the policy network, which can effectively model multimodal and asymmetric action distributions. It accurately generates navigation actions that conform to dynamic constraints through a multi-step denoising process, which can overcome the representation limitations of traditional reinforcement learning strategies. By constructing a composite loss function that includes imitation generation terms, value enhancement terms, and safety cost control terms, it can achieve policy optimization and risk constraints on the policy network, thereby improving the efficiency and safety of autonomous navigation of underwater vehicles and reducing the risks of online exploration and collisions.

[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An autonomous navigation method for underwater vehicles based on security diffusion reinforcement learning, characterized in that, The method includes the following steps: Obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information; The state vector to be trained is input into the initial conditional diffusion policy network to generate a smooth action to be trained, and the smooth action to be trained is post-processed to obtain the target action to be trained. Based on the target action to be trained and the state vector to be trained, a composite loss function is constructed; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; The initial conditional diffusion policy network is jointly optimized and trained according to the composite loss function to obtain the target conditional diffusion policy network. The current state vector corresponding to the current underwater vehicle is input into the target conditional diffusion strategy network to generate the current smooth action; The current smooth action is post-processed to obtain the current target action, and the current target action is output to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

2. The method according to claim 1, characterized in that, The method further includes: During the process of the current underwater vehicle performing the current target action, environmental interaction data is collected; When the preset offline training conditions are met, the environmental perception state information to be trained is updated according to the environmental interaction data. Based on the updated environment-aware state information to be trained, return to the step of constructing the state vector to be trained based on the environment-aware state information to be trained, until the updated target conditional diffusion policy network is obtained.

3. The method according to claim 1, characterized in that, The construction of a composite loss function based on the target action to be trained and the state vector to be trained includes: The target action to be trained and the state vector to be trained are concatenated to obtain the state-target action vector; The state-target action vector is input into the value critic network to generate state-action value; The state-target action vector is input into the cost critic network to generate the expected cumulative cost mean and cumulative cost variance. The imitation learning loss is determined based on the denoising loss of the initial conditional diffusion strategy network. Based on the state-action value, determine the value enhancement loss; The safety constraint loss is determined based on the expected average cumulative cost and the cumulative cost variance. The imitation learning loss, the value enhancement loss, and the security constraint loss are weighted and fused to obtain the composite loss function.

4. The method according to claim 1, characterized in that, The method further includes: In constructing the composite loss function, the Lagrange multiplier method is used to dynamically update the safety weights corresponding to the safety constraint loss.

5. The method according to claim 1, characterized in that, The step of inputting the current state vector corresponding to the current underwater vehicle into the target conditional diffusion policy network to generate the current smooth action includes: Obtain the current environmental perception state information corresponding to the current underwater vehicle, and construct the current state vector based on the current environmental perception state information; In the target conditional diffusion strategy network, the current state vector and the current noise action are concatenated to obtain the current state-noise action vector; The current state-noise action vector is input into the noise prediction network in the target conditional diffusion strategy network to generate the current estimated noise. The current estimated noise is denoised according to the denoising formula by a preset number of diffusion steps to obtain the current smoothing action.

6. The method according to claim 1, characterized in that, The step of performing post-processing on the current smoothed action to obtain the current target action, and outputting the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle executes the current target action, includes: The current smooth action is processed by instruction mapping to obtain the current control instruction corresponding to the current underwater vehicle; wherein, the current control instruction includes thrust instruction and rudder angle instruction; Based on the dynamic constraints of the current underwater vehicle, the current control command is trimmed to obtain the target control command; The target control command is output as the current target action to the actuator of the current underwater vehicle, so that the current underwater vehicle performs the current target action.

7. An autonomous navigation device for underwater vehicles based on security diffusion reinforcement learning, characterized in that, The device includes the following modules: The training state vector construction module is used to obtain the training environment perception state information corresponding to the underwater vehicle, and construct the training state vector based on the training environment perception state information. The training target action generation module is used to input the training state vector into the initial conditional diffusion policy network, generate the training smooth action, and perform post-processing on the training smooth action to obtain the training target action. The composite loss function construction module is used to construct a composite loss function based on the target action to be trained and the state vector to be trained; wherein, the composite loss function includes the imitation learning loss corresponding to the initial conditional diffusion policy network, the value enhancement loss corresponding to the value critic network, and the security constraint loss corresponding to the cost critic network; The policy network joint optimization training module is used to perform joint optimization training on the initial conditional diffusion policy network according to the composite loss function to obtain the target conditional diffusion policy network. The current smooth action generation module is used to input the current state vector corresponding to the current underwater vehicle into the target conditional diffusion strategy network to generate the current smooth action; The current target action generation module is used to perform post-processing on the current smooth action to obtain the current target action, and output the current target action to the actuator of the current underwater vehicle so that the current underwater vehicle can execute the current target action.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images