Water download control method and computer device

Abstract

This disclosure relates to a method and computer device for controlling underwater vehicles. It pertains to the field of underwater navigation and addresses the safety and mission execution difficulties caused by underwater vehicles being detected. The method includes: constructing an underwater environment simulation model; acquiring passive detection data during the simulated operation of the underwater vehicle based on the underwater environment simulation model; training an explicit risk memory model based on the passive detection data from the underwater vehicle's operational history; generating a multi-dimensional observation vector indicating the current probability of detection based on the risk memory features output by the explicit risk memory model and instantaneous operational data; and performing memory fusion based on the multi-dimensional observation vector and the hidden state of the previous time step of the gated loop unit to output a normalized angular velocity control command to control the underwater vehicle. The technical solution provided by this disclosure is applicable to complex underwater navigation scenarios, achieving efficient and accurate autonomous decision-making for detection avoidance.

Description

Technical Field

[0001] This disclosure relates to the field of automatic control of underwater vehicles, and more particularly to an underwater vehicle control method and computer device. Background Technology

[0002] In scenarios such as underwater resource exploration, marine environmental monitoring, underwater equipment inspection, and scientific research observation, the autonomous operation safety and stealth of underwater vehicles are crucial. During missions, underwater vehicles often operate in environments with other underwater detection platforms (such as those used for marine monitoring and search and rescue). Because the ownership of these platforms is unknown, as are their detection objectives, the range of information they collect, and the frequency bands they use, these platforms, especially clusters of multiple platforms, can pose safety hazards to underwater vehicles. Furthermore, the detection signals emitted by these platforms can interfere with the underwater vehicle's data acquisition and control signal reception, causing control errors or data distortion, thus affecting the underwater vehicle's mission performance.

[0003] Therefore, the ability of underwater vehicles to avoid detection has a decisive impact on mission success rate, the effectiveness of collected data, and their own safety, and a reliable underwater vehicle detection avoidance mechanism is urgently needed. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an underwater vehicle control method and computer device. In a simulated underwater environment, based on an explicit risk memory model and dual-stream memory fusion, efficient and accurate autonomous decision-making for detection avoidance is achieved, solving the safety and mission execution difficulties of underwater vehicles caused by detection.

[0005] According to a first aspect of the embodiments of this disclosure, an underwater body control method is provided, comprising:

[0006] An underwater environment simulation model is constructed, which includes a basic scene, a partially observable space, and a dynamic model. The dynamic model contains modeling rules for underwater bodies and at least one detection vehicle, and multiple detection vehicles constitute a detection vehicle cluster.

[0007] Based on the underwater environment simulation model, passive detection data during the simulated operation of the underwater body is obtained;

[0008] Explicit risk memory model trained based on passive detection data during the operation history of the underwater vehicle;

[0009] Based on the risk memory features and instantaneous operational data output by the explicit risk memory model, a multidimensional observation vector indicating the current probability of being detected is generated. The instantaneous operational data includes at least the current navigation status of the underwater vehicle and / or the instantaneous sonar observation data of the underwater vehicle.

[0010] The memory is fused based on the multidimensional observation vector and the hidden state of the gated loop unit (GRU) at the previous moment, and a normalized angular velocity control command is output to control the underwater body to reduce the possibility of being detected.

[0011] Furthermore, the step of training the explicit risk memory model based on passive detection data during the underwater vehicle's operational history includes:

[0012] The space surrounding the underwater vehicle is divided into multiple sectors;

[0013] The probability of the underwater vehicle being detected by the detection carrier cluster in the direction pointed to by each sector is dynamically updated. The probability of being detected indicates the likelihood that the underwater vehicle will be detected by the detection source in the corresponding sector.

[0014] Furthermore, the step of dynamically updating the detection probability of the underwater vehicle being detected by the detection carrier cluster in the direction pointed to by each sector includes:

[0015] The detected probability is dynamically updated through multi-step simulation. Each simulation step includes:

[0016] The probability of being detected in all of the aforementioned sectors at the previous time step is exponentially decayed.

[0017] For the sector where a probe signal is detected, the corresponding probe probability is updated with enhancement according to the following expression:

[0018] ,

[0019] Where t is the sequence number of the current simulation step, To update the strength coefficient, Let be the detected probability of the k-th sector after the t-th simulation step. Let be the detected probability of the k-th sector after the t-th simulation step.

[0020] Furthermore, the step of performing memory fusion based on the multidimensional observation vector and the hidden state of the GRU at the previous time step, and outputting a normalized angular velocity control command to control the underwater vehicle includes:

[0021] The multidimensional observation vector is encoded and its features are extracted by the feature extraction layer of a dual-stream memory deep neural network to obtain a feature-encoded vector.

[0022] The temporal reasoning layer of the dual-stream memory deep neural network performs temporal reasoning based on the feature encoding vector and the hidden state of the GRU at the previous time step to obtain the hidden state at the current time step.

[0023] The decision output layer of the dual-stream memory deep neural network outputs the hidden state at the current moment, and generates and outputs the normalized angular velocity control command.

[0024] Furthermore, after the step of obtaining the hidden state at the current time by performing temporal inference through the temporal inference layer of the dual-stream memory deep neural network based on the feature encoding vector and the hidden state of the GRU at the previous time step, the method further includes:

[0025] Based on the hidden state at the current moment, the decision output layer of the dual-stream memory deep neural network outputs a value estimate of the current state of the underwater vehicle.

[0026] Furthermore, the method also includes:

[0027] During the deployment phase, the branch that outputs the value estimate in the decision output layer is removed.

[0028] Furthermore, the method also includes:

[0029] Calculate the propulsion bonus according to the following expression. :

[0030] ,

[0031] in, t is the sequence number of the current simulation step, and t-1 is the sequence number of the previous simulation step. The speed of the underwater vehicle. For simulating step size, The distance between the underwater vehicle and the finish line at the previous moment is the longitudinal distance. The distance between the underwater vehicle and the finish line at the current moment is the longitudinal distance. H represents the position of the finish line. The current position of the underwater vehicle;

[0032] The penalty for secondary detection is calculated using the following expression. :

[0033] ,

[0034] in, For indicator functions, Basic penalty coefficient, For cumulative penalty coefficient, The maximum allowable detection threshold;

[0035] The following expressions are used to calculate the computer dynamic smoothing and energy consumption regularization terms. :

[0036] ,

[0037] in, For smoothing coefficients, The normalized angular velocity at the current moment;

[0038] Based on the execution result of the underwater vehicle reaching the finish line, a sparse termination reward is generated. ;

[0039] Calculate the composite reward function based on the following expression. :

[0040] .

[0041] Furthermore, the method also includes:

[0042] Based on the observation data of the simulation environment, the operation data of the underwater vehicle, and the composite reward function, the dual-stream memory deep neural network for memory fusion is iteratively trained.

[0043] Furthermore, the step of iteratively training the dual-stream memory deep neural network for memory fusion includes:

[0044] The simulation environment is randomized and reset in each training round;

[0045] Different training scenarios are set according to the stages of iterative training, and the policy update magnitude is limited by the proximal policy optimization algorithm to perform multiple rounds of iterative training on the dual-stream memory deep neural network.

[0046] According to a second aspect of the embodiments of this disclosure, a computer apparatus is provided, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to execute the above-described underwater body control method.

[0050] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: constructing an underwater environment simulation model, then acquiring passive detection data during the simulated operation of the underwater vehicle based on the underwater environment simulation model, training an explicit risk memory model based on the passive detection data during the underwater vehicle's operational history, generating a multi-dimensional observation vector indicating the current probability of detection based on the risk memory features output by the explicit risk memory model and instantaneous operational data, and finally performing memory fusion based on the multi-dimensional observation vector and the hidden state of the GRU at the previous moment, outputting a normalized angular velocity control command to control the underwater vehicle to reduce the probability of detection. This achieves efficient and accurate autonomous decision-making for detection avoidance, solving the safety and mission execution difficulties caused by detection of underwater vehicles.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0053] Figure 1 This is a flowchart illustrating an underwater body control method according to an exemplary embodiment.

[0054] Figure 2 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0055] Figure 3 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0056] Figure 4 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0057] Figure 5 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0058] Figure 6 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0059] Figure 7 This is a flowchart illustrating yet another underwater body control method according to an exemplary embodiment.

[0060] Figure 8 This is a schematic diagram illustrating the principle of an explicit risk memory model according to an exemplary embodiment.

[0061] Figure 9 This is a schematic diagram illustrating the principle of a dual-stream memory deep neural network according to an exemplary embodiment.

[0062] Figure 10 This is a schematic diagram illustrating the principle of a training process according to an exemplary embodiment. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0064] The ability of underwater vehicles to avoid detection has a decisive impact on mission success rate, the effectiveness of collected data, and their own safety, thus a reliable underwater vehicle detection avoidance mechanism is urgently needed.

[0065] To address the aforementioned problems, embodiments of this disclosure provide an underwater vehicle control method and computer device. In a simulated underwater environment, based on an explicit risk memory model and dual-stream memory fusion, efficient and accurate autonomous decision-making for detection avoidance is achieved, resolving the safety and mission execution difficulties caused by underwater vehicles being detected.

[0066] An exemplary embodiment of this disclosure provides an underwater vehicle control method. This method constructs an explicit risk memory model to transform sparse passive direction-finding signals into a potential field of detectable probability with spatiotemporal continuity. It combines GRU and Proximal Policy Optimization (PPO) algorithms to achieve adaptive control of the detectable probability and global optimization of the concealed navigation trajectory within a deep reinforcement learning framework.

[0067] This method enables underwater vehicles and other underwater vehicles to actively control themselves based on the detection platform, thereby avoiding detection processes such as... Figure 1 As shown, it includes:

[0068] Step 101: Construct an underwater environment simulation model.

[0069] The underwater environment simulation model includes a basic scene, a partially observable space, and a dynamic model. The dynamic model contains modeling rules for underwater vehicles and at least one detection vehicle, and multiple detection vehicles constitute a detection vehicle cluster.

[0070] According to one exemplary embodiment, the underwater vehicle is an underwater vehicle, such as an autonomous underwater vehicle (AUV), a remotely operated underwater vehicle (ROV), etc.

[0071] The detection carrier includes, but is not limited to:

[0072] Marine environmental monitoring buoys and underwater monitoring stations, seabed observation platforms, underwater vehicles, marine surveying and exploration equipment, waterway safety monitoring equipment, fishery resource monitoring equipment, offshore wind power equipment, underwater engineering inspection equipment, etc. Detection vehicles can be fixed or moved underwater, achieving target detection and environmental perception through acoustic and underwater acoustic methods.

[0073] In this step, a simulation environment is constructed in a two-dimensional plane for the underwater vehicle and multiple moving probe carriers. The underwater vehicle is set to perform a navigation mission from the starting point to the finish line in the two-dimensional plane, and multiple probe carriers with active detection capabilities are deployed in the environment.

[0074] According to one exemplary implementation, the modeling rules for the underwater vehicle include the kinematic equations of the underwater vehicle, which are defined as an underactuated moving point mass that maintains a constant speed. Its action space is defined as a continuous angular velocity control command, which is strictly constrained by the maximum physical turning angular velocity.

[0075] Simultaneously, a partially observable space of the system is defined, which simulates the characteristics of passive sonar and only includes the azimuth information of the probe relative to the underwater vehicle, without including distance and specific quantity information of the probe, thereby constructing a partially observable Markov decision process (POMDP).

[0076] According to one exemplary embodiment, the underwater hull is modeled as a two-dimensional moving point mass, maintaining a fixed speed V, and only changing its heading angle. To perform maneuvers. Its kinematic model is as follows:

[0077]

[0078] in, Let be the instantaneous velocity component of the underwater vehicle in the x-axis direction. Let be the instantaneous velocity component of the underwater vehicle in the y-axis direction. The heading angle of the underwater vehicle. It is the angular velocity control input, and is limited by the maximum physical steering angular velocity, satisfying... .

[0079] According to an exemplary embodiment, in this step, the underwater vehicle relies on passive sonar to sense the detection carrier, specifically, to sense the detection signal emitted by the detection carrier. A butterfly-shaped sector centered on the underwater vehicle is the passive sonar observation area of ​​the underwater vehicle, divided into a forward sector and a lateral sector. When the detection carrier's active sonar pulse is within this area, the underwater vehicle can acquire a relative azimuth angle measurement value with Gaussian white noise; if the detection carrier is in a blind zone or does not emit a pulse, no signal is observed.

[0080] Step 102: Obtain passive detection data during the simulated operation of the underwater body based on the underwater environment simulation model.

[0081] In this step, an underwater environment simulation model is used to simulate the actual operation of the underwater submersible and the detection capabilities of the probe. Data is continuously collected to obtain passive detection data during the submersible's operation.

[0082] Step 103: Train an explicit risk memory model based on passive detection data from the underwater vehicle's operational history.

[0083] To address the issues of state ambiguity and information forgetting caused by pure orientation observation, this step involves constructing an explicit risk memory model in the local coordinate system of the underwater body.

[0084] According to one exemplary implementation, a sector-based explicit risk memory model is constructed to include the area around the underwater vehicle. The panoramic space is evenly divided into Each sector is assigned a unique, dynamically updated probability of detection over time. The probability of being detected is used to quantify the likelihood that the sector direction will be detected by a detection source such as a detection vehicle in the near future.

[0085] According to one exemplary implementation, the number of sectors Set to 16, meaning each sector covers The directional range;

[0086] Global attenuation coefficient The range of values ​​is For example, 0.05, to ensure that historical information gradually fades away within a few seconds to tens of seconds, so as to avoid outdated threat information interfering with current decision-making;

[0087] Update intensity coefficient The range of values ​​is For example, 0.5, to achieve rapid response and smooth recording of sudden threat signals.

[0088] Step 104: Based on the risk memory features and instantaneous running data output by the explicit risk memory model, generate a multi-dimensional observation vector indicating the current probability of being detected.

[0089] The instantaneous operational data includes at least the current navigation status of the underwater vehicle and / or the instantaneous sonar observation data of the underwater vehicle.

[0090] Taking the sector-based explicit risk memory model as an example, in order to transform the discrete sector probability distribution into continuous spatial features that can be understood by neural networks, this step calculates a synthetic detection direction vector that reflects the current overall detection situation based on the detection probability of each sector. The synthesized detection direction vector is obtained by weighted summation and normalization of the center angle of each sector and its corresponding detection probability, indicating the statistically densest detection direction at the current moment after comprehensively considering historical observation information.

[0091] In this step, a multidimensional observation vector is constructed for inputting the reinforcement learning policy network. According to one exemplary implementation, the multidimensional observation vector may specifically be a multidimensional normalized observation vector. The multidimensional normalized observation vector It is composed of the following three features:

[0092] (1) The underwater vehicle's own navigation status: including the normalized coordinates of the underwater vehicle's own position. and the sine and cosine values ​​of the heading angle Where W is the scene width, i.e., the range of the simulation environment in the x-axis direction, and H is the scene height, i.e., the range of the simulation environment in the y-axis direction. , () represents the global coordinates of the underwater object.

[0093] (2) Instantaneous sonar observation: including signal presence flags And the sine and cosine values ​​of the relative azimuth of the main detection source detected at the current moment;

[0094] (3) Sector threat memory characteristics: including all The real-time detected probability sequence of each sector and the calculated synthetic detected direction vector.

[0095] According to an exemplary implementation, the synthesized detected direction vector is calculated based on the following expression. :

[0096] First, calculate the unnormalized weighted sum vector. :

[0097] ,

[0098] in, For the first The center angle of each sector Let be the probability of the k-th sector being detected at time t or at the t-th simulation step.

[0099] like Then calculate the normalized vector. ;like If the vector is zero, then set it to zero. This vector intuitively indicates the statistical direction that is most densely detected at the current moment after integrating historical observations.

[0100] Step 105: Perform memory fusion based on the multidimensional observation vector and the hidden state of the GRU at the previous time step, and output a normalized angular velocity control command to control the underwater vehicle.

[0101] In this step, a dual-stream memory deep neural network is used to fuse explicit spatial control memory with implicit temporal situational memory. This allows the neural network to perceive the current spatial risks being detected while also inferring the historical operational trends of the probe. Based on the dual-stream memory deep neural network, control decisions are made regarding underwater probes, moving them away from probes and / or areas where the probability of detection is high, thereby reducing the likelihood of detection.

[0102] According to one exemplary implementation, the architecture of a dual-stream memory deep neural network includes a feature extraction layer and a temporal inference layer. Specifically, the feature extraction layer can be a multilayer perceptron (MLP), which maps a 25-dimensional multidimensional observation vector into a 128-dimensional high-dimensional feature encoding vector.

[0103] The temporal inference layer can specifically be a GRU, which receives the feature encoding vector and the hidden state from the previous time step. Output the hidden state at the current time. Implicitly inferring the motion state of the detection vehicle.

[0104] According to one exemplary implementation, the architecture of the dual-stream memory deep neural network also includes dual-head output ends. Specifically, these include a policy head (Actor) and a value head (Critic).

[0105] The dual-stream memory deep neural network first uses an MLP as a feature extractor to perform preliminary feature encoding and extraction on multi-dimensional observation vectors, thereby obtaining high-dimensional feature encoding vectors.

[0106] Subsequently, a GRU (Geostationary Recurrent Memory Unit) is introduced as the core to receive the feature encoding vector and the hidden state of the previous time step as input, and output the hidden state of the current time step. Through the GRU's unique gating mechanism, long-term historical observation information is accumulated in the hidden state vector, thereby realizing the inference of the motion trajectory, speed, and cooperative encirclement situation of the detection vehicle at the implicit level.

[0107] Finally, the output is provided through the policy head (Actor) and value head (Critic) of the output layer. The policy head is responsible for outputting the Gaussian parameters (i.e., mean and standard deviation) of the action probability distribution, which are used to sample and generate normalized angular velocity control commands that conform to physical constraints; the value head is responsible for outputting the value estimate of the current state, which is used to calculate the advantage function during training to assist in policy updates.

[0108] An exemplary embodiment of this disclosure also provides an underwater vehicle control method, in which an explicit risk memory model is trained based on passive detection data from the underwater vehicle's operational history, as follows: Figure 2 As shown, it includes:

[0109] Step 201: Divide the space around the underwater vehicle into multiple sectors.

[0110] To address the state ambiguity and information forgetting issues arising from pure orientation observations, this step constructs an explicit sector threat memory mechanism within the local coordinate system of the underwater vehicle. This mechanism is used to define the threat memory of the surrounding areas of the underwater vehicle. The panoramic space is evenly divided into Each sector.

[0111] Step 202: Dynamically update the detection probability of the underwater vehicle being detected by the detection carrier cluster in the direction pointed to by each sector.

[0112] The probability of being detected indicates the likelihood that the underwater object will be detected by a detection source in the corresponding sector.

[0113] In this step, a dynamically updated detection probability is maintained for each sector. The probability of being detected is used to quantify the likelihood that the sector direction will be scanned by a detection source or that a detection source exists in the near future.

[0114] Then, the detected probability is continuously and dynamically updated. A dual-parallel mechanism of global attenuation and hit enhancement is employed to transform the discrete, sparse azimuth signal into a continuous spatial probability distribution with historical attenuation characteristics. The specific process is as follows: Figure 3 As shown, it includes:

[0115] The detected probability is dynamically updated through multi-step simulation. Each simulation step includes:

[0116] Step 301: Perform exponential decay on the detection probability of all sectors at the previous time step.

[0117] In this step, global decay is performed. In each simulation step, the probe probability of all sectors is exponentially decayed to simulate the gradual loss of historical information. This can be done using the following expression:

[0118] ,

[0119] in This is the attenuation coefficient.

[0120] Step 302: For the sector where the detection signal is detected, the detection probability is enhanced and updated.

[0121] In this step, within a simulation step, selective enhancement updates are performed on certain sectors. Specifically, hit enhancement is performed on sectors where detection signals are detected. For example, when the passive sonar is at the current moment... ( It also represents the sequence number of the current simulation step. (The previous time step "t-1" represents the (t-1)th simulation step.) When the detection signal emitted by the probe carrier is detected, the relative azimuth angle of the detection signal is calculated. Index of the sector to which it falls For this sector The threat probability has been enhanced and updated.

[0122] The corresponding detected probability is enhanced and updated according to the following expression:

[0123] ,

[0124] Where t is the sequence number of the current simulation step, To update the strength coefficient, Let k be the detected probability of the k-th sector after the simulation step is updated. The probability of being detected is updated for the k-th sector in the (t-1)-th simulation step.

[0125] An exemplary embodiment of this disclosure also provides an underwater vehicle control method, which uses a dual-stream memory deep neural network for memory fusion. Through the dual-stream fusion of explicit spatial detected memory and implicit temporal situational memory, the method achieves understanding and reasoning of the underwater detected or discovered situation. The process of controlling the underwater vehicle by performing memory fusion based on the multidimensional observation vector and the hidden state of the GRU at the previous time step, and outputting a normalized angular velocity control command, is as follows: Figure 4 As shown, it includes:

[0126] Step 401: Encode and extract features from the multidimensional observation vector using the feature extraction layer of a dual-stream memory deep neural network to obtain a feature encoding vector.

[0127] In this step, multidimensional observation vectors are received, the distribution of the current underwater space probability of being detected is memorized, and the detection probability information of discrete sectors is transformed into continuous features that can be processed by a dual-stream memory deep neural network to achieve accurate perception in the spatial dimension.

[0128] This step can be implemented using MLP to extract and integrate features from multidimensional observation vectors to obtain high-dimensional feature encoding vectors.

[0129] Step 402: The temporal reasoning layer of the dual-stream memory deep neural network performs temporal reasoning based on the feature encoding vector and the hidden state of the GRU at the previous time step to obtain the hidden state at the current time step.

[0130] In this step, to adapt to the application requirements of underwater long-term navigation that may be detected, the GRU gating mechanism is adopted based on feature vector encoding to dynamically perform temporal reasoning to obtain the hidden state at the current moment. The position and / or motion trend of the detection vehicle are inferred and predicted in the time dimension to ensure the accuracy of underwater vehicle control.

[0131] Step 403: Output the hidden state at the current moment through the decision output layer of the dual-stream memory deep neural network, and generate and output the normalized angular velocity control command.

[0132] In this step, the feature encoding vector and the hidden state are fused, and the fusion result is fed into the output layer. The output layer contains a decision head branch, which can be an Actor head. The decision head can output a normalized angular velocity control command, instructing the underwater vehicle to control its subsequent navigation according to the normalized angular velocity control command, so as to avoid or minimize the possibility of being detected.

[0133] According to one exemplary implementation, in this step, the parameters of the Gaussian distribution are output. (mean) and (Standard deviation), sampled and cropped to generate normalized angular velocity control commands. .

[0134] An exemplary embodiment of this disclosure also provides an underwater vehicle control method. This method, through a dual-headed output layer, outputs control commands for the underwater vehicle's navigation, as well as a value estimate of the current state. The specific process is as follows: Figure 5 As shown, it includes:

[0135] Step 501: Construct an underwater environment simulation model.

[0136] Step 502: Obtain passive detection data during the simulated operation of the underwater body based on the underwater environment simulation model.

[0137] Step 503: Train an explicit risk memory model based on passive detection data from the underwater vehicle's operational history.

[0138] Step 504: Based on the risk memory features and instantaneous running data output by the explicit risk memory model, generate a multi-dimensional observation vector indicating the current probability of being detected.

[0139] Step 505: Perform memory fusion based on the multidimensional observation vector and the hidden state of the GRU at the previous time step, and output a normalized angular velocity control command to control the underwater vehicle.

[0140] The implementation principles of steps 501 to 505 are the same as those of steps 101 to 105, and will not be repeated here.

[0141] Step 506: Based on the hidden state at the current moment, output the value estimate of the current state of the underwater body through the decision output layer of the dual-stream memory deep neural network.

[0142] In this embodiment, the output layer includes two branches: a decision head and a value head. The value head branch is specifically the Critic head, which outputs the value estimate of the current state. This provides valuable reference for the training phase.

[0143] According to one exemplary implementation, the current hidden state of the GRU can be... Actor head motion distribution parameters (μ, σ), Critic head value estimation This serves as the basis for subsequently developing the reward function and iteratively updating the parameters of the dual-stream memory deep neural network.

[0144] An exemplary embodiment of this disclosure also provides an underwater body control method, which constructs a composite reward function to avoid the inability of the subsequent training process to converge stably due to a single penalty or reward constraint.

[0145] To guide the policy network in finding the optimal balance between safe underwater navigation and increasing the probability of not being detected, this embodiment employs a composite reward function that includes a stealth navigation incentive, a cumulative exposure penalty, a maneuvering smoothing regularization term, and a sparse termination reward. .

[0146] Specifically, it includes:

[0147] 1. Calculate the propulsion bonus according to the following expression. :

[0148] ,

[0149] in, t is the sequence number of the current simulation step, and t-1 is the sequence number of the previous simulation step. The speed of the underwater vehicle. For simulating step size, This refers to the longitudinal distance between the underwater body and the endpoint line, collected at the previous moment or in the previous simulation step. This refers to the longitudinal distance between the underwater body and the finish line, collected at the current moment or in the current simulation step. H represents the position of the finish line. This represents the current position of the water-based body. This is the normalized longitudinal distance increment for moving towards the target defense line.

[0150] 2. Calculate the penalty for secondary detection based on the following expression. :

[0151] ,

[0152] in, For indicator functions, Basic penalty coefficient, For cumulative penalty coefficient, The maximum allowable detection threshold, For the duration of continuous detection, This indicates that the underwater object was detected or irradiated at the t-th simulation step or time t.

[0153] According to an exemplary implementation:

[0154] For the indicator function, the current time A value of 1 indicates that the device is detected by any detection vehicle; otherwise, a value of 0 indicates that the device is detected by any detection vehicle.

[0155] The cumulative duration of continuous detection up to the current moment (in seconds);

[0156] The maximum allowable detection threshold is set to 120 seconds. If the duration of continuous exposure / detection exceeds this time, the task is considered to have failed.

[0157] The base penalty coefficient is the single-step hit penalty coefficient, used to provide basic instantaneous negative feedback;

[0158] This is the cumulative penalty coefficient, used to generate a sharply increased penalty signal when the detected time approaches the maximum permissible detected threshold.

[0159] 3. Based on the following expressions, calculate the computer's dynamic smoothing and energy consumption regularization terms. :

[0160] ,

[0161] in, For smoothing coefficients, This is the normalized angular velocity at the current moment.

[0162] The norm 2 penalty for the angular velocity command, i.e. To reduce high-frequency jitter.

[0163] 4. Based on the execution result of the underwater vehicle reaching the finish line, generate a sparse termination reward. .

[0164] 5. Calculate the composite reward function based on the following expression. :

[0165] .

[0166] Among them, cumulative exposure penalty item Introduced the variable of continuous detection duration And established a kind of A dynamic penalty mechanism exhibiting a non-linear growth relationship. Specifically, when the underwater object is under detection, the penalty increases with continuous exposure time. Gradually approaching the maximum permissible exposure threshold , This will increase dramatically, quadratically. This forces the strategy network to automatically suppress the pursuit of the shortest path in critical moments, prioritizing large-scale evasive maneuvers to break the enemy's continuous lock-on, thereby significantly reducing the risk of being destroyed.

[0167] An exemplary embodiment of this disclosure also provides an underwater vehicle control method, which iteratively trains a dual-stream memory deep neural network for memory fusion based on observation data from a simulation environment, operational data of the underwater vehicle, and the composite reward function. By combining proximal policy optimization (PPO) with curriculum learning, iterative training is achieved in complex scenarios with amplitude limiting and gradient boosting to increase the difficulty of detection or discovery, thereby improving the training convergence speed and the performance of the final policy model.

[0168] The specific process is as follows: Figure 6 As shown, it includes:

[0169] Step 601: Randomize the simulation environment in each training round.

[0170] This step involves randomizing the simulation environment at the start of each training round (e.g., Episode).

[0171] According to one exemplary implementation, randomization environment reset involves at least one or more of the following parameters:

[0172] The initial abscissa of the underwater vehicle and the initial distribution position of at least one detection vehicle are randomly initialized at the starting line and the ending line, respectively.

[0173] The search strategy type for the detection vehicle cluster is randomly selected, including but not limited to constant speed cruise search, zigzag maneuver search, and cooperative encirclement search;

[0174] By injecting Gaussian noise of random intensity into the observed signal to simulate the uncertainty of the real underwater acoustic environment, the reinforcement learning agent is forced to make decisions not by over-relying on the accuracy of single-frame observations, but by making more use of the spatiotemporal features extracted by GRU and sector memory.

[0175] Furthermore, the number of simulation steps within each training round and the step size of each simulation step can be set at the beginning of each training round to avoid ineffective long-duration detection.

[0176] Step 602: Set different training scenarios according to the stage of iterative training, and limit the policy update range through the proximal policy optimization algorithm to perform multiple rounds of iterative training on the dual-stream memory deep neural network.

[0177] In this step, the PPO algorithm is used to train the policy network end-to-end. During the training process, truncated backpropagation (TBPTT) is used to handle long temporal dependency updates of GRU.

[0178] To enhance the model's generalization ability and robustness, a curriculum learning strategy is adopted: a low threat level is set in the early stages of training (e.g., few probes and low maneuverability). As the number of training rounds increases, the number of probes, their maneuverability, and the randomness of their initial deployment are gradually increased until a preset high-intensity coverage scenario is achieved. After training, the converged policy network is deployed on the underwater vehicle's control system to achieve real-time online simulation and control.

[0179] According to an exemplary implementation, in this step, the magnitude of parameter updates in each iteration is limited by the PPO truncation mechanism, and then optimized by combining the value loss of the output of the dual-stream memory deep neural network, and all parameters of the dual-stream memory deep neural network are updated by backpropagation.

[0180] According to one exemplary implementation, a batch update mechanism is adopted in this step. When the number of collected samples reaches a preset batch threshold, a training round is started to ensure a sufficient number of samples, improve the stability of iterative updates, and reduce the impact of noise.

[0181] According to one exemplary implementation, a training termination condition can be set. When the iterative training reaches the training termination condition, it indicates that the training has converged, and policy updates can be stopped. The training termination condition can be a performance indicator, stability indicator, etc., and can be configured according to application requirements and processing capabilities.

[0182] According to one exemplary implementation, during the deployment phase, the branch outputting the value estimate in the decision output layer is removed. That is, after training, the parameters in the policy network used to output decision actions are saved. When the policy network needs to be deployed to the underwater vehicle, a lightweighting process is performed, pruning the value head branch. The policy network includes a two-stream memory deep neural network, which is a complete decision system that receives observation data and outputs normalized angular velocity control commands and value estimates. The lightweighted policy network can be deployed on the underwater vehicle's computing platform, receiving observation data from a simulation environment or actual sensors, generating normalized angular velocity control commands, and converting them into voltage or PWM signals to drive the underwater vehicle's servo motors.

[0183] An exemplary embodiment of this disclosure also provides an underwater vehicle control method, which can be applied to any electronic device with computing capabilities, such as a shore-based command and control computer, a mission planning workstation mounted on a mother ship, or preferably, a high-performance embedded control computer directly deployed inside an underwater vehicle (UUV). The electronic device includes:

[0184] (1) Processor: At least one processor (CPU / GPU) is used to perform inference calculations, update the environment state, and generate control instructions for the deep neural network. Considering the computational characteristics of the reinforcement learning policy network (Actor-Critic), a processor with tensor acceleration capability is preferred.

[0185] (2) Memory: The memory (RAM and ROM / Flash) is connected to the processor and is used to store computer program instructions, pre-trained policy network parameters (Weights), sector detection memory matrix and real-time sensor data.

[0186] (3) Communication interface: used to receive real-time data from underlying sensors (such as passive sonar array, inertial navigation system INS, Doppler log DVL) and send the calculated rudder angle or angular velocity command to the underlying power system (such as servo controller or thruster driver).

[0187] like Figure 7 As shown, the method in this embodiment mainly includes:

[0188] Step 701: Construct an underwater dynamic model of the adversarial environment and a partial observable space.

[0189] In reinforcement learning tasks, constructing a high-fidelity simulated interactive environment is a prerequisite for training robust policies. This embodiment first builds a two-dimensional underwater environment simulation model in a simulation computer, incorporating multi-agent game dynamics, complex underwater acoustic detection characteristics, and strict physical constraints. This step specifically includes:

[0190] S1.1 Define the simulation area and task objectives.

[0191] Define simulation area The spatial extent is defined as a two-dimensional rectangular sea area. According to one exemplary embodiment, the width of the underwater simulation environment The length of the underwater simulation environment .

[0192] (1) Submersible

[0193] The mission starting area is set at Near the boundary (e.g.) The finish line is set as (Right now For underwater vehicles, within the specified mission time... Inside, starting from the starting area, it successfully reached the finish line without being detected by any detection vehicle.

[0194] (2) Detection carrier cluster

[0195] Depend on It consists of a detection vehicle (such as a mobile UUV) equipped with active sonar detection capabilities. The detection vehicle patrols collaboratively within the simulated area, continuously illuminating and locking onto underwater vehicles using active sonar.

[0196] (3) Rules for determining whether a task is successful

[0197] Current longitudinal position of the underwater body Furthermore, since the conditions for mission failure were not triggered, the underwater navigation mission was considered successful, and the route was successfully completed without being detected, interfered with, or threatened.

[0198] The conditions for determining that the underwater download task has failed are as follows:

[0199] 1) Detected: The duration of continuous irradiation of the underwater object detected by any detection carrier. Reaching the threshold ;

[0200] 2) Timeout: The number of simulation steps exceeds the maximum step limit (e.g., the maximum frequency limit is set to 10,000 steps, corresponding to a simulation time of...). );

[0201] 3) Boundary crossing: The underwater vehicle drives out of the left or right boundary of the simulation area ( or ) or return to the starting point ( ).

[0202] S1.2 Dynamic modeling of underwater bodies.

[0203] To simulate the underactuated characteristics and maneuvering inertia of a real underwater vehicle, this embodiment uses a two-dimensional mass model subject to kinematic constraints.

[0204] In this embodiment, the speed V of the underwater vehicle is kept constant (simulating a constant-speed cruise stealth navigation mode). Available Configuration is performed within this range. Underwater vehicles can only be configured by changing their heading angle. To carry out maneuvers.

[0205] The kinematic model in continuous time is defined as follows:

[0206]

[0207] in, Let be the instantaneous velocity component of the underwater vehicle in the x-axis direction. Let be the instantaneous velocity component of the underwater vehicle in the y-axis direction. The heading angle of the underwater vehicle. It is the angular velocity control input, and is limited by the maximum physical steering angular velocity, satisfying... .

[0208] Important constraint: Real-world UUVs cannot achieve instantaneous large-angle turning, therefore the angular velocity... Apply hard constraints:

[0209] .

[0210] In this embodiment, the maximum steering angular velocity Set as This strict constraint means that if an underwater vehicle wants to turn around (change course)... ), at least This enables the strategy network to be forward-looking, generating precise control commands as early as possible based on predictions of movement trends.

[0211] In the simulation, the Euler method is used to discretize the above model, and the simulation step size is... Specifically, discretization can be performed using the following discrete update formula:

[0212]

[0213] in," , "These are the global coordinates of the underwater object at the current time t or the t-th simulation step." , "V" represents the global coordinates of the underwater vehicle at time t+1 or the (t+1)th simulation step, and "V" represents the forward velocity of the underwater vehicle. It is the heading angle at the current time t or at the t-th simulation step. It is the heading angle at the current time t or at the t-th simulation step. It is the normalized action value output by the policy head of the neural network. It is the maximum angular velocity that a submarine can achieve.

[0214] S1.3 Collaborative search of detection vehicles and modeling of sonar detection.

[0215] This step involved detailed modeling of the probe's behavior, including:

[0216] (1) Detection vehicle motion strategy

[0217] The probe is not stationary; rather, it employs a strategy combining a pre-set search route with localized maneuvers. Specifically, this includes:

[0218] 1) Z-shaped cruise: The detection vehicle sweeps the sea in a large Z-shaped pattern along an axis perpendicular to the direction of the underwater vehicle's travel.

[0219] 2) Cooperative Encirclement: The detection carriers maintain a certain level of communication coordination to avoid excessive overlap of the search areas and form a dynamic detection area network.

[0220] 3) Detection vehicle speed: set to Slightly below the maximum speed of the underwater vehicle, the numerical advantage of the detection carrier cluster increases the likelihood of successfully detecting the passing underwater vehicle.

[0221] (2) Sonar detection model (sensor interaction)

[0222] The detection vehicle is equipped with an active sonar, and the detection area is modeled as a fan-shaped region with left and right detection radii. Detecting the opening angle Front and rear detection radius Detecting the opening angle .

[0223] The underwater vehicle is equipped with a passive sonar, whose receiving model follows the "bearing-only" characteristic. The detection area is also modeled as a fan-shaped region, with left and right detection radii. Detecting the opening angle Front and rear detection radius Detecting the opening angle .

[0224] At each simulation moment The system determines the underwater body observation based on the following logic:

[0225] 1) Calculate relative geometric relationships: for each probe carrier target Calculate its Euclidean distance from the underwater body using the following expression. ,

[0226] in," , "These are the global coordinates of the i-th probe carrier." , "These are the global coordinates of the underwater object."

[0227] 2) Physical Detection Determination: If the underwater object is located within the scanning sector of the sonar on the detection carrier, it is determined that the underwater object has been illuminated or detected (hit). At this time, the continuous detection duration timer for the underwater object is activated. Increase:

[0228]

[0229] This embodiment incorporates a slow cooling mechanism during the exposure time. The uncertainty decay of the simulated probe carrier after detecting and then losing the target is generally much slower than the accumulation rate.

[0230] 3) Generating passive observation signals: If the probe carrier Within the receiving range of the underwater vehicle's passive sonar, the underwater vehicle obtains the detection carrier. relative azimuth observations To simulate the uncertainties of a real underwater acoustic environment, Gaussian white noise is superimposed on the observations according to the following expression:

[0231] ,

[0232] in, If the probe is too far away or is in the blind zone of the underwater probe, the observation channel will be NaN (no signal).

[0233] Step 702: Construct an explicit risk memory model based on sector partitioning.

[0234] Because passive sonar observations are sparse (the other party may not make a sound), intermittent (the scanning beam moves away), and noisy, this embodiment provides an explicit risk memory model, specifically an explicit sector threat memory (STM) module.

[0235] This step includes:

[0236] S2.1 Sector space division.

[0237] In the body-fixed frame of the underwater body, the horizontal plane is... Evenly divided into A sector-shaped region. In this embodiment, a sector-shaped region is defined. That is, the angular span of each sector is .

[0238] No. Each sector ( ) Coverage range of angles for:

[0239] .

[0240] Each sector maintains a scalar This is called the probability of being detected, also known as the threat probability, and is used to characterize the likelihood of a detection source, such as a detection vehicle, existing in that direction. Initially, all... .

[0241] S2.2 Dynamic update mechanism for threat probability.

[0242] The STM module update involves two parallel processes: memory decay and sensory enhancement, simulating the short-term memory characteristics of organisms in response to danger.

[0243] (1) Global Decay.

[0244] Regardless of whether a signal is detected at the current moment, the probability of all sectors being detected will naturally decline over time, adhering to the principle that "the older the information, the lower its reference value." The update formula is:

[0245] ,

[0246] in, This is the attenuation coefficient. In this embodiment, it is taken as... If no new probe signals are injected into a sector, its threat value will decay to its original value after approximately 20 simulation steps. This allows for the automatic filtering of outdated interference.

[0247] (2) Hit Update

[0248] When passive sonar is at time Received valid azimuth signal of the probe carrier (i.e., the aforementioned) When this occurs, the system performs the following operations:

[0249] 1) Index Calculation: First, calculate the sector index to which the azimuth belongs. :

[0250] .

[0251] 2) Hit Enhancement: Improves hit rate in hit sectors and its adjacent sectors ( The probability of detection (considering noise diffusion) is updated smoothly based on the following expression:

[0252] ,

[0253] Where t is the sequence number of the current simulation step, To update the strength coefficient, is the probability of being detected after the k-th sector is updated in step t.

[0254] This embodiment takes It employs an exponentially weighted moving average (EWMA), enabling threat values ​​to respond quickly to new detections without being affected by sudden noise spikes.

[0255] Figure 8 An example of an explicit risk memory model integrating fragmented information in the temporal and spatial domains is shown. Before updating the probe probability, a probe signal is detected in the 6th sector. Then, global memory decay is applied to all sectors, and the probe probability corresponding to the 6th sector where the probe signal was detected is boosted to obtain the final probe probabilities for each sector. The bar probability chart below each sector (vertical axis represents the probe probability value, horizontal axis represents the sector number) shows that after the global probe probability is reduced by memory decay, the probe probability of the 6th sector increases due to the boost.

[0256] If multiple targets from different probes are detected at the same time, the above update is performed sequentially for each target.

[0257] S2.3 Calculation of the synthesized direction vector of the probe.

[0258] To transform the discrete probability values ​​of the 16 sectors into continuous spatial physical quantities that are easier for neural networks to understand, this step further calculates the synthesized probe direction vector. .

[0259] The synthesized detected direction vector represents the direction of the combined repulsive force exerted by the current environment on the underwater object. The calculation steps are as follows:

[0260] (1) Calculate the sector center vector

[0261] For the Each sector has a unit direction vector of... ,in The center angle of the sector.

[0262] (2) Weighted summation

[0263] ,

[0264] (3) Vector normalization

[0265] ,

[0266] in This is to prevent the division by zero of minute quantities.

[0267] It points to the area with the densest detection signal. For example, if the left side ( ) and left front ( If the sector probabilities are all high, then the synthesized vector will point to the left front. This feature not only compresses the input dimension but also provides a clear gradient avoidance direction for the underwater vector.

[0268] Step 703: Multidimensional observation vector encoding and normalization.

[0269] To ensure efficient convergence of the reinforcement learning policy network, sensor data with different physical dimensions are mapped to a unified numerical range. This embodiment constructs a multidimensional observation vector. Include Dimensional features, specifically encoded as follows:

[0270] S3.1 Self-navigation status coding (4-dimensional).

[0271] (1) Location information

[0272] The global coordinates of the underwater vehicle are calculated using the following expression. Normalization to Interval:

[0273] ,

[0274] Normalization can prevent gradient explosion in neural networks caused by excessively large coordinate values.

[0275] (2) Heading information

[0276] To avoid the angle at and To address discontinuities (i.e., numerical jumps) at certain points, instead of directly inputting angle values, input their trigonometric function values:

[0277] , .

[0278] S3.2 Instantaneous sonar observation coding (3D).

[0279] Although sector memory aggregates historical information, current instantaneous observations remain crucial for emergency collision avoidance. This embodiment selects the relative azimuth of the probe closest to the underwater object (with the strongest signal strength). Encode:

[0280] (1) Signal flag bit

[0281] If any detection signal is detected at the current moment, set it to 1; otherwise, set it to 0.

[0282] (2) Relative orientation

[0283] The following triangular encoding is also used:

[0284] ,

[0285] when At this time, the azimuth feature is forcibly set to zero to avoid inputting noise into the network.

[0286] S3.3 Sector Threat Memory Feature Encoding (18 Dimensions).

[0287] It contains information on spatiotemporal evolution:

[0288] (1) Sector probability sequence

[0289] Enter directly Real-time probabilities of each sector being detected This enables the policy network to detect gaps in the detection capabilities of multiple detection vehicles.

[0290] (2) Synthetic threat vector

[0291] Input the two-dimensional vector calculated in step S2.3 .

[0292] S3.4 Auxiliary status information.

[0293] To help the network determine task progress, a normalized continuous irradiation duration was added. According to one example implementation, the duration of continuous irradiation... This is considered a hidden state and is primarily fed back to the Critic head through a reward function. The feature vectors output from S3.1 to S3.3 are concatenated to obtain the final multidimensional observation vector. .

[0294] Step 704: Construct a dual-stream memory deep neural network that integrates temporal reasoning and explicit memory.

[0295] To achieve end-to-end mapping from sensor input to control command output in a partially observable environment (POMDP), this embodiment constructs a special deep neural network architecture: a dual-stream memory deep neural network. This architecture not only utilizes the explicit risk memory from step 702 as input features, but also introduces implicit recurrent memory units within the network.

[0296] like Figure 9 As shown, this dual-stream memory deep neural network mainly consists of three parts: an embedding layer, a recurrent layer, and output heads.

[0297] S4.1 Feature Extraction Layer (MLP Encoder).

[0298] The input layer receives the 25-dimensional multidimensional observation vector generated in step 703. .because The physical quantities contained within (such as position coordinates, probability values, and Boolean flags) have different distribution characteristics, and directly inputting them into a recurrent network may lead to training instability. Therefore, the observations are first mapped to a high-dimensional latent space through a fully connected MLP.

[0299] The specific network configuration is as follows:

[0300] Input dimension: 25.

[0301] Hidden layer 1: 128 neurons, activation function is ReLU.

[0302] Hidden layer 2: 128 neurons, activation function is ReLU.

[0303] Output (feature encoding vector) ): 128 dimensions.

[0304] The mathematical expression is:

[0305] ,

[0306] in These are the trainable weight parameters for this layer.

[0307] S4.2 Sequential Inference Layer (GRU Memory)

[0308] Considering the movement of the probe, a single frame observation cannot infer its velocity vector and motion intention. This embodiment uses a gated recurrent unit (GRU) to process time-series information. GRUs have few parameters and high computational efficiency, making them suitable for deployment on embedded platforms of underwater vehicles such as underwater vehicles with limited computing power.

[0309] The GRU unit receives the feature code at the current time. and the hidden state of the previous moment (With the dimension also set to 128), the hidden state is updated using the following gating mechanism:

[0310] (1) Reset Gate Calculation

[0311] Reset door Its main function is to determine how much of the hidden state information from the previous time step needs to be forgotten or discarded when generating new candidate memories. In complex underwater acoustic environments, this mechanism can effectively filter out false historical signals caused by multipath effects or sporadic noise, preventing outdated and erroneous threat information from interfering with current situation assessment. Its calculation formula is as follows:

[0312] .

[0313] in:

[0314] For the current moment The reset gate output vector, whose element values ​​are in Within the range. The closer the value is to 0, the more the system tends to discard the corresponding historical information;

[0315] The sigmoid non-linear activation function is used to map the input to... interval;

[0316] , These are the current input features. and the hidden state at the previous moment The weight matrix mapped to the reset gate space is a parameter that the network needs to optimize during training.

[0317] , These are the corresponding bias vectors.

[0318] (2) Update Gate Calculation

[0319] Update Gate Its main function is to determine the final hidden state. How much of the old memory from the previous moment should be retained? And how much of the current moment's new memory (i.e., candidate hidden states) should be mixed in. In underwater stealth penetration missions, when the red team's detection beam temporarily moves out of the blue team's position (i.e., a signal blind zone appears), the update gate can enable the network to maintain a long-term memory of the red team's position, preventing the vehicle from making blind maneuvers due to the loss of target signals. Its calculation formula is:

[0320] .

[0321] in:

[0322] For the current moment The update gate output vector, whose element values ​​are also in Within the range. The closer the value is to 1, the more the system tends to retain historical memories; The closer the value is to 0, the more the system tends to adopt the current new observation;

[0323] , Input features and the hidden state at the previous moment The weight matrix is ​​mapped to the updated gate space;

[0324] , These are the corresponding bias vectors.

[0325] (3) Calculation of Candidate State

[0326] Candidate hidden state It is a "proposed memory" generated based on current observations and combined with valid historical information filtered through a reset gate. It represents the network's latest understanding of environmental characteristics based on the current situation. Its calculation formula is:

[0327] .

[0328] in:

[0329] For the current moment The candidate hidden state vector contains all the new features that may need to be updated into memory.

[0330] The hyperbolic tangent activation function is used to map data to... The interval ensures stable gradient propagation and increases nonlinear expressiveness;

[0331] This represents the Hadamard product, which is the element-wise multiplication of two matrices or vectors of the same dimension. A reset gate is used here. The linear transformation result of the previous state is multiplied element by element to achieve selective utilization of historical features;

[0332] , These are the weight matrices corresponding to the generated candidate hidden states;

[0333] , These are the bias vectors corresponding to the generated candidate hidden states.

[0334] (4) Final Hidden State Update

[0335] Finally, GRU utilizes the update gate. Linear interpolation is performed on the retained old memories and the introduced new memories to obtain the final hidden state at the current time step. This hidden state vector not only serves as the feature input to the output layer at the current time step, but is also passed to the GRU unit at the next time step to continue participating in the loop. Its calculation formula is:

[0336] ,

[0337] in, It is the Sigmoid activation function. This is the Hadamard product (i.e., element-wise multiplication). It serves not only as input for the next moment but also as input features for subsequent decision heads. It implicitly encodes the trajectory features of the enemy target.

[0338] Through the above four steps, the hidden state vector output by the temporal inference layer is obtained. In this process, the motion state of the underwater vehicle itself, the instantaneous azimuth information of the passive sonar, and the dynamic spatiotemporal evolution characteristics of the risk of the sector being detected have been deeply coupled, providing extremely rich and denoised situational awareness data for the subsequent decision output layer.

[0339] S4.3 Decision Output Layer (Actor-Critic Heads)

[0340] The network's final output layer is divided into two independent branches, corresponding to the policy (Actor) and value (Critic) in reinforcement learning, respectively.

[0341] (1) Actor Head, used for action generation.

[0342] 1) Input: Hidden state .

[0343] 2) Structure: Two fully connected layers (128 -> 64 -> 2).

[0344] 3) Output: Parameters of the Gaussian distribution (mean) and (Standard deviation). To ensure numerical stability, for Mapped to using the Tanh activation function ,right Use the Softplus activation function to ensure non-negativity.

[0345] 4) Action sampling: based on Sampling yields the original action And restricted to using Clip operation The interval serves as the normalized angular velocity command. .

[0346] 5) Actual controlled quantity: .

[0347] (2) Critic Head, used to assist in training.

[0348] 1) Input: Hidden state .

[0349] Structure: Two fully connected layers (128 -> 64 -> 1).

[0350] Output: scalar , indicating the current state Below is the expected cumulative discount reward sum. This value is only used to calculate the advantage function during the training phase; in online applications, this branch can be pruned to save computing power.

[0351] Step 705: Design a composite reward function that includes a double cumulative exposure penalty.

[0352] The reward function determines the agent's behavioral pattern. To reflect sensitivity to the "cumulative probability of being detected," this embodiment designs the following composite reward function. :

[0353] .

[0354] The physical meaning and mathematical expression of each award are defined in detail below:

[0355] 1. Navigation propulsion reward.

[0356] Navigation propulsion rewards Encourage the underwater vehicle to overcome resistance and move towards the finish line. The movement and evaluation expression are as follows:

[0357] ,

[0358] in, t is the sequence number of the current simulation step, and t-1 is the sequence number of the previous simulation step. The speed of the underwater vehicle. For simulating step size, The distance between the underwater vehicle and the finish line at the previous moment is the longitudinal distance. The distance between the underwater vehicle and the finish line at the current moment is the longitudinal distance. H represents the position of the finish line. This represents the current position of the water-based body.

[0359] The propulsion bonus is a normalized longitudinal velocity. For example, when sailing north at full speed, the bonus is approximately 0.5; if spinning in place or sailing sideways, the bonus is 0; if sailing backward, the bonus is negative.

[0360] 2. Penalty for being detected a second time.

[0361] To avoid the agent failing to sense the urgency of being detected or locked, this embodiment introduces a mechanism based on the duration of continuous detection. Nonlinear penalty. Penalty for secondary detection. The calculation expression is:

[0362] ,

[0363] in, For indicator functions, Basic penalty coefficient, For cumulative penalty coefficient, The maximum allowable detection threshold, The duration of continuous detection is a variable.

[0364] According to one exemplary implementation, secondary detection penalty The parameters are defined as follows:

[0365] : Indicator function, which is 1 if the current time is being detected, and 0 otherwise;

[0366] The base penalty coefficient provides slight negative feedback, preventing the agent from being detected or discovered.

[0367] : Cumulative penalty coefficient, with a relatively large value;

[0368] : Maximum allowed detection threshold.

[0369] According to one exemplary implementation, when hour, The value is The penalty effect is minimal, at which point the agent dares to risk continuing to traverse the detection area; while when hour, The value is The punishment increased significantly. When hour, The value is close to Superposition coefficient This then generates huge negative rewards ( This forces the policy network to forgo advancement rewards (maximum only) during near-death states. They will make every effort to maneuver and evade in order to interrupt the process of being discovered or locked on.

[0370] 3. Motion smoothing and energy consumption regularization terms.

[0371] Maneuver smoothing and energy consumption regularization This setting is to prevent high-frequency bang-bang control of the underwater servo motor and reduce fluid resistance. The calculation expression is as follows:

[0372] ,

[0373] in, For smoothing coefficients, This is the normalized angular velocity at the current moment. According to one exemplary implementation, . Encourage maintaining a straight course when not necessary. ).

[0374] 4. Sparse termination reward.

[0375] Sparse Termination Reward A large one-time reward signal will be given at the end of the training round, as defined below:

[0376]

[0377] Large positive / negative rewards ensure that the ultimate goal of strategy optimization is to complete the task, rather than simply to accumulate process points.

[0378] Step 706: Training and deployment process based on PPO.

[0379] This step includes:

[0380] S6.1 Training Algorithm Selection and Configuration.

[0381] PPO is an on-policy algorithm that uses a "Clipping" mechanism to limit the magnitude of each parameter update, thus ensuring training stability in complex continuous control tasks.

[0382] To address the long-term nature of GRU training, the TBPTT technique is employed:

[0383] (1) Divide the collected long trajectory (e.g., 1000 steps) into multiple lengths. Small sequence blocks.

[0384] (2) Perform gradient backpropagation within each sequence block, truncate gradients between blocks, but pass hidden states. This ensures the continuity of memory.

[0385] S6.2 Course Learning and Training Strategies

[0386] To address the cold-start difficulty caused by "sparse rewards" (i.e., the agent is unlikely to accidentally reach the finish line at the beginning, thus hindering learning), this embodiment designs a three-stage learning scheme:

[0387] (1) Phase 1 (Navigation Training, 0-1000 Iterations)

[0388] 1) Environment setup: No risk to the detection carrier, or the detection carrier is stationary.

[0389] 2) Objective: To enable the body to learn basic kinematic control and the potential to score points by walking towards the finish line.

[0390] (2) Phase Two (Obstacle Avoidance Training, 1000-3000 Iterations)

[0391] 1) Environmental setup: Number of probes It performs simple reciprocating motion, halving the detection radius.

[0392] 2) Objective: To activate the sector threat memory module, enabling underwater targets to learn to bypass explicit detection risks.

[0393] (3) Phase Three (Combat Exercises, 3000+ Iterations)

[0394] 1) Environmental setup: Number of probes Full-speed Z-shaped collaborative search, enabling full secondary exposure penalty.

[0395] 2) Objective: To train extreme survivability and multi-objective game strategies.

[0396] Figure 10 An example of an interactive training agent is shown. In the underwater environment simulated by the agent, triangles represent underwater vehicles, denoted as B0; dots represent detection vehicles with sensing capabilities, and the number at the center of the dot is the identification number of the detection vehicle. Figure 10 The image shows five detection vehicles numbered 0 to 4, abbreviated as R0 to R4, forming a detection vehicle cluster. The butterfly-shaped sector surrounding the triangle represents the passive sonar observation area of ​​the underwater vehicle. The butterfly-shaped sector surrounding the dot represents the detection area of ​​the detection vehicle or the area where the detection signal can reach.

[0397] Still with Figure 10 Taking the scenario shown as an example, the information during the training process is as follows:

[0398] a. Unit situation

[0399] The details are shown in Table 1.

[0400] Table 1. Situational Information of Each Unit During Training

[0401]

[0402] b. Information received by the sonar of the underwater vehicle

[0403] Front / Back: 9000m @ 60°;

[0404] Lateral: 13500m @ 120°;

[0405] Main detection source direction: 322°;

[0406] Recommended escape / running direction: 142°;

[0407] Targets contacted: 2;

[0408] Immediate goal: R2;

[0409] Distance: 10899m;

[0410] Angle: 292°;

[0411] Direction: Lateral.

[0412] S6.3 Deployment and Online Simulation.

[0413] After training, save the policy network parameters as a .pth file. When deploying the policy network to an underwater vehicle, you can choose to perform the following lightweight processing:

[0414] (1) Pruning: Remove the Critic value head branch and retain only the feature extraction layer, GRU layer and Actor strategy head.

[0415] (2) Reasoning process:

[0416] 1) Initialize GRU hidden state .

[0417] 2) Read passive sonar data and INS data within each control cycle (e.g., 200ms).

[0418] 3) Execute steps S2 (update sector memory) and S3 (generate observations). ).

[0419] 4) and the previous moment Input the network and get and .

[0420] 5) It is converted into a voltage or PWM signal to drive the servo motor.

[0421] An exemplary embodiment of this disclosure also provides a computer apparatus, including:

[0422] processor;

[0423] Memory used to store processor-executable instructions;

[0424] The processor is configured to execute the underwater body control method provided in the embodiments of this disclosure.

[0425] An exemplary embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a computer's processor, enable the computer to perform the underwater body control method provided in the embodiments of this disclosure.

[0426] This disclosure provides an underwater vehicle control method and computer device. First, an underwater environment simulation model is constructed. Then, passive detection data during the simulated operation of the underwater vehicle is acquired based on the underwater environment simulation model. An explicit risk memory model is trained based on the passive detection data from the underwater vehicle's operational history. Next, a multi-dimensional observation vector indicating the current probability of detection is generated based on the risk memory features output by the explicit risk memory model and instantaneous operational data. Finally, memory fusion is performed based on the multi-dimensional observation vector and the hidden state of the gated recurrent unit (GRU) at the previous moment, and a normalized angular velocity control command is output to control the underwater vehicle, thereby reducing the probability of detection. This achieves efficient and accurate autonomous decision-making for detection avoidance, solving the safety and mission execution difficulties caused by detection of underwater vehicles.

[0427] In scenarios where the passive sonar of an underwater vehicle can only obtain noisy azimuth angles, it can accurately predict the movement trend of the detection vehicle and output actions to avoid detection in advance, which can significantly reduce the probability of the vehicle being falsely detected or misidentified and improve the reliability of operation.

[0428] By constructing a physically interpretable explicit risk memory model, combining the nonlinear fitting capability of deep reinforcement learning, and designing a targeted cumulative risk reward mechanism, the problem of normal navigation of underwater vehicles under passive observation conditions was successfully solved.

[0429] The PPO algorithm combined with a curriculum-based training mode is employed. By limiting the policy update magnitude through pruning probability ratios, training crashes are avoided. Simultaneously, the difficulty of the scenario is increased in stages (from single-source straight-line cruising to multi-source cooperative encirclement), solving the cold-start problem in complex underwater scenarios. The trained policy can adapt to scenarios with different numbers of detectors, motion patterns, and noise intensities, exhibiting strong generalization ability and meeting the application needs of multiple scenarios.

[0430] Taking into account the operational characteristics of actual underwater vehicles such as AUVs, an underactuated dynamics model was constructed, strictly constraining physical limits such as turning angular velocity and speed. Simultaneously, the noise and blind zone characteristics of underwater passive sonar were simulated to generate observation data consistent with real-world scenarios. The Actor head directly outputs normalized turning angular velocity commands, which, after linear mapping, can drive the actuators, achieving seamless integration between simulation training and engineering deployment.

[0431] Balancing stealth and mission efficiency, this scheme designs a composite reward function with a secondary cumulative exposure penalty. Through a combination of positive incentives and negative penalties, it encourages the vehicle to move away from detection sources and avoid high-threat sectors, while also guiding it to efficiently advance towards the target area, thus achieving a balance between safety and mission efficiency.

[0432] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0433] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0434] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0435] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0436] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for controlling underwater vehicles, characterized in that, include: An underwater environment simulation model is constructed, which includes a basic scene, a partially observable space, and a dynamic model. The dynamic model contains modeling rules for underwater vehicles and at least one detection vehicle, and multiple detection vehicles constitute a detection vehicle cluster. Passive detection data during the simulated operation of the underwater body are obtained based on the underwater environment simulation model; Explicit risk memory model trained based on passive detection data during the operation history of the underwater vehicle; Based on the risk memory features and instantaneous operational data output by the explicit risk memory model, a multidimensional observation vector indicating the current probability of being detected is generated. The instantaneous operational data includes at least the current navigation status of the underwater vehicle and / or the instantaneous sonar observation data of the underwater vehicle. The underwater body is controlled by memory fusion based on the multidimensional observation vector and the hidden state of the gated loop unit (GRU) at the previous time step, and a normalized angular velocity control command is output to reduce the possibility of being detected. Calculate the propulsion bonus according to the following expression. : ， in, t is the sequence number of the current simulation step, and t-1 is the sequence number of the previous simulation step. The speed of the underwater vehicle. For simulating step size, The distance between the underwater vehicle and the finish line at the previous moment is the longitudinal distance. The distance between the underwater vehicle and the finish line at the current moment is the longitudinal distance. H represents the position of the finish line. The current position of the underwater vehicle; The penalty for secondary detection is calculated using the following expression. : ， in, For indicator functions, Basic penalty coefficient, For cumulative penalty coefficient, The maximum allowable detection threshold, The duration of continuous detection is a variable; The following expressions are used to calculate the computer dynamic smoothing and energy consumption regularization terms. : ， in, For smoothing coefficients, The normalized angular velocity at the current moment; Based on the execution result of the underwater vehicle reaching the finish line, a sparse termination reward is generated. ; Calculate the composite reward function based on the following expression. : 。 2. The underwater body control method according to claim 1, characterized in that, The steps for training the explicit risk memory model based on passive detection data during the underwater vehicle's operational history include: The space surrounding the underwater vehicle is divided into multiple sectors; The probability of the underwater vehicle being detected by the detection carrier cluster in the direction pointed to by each sector is dynamically updated. The probability of being detected indicates the likelihood that the underwater vehicle will be detected by the detection source in the corresponding sector.

3. The underwater body control method according to claim 2, characterized in that, The step of dynamically updating the detection probability of the underwater vehicle being detected by the detection carrier cluster in the direction pointed to by each sector includes: The detected probability is dynamically updated through multi-step simulation. Each simulation step includes: The probability of being detected in all of the aforementioned sectors at the previous time step is exponentially decayed. For the sector where a probe signal is detected, the corresponding probe probability is updated with enhancement according to the following expression: ， Where t is the sequence number of the current simulation step, To update the strength coefficient, Let be the detected probability of the k-th sector after the t-th simulation step. The probability of being detected is updated for the k-th sector in the (t-1)-th simulation step.

4. The underwater body control method according to claim 1, characterized in that, The step of performing memory fusion based on the multidimensional observation vector and the hidden state of the GRU at the previous time step, and outputting a normalized angular velocity control command to control the underwater vehicle includes: The multidimensional observation vector is encoded and its features are extracted by the feature extraction layer of a dual-stream memory deep neural network to obtain a feature-encoded vector. The temporal reasoning layer of the dual-stream memory deep neural network performs temporal reasoning based on the feature encoding vector and the hidden state of the GRU at the previous time step to obtain the hidden state at the current time step. The decision output layer of the dual-stream memory deep neural network outputs the hidden state at the current moment, and generates and outputs the normalized angular velocity control command.

5. The underwater body control method according to claim 4, characterized in that, After obtaining the hidden state at the current time step through temporal reasoning based on the feature encoding vector and the hidden state of the GRU at the previous time step using a gate control mechanism, the method further includes: Based on the hidden state at the current moment, the decision output layer of the dual-stream memory deep neural network outputs a value estimate of the current state of the underwater vehicle.

6. The underwater body control method according to claim 5, characterized in that, The method further includes: During the deployment phase, the branch that outputs the value estimate in the decision output layer is removed.

7. The underwater body control method according to claim 1, characterized in that, The method further includes: Based on the observation data of the simulation environment, the operation data of the underwater vehicle, and the composite reward function, the dual-stream memory deep neural network for memory fusion is iteratively trained.

8. The underwater body control method according to claim 7, characterized in that, The step of iteratively training the dual-stream memory deep neural network for memory fusion includes: The simulation environment is randomized and reset in each training round; Different training scenarios are set according to the stages of iterative training, and the policy update magnitude is limited by the proximal policy optimization algorithm to perform multiple rounds of iterative training on the dual-stream memory deep neural network.

9. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the underwater body control method as described in any one of claims 1 to 8.

Images