A World Model-Based Control Method for Underwater Autonomous Vehicles

Abstract

This invention discloses a control method for an underwater autonomous vehicle based on a world model, belonging to the field of intelligent control for underwater robots. First, multimodal observation data from multiple historical moments of the vehicle are acquired and encoded into latent state vectors by a hybrid encoder. Next, the actions output by the policy network and the latent state vectors are input together into the world model to predict the latent state and reward function for the next moment. Based on the prediction results of the latent state vectors and the world model, the parameters of the world model, hybrid encoder, policy network, and value network are updated. Subsequently, using the updated world model, an imagined trajectory sequence is generated in the latent space. Combining the reward and advantage function, the parameters of the policy network and value network are updated using a proximal policy optimization until the sum of the comprehensive rewards for all moments of the imagined trajectory sequence converges, resulting in a trained hybrid encoder and policy network. This invention achieves efficient and stable policy learning in the end-to-end control of underwater autonomous vehicles.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of underwater robots, specifically relating to a control method for an underwater autonomous vehicle based on a world model. Background Technology

[0002] Autonomous underwater vehicles (AUVs) play a crucial role in ocean exploration, underwater equipment maintenance, and subsea pipeline inspection. The underwater environment is characterized by high complexity, strong dynamic uncertainty, and high observation noise. Traditional rule-based control methods or control methods based on precise dynamic models are insufficient to meet the autonomous control requirements of complex underwater operations. Furthermore, the dynamic systems of AUVs and remotely operated vehicles (ROVs) are highly nonlinear and complex, making it difficult to directly obtain precise models suitable for control.

[0003] Deep reinforcement learning has made significant progress in the field of robot control. Model-based reinforcement learning (MBRL) learns the internal dynamics of the environment, enabling trajectory visualization in the latent space and thus policy optimization without frequent environmental interactions. The Dreamer series of algorithms is a representative work in the field of MBRL, achieving efficient sample utilization and policy learning by learning latent state representations and state transition models in the latent space.

[0004] However, the existing related technologies still have the following problems:

[0005] (1) The state observation of the underwater environment includes multimodal information such as visual images and body perception vectors. Existing control methods lack an effective heterogeneous sensor data fusion and coding mechanism.

[0006] (2) Existing training strategies have insufficient accuracy in estimating the reward function, which further deteriorates the training stability of world model-based control methods when the reward distribution is dense and has high variance in the problem of autonomous control in underwater environment.

[0007] (3) The existing Dreamer method uses exponential moving average to soft update the target network, but in the high-frequency interference scenario of underwater control, the response speed of soft update is insufficient, and policy learning is prone to synchronization deviation between the target network and the online network.

[0008] Therefore, there is an urgent need to propose a new method for controlling autonomous underwater vehicles. Summary of the Invention

[0009] To address the problems of insufficient multimodal perception fusion, unstable training, and low sample efficiency in existing technologies for complex underwater environments, this invention provides a control method for underwater autonomous vehicles based on a world model.

[0010] The specific technical solution is as follows:

[0011] S1: Acquire multimodal observation data of the underwater autonomous vehicle at multiple historical moments, and construct a hybrid encoder, online policy network, online value network, and world model;

[0012] A hybrid encoder is used to encode multimodal observation data into latent state vectors; an online policy network generates actions based on the latent state vectors, the actions including the thrust of the underwater autonomous vehicle's propulsion system; an online value network is used to calculate the value of the latent state vectors; and a world model is used to generate predicted rewards and predicted latent state vectors for the next time step.

[0013] S2 calculates the comprehensive reward for each historical moment and updates the parameters of the world model, hybrid encoder, online policy network, and online value network.

[0014] S3: Select the potential state vector at a certain moment, and use the updated online value network, online policy network and world model in S2 to obtain the predicted potential state vector sequence, predicted reward sequence, action sequence and value sequence for a specified duration, thereby forming the imagined trajectory sequence, and further updating the parameters of the online policy network and online value network.

[0015] S4, repeat S2-S3 until the sum of the comprehensive rewards at all times in the imagined trajectory sequence converges, and the trained hybrid encoder and online policy network are obtained;

[0016] S5 uses a trained hybrid encoder and online policy network to obtain the actions corresponding to the multimodal observation data at the current moment, thereby controlling the underwater autonomous vehicle.

[0017] Further, in S1, the multimodal observation data includes a body perception vector and an RGB image of the front of the underwater autonomous vehicle; the body perception vector includes the relative position difference between the position of the underwater autonomous vehicle and the target position, the relative attitude angle difference between the attitude of the underwater autonomous vehicle and the attitude of the target, the attitude angle of the underwater autonomous vehicle, the velocity of the underwater autonomous vehicle, and the sensor data of the underwater autonomous vehicle's inertial measurement unit.

[0018] Furthermore, in S1, the hybrid encoder consists of a convolutional neural network branch and a multilayer perceptron branch; the process of obtaining the latent state vector using the hybrid encoder is specifically as follows:

[0019] The RGB image at each historical moment is input into a convolutional neural network branch, and the ontology perception vector is input into a multilayer perceptron branch. The outputs of all branches are cascaded to form the latent state vector at that historical moment.

[0020] Furthermore, in S2, the comprehensive reward at each historical moment is calculated using a comprehensive reward function, the formula for which is:

[0021] ;

[0022] Where R represents the overall reward. For distance-based rewards, if the distance between the underwater autonomous vehicle's position and the target position is less than 0.1, the value is set to [value missing]. Otherwise, take the negative of the distance; For attitude reward, if the total attitude angle deviation between the underwater autonomous vehicle's attitude and the target attitude is less than 5°, the value is taken as follows: Otherwise, it is a negative number for the total attitude deviation; The motion smoothing penalty term is obtained by subtracting the motions at two consecutive moments and taking the modulus. It comes at the cost of time.

[0023] Furthermore, in S2, the world model and the hybrid encoder update their parameters through a joint loss function, which is obtained by adding the dynamic prediction loss, the representation learning loss, the reward prediction loss, and the scaled decoding and reconstruction loss.

[0024] The dynamic prediction loss is obtained by calculating the root mean square error between the predicted latent state vector after gradient truncation and the true latent state vector; the representation learning loss is obtained by calculating the root mean square error between the predicted latent state vector and the true latent state vector after gradient truncation; the reward prediction loss is obtained by calculating the root mean square error between the reward function prediction and the comprehensive reward function; and the decoding reconstruction loss is obtained by calculating the root mean square error between the decoded ontology perception vector predicted by the latent state vector and the true ontology perception vector.

[0025] Furthermore, in step S2, the process of updating the parameters of the online value network and the online policy network specifically involves:

[0026] S201, update world model and hybrid encoder parameters;

[0027] S202, take the hybrid encoder before the update in S201 as the target hybrid encoder, input the multimodal observation data of each historical moment into the target hybrid encoder and the hybrid encoder after the update in S201 respectively, and output their respective potential state vectors;

[0028] S203, input the latent state vector output by the target hybrid encoder into the target policy network and the target value network respectively to obtain the target action and the target value; input the updated latent state vector output by the hybrid encoder into the online policy network and the online value network respectively to obtain the action and the value.

[0029] S204: Calculate the KL divergence between target actions and actions as the policy compensation loss, and update the online policy network parameters; and calculate the mean square error between target value and value as the value compensation loss, and update the online value network parameters.

[0030] Furthermore, the parameters of the target policy network and the target value network are updated using a periodic hard update method; the periodic hard update strategy is to completely copy the parameters of the online policy network and the online value network into their respective target networks every specified number of iterations, thereby completing the parameter update of the target policy network and the target value network.

[0031] Furthermore, in step S3, the process of obtaining the imagined trajectory sequence is specifically as follows:

[0032] S301, Select the potential state vector at a certain moment as the initial potential state vector;

[0033] In S302, the initial latent state vector is input into the updated online value network and the updated online policy network in S2 respectively to obtain the action and value corresponding to this moment; further, the latent state vector and action are input into the updated world model in S2 to obtain the predicted reward and the predicted latent state vector for the next moment.

[0034] S303, repeat S302 using the predicted latent state vector of the next moment until a specified duration is reached, to obtain a predicted latent state vector sequence, a predicted reward sequence, an action sequence, and a value sequence, all of which constitute an imagined trajectory sequence; the predicted latent state vector sequence includes an initial latent state vector and a predicted latent state vector.

[0035] Furthermore, in step S3, the process of updating the parameters of the online policy network and the online value network based on the imagined trajectory sequence specifically involves:

[0036] S31, using the value sequence in the imagined trajectory sequence, calculate the value at each time step in the imagined trajectory sequence. The return is further subtracted from the value at that moment to obtain the advantage function at that moment;

[0037] S32, based on all moments in the imagined trajectory sequence The return is calculated using the exponential moving average.

[0038] S33, using the exponential moving average and the dominance function at each time step in the imagined trajectory sequence, calculates the normalized dominance function at each time step;

[0039] S34, combining the imagined trajectory sequence and the normalized advantage function at all times in the imagined trajectory sequence, the PPO loss function is used to update the online policy network parameters; and based on the online value at all times and The mean squared error between returns is used to update the parameters of the online value network.

[0040] Furthermore, S4 also includes a model rollback mechanism based on task performance; the specific content of the model rollback mechanism is as follows:

[0041] At each iteration, the sum of the comprehensive rewards for all moments in the imagined trajectory sequence in the current iteration is calculated, and the historical best comprehensive reward is updated. If the difference between the historical best comprehensive reward and the sum of the comprehensive rewards for all moments in the current iteration is greater than or equal to 50% of the absolute value of the historical best comprehensive reward, then the parameters of the hybrid encoder, world model, online policy network, online value network, target policy network, and target value network in the current iteration are all replaced with the parameters corresponding to the historical best comprehensive reward.

[0042] The beneficial effects of this invention are:

[0043] (1) This invention takes into account the multimodal observation characteristics of the underwater environment and the stability requirements during the model training process, and constructs a hybrid encoder composed of convolutional neural network branches and multilayer perceptron branches in parallel;

[0044] (2) This invention decouples the world model update frequency from the complete update frequency of the policy network and the value network, which solves the problem of optimism bias and policy degradation caused by the synchronous update strategy of the world model and the policy network in existing model-based reinforcement learning methods. Thus, it achieves a more stable training process and better control performance under the complex reward distribution conditions of underwater autonomous vehicle control.

[0045] (3) This invention further enhances the robustness of training by periodic target network hard update and model rollback mechanism based on task performance, overcoming the problem of insufficient response speed leading to synchronization deviation in existing methods.

[0046] (4) In view of the problems of insufficient multimodal perception fusion, unstable training and low sample efficiency of underwater autonomous vehicles in complex underwater environments, this invention constructs a world model system including a hybrid encoder, a gated recurrent unit world model, an Actor policy network and a Critic value network. The prediction accuracy of the world model is improved by the joint loss function. The stability of policy learning is guaranteed by the percentile difference adaptive normalization method based on exponential moving average and the truncated surrogate objective function of near-end policy optimization. The robustness of training is further enhanced by the periodic hard update of the objective network and the model rollback mechanism based on task performance, thereby improving the performance and stability of underwater autonomous vehicle control. Attached Figure Description

[0047] Figure 1 This is a flowchart of a control method for an underwater autonomous vehicle based on a world model.

[0048] Figure 2 This is a schematic diagram of the control inputs for an autonomous underwater vehicle.

[0049] Figure 3 This is a schematic diagram illustrating the changes in the state error of an underwater autonomous vehicle.

[0050] Figure 4 This is a comparison chart of reward curves during the training process. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0052] The overall flow of the underwater autonomous vehicle control method based on a world model proposed in this invention is as follows: Figure 1 As shown, each step is as follows:

[0053] Step S1: Construct a hybrid encoder to encode the multimodal observation data of the underwater autonomous vehicle in a unified manner.

[0054] The hybrid encoder consists of parallel branches of a convolutional neural network and a multilayer perceptron. It encodes two heterogeneous types of data—visual images and ontology perception vectors—into a unified latent space, enabling the world model to capture the intrinsic correlations between multimodal information in a unified representation. Finally, a unified latent state vector is formed through cascading operations. , represented as:

[0055]

[0056] in, For image, It is a convolutional neural network. For ontology perception vectors, It is a multilayer perceptron.

[0057] Specifically, the convolutional neural network branch processes the visual image input, which is an RGB image captured by a monocular camera mounted on the underwater autonomous vehicle. This image represents the view directly in front of the vehicle (in one specific embodiment of the invention, the RGB image size is 64×64×3, provided by a camera sensor in the HoloOcean simulation environment). Spatial features are extracted layer by layer through convolutional layers (in one specific embodiment of the invention, the stride of the convolutional layer is 2 and the kernel size is 4). Each layer includes image channel normalization and... The activation function ultimately compresses the spatial dimension to 1×1 through adaptive average pooling, outputting a feature vector of fixed dimensions.

[0058] The multilayer perceptron branch is used to process the ontology sensing vector input, which is a 26-dimensional ontology sensing vector containing the relative position difference between the current position and the target position. The relative attitude angle difference between the current attitude and the target attitude Current attitude angle Current speed And inertial measurement unit (IMU) sensor data.

[0059] The ontology perception vector is processed through a multi-layer fully connected network (in one specific embodiment of the invention, a 3-layer fully connected network) via a multi-layer perceptron branch for feature extraction. Each fully connected network layer contains... Activation function.

[0060] Step S2: Construct a policy network and value network with an "online-goal" dual structure.

[0061] This invention uses MLP as the basic network architecture, constructs a policy network and a value network respectively, and sets up an online network and its corresponding target network for each network. The network parameters of the online policy network and the online value network adopt the latest network parameters, while the target policy network and the target value network adopt the online policy network and the online value network parameters from the last update.

[0062] The policy network is used to determine the potential state representation vector. Generate predicted actions In this invention, actions represent the thrust of each propeller of the aircraft, and a value network is used to compute the latent state representation vector. The value of.

[0063] In one specific embodiment of the present invention, the policy network adopts a Gaussian policy distribution, and the action mean is used. Nonlinear activation constraints to Interval, standard deviation of action passing Activate and apply positive bias To ensure positive values, the output vector dimension of the policy network is... , The magnitude of the action vector is used. After the output vector is split, it is divided into a mean vector and a standard deviation vector. A normal distribution is constructed for sampling.

[0064] Step S3: Construct a world model based on gated recurrent units to jointly model the state transition dynamics and reward function in the latent space.

[0065] The world model is the core of this invention, responsible for learning the dynamics of environmental transitions and reward functions in the latent space, thereby improving sample training efficiency.

[0066] The model will use the potential state vector at the current time. and actions As input, the action is first mapped to a space of the same dimension as the latent state through the action projection layer, and then the hidden state is updated through the GRU gated recurrent unit. Finally, the potential state vector for the next time step is predicted using a priori head network. And predict instant rewards using reward head networks. .

[0067] Specifically, the model includes the following components:

[0068] (1) Action projection layer: consisting of a linear transformation layer composition, This represents the weight matrix of the linear transformation layer, used to map the 8-dimensional action vector to the same dimensional space as the potential state vector;

[0069] (2) Gated recurrent unit: The hidden state size is twice the dimension of the latent state vector, and the input is the latent state. With projection action spliced ​​vector The output is the updated hidden state. ;

[0070] (3) Prior head network: 3-layer MLP, which hides the state Mapped to the predicted latent state vector at the next time step ;

[0071] (4) Reward Head Network: A 3-layer MLP used to calculate the comprehensive reward of the hidden state through the comprehensive reward function during the training phase, and at the same time, the hidden state Mapped to scalar prediction reward The expression for each component of the world model is as follows:

[0072]

[0073]

[0074]

[0075] in, Let be the hidden state of the GRU at time t+1. Let be the potential state vector at time t. The projected action is the result of projecting the action. For gated loop unit, For the prior head network, To reward the head network, Let be the reward function at time t.

[0076] During the training phase, the formula for calculating the comprehensive reward function is as follows:

[0077]

[0078] Where R represents the overall reward. As a distance bonus, if the distance between the vehicle's current position and the target position is less than 0.1, then... Otherwise, take the negative of the distance; For attitude reward, if the total attitude angle deviation between the current attitude and the target attitude is less than 5°, then... Otherwise, it is a negative number for the total attitude deviation; This is a penalty for smooth motion. Let be the magnitude of the vector. For a fixed time cost per step, the optimal choice is... .

[0079] Step S4: Train the world model and hybrid encoder using a joint loss function based on real-world interactive data.

[0080] During the training phase, the world model is trained using samples stored in the experience buffer pool. The experience buffer pool uses a fixed-capacity circular buffer to maintain the most recent experience interaction samples in a first-in-first-out manner. During sampling, B starting positions are randomly selected from the buffer index range. For each starting position, T consecutive steps of sequence data are extracted to form B×T transition samples for batch training of the world model. Each transition sample contains: the current multimodal latent state vector, the action vector executed, the reward obtained, the state at the next moment, and the termination flag.

[0081] B×T transition samples are sampled from the experience buffer pool, and the latent state sequence is obtained through a hybrid encoder, reshaping the dimensions. A time-step unfolding is performed using a world model: at each time step... ,enter and The forward propagation of the gated loop unit is obtained The predicted value is obtained through the prior head network and the reward head network. and .

[0082] The joint loss function is calculated by stacking the predicted values ​​from all time steps. This joint loss function decouples dynamic learning from representation learning through gradient truncation and prevents non-steady-state optimization caused by negative loss values ​​through zero-value lower bound truncation. The parameters of the world model and the hybrid encoder are then updated based on the joint loss function.

[0083] Joint loss function The specific calculation formula is as follows:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] in, For dynamic loss prediction, SG represents the gradient truncation operation. This indicates that gradient backpropagation to the hybrid encoder is prevented; To characterize the learning loss, This indicates that gradients are prevented from being backpropagated to the world model; To reward predicted losses; For decoding and reconstruction loss; This is the function for calculating the root mean square error; This represents the set of predicted latent state vectors composed of all training samples. This represents the set of true latent state vectors for all training samples. This represents the true combined reward across all training samples. Represents the set of scalar reward predictions for all training samples. Indicates from The reconstructed ontology perception vector obtained from decoding This represents the set of true ontology-aware vectors for all training samples. This represents the scaling factor, used to balance the contribution weights of the representation learning loss.

[0090] In one specific embodiment of the invention, the parameters of the world model and the hybrid encoder are jointly updated using the same Adam optimizer (learning rate 3e-4). The world model is updated 10 times per training epoch.

[0091] Step S5: Perform latent spatial representation offset compensation on the policy network and value network.

[0092] After each round of parameter updates in the world model, the hybrid encoder from the previous update is used as the target encoder network. The target hybrid encoder network is then used to encode the current state to obtain the target latent state representation. ;Utilizing target policy networks based on Generate target policy distribution ;Utilizing the latent state representation based on the output of the target hybrid encoder using the target value network Assess the value of the target state ;Utilizing an online policy network to represent the latent state based on the current hybrid encoder output Generate online policy distribution .

[0093] Distribution of target strategies Online strategy distribution The KL divergence between them serves as the strategy compensation loss. ,Right now:

[0094]

[0095] The gradient of the policy network parameters is calculated along the policy compensation loss and the policy network parameters are updated so that the online policy distribution remains consistent with the target policy distribution even after the latent space representation changes.

[0096] Representing the current potential state using an online value network Value estimate The target state value output by the target value network The mean square error between them serves as a value compensation loss. ,Right now:

[0097]

[0098] The gradient of the value network parameters is calculated along the value compensation loss and the value network parameters are updated to ensure that the value estimate remains consistent with the target value estimate after the latent space representation changes.

[0099] Step S6: Perform multi-step imagined trajectory generation in the latent space, and perform forward prediction through world model iteration to obtain a complete imagined trajectory sequence.

[0100] After the world model has iterated a specified number of times, proceed to S6 to update the parameters of the policy network and value network.

[0101] Given the current potential state vector As the initial potential state vector, the hidden state of the gated recurrent unit is initialized with zero. In the beginning, Within the imaginary range of steps, the following process is repeated: The online policy network is based on the k-th time step. Output motion distribution parameters and sample motion. (mean) and standard deviation ), world model reception Predict the potential state vector at the next time step. Predicted Rewards And new hidden states Online value network assessment State value .

[0102] Finally, a complete sequence of imagined trajectories is obtained: a sequence of latent state vectors. Action sequence Predicting reward sequences and value sequence .

[0103] Step S7: Update the policy network and value network based on the imagined trajectory.

[0104] Based on the imagined trajectory generated in S6, this invention performs joint updates to the policy network and the value network. This process includes... The process consists of four parts: return calculation, adaptive advantage normalization, PPO strategy update, and Critic value network update.

[0105] (1) Return calculation

[0106] Using λ-returns allows for a flexible balance between bias and variance. This invention, based on the imagined trajectory generated in S4, from... arrive 0 Reverse recursive calculation Return :

[0107]

[0108] in, As a discount factor, The preferred weighting coefficient for time difference (TD) bias is... , ; Calculated based on imagined trajectory Rewards .

[0109] (2) Adaptive advantage normalization

[0110] The dominance function is estimated using an adaptive normalization method based on percentile differences of exponential moving average.

[0111] First, calculate the results obtained from the current batch. 5th percentile of returns and the 95th percentile Then, use exponential moving average to... The fluctuation range of returns is smoothed out:

[0112]

[0113] Where D-1 represents the D-1th training batch, i.e. the previous training batch; This is the exponential moving average of the current batch. The smoothing factor for the exponential moving average (EMA) is preferably 0.99; and These are the current batches. Return The 95th and 5th percentiles.

[0114] Finally, the normalized advantage function is calculated:

[0115]

[0116]

[0117] in, For the dominant function, Used to constrain the lower bound of the normalization factor to 1 to prevent division by zero; This is the normalized dominance function.

[0118] (3) PPO strategy update

[0119] The policy network is trained using a truncated objective function optimized for near-end policy. This truncation mechanism limits the policy update magnitude and prevents policy collapse. The PPO loss function for the policy network is:

[0120]

[0121]

[0122]

[0123] in, Importance sampling ratio; To truncate the scope and prevent excessively large policy updates; The policy distribution entropy is used to encourage exploration; The KL divergence of the old and new policy distributions is used to constrain the policy update magnitude. The strategy entropy coefficient is preferably 0.0003; The KL divergence coefficient is preferably 0.01. This is the set of normalized advantage functions for all samples in the current training batch. The set of actions for all samples in the current training batch. For mathematical expectation, and These are the parameters after the online policy network update and the parameters before the update, respectively. and These are the unconstrained target and the clipped target, respectively. The function that takes the minimum value Let be the loss function for the online policy network.

[0124] Unlike standard policy optimization (PPO), this invention applies PPO updates in the latent space rather than the original observation space, leveraging the latent representations provided by the world model for more efficient policy learning. In one specific embodiment of this invention, the policy network employs the Adam optimizer with a learning rate of [missing information]. Each update involves 10 iterations.

[0125] (4) Critic value network update

[0126] By minimizing the value estimate and The mean squared error loss between rewards is used to update the online value network, providing an accurate value benchmark for policy optimization in the online policy network. The loss function of the value network is:

[0127]

[0128] in, For the value network to estimate the value of the state, The loss function for the online value network, It is the set of all λ rewards for the current training batch.

[0129] In one specific embodiment of the present invention, the value network also employs the Adam optimizer, with a learning rate of... Each update involves 10 iterations.

[0130] Step S8: Periodically perform hard updates to the target network and use a model rollback mechanism based on task performance to prevent policy degradation.

[0131] This invention employs a periodic hard update strategy: every fixed number of rounds (e.g., every 8 rounds), the online network parameters are completely copied to the target network, specifically including the hybrid encoder network, the Actor policy network, and the Critic value network. Before updating the target network, the old policy distribution and old value estimate are calculated using the current target network before copying, and used in the optimization process of the value network and the policy network to ensure the consistency of the optimization.

[0132] To further enhance the robustness of training and prevent policy degradation in the later stages of training, this invention further introduces a model rollback mechanism based on task performance, specifically:

[0133] During training, the sum of the comprehensive reward function calculated in each training round is continuously monitored, and the historical best reward is updated synchronously. When the reward in a single round decreases by more than 50% relative to the historical best reward, i.e. It automatically triggers a rollback operation, immediately loading the best model parameters corresponding to the previously saved historical best reward, including the parameters and optimizer states of the encoder, world model, policy network, value network and its target network, so that the training can be restored from the stable state before degradation.

[0134] Finally, when the sum of the comprehensive reward functions calculated at all time steps converges, the training is considered complete, and the trained hybrid encoder and policy network are obtained.

[0135] During the inference phase, the RGB image and the ontology perception vector at the current moment are input into the trained hybrid encoder to obtain the latent state vector, which is then input into the trained policy network to obtain the action. This allows the attitude and movement of the underwater autonomous vehicle to be controlled by adjusting its thrusters.

[0136] As can be seen from the above embodiments, this invention addresses the problem of unstable control of underwater autonomous vehicles in complex underwater environments. It considers both the multimodal observation characteristics of the underwater environment and the stability requirements of model training, constructing a lightweight world model system comprising a hybrid encoder, a world model based on gated recurrent units, a policy network, and a value network. The prediction accuracy of the world model is improved by truncating the joint loss function and using a hidden state reset mechanism based on termination signals. The stability of policy learning is ensured by using an adaptive normalization method based on percentile differences of exponential moving averages and a truncated surrogate objective function for near-end policy optimization. The robustness of training is further enhanced by periodic hard updates to the objective network and a model rollback mechanism based on task performance, thereby improving the performance and stability of underwater autonomous vehicle control.

[0137] To verify the effectiveness of the method proposed in this invention, further experiments were conducted based on a six-degree-of-freedom autonomous control scenario for an underwater autonomous vehicle constructed using the HoloOcean underwater simulation platform, in order to verify the effectiveness of this invention.

[0138] The simulation environment is based on a HoloOcean custom scene. The maximum number of steps per round for the underwater autonomous vehicle is 120, and the maximum thrust is 15N. At the beginning of each round, the target pose (position and attitude angle) is set. The observation space includes a 26-dimensional body perception vector and a 64×64×3 RGB image, and the motion space is an 8-dimensional continuous control signal, which corresponds to the eight thrusters of the underwater autonomous vehicle.

[0139] The system hyperparameter is: discount factor. , coefficient Exponential moving average coefficient Near-end strategy optimization cutoff range Entropy coefficient KL coefficient The world model learning rate is set to The learning rates for the policy network and the value network are set to... The buffer capacity is 50,000, and the step size is imagined to be H=4.

[0140] Figure 2 To illustrate the changes in control input of the underwater autonomous vehicle under the training action strategy, each colored curve represents the thrust change of a thruster over time. It can be seen that the thrust difference between two adjacent times is small, proving that the control input of the underwater autonomous vehicle changes smoothly.

[0141] Figure 3 The figure shows the changes in the state error of the underwater autonomous vehicle during the control process, where Figure 3 (a) represents the change of the underwater autonomous vehicle's displacement in the X direction with time step. Figure 3 (b) shows the change of the roll angle of the autonomous underwater vehicle with time step. Figure 3 (c) represents the change of the underwater autonomous vehicle's displacement in the Y direction with time step. Figure 3 (d) represents the change in the pitch angle of the autonomous underwater vehicle with time step. Figure 3 (e) represents the change of the underwater autonomous vehicle's displacement in the Z direction with time step. Figure 3 (f) represents the change of the yaw angle of the autonomous underwater vehicle with time step.

[0142] It can be seen that, under the action of control input, as time steps progress, the state error of the underwater autonomous vehicle tends to 0 and is controlled within an acceptable error range, which demonstrates the effectiveness of the underwater autonomous vehicle control method based on the world model proposed in this invention.

[0143] Figure 4 The figure shows a comparison of the reward changes with the number of training steps between the world model-based underwater autonomous vehicle control method and the traditional PPO method. It can be found that, compared with the traditional PPO method, the present invention effectively improves the sample training efficiency and reduces the training time cost.

[0144] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A control method for an underwater autonomous vehicle based on a world model, characterized in that, include: S1: Acquire multimodal observation data of the underwater autonomous vehicle at multiple historical moments, and construct a hybrid encoder, online policy network, online value network, and world model; A hybrid encoder is used to encode multimodal observation data into latent state vectors; an online policy network generates actions based on the latent state vectors, the actions including the thrust of the underwater autonomous vehicle's propulsion system; an online value network is used to calculate the value of the latent state vectors; and a world model is used to generate predicted rewards and predicted latent state vectors for the next time step. S2 calculates the comprehensive reward for each historical moment and updates the parameters of the world model, hybrid encoder, online policy network, and online value network. S3: Select the potential state vector at a certain moment, and use the updated online value network, online policy network and world model in S2 to obtain the predicted potential state vector sequence, predicted reward sequence, action sequence and value sequence for a specified duration, thereby forming the imagined trajectory sequence, and further updating the parameters of the online policy network and online value network. S4, repeat S2-S3 until the sum of the comprehensive rewards at all times in the imagined trajectory sequence converges, and the trained hybrid encoder and online policy network are obtained; S5 uses a trained hybrid encoder and online policy network to obtain the actions corresponding to the multimodal observation data at the current moment, thereby controlling the underwater autonomous vehicle.

2. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In S1, the multimodal observation data includes a body perception vector and an RGB image of the front of the underwater autonomous vehicle; the body perception vector includes the relative position difference between the position of the underwater autonomous vehicle and the target position, the relative attitude angle difference between the attitude of the underwater autonomous vehicle and the attitude of the target, the attitude angle of the underwater autonomous vehicle, the velocity of the underwater autonomous vehicle, and the sensor data of the underwater autonomous vehicle's inertial measurement unit.

3. The underwater autonomous vehicle control method based on a world model according to claim 2, characterized in that, In S1, the hybrid encoder consists of a convolutional neural network branch and a multilayer perceptron branch; the process of obtaining the latent state vector using the hybrid encoder is as follows: The RGB image at each historical moment is input into a convolutional neural network branch, and the ontology perception vector is input into a multilayer perceptron branch. The outputs of all branches are cascaded to form the latent state vector at that historical moment.

4. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In step S2, the comprehensive reward at each historical moment is calculated using a comprehensive reward function, the formula for which is: ； Where R represents the overall reward. For distance-based rewards, if the distance between the underwater autonomous vehicle's position and the target position is less than 0.1, the value is set to [value missing]. Otherwise, take the negative of the distance; For attitude reward, if the total attitude angle deviation between the underwater autonomous vehicle's attitude and the target attitude is less than 5°, the value is taken as follows: Otherwise, it is a negative number for the total attitude deviation; The motion smoothing penalty term is obtained by subtracting the motions at two consecutive moments and taking the modulus. It comes at the cost of time.

5. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In S2, the world model and the hybrid encoder update their parameters through a joint loss function, which is obtained by adding the dynamic prediction loss, the representation learning loss, the reward prediction loss, and the scaled decoding and reconstruction loss. The dynamic prediction loss is obtained by calculating the root mean square error between the predicted potential state vector after gradient truncation and the actual potential state vector. The representation learning loss is obtained by calculating the root mean square error between the predicted latent state vector and the true latent state vector after gradient truncation; the reward prediction loss is obtained by calculating the root mean square error between the reward function prediction and the comprehensive reward function; and the decoding reconstruction loss is obtained by calculating the root mean square error between the decoded ontology perception vector predicted by the latent state vector and the true ontology perception vector.

6. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In step S2, the process of updating the parameters of the online value network and the online policy network is as follows: S201, update world model and hybrid encoder parameters; S202, take the hybrid encoder before the update in S201 as the target hybrid encoder, input the multimodal observation data of each historical moment into the target hybrid encoder and the hybrid encoder after the update in S201 respectively, and output their respective potential state vectors; S203, input the latent state vector output by the target hybrid encoder into the target policy network and the target value network respectively to obtain the target action and the target value; input the updated latent state vector output by the hybrid encoder into the online policy network and the online value network respectively to obtain the action and the value. S204: Calculate the KL divergence between target actions and actions as the policy compensation loss, and update the online policy network parameters; and calculate the mean square error between target value and value as the value compensation loss, and update the online value network parameters.

7. The underwater autonomous vehicle control method based on a world model according to claim 6, characterized in that, The parameters of the target policy network and the target value network are updated using a periodic hard update method; The periodic hard update strategy involves completely copying the parameters of the online policy network and the online value network to their respective target networks every specified number of iterations, thereby completing the parameter update of the target policy network and the target value network.

8. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In step S3, the process of obtaining the imagined trajectory sequence is as follows: S301, Select the potential state vector at a certain moment as the initial potential state vector; In S302, the initial latent state vector is input into the updated online value network and the updated online policy network in S2 respectively to obtain the action and value corresponding to this moment; further, the latent state vector and action are input into the updated world model in S2 to obtain the predicted reward and the predicted latent state vector for the next moment. S303, repeat S302 using the predicted latent state vector of the next moment until a specified duration is reached, to obtain a predicted latent state vector sequence, a predicted reward sequence, an action sequence, and a value sequence, all of which constitute an imagined trajectory sequence; the predicted latent state vector sequence includes an initial latent state vector and a predicted latent state vector.

9. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, In step S3, the process of updating the parameters of the online policy network and the online value network based on the imagined trajectory sequence is as follows: S31, using the value sequence in the imagined trajectory sequence, calculate the value at each time step in the imagined trajectory sequence. The return is further subtracted from the value at that moment to obtain the advantage function at that moment; S32, based on all moments in the imagined trajectory sequence The return is calculated using the exponential moving average. S33, using the exponential moving average and the dominance function at each time step in the imagined trajectory sequence, calculates the normalized dominance function at each time step; S34, combining the imagined trajectory sequence and the normalized advantage function at all times in the imagined trajectory sequence, the PPO loss function is used to update the online policy network parameters; and based on the online value at all times and The mean squared error between returns is used to update the parameters of the online value network.

10. The underwater autonomous vehicle control method based on a world model according to claim 1, characterized in that, S4 also includes a model rollback mechanism based on task performance; the specific content of the model rollback mechanism is as follows: At each iteration, the sum of the comprehensive rewards for all moments in the imagined trajectory sequence in the current iteration is calculated, and the historical best comprehensive reward is updated. If the difference between the historical best comprehensive reward and the sum of the comprehensive rewards for all moments in the current iteration is greater than or equal to 50% of the absolute value of the historical best comprehensive reward, then the parameters of the hybrid encoder, world model, online policy network, online value network, target policy network, and target value network in the current iteration are all replaced with the parameters corresponding to the historical best comprehensive reward.

Images