Large language model guided underwater robot hierarchical reinforcement learning method and system

Abstract

The application provides a large language model guided underwater robot hierarchical reinforcement learning method and belongs to the technical field of underwater robot control. In view of the problems of sparse rewards in the underwater environment and difficult decision-making of long sequence tasks, the application constructs a multi-modal environment state space and a hierarchical action space decoupled from skills and target solutions; the semantic reasoning ability of the large language model is used to convert the environment state into a natural language description and generate a prior probability distribution of skill selection; in the hierarchical reinforcement learning training, a regularization constraint based on KL divergence is introduced to guide the optimization of the high-level policy network by using the prior distribution. Through the deep integration of semantic guidance and decoupling architecture, the cold start and dimension disaster problems in the complex underwater environment are effectively solved, and the autonomous operation planning efficiency and robustness of the underwater robot are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for underwater robots, and in particular relates to a hierarchical reinforcement learning method and system for underwater robots guided by a large language model. Background Technology

[0002] With the rapid development of marine resource exploration, underwater environmental monitoring, and subsea engineering operations, autonomous underwater vehicles (AUVs), as core execution vehicles, directly determine operational efficiency and mission success rates through their intelligent control capabilities. In complex underwater scenarios, AUVs often need to complete long-sequence decision-making tasks (such as multi-target sampling, continuous obstacle avoidance, and precise path tracking). These tasks are characterized by significant features such as dynamic environmental uncertainty, sparse reward signals, and high dimensionality of the action space, placing stringent demands on the reasoning ability, convergence speed, and robustness of control algorithms.

[0003] Reinforcement learning (RL), as an intelligent algorithm that autonomously learns optimal policies through environmental interaction, has been widely explored in the field of robot control. However, traditional reinforcement learning faces significant bottlenecks in long-sequence AUV tasks: on the one hand, the high-dimensional state and action spaces of the underwater environment lead to low algorithm exploration efficiency and a tendency to get trapped in local optima; on the other hand, sparse reward mechanisms make it difficult for the algorithm to quickly establish an effective correlation between actions and rewards, resulting in long training cycles and poor convergence stability, making it difficult to meet the real-time requirements of actual operations.

[0004] To address these issues, researchers proposed a hierarchical reinforcement learning (HRL) framework. This framework breaks down complex tasks into two levels: high-level decision-making and low-level execution, reducing the decision horizon and improving exploration efficiency. However, existing HRL methods often train high-level policies from scratch, lacking effective prior knowledge guidance. This results in slow convergence and insufficient policy robustness in long underwater sequence tasks. Furthermore, the poor compatibility between low-level control logic and high-level decision-making makes it difficult to fully leverage the advantages of a hierarchical architecture.

[0005] In recent years, Large Language Models (LLMs), with their powerful semantic reasoning capabilities and massive amounts of general knowledge, have provided a new path for optimizing reinforcement learning. LLMs can generate high-level decision guidance through natural language understanding and logical reasoning, helping reinforcement learning algorithms quickly locate effective exploration directions. However, directly applying LLMs to AUV control still has key drawbacks: First, LLMs can only process text input and cannot directly perceive multimodal observation data of the underwater environment (such as acoustic sensor data and visual image data); second, LLMs lack task-specific knowledge, and the generated decision commands may not match the actual execution capabilities of the AUV, leading to control failure; third, the online inference latency of LLMs is high, making it difficult to meet the needs of real-time AUV control and limiting their deployment in practical scenarios.

[0006] Furthermore, existing methods that integrate LLM and reinforcement learning either rely solely on LLM to design reward functions, resulting in redundant reward design and high computational costs; or require LLM to provide continuous online guidance during the deployment phase, making it impossible to achieve independent control without LLM and difficult to adapt to the limited computing resources and underwater communication constraints of AUVs. At the same time, considering the unique characteristics of AUV underwater operations, existing methods do not fully consider dynamic interference from the underwater environment, sensor noise, and other factors, leading to poor practical adaptability of the strategies.

[0007] In summary, the existing technology has the following shortcomings:

[0008] 1. Traditional reinforcement learning has low exploration efficiency and slow convergence speed in AUV long sequence sparse reward tasks, which makes it difficult to meet the needs of actual operation.

[0009] 2. Existing hierarchical reinforcement learning (HRL) strategies lack effective prior guidance, making training difficult and resulting in insufficient policy robustness.

[0010] 3. The fusion method of LLM and reinforcement learning has problems such as poor environmental perception adaptability, insufficient task specificity, and high deployment delay, and cannot be directly applied to AUV control.

[0011] 4. Existing methods are not fully adapted to the dynamic characteristics of underwater environments and the hardware constraints of AUVs, resulting in low feasibility for practical deployment.

[0012] Therefore, there is an urgent need for an intelligent control method that can integrate the advantages of LLM semantic reasoning with the characteristics of hierarchical reinforcement learning architecture and adapt to AUV underwater operation scenarios, so as to solve the core technical problems such as sparse rewards, large action space, slow training convergence, and poor real-time deployment in long sequence tasks. Summary of the Invention

[0013] To address the shortcomings of existing technologies, such as the low efficiency of traditional reinforcement learning in exploring long-sequence sparse reward tasks for AUVs, the lack of effective prior knowledge guidance for high-level strategies in hierarchical reinforcement learning, poor adaptability of large language models (LLMs) to the environmental perception and execution capabilities of underwater robot control scenarios, and insufficient real-time performance during deployment, this invention provides a hierarchical reinforcement learning method for underwater robots guided by a large language model. By constructing a collaborative architecture between the semantic guidance end and the decision execution end, and combining action space decoupling, hybrid loss function training, and a separate training and deployment process, this method enables efficient training and independent real-time control of AUVs in long-sequence tasks in complex underwater environments. It improves the convergence speed, robustness, and adaptability of control strategies to actual operations, solving the core problems of training difficulties, weak generalization, and deployment limitations faced by existing AUV intelligent control technologies.

[0014] The objective of this invention is achieved through the following technical solution:

[0015] This invention discloses a hierarchical reinforcement learning method for underwater robots guided by a large language model, comprising the following steps:

[0016] S1: The collected multimodal data is transformed into underwater environment observation data at the current moment and input into the high-level policy network. After action space decoupling, the predicted skill and target probability distribution are output. Effective decision combinations are selected and mapped to the low-level control primitive library to execute local tasks. When the local task termination condition is met, the basic feedback set and training supplement set are collected and stored in the same policy trajectory buffer. When the preset batch capacity is reached, S2 is triggered.

[0017] S2: Extract image-class observation data from the training supplement set, generate environmental state description text through the visual language model, combine it with the total task description text and input it into the large language model to generate multiple sets of joint probability distributions and take the mean. After edge decoupling, obtain the guidance skills and target probability distributions.

[0018] S3: Calculate the first KL divergence between the predicted and guided skill probability distributions, and the second KL divergence between the predicted and guided target probability distributions, respectively. Construct the imitation learning loss, and after annealing weight decay, construct a hybrid loss function with the reinforcement learning loss. Use the proximal policy optimization algorithm to update the high-level policy network and clear the same policy trajectory buffer. If the overall task termination condition and convergence criterion are met, the frozen high-level policy network is obtained; otherwise, repeat S1-S3 until the preset maximum number of training rounds.

[0019] S4: Deploy the frozen high-level strategy network and the low-level control primitive library, output effective skills and target decisions based on the underwater environment observation data at the current moment, map them to the low-level control primitive library to drive the robot's actuators, until the overall task is completed and the entire trajectory data is archived.

[0020] Furthermore, the multimodal data includes underwater terrain data, obstacle distance data, mission target coordinate data, aquatic environmental parameters, and image-based observation data; the multimodal sensor includes an acoustic sensor, a vision camera, and an inertial measurement unit, as well as an edge computing board connected to the aforementioned sensors; the edge computing board has an embedded intelligent processing chip for deploying fully connected layers or convolutional neural networks, the acoustic sensor and the vision camera acquire multimodal data, and the inertial measurement unit acquires the autonomous underwater robot's state data;

[0021] The multimodal data is converted into current-moment underwater environment observation data, including:

[0022] The underwater terrain data, obstacle distance data, target coordinate data, and aquatic environmental parameters in the multimodal data are used as numerical observation data. These data are mapped through a fully connected layer and transformed into numerical physical feature vectors. The image-type observation data are processed by a convolutional neural network to extract features and transform them into visual spatial feature vectors that match the dimension of the numerical physical feature vectors. The numerical physical feature vectors and the visual spatial feature vectors are then concatenated to form unified underwater environmental observation data for the current moment.

[0023] Furthermore, the process of decoupling the action space and outputting the predicted skill probability distribution and the predicted target probability distribution in parallel, and then filtering for effective decisions using a preset probability threshold, includes:

[0024] The current underwater environment observation data is input into the high-level policy network. The associated features are extracted through the shared convolutional neural network of the high-level policy network, and the nonlinear transformation is completed by the ReLU activation function to output multimodal associated shared features.

[0025] The multimodal association shared features are input in parallel into the skill sub-module and target sub-module of the high-level policy network. After being mapped by fully connected hidden layers containing multiple ReLU activation functions in both the skill sub-module and the target sub-module, the predicted skill probability distribution and the predicted target probability distribution are output by the terminal softmax function, respectively.

[0026] The predicted skill probability distribution and the predicted target probability distribution are filtered according to a preset probability threshold. Options with a probability value not lower than the preset probability threshold are selected as valid options. The option with the highest probability among the valid options is selected as the valid skill decision and the valid target decision, respectively. They are combined into a valid decision combination and mapped to the underlying control primitive library to drive the execution of local tasks. If the probability values ​​of all options in the predicted skill probability distribution and the predicted target probability distribution are lower than the preset probability threshold, then hovering in the optional skill set is selected as the valid skill decision, and the autonomous underwater robot itself in the optional target set is selected as the valid target decision. They are combined into a valid decision combination and mapped to the underlying control primitive library to execute a safe hovering action.

[0027] Furthermore, the guidance strategy distribution is decoupled at the edge, splitting the joint probability distribution containing skills and goals into independent guidance skill probability distributions and guidance goal probability distributions. The method is as follows:

[0028] By traversing the set of optional targets All targets The joint probabilities are summed to separate the independent probability distribution of the guidance skill from the guidance strategy distribution. The calculation formula is as follows:

[0029]

[0030] By traversing the set of optional skills All skills The joint probabilities are summed to separate the independent guidance target probability distribution from the guidance policy distribution. The calculation formula is as follows:

[0031]

[0032] in, For an optional set of targets, A set of optional skills.

[0033] Furthermore, the reinforcement learning loss is based on the assumption that the probability distribution of the predicted skill and the probability distribution of the predicted target are independent under the action space decoupling architecture. The reinforcement learning loss is calculated by the logarithm of the joint probability, and the expression is:

[0034]

[0035] in, Indicated in the empirical data distribution Find the expected value; Given the observed data, predict the probability of selecting a corresponding effective skill decision from the skill probability distribution; To predict the probability of selecting a corresponding effective target decision from the target probability distribution; the empirical data distribution The trajectory buffer of the same strategy consists of a set of historical samples composed of multiple sets of underwater environment observation data at the current moment and their corresponding effective skill decisions and effective target decisions;

[0036] The reinforcement learning loss is decoupled into a linear superposition of the skill reinforcement learning loss and the objective reinforcement learning loss, i.e.:

[0037] .

[0038] Furthermore, when constructing the hybrid loss function, the reinforcement learning loss is fused with the imitation learning loss dynamically weighted by the annealing weight parameters;

[0039] The imitation learning loss refers to the first difference between the predicted skill probability distribution and the guided skill probability distribution under the empirical data distribution. Divergence and the second divergence between the predicted target probability distribution and the guiding target probability distribution The mathematical expectation of the sum of the divergences;

[0040] The annealing weight parameters are used in the early stages of training to force the high-level policy network to update its direction by using the guidance policy distribution generated by the large language model. Subsequently, the proportion of imitation learning loss is gradually reduced until it transitions to autonomous exploration that relies entirely on reinforcement learning loss.

[0041] Furthermore, the method for adjusting the annealing weight parameters during the training process is as follows:

[0042] The initial value of the annealing weight parameter is determined by the ratio of the reinforcement learning loss to the imitation learning loss during the initial training phase, and the calculation formula is as follows:

[0043]

[0044] To reinforce learning loss, Learning loss due to imitation;

[0045] During the training phase, the annealing weight parameters As it decays linearly to 0, its piecewise function expression is:

[0046]

[0047] Where i represents the current training step, The preset decay termination step is used to characterize the training critical node where the imitation learning loss weights return to zero and the high-level policy network is transformed into a system that relies entirely on reinforcement learning loss for updates.

[0048] Furthermore, the high-level policy network is updated using a near-end policy optimization algorithm, including: during the training phase, after each local task termination condition is met and the basic feedback set and training supplement set are stored in the same policy trajectory buffer, if it is determined that the amount of data in the same policy trajectory buffer has reached the preset batch capacity, then after the construction of the hybrid loss function is completed, the gradient of the hybrid loss function is extracted.

[0049] The near-end policy optimization algorithm is adopted to update the network parameters of the high-level policy network once according to the gradient of the hybrid loss function, and the same policy trajectory buffer is cleared after the update is completed. After the parameter update is completed, the high-level policy network receives the basic feedback set generated when the local task termination condition is met, outputs effective skill decision and effective target decision, and maps them to the low-level control primitive library to drive the robot actuator to perform the local task.

[0050] The execution of the aforementioned local tasks and the network parameter update steps are repeated until the overall task termination condition is met, at which point the current complete training round ends. It is then determined whether consecutive complete training rounds meet the convergence judgment criteria. If they do, the training phase ends. If they do not meet the criteria and the preset maximum number of training rounds has not yet been reached, the next complete training round begins.

[0051] Furthermore, the termination condition and convergence criterion for the overall task are as follows:

[0052] The overall mission termination conditions include: the autonomous underwater robot successfully reaching the global final target tolerance range, a collision that violates the mission safety constraints, or the total execution time exceeding the preset global maximum time limit.

[0053] The convergence criteria include: the task success rate calculated statistically based on local task trajectory data within multiple consecutive complete training rounds is not lower than a preset success rate threshold; the cumulative value fluctuation of all environmental feedback reward signals within a single training round is less than a preset fluctuation threshold; and the numerical decrease of the imitation learning loss tends to stabilize and converge.

[0054] This invention discloses a hierarchical reinforcement learning method for underwater robots guided by a large language model, used to execute the method, comprising:

[0055] The multimodal data acquisition module is used to acquire multimodal data and autonomous underwater robot status data in real time, and convert the multimodal data into underwater environment observation data at the current moment; during the deployment phase, it is used to package and generate a basic feedback set and feed it back to the high-level policy network; during the training phase, it outputs the basic feedback set and multimodal data to form a training supplementary set.

[0056] The decision execution end has a built-in high-level strategy network module and a low-level control primitive library. The high-level strategy network module is used to receive the underwater environment observation data at the current moment, and output the predicted skill probability distribution and the predicted target probability distribution in parallel after decoupling from the action space. Effective decision combinations are then selected and mapped to the low-level control primitive library to generate control signals.

[0057] The underwater robot actuator is used to receive the control signals output by the underlying control primitive library and complete the corresponding physical interaction actions;

[0058] The same-strategy trajectory buffer storage module is configured to collect the basic feedback set and the training supplement set and store them in the same-strategy trajectory buffer when the local task termination condition is met, and to trigger the semantic guidance module when the data in the same-strategy trajectory buffer reaches a preset batch; The semantic guidance module has a built-in visual language model and a large language model, which are used to extract image-type observation data from the multimodal data in the training supplement set, generate environmental state description text through the visual language model, combine it with the total task description text and input it into the large language model to generate multiple sets of joint probability distributions and take the mean, and obtain the guidance skill probability distribution and guidance target probability distribution through edge decoupling;

[0059] The loss calculation module is configured to calculate the first KL divergence between the predicted skill probability distribution and the guided skill probability distribution, and the second KL divergence between the predicted target probability distribution and the guided target probability distribution. Under the empirical data distribution, the mathematical expectation of the sum of the first KL divergence and the second KL divergence is the imitation learning loss. After annealing weight decay, it is combined with the reinforcement learning loss to construct a hybrid loss function.

[0060] The policy update module is used to update the parameters of the high-level policy network based on the gradient of the hybrid loss function using a near-end policy optimization algorithm, and clear the same policy trajectory buffer after the update is completed.

[0061] The deployment and execution module is configured to deploy the high-level policy network and the low-level control primitive library that are frozen during training. Based on the underwater environment observation data at the current moment, it outputs effective skill decisions and effective target decisions, which are mapped to the low-level control primitive library and transformed into control signals to drive the autonomous underwater robot's actuators until the overall task is completed and the entire trajectory data of the overall task is archived.

[0062] The beneficial effects of this invention are as follows:

[0063] 1. This invention provides high-level decision-making priors by leveraging the semantic reasoning capabilities of LLM at the semantic guidance end, thus solving the exploration efficiency problem of hierarchical reinforcement learning; it achieves decoupling of the action space to reduce the decision-making dimension through the hierarchical architecture of high-level strategies and low-level control at the decision execution end; and it balances training efficiency and deployment real-time performance through a separation mode of guided learning in the training phase and independent operation in the deployment phase, ultimately forming an intelligent control closed loop adapted to AUV underwater operations.

[0064] 2. This invention introduces prior knowledge and logical reasoning capabilities of LLM through semantic guidance, generating a guidance policy distribution that provides a clear exploration direction for high-level policy networks, avoiding the blind exploration of traditional reinforcement learning. At the same time, it constructs a hybrid loss function and combines it with the dynamic adjustment of the annealing weight parameter λ to achieve a smooth transition between guided learning and autonomous learning, quickly establishes the association between actions and rewards, efficiently solves the core problem of slow training convergence in sparse reward scenarios, effectively overcomes the training dilemma of long sequences of sparse rewards in AUV, and significantly improves the convergence speed and learning efficiency of control policies.

[0065] 3. The high-level strategy network of this invention adopts a skill and objective dual-module action space decoupling design to reduce decision-making complexity; the low-level control primitive library integrates RRT* path planning and PID closed-loop control to adapt to unknown underwater obstacles and dynamic disturbances, and with the softmax probability distribution threshold screening mechanism, it significantly enhances the robustness and environmental adaptability of the AUV control strategy, ensuring the reliability and stability of operations in complex underwater scenarios.

[0066] 4. In underwater operations, this invention employs a training and deployment separation design. During the training phase, only LLM is used to complete policy optimization. During the deployment phase, only a mature high-level policy network and a low-level control primitive library are loaded. The high-level policy network (reinforcement learning model) performs task planning, while the low-level control primitive library performs path planning and motion control. This eliminates the need for online inference with LLM, allowing for independent real-time control without the high latency and high computational consumption of LLM. This perfectly adapts to the low latency and computational constraints of AUV underwater operations, satisfying the real-time control and hardware resource constraints of AUV underwater operations.

[0067] 5. This invention designs a dedicated feature processing scheme for AUV multimodal observation data. Features are extracted from numerical data and image data respectively and then concatenated into a unified input. Multimodal environmental data is transformed into text that LLM can understand through VLM, optimizing the multimodal environment perception and semantic decision adaptation logic, realizing seamless adaptation of environment perception and semantic decision, and improving the AUV's perception response accuracy and adaptability to complex environments.

[0068] 6. The underlying control primitive library of this invention is built based on hard-coded expert knowledge, forming standardized and reusable control logic without the need for repeated design; LLM prior guidance reduces the dependence on massive experience data, shortens the training cycle and resource consumption; the core architecture does not depend on specific AUV models or mission scenarios, and can be adapted to various underwater operation requirements by adjusting parameters, reducing development and implementation costs, and has strong scenario adaptability and practical value. Attached Figure Description

[0069] Figure 1 This is a general framework diagram for the application of the present invention.

[0070] Figure 2This is a schematic diagram of the deployment process of the present invention.

[0071] Figure 3 This is a schematic diagram of the action space decoupling process of the present invention.

[0072] Figure 4 This is a schematic diagram of the execution logic of the underlying control primitive library of the present invention.

[0073] Figure 5 This is a schematic diagram of the training process of the present invention.

[0074] Figure 6 This is a flowchart illustrating the semantic guidance module of the present invention.

[0075] Figure 7 This is a system block diagram of the present invention. Detailed Implementation

[0076] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0077] Example: Figure 1 As shown, a hierarchical reinforcement learning method for underwater robots guided by a large language model includes a training phase and a deployment phase.

[0078] During the training phase, such as Figures 1-6 As shown:

[0079] S1: System Initialization: At the start of the training phase, the system instantiates and pre-configures the hardware and software modules. Specifically, the high-level policy network is initialized at the airborne decision execution end, and the low-level control primitive library is loaded. This primitive library has a pre-installed path planning module and motion control module. The same policy trajectory buffer is initialized to temporarily store interactive data, and the overall task description text corresponding to the current task is loaded. The overall task description text is pre-input by shore-based operators through a human-machine interface, or automatically generated by the upper-level task scheduling system based on operational requirements. This text encodes the final operational purpose, working sea area boundary, and global safety constraints of the autonomous underwater vehicle in natural language. In addition, the system calculates the initial values ​​of the annealing weight parameters based on the manually set initial network hyperparameters and the expected value of the prior loss, which serve as the numerical benchmark for the subsequent construction of the hybrid loss function.

[0080] The same-policy trajectory buffer is deployed in the training server on shore or the mother ship to temporarily store the basic feedback set and the training supplement set collected after the execution of local tasks, thus forming the empirical data distribution D. To adapt to the same-policy characteristics of the near-end policy optimization algorithm, when the data volume in the buffer reaches the preset batch capacity and the high-level policy network parameters are updated, the buffer needs to be cleared to ensure that subsequent training data is generated by the updated high-level policy network.

[0081] The initial network hyperparameters refer to the empirical configuration parameters that are pre-set before the environmental interaction begins for the high-level policy network structure and its supporting near-end policy optimization algorithm. Specifically, they include the learning rate, discount factor, pruning threshold, and weight initialization constants of the fully connected layers of the network.

[0082] The aforementioned prior loss expectation value refers to the initial magnitude value estimated by humans based on expert experience or pre-training benchmarks for the reinforcement learning loss and imitation learning loss that may occur in the initial state during the cold start phase when there is a lack of real feedback from the underwater environment.

[0083] S2: Local task execution and data acquisition: The autonomous underwater robot performs local tasks in the underwater environment, and collects multimodal data and autonomous underwater robot status data in real time through multimodal sensors, and transforms the multimodal data into unified-dimensional underwater environment observation data at the current moment;

[0084] The current underwater environment observation data is input into a high-level policy network. After action space decoupling processing, the predicted skill probability distribution and predicted target probability distribution are output in parallel. Effective skill decisions and effective target decisions are generated by filtering through preset probability thresholds and combined into an effective decision combination. Mapped to the underlying control primitive library, the underlying control primitive library will effectively make decisions on combination. Ultimately, this translates into control signals that control the underwater robot's actuators. If the probability values ​​of all options in both the predicted skill probability distribution and the predicted target probability distribution are lower than a preset probability threshold, then hovering in the set of selectable skills is taken as an effective skill decision, and the autonomous underwater robot itself in the set of selectable targets is taken as an effective target decision. These are combined into an effective decision combination and mapped to the underlying control primitive library to execute a safe hovering action.

[0085] The preset probability threshold is the minimum decision confidence scalar set during system initialization. It has a dual safety mechanism in the decision-making process: first, it serves as a hard filtering boundary to eliminate low-confidence noise in the prediction distribution, ensuring that the issued effective decisions have sufficient reasoning certainty; second, it serves as an anomaly fallback benchmark, directly triggering the autonomous underwater robot's safe hovering action when all predicted options are below this threshold.

[0086] When the local task termination condition is detected, i.e., the state boundary condition is triggered or the maximum number of bottom-level action steps is reached, the generated interaction data is classified, packaged, and stored. The interaction data specifically includes:

[0087] (1) Basic feedback set: It consists of the state data of the autonomous underwater robot synchronously acquired by the multi-modal sensors when the local task termination condition is met, and the underwater environment observation data at the current moment after conversion;

[0088] (2) Training Supplement Set: It is composed of multimodal data collected by multimodal sensors when the local task termination condition is met, the predicted skill probability distribution and predicted target probability distribution output by the high-level policy network, the local task trajectory data fed back by the low-level control primitive library (including task success flag, collision violation flag, forced termination flag and actual action steps), and the environmental feedback reward signal calculated by the high-level policy network.

[0089] The high-level policy network sends back the generated basic feedback set and training supplement set together and stores them in the same policy trajectory buffer. The process of executing local tasks and sending back data is repeated until the total amount of data in the same policy trajectory buffer reaches the preset batch capacity.

[0090] S3: Semantic guidance, such as Figure 6 As shown, when the amount of data in the trajectory buffer of the same strategy reaches the preset batch capacity, the semantic guidance module is activated;

[0091] The semantic guidance module extracts image-type observation data from the multimodal data in the training supplement set and inputs it into the visual language model to generate environmental state description text. This text, along with the pre-loaded total task description text, is then input into the large language model. After multiple rounds of queries, multiple sets of joint probability distributions are generated, and the average value is taken to obtain the guidance policy distribution. This guidance policy distribution is decoupled through edge detection and outputs independent guidance skill probability distributions and guidance target probability distributions.

[0092] S4 Policy Update: Under the empirical data distribution, calculate the first KL divergence between the predicted skill probability distribution and the guided skill probability distribution, and the second KL divergence between the predicted target probability distribution and the guided target probability distribution. The expected value of the sum of these two is the imitation learning loss. After annealing and weight decay, construct a hybrid loss function with the reinforcement learning loss calculated based on the environmental feedback reward signal in the training appendix. Using the gradient of the hybrid loss function, update the high-level policy network using the proximal policy optimization algorithm and clear the same policy trajectory buffer. When the overall task termination condition is met, further determine whether the convergence judgment criterion is met. If it is met, end the training phase and obtain the high-level policy network with training frozen; otherwise, repeat S2 to S4 until the preset maximum number of training rounds.

[0093] After the high-level strategy network parameters are updated and the same strategy trajectory buffer is cleared, the airborne high-level strategy network, based on the updated parameters, receives the basic feedback set generated when the local task termination condition is met, and continues to output effective skill decisions and effective target decisions, which are mapped to the low-level control primitive library to drive the underwater robot actuator to execute new local tasks; until the above-mentioned total task termination condition is met.

[0094] The overall task termination condition is used to characterize the physical or logical end of a single global operation, specifically including any of the following situations: the autonomous underwater vehicle successfully reaches the global final target tolerance range; a serious collision occurs that violates the task safety constraints; or the total execution time exceeds the preset global maximum time limit.

[0095] After the current complete training round ends, it is determined whether the convergence criterion is met. If it is met, the training is considered successful and the training phase ends normally. If the convergence criterion is not met, it is further determined whether the number of complete training rounds executed has reached the preset maximum number of training rounds. If not, the autonomous underwater vehicle begins the exploration of the next complete training round; if the preset maximum number of training rounds has been reached and the convergence criterion is still not met, the training phase is forcibly terminated.

[0096] The preset maximum number of training rounds is used to characterize the maximum number of complete training rounds allowed to be executed during the training phase. As a mandatory fallback mechanism to prevent the high-level policy network from getting stuck in infinite exploration due to difficulty in convergence, when the actual number of complete training rounds executed reaches this value and the convergence judgment criterion is still not met, the system is triggered to forcibly terminate the training phase.

[0097] The complete training round is used to characterize a single data sampling period, that is, the complete environmental interaction process of the autonomous underwater robot during the training phase, from the initial execution of the task to the triggering and satisfaction of the total task termination condition.

[0098] The convergence criterion is a statistical condition used to characterize the achievement of training goals for high-level policy networks. Specifically, it includes: the task success rate calculated statistically based on local task trajectory data within multiple consecutive complete training rounds is not lower than a preset success rate threshold; the cumulative value fluctuation of all environmental feedback reward signals within a single training round is less than a preset fluctuation threshold; the numerical decrease of the imitation learning loss tends to stabilize and converge; when a preset number of consecutive complete training rounds meet the above conditions, it is determined that the convergence criterion is met and the training phase ends.

[0099] During the deployment phase, such as Figure 2 As shown:

[0100] Once the training phase has successfully concluded and the convergence criteria have been met, the deployment phase based on the airborne operating environment will commence.

[0101] Deploying a high-level policy network and a pre-set low-level control primitive library on an underwater robot marks the official start of autonomous underwater robot operation planning. Based on the underwater environment observation data at the current moment, it outputs effective skill decisions and effective target decisions, which are combined and mapped to the low-level control primitive library to be transformed into control signals to drive the underwater robot's actuators until the overall task is completed and the entire trajectory data of the overall task is archived.

[0102] The multimodal data includes underwater topography data, obstacle distance data, mission target coordinate data, water environment parameters, and image-based observation data; the multimodal sensors include acoustic sensors, a vision camera, and an inertial measurement unit, as well as an edge computing board connected to the aforementioned sensors; the edge computing board has an embedded intelligent processing chip for deploying fully connected layers or convolutional neural networks, the acoustic sensors and the vision camera acquire multimodal data, and the inertial measurement unit acquires the autonomous underwater robot's state data;

[0103] The edge computing board is embedded with an intelligent processing chip for deploying fully connected layers or convolutional neural networks, thereby directly completing the feature transformation from multimodal data to unified-dimensional underwater environment observation data at the underlying hardware level. This avoids communication delays and computing power congestion caused by directly transmitting massive amounts of raw data to the high-level policy network.

[0104] The multimodal data is converted into underwater environmental observation data at the current moment, including:

[0105] The underwater terrain data, obstacle distance data, target coordinate data, and aquatic environmental parameters in the multimodal data are used as numerical observation data. These data are mapped through a fully connected layer and transformed into numerical physical feature vectors. The image-type observation data are processed by a convolutional neural network to extract features and transform them into visual spatial feature vectors that match the dimension of the numerical physical feature vectors. The numerical physical feature vectors and the visual spatial feature vectors are then concatenated to form unified underwater environmental observation data for the current moment.

[0106] The specific process involves nonlinearly mapping the numerical observation data through a fully connected layer to transform it into a one-dimensional numerical physical feature vector; performing deep feature extraction and spatial dimension compression on the image-type observation data through a convolutional neural network to transform it into a one-dimensional visual spatial feature vector that matches the dimension of the numerical physical feature vector; and then performing feature-level concatenation and fusion of the numerical physical feature vector and the visual spatial feature vector to construct the current-moment underwater environment observation data that characterizes the multimodal semantic associations of the underwater environment.

[0107] In both the training and deployment phases, a basic feedback set is generated when the termination condition of a local task is met; only in the training phase, when the termination condition of a local task is met, an additional training set is generated.

[0108] like Figure 3 As shown, the process of decoupling the action space and outputting the predicted skill and target probability distribution in parallel to obtain effective decisions includes:

[0109] The current underwater environment observation data is input into the high-level policy network. The associated features are extracted through the shared convolutional neural network of the high-level policy network, and then nonlinearly transformed by the ReLU activation function to output multimodal associated shared features. These multimodal associated shared features are used to characterize the deep coupling relationship between underwater physical state and spatial visual information.

[0110] The multimodal association shared features are input in parallel into the skill sub-module and target sub-module of the high-level policy network. After being mapped by fully connected hidden layers containing multiple ReLU activation functions in both the skill sub-module and the target sub-module, the predicted skill probability distribution and the predicted target probability distribution are output by the terminal softmax function, respectively.

[0111] The predicted skill probability distribution and the predicted target probability distribution are filtered according to preset probability thresholds. Options with probability values ​​not lower than the preset probability thresholds are selected as valid options. The option with the highest probability among the valid options is selected as the valid skill decision and the valid target decision, respectively. These are combined into a valid decision combination and mapped to the underlying control primitive library to drive the execution of local tasks. If the probability values ​​of all options in the predicted skill probability distribution or the predicted target probability distribution are lower than the preset probability thresholds, a safe hovering action is triggered. That is, hovering is taken as a valid skill decision and the autonomous underwater robot is taken as a valid target decision. The two are combined into a valid decision combination and mapped to the underlying control primitive library for execution.

[0112] The local task termination conditions include state boundary trigger conditions and the maximum number of bottom-level action steps. The state boundary trigger conditions are used to characterize the interaction between the autonomous underwater robot and the environment, which produces a clear physical result, covering the normal target arrival tolerance range and collisions that violate the task safety constraints. The maximum number of bottom-level action steps limit is used to characterize the execution process getting stuck in an infinite loop or being too inefficient to produce a clear result.

[0113] When the local task termination condition is met, and during the training phase, the lower-level control primitive library feeds back the local task trajectory data to the higher-level policy network. The local task trajectory data consists of a task success flag, a collision violation flag, a forced termination flag, and the actual number of action steps N during the execution of the local task. The task success flag is set when the target tolerance range is reached normally. The collision violation flag is set when the autonomous underwater robot violates safety constraints or a collision occurs. The forced termination flag is set when the maximum number of lower-level action steps is reached.

[0114] Subsequently, the multimodal sensors synchronously output the autonomous underwater robot's state data when the local task termination condition is met and the underwater environment observation data at the current moment to the high-level policy network.

[0115] The high-level policy network combines the above data to calculate the environmental feedback reward signal. The calculation formula is as follows:

[0116]

[0117] in, As a constant reward for success, Let be the penalty constant. The time step penalty coefficient, The actual number of steps in the action; , , These are mutually exclusive Boolean indicator variables, corresponding to the activation status of the task success flag, collision violation flag, and forced termination flag, respectively. If activated, they are set to 1; otherwise, they are set to 0. After calculation, the environmental feedback reward signal is sent. Store in the trajectory buffer with the same strategy;

[0118] When the amount of data in the same strategy trajectory buffer reaches the preset batch capacity, the estimated action advantage function is calculated based on the environmental feedback reward signal. ;

[0119] The action advantage function valuation The formula used to measure the return advantage of the current decision portfolio relative to the average batch level is as follows:

[0120]

[0121] in, The environmental feedback reward signal calculated for the current local task has been internally processed using the actual number of action steps. A time penalty cost was introduced; The mathematical mean of all environmental feedback reward signals within the preset batch capacity;

[0122] The preset batch capacity is used to characterize the sample accumulation threshold that triggers the high-level policy network to perform a single parameter update; its physical meaning is the total amount of joint data of the basic feedback set and the training supplement set continuously received and accumulated in the same policy trajectory buffer, generated by the execution of local tasks multiple times in succession.

[0123] like Figure 6 As shown, the large language model generates multiple sets of joint probability distributions through multiple rounds of queries and takes the average to obtain the guiding policy distribution. The calculation formula is as follows:

[0124]

[0125] in, For the number of queries, For the first The second query outputs the set of optional skills from the large language model. With optional target set The joint probability distribution of ; For the joint probability distribution Select Skill With the goal The joint probability at time; For the optional skill set Any skill in the game, For the optional target set Any target in the process.

[0126] like Figure 5 As shown, to adapt to the action space decoupling built into the high-level policy network, the guidance policy distribution is decoupled at the edge, splitting the joint probability distribution containing skills and targets into independent guidance skill probability distributions and guidance target probability distributions;

[0127] Specifically, by traversing the set of optional targets All targets The joint probabilities are summed to separate the independent probability distribution of the guidance skill from the guidance strategy distribution. The calculation formula is as follows:

[0128]

[0129] Similarly, by traversing the set of optional skills... All skills The joint probabilities are summed to separate the independent guidance target probability distribution from the guidance policy distribution. The calculation formula is as follows:

[0130]

[0131] in, For an optional set of targets, A set of optional skills.

[0132] When constructing the hybrid loss function, the reinforcement learning loss and the imitation learning loss dynamically weighted by the annealing weight parameters are fused together.

[0133] The imitation learning loss refers to the first difference between the predicted skill probability distribution and the guided skill probability distribution under the empirical data distribution. Divergence and the second divergence between the predicted target probability distribution and the guiding target probability distribution The mathematical expectation of the sum of the divergences.

[0134] The first difference between the predicted skill probability distribution and the guidance skill probability distribution divergence The calculation formula is:

[0135]

[0136] in, To extract the current underwater environment observation data from the trajectory buffer of the same strategy, For underwater environmental observation data at the given current time Under the condition that the guidance skill probability distribution is selected, the skill is selected. The probability, For underwater environmental observation data at the current moment Under the condition that the predicted skill probability distribution selects a skill The probability, For the set of optional skills;

[0137] The second KL divergence between the predicted target probability distribution and the guiding target probability distribution The calculation formula is:

[0138]

[0139] in, For underwater environmental observation data at the given current time Under the condition that the target is selected from the guidance target probability distribution. The probability, For underwater environmental observation data at the given current time Under the condition that the predicted target probability distribution is selected, the target is selected. The probability, The set of optional targets;

[0140] Under the empirical data distribution, the expected value of the sum of the calculated first KL divergence and the second KL divergence constitutes the imitation learning loss term. .

[0141] Annealing weight parameters with dynamic decay The acquired skills will be used to reinforce learning loss. With target reinforcement learning loss Compared with the first and second ones mentioned above Divergence is weighted and fused to construct a hybrid loss function. Its expression is:

[0142]

[0143] in, The annealing weight parameter is... For the distribution of the empirical data, Indicated in the empirical data distribution The following is an analysis of the underwater environment observation data at the current moment. Determine the expected value of the empirical data distribution; Constituting reinforcement learning loss The mixture term is formed by multiplying the imitation learning loss by the annealing weight parameter. , used to characterize the imitation learning loss of dynamic weight decay.

[0144] The annealing weight parameters are initially used to force the high-level policy network to update its direction based on the guidance policy distribution generated by the large language model. Subsequently, the proportion of imitation learning loss is gradually reduced until it transitions to autonomous exploration entirely reliant on reinforcement learning loss. That is, the guidance policy distribution generated by the large language model is used in the initial training phase. As a benchmark, the update direction of the higher-level policy network is forcibly guided;

[0145] Subsequently, with the annealing weight parameter The decay gradually reduces the proportion of the imitation learning loss in the hybrid loss function until it smoothly transitions to autonomous exploration that relies entirely on the reinforcement learning loss.

[0146] The method for adjusting the annealing weight parameters during training is as follows:

[0147] The initial value of the annealing weight parameter is determined by the ratio of the reinforcement learning loss to the imitation learning loss during the initial training phase, and the calculation formula is as follows:

[0148]

[0149] To reinforce learning loss, Learning loss due to imitation;

[0150] During the training phase, the annealing weight parameters As it decays linearly to 0, its piecewise function expression is:

[0151]

[0152] in, For the current training step, The preset decay termination step is used to characterize the training critical node where the imitation learning loss weights return to zero and the high-level policy network is transformed into a system that relies entirely on reinforcement learning loss for updates.

[0153] The Proximal Policy Optimization (PPO) algorithm is employed to update the parameters of the high-level policy network using the gradient of a hybrid loss function. Based on the gradient direction and stride of the hybrid loss function, the network parameters of the shared convolutional neural network and the fully connected layers of the skill and target submodules within the high-level policy network are updated. Figure 5 As shown. Includes:

[0154] During the training phase, after each local task termination condition is met and the basic feedback set and training supplement set are stored in the same policy trajectory buffer, if it is determined that the amount of data in the same policy trajectory buffer has reached the preset batch capacity, then after the construction of the hybrid loss function is completed, the gradient of the hybrid loss function is extracted.

[0155] The near-end policy optimization algorithm is used to update the network parameters of the high-level policy network once based on the gradient of the hybrid loss function, and the same policy trajectory buffer is cleared after the update is completed.

[0156] After the parameters are updated, the high-level policy network receives the basic feedback set generated when the local task termination condition was met, outputs effective skill decisions and effective target decisions, and maps them to the low-level control primitive library to drive the robot actuator to execute subsequent local tasks. The above-mentioned local task execution and network parameter update steps are repeated until the overall task termination condition is met. At this point, the current complete training round ends. After the current complete training round ends, it is further determined whether consecutive complete training rounds meet the convergence judgment criteria. If the criteria are met, the training phase ends; if the criteria are not met and the preset maximum number of training rounds has not been reached, the next complete training round begins.

[0157] The reinforcement learning loss is based on the assumption that the probability distributions of predicted skills and predicted targets are independent under the action space decoupling architecture. The reinforcement learning loss is calculated using the logarithm of the joint probability, and its expression is:

[0158]

[0159] in, This represents the mathematical expectation obtained under the empirical data distribution D; This is the current underwater environment observation data. In order to provide the observation data Under the given conditions, select the corresponding effective skill decision from the predicted skill probability distribution. The probability of; To predict the probability of selecting the corresponding effective target in the target probability distribution;

[0160] The distribution of empirical data The trajectory buffer of the same strategy consists of a set of historical samples composed of multiple sets of underwater environment observation data at the current moment and their corresponding effective skill decisions and effective target decisions;

[0161] The reinforcement learning loss is decoupled into a linear superposition of the skill sub-reinforcement learning loss and the target reinforcement learning loss, i.e.;

[0162] ;

[0163] Among them, the skill reinforcement learning loss The equivalent calculation term is:

[0164]

[0165] The target reinforcement learning loss The equivalent calculation is:

[0166] .

[0167] Due to the input of underwater environmental observation data at the current moment First, multimodal association shared features are extracted using a shared convolutional neural network. These multimodal association shared features are then input in parallel into independent skill and target sub-modules. Given the current observation state, the selection of skill options and target options do not interfere with each other, thus satisfying the assumption that the predicted skill probability distribution and the predicted target probability distribution are independent under the action space decoupling architecture.

[0168] The high-level policy network is constructed based on the Actor-Critic framework using a proximal policy optimization algorithm. Its network structure specifically includes a shared feature extraction network, an action decoupling output layer, and a state value evaluation layer, as detailed below:

[0169] The shared feature extraction network comprises multiple layers of one-dimensional convolutional neural networks and fully connected layers. Specifically, the network consists of at least two layers of one-dimensional convolutional layers and pooling layers stacked alternately, with a ReLU activation function and a batch normalization layer connected after each convolutional layer. The network takes uniform-dimensional underwater environmental observation data at the current moment as input, and uses one-dimensional convolutional kernels to deeply extract the local implicit correlations between numerical physical features and visual spatial features. After the feature map is flattened, it passes through a fully connected layer with a ReLU activation function to output high-dimensional multimodal correlation shared features.

[0170] The skill submodule (Actor network skill branch) consists of at least two fully connected hidden layers with ReLU activation functions. It receives the multimodal association shared features as input, progressively performs dimensionality reduction and nonlinear mapping of the feature space, and terminates with a fully connected output layer whose number of nodes is equal to the dimension of the optional skill set S. The output layer then undergoes probability normalization using a softmax function to output the predicted skill probability distribution.

[0171] The target submodule (Actor network target branch) is an independent network branch running parallel to the skill submodule. It consists of at least two fully connected hidden layers with ReLU activation functions. It also receives the multimodal association shared features as input, and its terminal is the number of nodes and the set of selectable targets. The output layer is a fully connected layer with equal dimensions, and the probability is normalized by the softmax function to output the probability distribution of the predicted target.

[0172] State value assessment submodule (Critic network branch): Used to assist the proximal policy optimization algorithm in calculating the estimated value of the action advantage function. This module consists of at least two fully connected hidden layers with ReLU activation functions. It receives the multimodal association shared features as input and terminates at a single-neuron linear output layer without activation functions, outputting a scalar estimate of the state value under the current observation state.

[0173] like Figure 4 As shown, the underlying control primitive library is based on the operational requirements of autonomous underwater robots and incorporates expert knowledge in the field of marine engineering. It is pre-built through hard-coded standardized basic control logic and serves as an executable carrier for the decision-making execution end to perform local tasks. The underlying control primitive library contains a path planning module and a motion control module.

[0174] A predefined set of optional skills S that perfectly matches the output dimension of the skill submodule. This set of optional skills S contains standardized operational skills covered by the underlying control primitive library, including at least approach, avoidance, sampling, hovering, and exploration.

[0175] Synchronize a predefined set of optional targets T that perfectly matches the output dimension of the target submodule. The core external objects that the autonomous underwater robot needs to interact with or avoid during its operation, as well as the autonomous underwater robot itself, include at least the mission target area, obstacles, sampling points, underwater markers, and the autonomous underwater robot.

[0176] The predicted skill probability distribution and predicted target probability distribution output by the high-level policy network have dimensions corresponding to the set of available skills, respectively. Optional target set One-to-one correspondence ensures that the effective skill decisions and effective goal decisions obtained from the screening can be accurately mapped to the underlying control primitive library to complete the execution of a local task;

[0177] The path planning module matches a set of optional skills based on the effective skill decisions and effective goal decisions obtained from the mapping. The corresponding operation logic adopts an improved fast exploration random tree algorithm, combined with real-time environmental obstacle data to generate a collision-free reference trajectory that meets the kinematic constraints of the autonomous underwater robot;

[0178] The motion control module constructs closed-loop control logic based on the PID algorithm, converts the collision-free reference trajectory into control signals, and drives the robot actuator to complete the corresponding physical actions to perform local tasks.

[0179] When the aforementioned safe hovering action is triggered, it is broken down into a set of optional skills. Hovering skills and selectable target set The autonomous underwater robot in the model is mapped to the underlying control primitive library as a combination of effective skill decision-making and effective target decision-making. This decomposition mechanism ensures the uniformity of the control interface dimension, so that when performing the safe hovering action, it follows the same execution rules, local task termination conditions, and data acquisition and transfer processes as other local tasks.

[0180] In actual deployment, the specific process of this invention is as follows: Figure 2 As shown, the multimodal sensors simultaneously acquire the autonomous underwater robot's state data and the current underwater environment observation data to form a basic feedback set. The high-level policy network receives and extracts the current underwater environment observation data, processes it in parallel through decoupling in the action space, and independently outputs the predicted skill probability distribution and the predicted target probability distribution. A preset probability threshold is used to filter out targets with a probability not lower than the preset threshold. The system selects the options and chooses the one with the highest probability from those that meet the conditions. These are established as the effective skill decision and the effective target decision, respectively. Together, they constitute the effective decision combination for the current control cycle and are mapped to the underlying control primitive library. If the probability values ​​of all options in the predicted skill probability distribution and the predicted target probability distribution are both lower than a preset probability threshold, then hovering in the optional skill set is taken as the effective skill decision, and the autonomous underwater robot itself in the optional target set is taken as the effective target decision. These are combined into an effective decision combination and mapped to the underlying control primitive library. After initialization, the underlying control primitive library receives the effective decision combination input from the high-level policy network and transforms it into a specific physical operation requirement based on predefined rules. Based on this operation requirement, an improved fast exploration random tree algorithm is used, combined with real-time environmental obstacle data, to generate a collision-free reference trajectory that satisfies the kinematic constraints of the autonomous underwater robot. Subsequently, the motion control module constructs a closed-loop control logic based on the PID algorithm, transforming the above collision-free reference trajectory into a precise control signal, which is input to the robot actuator to drive the underwater robot to perform the corresponding physical actions.

[0181] During the operation of the executing agency, a real-time determination is made as to whether the partial task termination conditions are met. If the partial task termination conditions are not yet met, then... Figure 4 The closed-loop logic shown maintains the current collision-free reference trajectory and outputs control signals. If the local task termination condition is met, such as reaching the target tolerance range normally or meeting the state boundary conditions and the maximum number of steps at the lower level, the lower-level action execution terminates and proceeds to the next level's determination. Further determination is made regarding whether the overall task termination condition is met. If the overall task is not completed, the higher-level policy network uses the basic feedback set at the local task termination moment as input to enter the execution flow of the next local task, while the lower-level control primitive library enters a state of waiting for the next decision combination. If the overall task termination condition is fully met, the entire trajectory data of the overall task is archived locally offline.

[0182] The overall mission trajectory data refers to the continuous set of records with a complete time sequence generated by the autonomous underwater vehicle from the moment it enters the water and independently executes its first local task until the termination condition of the overall mission is triggered and met. This includes a sequence of underwater environmental observation data continuously converted from multimodal sensors at the current moment, and a sequence of effective decision combinations output by the high-level policy network based on the aforementioned underwater environmental observation data at the current moment. The underlying control primitive library executes the corresponding actual physical motion trajectory, as well as the safety and status logs throughout the task process.

[0183] After the entire mission trajectory data is archived offline, it can be used for post-mission playback of underwater robot operating conditions and precise fault tracing after the mission is completed.

[0184] like Figure 7As shown, this invention provides a hierarchical reinforcement learning method for underwater robots guided by a large language model, used to execute the method described in this invention, comprising:

[0185] The multimodal data acquisition module is used to acquire multimodal data and autonomous underwater robot status data in real time, and convert the multimodal data into underwater environment observation data at the current moment; during the deployment phase, it is used to package and generate a basic feedback set and feed it back to the high-level policy network; during the training phase, it outputs the basic feedback set and multimodal data to form a training supplementary set.

[0186] The decision execution end has a built-in high-level strategy network module and a low-level control primitive library. The high-level strategy network module is used to receive the underwater environment observation data at the current moment, decouple and output the predicted skill probability distribution and the predicted target probability distribution in parallel through action space decoupling, filter to obtain effective decisions, and map the effective decisions to the low-level control primitive library to generate control signals.

[0187] The underwater robot actuator is used to receive the control signals output by the underlying control primitive library and complete the corresponding physical interaction actions;

[0188] The same-strategy trajectory buffer storage module is configured to collect the basic feedback set and the training supplement set and store them in the same-strategy trajectory buffer when the local task termination condition is met, and to trigger the semantic guidance module when the data in the same-strategy trajectory buffer reaches a preset batch; The semantic guidance module has a built-in visual language model and a large language model, which are used to extract image-type observation data from the multimodal data in the training supplement set, generate environmental state description text through the visual language model, combine it with the total task description text and input it into the large language model to generate multiple sets of joint probability distributions and take the mean, and obtain the guidance skill probability distribution and guidance target probability distribution through edge decoupling;

[0189] The loss calculation module is configured to calculate the first KL divergence between the predicted skill probability distribution and the guided skill probability distribution, and the second KL divergence between the predicted target probability distribution and the guided target probability distribution. Under the empirical data distribution, the mathematical expectation of the sum of the first KL divergence and the second KL divergence is the imitation learning loss. After annealing weight decay, it is combined with the reinforcement learning loss to construct a hybrid loss function.

[0190] The policy update module is used to update the parameters of the high-level policy network based on the gradient of the hybrid loss function using a proximal policy optimization algorithm, and clear the same policy trajectory buffer after the update is completed. At the same time, it determines whether the overall task termination condition is met. If the overall task termination condition is met, it further determines whether the convergence judgment criterion is met. If it is met, the training phase ends and the high-level policy network with training frozen is obtained. Otherwise, the process from "acquiring multimodal data" to "updating parameters" in the figure is executed repeatedly until the preset maximum number of training rounds is reached.

[0191] The deployment and execution module is configured to deploy the high-level policy network and the low-level control primitive library that are frozen during training. Based on the underwater environment observation data at the current moment, it outputs effective skill decisions and effective target decisions, which are mapped to the low-level control primitive library and transformed into control signals to drive the underwater robot's actuators until the overall task is completed and the entire trajectory data of the overall task is archived.

[0192] The parts not described in detail in this application are all existing conventional technologies and will not be elaborated here.

[0193] It is understood that the above specific description of the present invention is only for illustrating the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention to achieve the same technical effect; as long as the use needs are met, they are all within the protection scope of the present invention.

Claims

1. A hierarchical reinforcement learning method for underwater robots guided by a large language model, characterized in that, Includes the following steps: S1: The collected multimodal data is transformed into underwater environment observation data at the current moment and input into the high-level policy network. After action space decoupling, the predicted skill and target probability distribution are output. Effective decision combinations are selected and mapped to the low-level control primitive library to execute local tasks. When the local task termination condition is met, the basic feedback set and training supplement set are collected and stored in the same policy trajectory buffer. When the preset batch capacity is reached, S2 is triggered. S2: Extract image-class observation data from the training supplement set, generate environmental state description text through the visual language model, combine it with the total task description text and input it into the large language model to generate multiple sets of joint probability distributions and take the mean. After edge decoupling, obtain the guidance skills and target probability distributions. S3: Calculate the first KL divergence between the predicted and guided skill probability distributions, and the second KL divergence between the predicted and guided target probability distributions, respectively. Construct the imitation learning loss, and after annealing weight decay, construct a hybrid loss function with the reinforcement learning loss. Use the proximal policy optimization algorithm to update the high-level policy network and clear the same policy trajectory buffer. Satisfy the overall task termination condition and convergence criterion to obtain the frozen high-level policy network. Otherwise, repeat S1-S3 until the preset maximum number of training rounds is reached; S4: Deploy the frozen high-level strategy network and the low-level control primitive library, output effective skills and target decisions based on the underwater environment observation data at the current moment, map them to the low-level control primitive library to drive the robot's actuators, until the overall task is completed and the entire trajectory data is archived.

2. The method according to claim 1, characterized in that: The multimodal data includes underwater topography data, obstacle distance data, mission target coordinate data, aquatic environmental parameters, and image-based observation data. Multimodal data is acquired through multimodal sensors, including acoustic sensors, a vision camera, and an inertial measurement unit (IMU), as well as an edge computing board connected to these sensors. The edge computing board embeds an intelligent processing chip for deploying fully connected layers or convolutional neural networks. The acoustic sensors and the vision camera acquire multimodal data, and the IMU acquires the autonomous underwater robot's state data. The multimodal data is converted into current-moment underwater environment observation data, including: The underwater terrain data, obstacle distance data, target coordinate data, and aquatic environmental parameters in the multimodal data are used as numerical observation data. These data are mapped through a fully connected layer and transformed into numerical physical feature vectors. The image-type observation data are processed by a convolutional neural network to extract features and transform them into visual spatial feature vectors that match the dimension of the numerical physical feature vectors. The numerical physical feature vectors and the visual spatial feature vectors are then concatenated to form unified underwater environmental observation data for the current moment.

3. The method according to claim 1, characterized in that: The process of decoupling the action space and outputting the predicted skill probability distribution and the predicted target probability distribution in parallel, and then filtering for effective decisions using a preset probability threshold, includes: The current underwater environment observation data is input into the high-level policy network. The associated features are extracted through the shared convolutional neural network of the high-level policy network, and the nonlinear transformation is completed by the ReLU activation function to output multimodal associated shared features. The multimodal association shared features are input in parallel into the skill sub-module and target sub-module of the high-level policy network. After being mapped by fully connected hidden layers containing multiple ReLU activation functions in both the skill sub-module and the target sub-module, the predicted skill probability distribution and the predicted target probability distribution are output by the terminal softmax function, respectively. The predicted skill probability distribution and the predicted target probability distribution are filtered according to a preset probability threshold. Options with a probability value not lower than the preset probability threshold are selected as valid options. The option with the highest probability among the valid options is selected as the valid skill decision and the valid target decision, respectively. They are combined into a valid decision combination and mapped to the underlying control primitive library to drive the execution of local tasks. If the probability values ​​of all options in the predicted skill probability distribution and the predicted target probability distribution are lower than the preset probability threshold, then hovering in the optional skill set is selected as the valid skill decision, and the autonomous underwater robot itself in the optional target set is selected as the valid target decision. They are combined into a valid decision combination and mapped to the underlying control primitive library to execute a safe hovering action.

4. The method according to claim 1, characterized in that, The decoupling of guidance skills and target probability distributions through edge detection involves splitting the joint probability distribution containing both skills and targets into independent guidance skill probability distributions and guidance target probability distributions. The method is as follows: By traversing the set of optional targets All targets The joint probabilities are summed to separate the independent probability distribution of the guidance skill from the guidance strategy distribution. The calculation formula is as follows: ， By traversing the set of available skills All skills The joint probabilities are summed to separate the independent target probability distribution from the policy distribution. The calculation formula is as follows: ， in, For an optional set of targets, A set of optional skills.

5. The method according to claim 1, characterized in that, The reinforcement learning loss is based on the assumption that the probability distributions of the predicted skill and the predicted target are independent under the action space decoupling architecture. The reinforcement learning loss is calculated using the logarithm of the joint probability, and its expression is: ， in, Indicating the distribution of empirical data Find the expected value; Given the observed data, predict the probability of selecting a corresponding effective skill decision from the skill probability distribution; To predict the probability of selecting a corresponding effective target decision from the target probability distribution; the empirical data distribution The trajectory buffer of the same strategy consists of a set of historical samples composed of multiple sets of underwater environment observation data at the current moment and their corresponding effective skill decisions and effective target decisions; The reinforcement learning loss is decoupled into a linear superposition of the skill reinforcement learning loss and the objective reinforcement learning loss, i.e.: 。 6. The method according to claim 1, characterized in that, When constructing the hybrid loss function, the reinforcement learning loss and the imitation learning loss dynamically weighted by the annealing weight parameters are fused together. The imitation learning loss refers to the first difference between the predicted skill probability distribution and the guided skill probability distribution under the empirical data distribution. Divergence and the second divergence between the predicted target probability distribution and the guiding target probability distribution The mathematical expectation of the sum of the divergences; The annealing weight parameters are used in the early stages of training to force the high-level policy network to update its direction by using the guidance policy distribution generated by the large language model. Subsequently, the proportion of imitation learning loss is gradually reduced until it transitions to autonomous exploration that relies entirely on reinforcement learning loss.

7. The method according to claim 6, characterized in that, The method for adjusting the annealing weight parameters during training is as follows: The initial value of the annealing weight parameter is determined by the ratio of the reinforcement learning loss to the imitation learning loss during the initial training phase, and the calculation formula is as follows: ， To reinforce learning loss, Learning loss due to imitation; During the training phase, the annealing weight parameters As it decays linearly to 0, its piecewise function expression is: ， Where i represents the current training step, The preset decay termination step is used to characterize the imitation learning loss weights returning to zero and the high-level policy network transforming into a training critical node that relies entirely on reinforcement learning loss for updates.

8. The method according to claim 1, characterized in that, Updating the high-level policy network using a near-end policy optimization algorithm includes: during the training phase, after each local task termination condition is met and the basic feedback set and training supplement set are stored in the same policy trajectory buffer, if it is determined that the amount of data in the same policy trajectory buffer has reached the preset batch capacity, then after the construction of the hybrid loss function is completed, the gradient of the hybrid loss function is extracted. The near-end policy optimization algorithm is adopted to update the network parameters of the high-level policy network once according to the gradient of the hybrid loss function, and the same policy trajectory buffer is cleared after the update is completed. After the parameter update is completed, the high-level policy network receives the basic feedback set generated when the local task termination condition is met, outputs effective skill decision and effective target decision, and maps them to the low-level control primitive library to drive the robot actuator to perform the local task. The execution of the aforementioned local tasks and the network parameter update steps are repeated until the overall task termination condition is met, at which point the current complete training round ends. Then, it is further determined whether consecutive complete training rounds meet the convergence judgment criteria. If they do, the training phase ends. If they do not meet the criteria and the preset maximum number of training rounds has not yet been reached, the next complete training round begins.

9. The method according to claim 8, characterized in that, The termination condition and convergence criterion for the overall task are as follows: The overall mission termination conditions include: the autonomous underwater robot successfully reaching the global final target tolerance range, a collision that violates the mission safety constraints, or the total execution time exceeding the preset global maximum time limit. The convergence criteria include: the task success rate calculated statistically based on local task trajectory data within multiple consecutive complete training rounds is not lower than a preset success rate threshold; the cumulative value fluctuation of all environmental feedback reward signals within a single training round is less than a preset fluctuation threshold; and the numerical decrease of the imitation learning loss tends to stabilize and converge.

10. A hierarchical reinforcement learning method for underwater robots guided by a large language model, used to perform the method according to any one of claims 1 to 9, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal data and autonomous underwater robot status data in real time, and convert the multimodal data into underwater environment observation data at the current moment; During the deployment phase, it is used to package and generate a basic feedback set and feed it back to the higher-level policy network; During the training phase, the basic feedback set and multimodal data are output to form a training supplement set; The decision execution end has a built-in high-level strategy network module and a low-level control primitive library. The high-level strategy network module is used to receive the underwater environment observation data at the current moment, and output the predicted skill probability distribution and the predicted target probability distribution in parallel after decoupling from the action space. Effective decision combinations are then selected and mapped to the low-level control primitive library to generate control signals. The underwater robot actuator is used to receive the control signals output by the underlying control primitive library and complete the corresponding physical interaction actions; The same strategy trajectory buffer storage module is configured to collect the basic feedback set and the training supplement set into the same strategy trajectory buffer when the local task termination condition is met, and to trigger the semantic guidance module when the data in the same strategy trajectory buffer reaches a preset batch. The semantic guidance module has a built-in visual language model and a large language model. It is used to extract image-type observation data from the multimodal data in the training supplement set, generate environmental state description text through the visual language model, and input the total task description text into the large language model to generate multiple sets of joint probability distributions and take the mean. After edge decoupling, the guidance skill probability distribution and guidance target probability distribution are obtained. The loss calculation module is configured to calculate the first KL divergence between the predicted skill probability distribution and the guided skill probability distribution, and the second KL divergence between the predicted target probability distribution and the guided target probability distribution. Under the empirical data distribution, the mathematical expectation of the sum of the first KL divergence and the second KL divergence is the imitation learning loss. After annealing weight decay, it is combined with the reinforcement learning loss to construct a hybrid loss function. The policy update module is used to update the parameters of the high-level policy network based on the gradient of the hybrid loss function using a near-end policy optimization algorithm, and clear the same policy trajectory buffer after the update is completed. The deployment and execution module is configured to deploy the high-level policy network and the low-level control primitive library that are frozen during training. Based on the underwater environment observation data at the current moment, it outputs effective skill decisions and effective target decisions, which are mapped to the low-level control primitive library and transformed into control signals to drive the autonomous underwater robot's actuators until the overall task is completed and the entire trajectory data of the overall task is archived.

Images