An auv hierarchical visual language action decision method and system for underwater restricted environment

Abstract

The application discloses an AUV layered visual language action decision method and system for underwater restricted environment, and relates to the field of autonomous underwater vehicles. The method maps user natural language instructions and underwater visual images into a structured task description file through an on-board visual-language model, generates a timestamped reference trajectory through task analysis and compilation, performs online verification and projection correction on the reference trajectory or bottom layer control quantity based on a hard safety constraint set by a safety supervisor during execution, and forcibly covers a safe fallback primitive when an anomaly is detected, and finally drives the AUV to execute through bottom layer dynamics closed loop control and thruster distribution. The method decouples semantic understanding and bottom layer control, combines hard constraint verification and forced fallback mechanism of the safety supervision module, and improves the explainability, safety and physical executability of AUV decision in underwater restricted environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous underwater vehicles, and in particular to a hierarchical visual language action decision-making method and system for AUVs in underwater confined environments. Background Technology

[0002] Underwater confined environments (such as shipwreck compartments, submarine utility tunnels, underwater caves, and narrow passages) are characterized by extremely narrow spaces, densely packed obstacles, rapid attenuation of light, and high water turbidity. When autonomous underwater vehicles (AUVs) perform tasks such as inspection, search and rescue, and geological sampling in these environments, they face significant challenges, including unstable sensing information, easy drift in acoustic / inertial positioning, and a sharp increase in collision risk due to spatial compression.

[0003] Existing end-to-end vision-language-action models directly map visual observations and natural language commands into thruster control variables, aiming to achieve direct language-driven control. However, this approach exposes the following core problems in underwater confined environments:

[0004] The reasoning process is opaque, lacking auditable intermediate decision-making outputs. The end-to-end model directly maps high-dimensional sensory inputs to low-dimensional control variables, making the internal decision-making process completely invisible. Operators cannot review the rationality of the task plan before execution, nor can they trace the cause when abnormal actions occur. Underwater operations have extremely limited fault tolerance; once an AUV exhibits dangerous behavior, engineers cannot locate the root cause of the failure from within the model, severely hindering post-event review and system reliability improvement.

[0005] The lack of deterministic hard safety constraints and independent interception mechanisms is a significant problem. Purely data-driven vision-language models, under conditions of insufficient training coverage such as underwater turbidity, backlighting, and interference from suspended particles, are highly prone to outputting dangerous maneuver commands that exceed the platform's physical limits. Examples include instructing the AUV to enter gaps smaller than its own size, issuing sharp turns beyond the thruster's capabilities, or providing extreme pitch angles. Existing solutions only apply soft constraints through a loss function during the training phase, lacking a safety gatekeeping mechanism independent of the inference module. Once a dangerous command is issued and executed, it could directly lead to an AUV collision, capsizing, or instability, causing irreversible equipment damage or even mission failure.

[0006] The output control quantity is not adequately matched to the physical capability boundaries of the underwater actuator. Underwater thrusters are constrained by strong nonlinear characteristics such as motor response bandwidth, thrust rate of change, and dead zone, meaning that not all desired force / torque commands can be accurately executed. Existing solutions often predict the normalized control quantity directly at the output layer without constraints, lacking explicit modeling of the thruster's saturation boundary and upper limit of the rate of change. This approach easily generates commands exceeding the thruster's physical capabilities, leading to a significant deviation of the actual thrust response from expectations. The AUV loses controllability due to continuous thruster saturation, often meaning collisions are unavoidable in confined spaces. Summary of the Invention

[0007] The purpose of this invention is to provide a hierarchical visual language action decision-making method and system for AUVs in underwater confined environments, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a hierarchical visual language action decision-making method for AUVs in underwater confined environments, comprising the following steps:

[0009] Step S1: Receive user's natural language control commands, synchronously acquire underwater visual images and fuse multi-sensor data to obtain AUV current state estimation and environmental obstacle distance information;

[0010] Step S2: Input the natural language control command and the visual image into the airborne vision-language model. After semantic understanding and constrained decoding, output a structured task description file. The structured task description file contains a sequence of action primitives arranged in time sequence and the target parameters and execution constraints corresponding to each action primitive.

[0011] Step S3: Perform syntax compliance verification and parameter boundary verification on the structured task description file. After the verification is passed, compile the action primitive sequence into a reference trajectory containing timestamps, track points and velocity profiles according to the preset primitive library.

[0012] Step S4: During the execution of the reference trajectory, the safety supervisor performs rolling time-domain violation prediction and online correction of the reference trajectory or real-time control quantity based on a pre-built set of hard safety constraints at a period no less than the underlying control frequency; when any state quantity is predicted or detected to break through the boundary of the hard safety constraint set, the safety supervisor immediately interrupts the upper-layer trajectory input and forcibly outputs a safety rollback command to replace the reference trajectory.

[0013] Step S5: The trajectory command corrected by the safety monitor or the safety reversal command is sent to the underlying dynamic closed-loop controller. The desired control force and torque are calculated based on the AUV's six-degree-of-freedom hydrodynamic model. After the thruster control allocation and saturation limiting processing, the AUV is driven to perform the corresponding action.

[0014] Preferably, the set of hard safety constraints in step S4 includes at least: minimum safety distance constraint, depth boundary constraint, attitude angle boundary constraint, and upper limit constraint of sailing speed; when the safety supervisor perceives the confidence level as lower than a preset threshold, it automatically increases the threshold of the minimum safety distance constraint and simultaneously decreases the threshold of the upper limit constraint of sailing speed, so that the system switches to a conservative operation mode.

[0015] Preferably, the constrained decoding in step S2 maintains a prefix tree based on structured data constraint rules. During the word-by-word decoding process of the visual-language model, the position of the currently generated sequence in the prefix tree is queried, and a mask set for the next legal token is dynamically generated. The probability value of the illegal token is set to negative infinity, forcing the model output path to always fall within the legal space defined by the structured data constraint rules.

[0016] Preferably, the structured task description file includes at least a primitive type field, a target parameter field, and a constraint field; the parameter boundary verification in step S3 includes at least: verifying whether the primitive type field belongs to the preset primitive library enumeration range, verifying whether the target parameter field is within the numerical range allowed by the platform, and verifying whether the constraint field is not lower than the platform's minimum safety margin. If any verification fails, the current task is rejected and a hover or floating back is triggered.

[0017] Preferably, the violation prediction in the rolling time domain in step S4 includes: starting from the current state estimate, recursively predicting the future state sequence based on the AUV discrete motion model within a preset prediction window; if the predicted value at any moment in the future state sequence falls into the violation region of the hard safety constraint set, the online correction is triggered; the online correction includes spatial pruning, velocity retiming, or local obstacle bypass reconstruction of the reference trajectory.

[0018] Preferably, in step S4, the safety monitor performs a correction based on safety filtering on the real-time control quantity: with the original control quantity as the center, it solves for the nearest Euclidean projection point in the convex constraint space formed by the feasible region of the thruster and the safety invariant set, and replaces the original control quantity output with the corrected control quantity corresponding to the nearest point; when the thruster's continuous saturation time exceeds a preset threshold, the safety monitor directly triggers the safety rollback command.

[0019] A hierarchical visual-language action decision-making system for AUVs in underwater confined environments, comprising the following execution methods:

[0020] The instruction receiving module is configured to receive user natural language control instructions;

[0021] The perception data collection module is configured to acquire underwater visual images and fuse data from the inertial measurement unit, depth gauge and Doppler velocimeter, and output AUV state estimation and environmental perception information;

[0022] The vision-language reasoning module is configured to run an onboard vision-language model and map the natural language control commands and the visual images into a structured task description file.

[0023] The task parsing module is configured to perform syntax and boundary checks on the structured task description file;

[0024] The task compilation module is configured to compile the validated sequence of action primitives into a timestamped reference trajectory;

[0025] The safety supervision module is configured to perform rolling verification and projection correction on the reference trajectory or real-time control quantity based on the hard safety constraint set, and to force the reference trajectory to be covered by a safety rollback command when a violation occurs;

[0026] The dynamic closed-loop control module is configured to calculate the desired control force and torque based on the AUV hydrodynamic model.

[0027] The thruster allocation module is configured to map the desired control force and torque into thrust commands for each thruster through constraint optimization.

[0028] Preferably, the safety monitoring module includes:

[0029] The constraint set generation unit is used to dynamically maintain and update the hard safety constraint set, which includes the lower limit of obstacle distance, the upper and lower limits of depth, the upper limit of attitude angle, and the upper limit of velocity.

[0030] The risk prediction unit is used to recursively predict the future state sequence within the prediction time domain and determine whether the boundary of the hard safety constraint set is touched.

[0031] The online projection correction unit is used to perform minimum norm projection on the non-compliant trajectory points or control quantities, pulling them back into the safety domain defined by the hard safety constraint set;

[0032] The overlay rollback unit has a higher output priority than the task compilation module. It is used to directly cut off the input channel of the reference trajectory when a hard safety condition is triggered, and output the safety rollback command to the dynamic closed-loop control module.

[0033] Preferably, the reasoning output cycle of the visual-language reasoning module is 0.5Hz to 5Hz, the control operation cycle of the dynamic closed-loop control module and the thruster allocation module is 20Hz to 200Hz, and the monitoring and intervention cycle of the safety supervision module is not less than the control operation cycle, so as to achieve real-time safety coverage of upper-level decisions.

[0034] Preferably, the safety rollback command includes at least one of emergency braking, hovering, controlled levitation, and low-speed rollback. The rollback coverage unit selects output according to the priority order of emergency braking over hovering, hovering over controlled levitation, and controlled levitation over low-speed rollback. The system state switches between at least normal tracking mode, conservative degradation mode, rollback coverage mode, and task recovery mode. When no violation of the hard safety constraint set is triggered within a preset duration, the system switches from the rollback coverage mode to the task recovery mode, resumes the execution of the remaining action primitives, or triggers replanning.

[0035] The technical effects and advantages of this invention are as follows:

[0036] 1. This hierarchical visual-language action decision-making system for AUVs in confined underwater environments differs from end-to-end models that directly output untraceable control quantities. This invention uses a visual-language reasoning module to transform natural language commands and visual observations into a structured task description file, containing action primitive types arranged in a time sequence, target parameters, and execution constraint fields. The task parsing module performs syntax and parameter boundary checks on the structured description before task execution, enabling operators to review the task's rationale before issuing commands and trace the decision-making basis after anomalies occur. This provides a data foundation for underwater operation review, fault location, and system iteration.

[0037] 2. This hierarchical visual-language action decision-making system for AUVs designed for underwater confined environments features an independently configured safety supervision module within the control link. This module does not rely on the output of the visual-language inference module; instead, it independently performs rolling verification of the reference trajectory and real-time control variables based on a hard safety constraint set constructed by the constraint set generation unit. When the risk prediction unit determines that the current command will result in collision with an obstacle or exceeding depth / attitude / velocity limits, the online projection correction unit first attempts to make minimal modifications to the trajectory or control variables. If the correction still fails to bring the system back to the safe domain, the overlay backoff unit directly cuts off the upper-level input with the highest priority, forcibly outputting emergency braking, hovering, or surfacing safety backoff commands to the dynamic closed-loop control module. This mechanism ensures that even if the upper-level visual-language inference produces a completely erroneous semantic understanding, the lower-level execution is still constrained by hard constraints.

[0038] 3. This hierarchical visual language action decision-making system for AUVs designed for underwater confined environments calculates the desired control force and torque based on a six-degree-of-freedom hydrodynamic model through a dynamic closed-loop control module. Then, the thruster allocation module, under the condition of explicitly including upper and lower limits of thrust, upper limit of rate of change, and dead zone compensation constraints, maps the desired control quantity to thrust commands for each thruster through constraint optimization, ensuring that the issued commands always fall within the physical capability boundaries of the thrusters. At the same time, when the system detects that the thrusters are continuously saturated beyond a preset threshold, the safety monitoring module directly triggers a safety backoff to prevent the system from operating at the edge of loss of controllability, making the tasks planned at the upper level truly executable at the physical level. Attached Figure Description

[0039] Figure 1 A schematic diagram of the structure of the hierarchical visual language action decision-making system for AUVs in underwater confined environments provided by the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the structured task description generation and syntax constraint verification provided by the present invention.

[0041] Figure 3 This is a flowchart illustrating the online verification, projection correction, and forced rollback of the security monitoring module provided by the present invention.

[0042] Figure 4 This is a schematic diagram of the workflow of the underlying dynamic closed-loop control and thruster allocation provided by the present invention;

[0043] Figure 5 This is a schematic diagram of the hard safety trigger and rollback state machine provided by the present invention.

[0044] In the diagram: 101, Command Receiving Module; 102, Perceptual Data Collection Module; 103, Visual-Language Reasoning Module; 104, Task Parsing Module; 105, Task Compilation Module; 106, Safety Supervision Module; 1061, Constraint Set Generation Unit; 1062, Risk Prediction Unit; 1063, Online Projection Correction Unit; 1064, Coverage Backoff Unit; 107, Dynamic Closed-Loop Control Module; 108, Thruster Allocation Module; 109, Actuator / Thruster Module. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention provides, for example Figures 1-5The illustrated hierarchical visual language action decision-making method for AUVs in underwater confined environments includes the following steps:

[0047] Step S1: Receive user's natural language control commands, synchronously acquire underwater visual images and fuse multi-sensor data to obtain AUV current state estimation and environmental obstacle distance information;

[0048] Step S2: Input the natural language control commands and visual images into the airborne vision-language model. After semantic understanding and constrained decoding, output a structured task description file. The structured task description file contains a sequence of action primitives arranged in time sequence and the target parameters and execution constraints corresponding to each action primitive.

[0049] Step S3: Perform syntax compliance and parameter boundary checks on the structured task description file. After the checks pass, compile the action primitive sequence into a reference trajectory containing timestamps, track points and velocity profiles according to the preset primitive library.

[0050] Step S4: During the execution of the reference trajectory, the safety supervisor performs rolling time-domain violation prediction and online correction of the reference trajectory or real-time control quantity based on the pre-built hard safety constraint set at a period no less than the lower control frequency; when any state quantity is predicted or detected to break through the boundary of the hard safety constraint set, the safety supervisor immediately interrupts the upper-level trajectory input and forcibly outputs a safety backoff command to replace the reference trajectory.

[0051] Step S5: The trajectory command or safety retreat command corrected by the safety monitor is sent to the underlying dynamic closed-loop controller. The desired control force and torque are calculated based on the AUV's six-degree-of-freedom hydrodynamic model. After the thruster control allocation and saturation limiting processing, the AUV is driven to perform the corresponding action.

[0052] Furthermore, the set of hard safety constraints in step S4 includes at least: minimum safe distance constraint, depth boundary constraint, attitude angle boundary constraint, and upper limit constraint of sailing speed; when the perception confidence level is lower than the preset threshold, the safety monitor automatically increases the threshold of the minimum safe distance constraint and simultaneously decreases the threshold of the upper limit constraint of sailing speed, so that the system switches to a conservative operation mode.

[0053] Furthermore, in step S2, the constrained decoding maintains a prefix tree based on structured data constraint rules. During the word-by-word decoding process of the visual-language model, the position of the currently generated sequence in the prefix tree is queried, and the mask set of the next legal token is dynamically generated. The probability value of the illegal token is set to negative infinity, forcing the model output path to always fall within the legal space defined by the structured data constraint rules.

[0054] Furthermore, the structured task description file includes at least a primitive type field, a target parameter field, and a constraint field; the parameter boundary verification in step S3 includes at least: verifying whether the primitive type field belongs to the preset primitive library enumeration range, verifying whether the target parameter field is within the numerical range allowed by the platform, and verifying whether the constraint field is not lower than the platform's minimum safety margin. If any verification fails, the current task is rejected and a hover or floating back is triggered.

[0055] Furthermore, the violation prediction in the rolling time domain in step S4 includes: starting from the current state estimate, recursively predicting the future state sequence based on the AUV discrete motion model within a preset prediction window; if the predicted value at any time in the future state sequence falls into the violation area of ​​the hard safety constraint set, online correction is triggered; online correction includes spatial pruning, velocity retiming, or local obstacle bypass reconstruction of the reference trajectory.

[0056] Furthermore, in step S4, the safety monitor performs a safety-based correction on the real-time control quantity: with the original control quantity as the center, it solves for the nearest Euclidean projection point in the convex constraint space formed by the feasible region of the thruster and the safety invariant set, and replaces the original control quantity output with the corrected control quantity corresponding to the nearest point; when the continuous saturation time of the thruster exceeds the preset threshold, the safety monitor directly triggers the safety back-off command.

[0057] A hierarchical visual-language action decision-making system for AUVs in underwater confined environments, comprising the following execution methods:

[0058] The instruction receiving module 101 is configured to receive user natural language control instructions;

[0059] The perception data collection module 102 is configured to acquire underwater visual images and fuse data from the inertial measurement unit, depth gauge and Doppler velocimeter, and output AUV state estimation and environmental perception information.

[0060] The visual-language reasoning module 103 is configured to run an onboard visual-language model to map natural language control commands and visual images into structured task description files;

[0061] Task parsing module 104 is configured to perform syntax and boundary checks on structured task description files;

[0062] Task compilation module 105 is configured to compile the validated action primitive sequence into a timestamped reference trajectory;

[0063] The safety supervision module 106 is configured to perform rolling verification and projection correction on the reference trajectory or real-time control quantity based on the hard safety constraint set, and to force the reference trajectory to be covered by a safety rollback command when a violation occurs.

[0064] The dynamic closed-loop control module 107 is configured to calculate the desired control force and torque based on the AUV hydrodynamic model.

[0065] The thruster allocation module 108 is configured to map the desired control force and torque into thrust commands for each thruster through constraint optimization.

[0066] Furthermore, the safety monitoring module 106 includes:

[0067] The constraint set generation unit 1061 is used to dynamically maintain and update the hard safety constraint set, which includes the lower limit of obstacle distance, the upper and lower limits of depth, the upper limit of attitude angle, and the upper limit of velocity.

[0068] Risk prediction unit 1062 is used to recursively predict future state sequences within the prediction time domain and determine whether the hard safety constraint set boundary is touched.

[0069] The online projection correction unit 1063 is used to perform minimum norm projection on the non-compliant trajectory points or control quantities, pulling them back into the safety domain defined by the hard safety constraint set;

[0070] The overlay back-off unit 1064 has a higher output priority than the task compilation module 105. It is used to directly cut off the input channel of the reference trajectory when the hard safety condition is triggered, and output a safety back-off instruction to the dynamic closed-loop control module 107.

[0071] Furthermore, the reasoning output cycle of the visual-language reasoning module 103 is 0.5Hz to 5Hz, the control operation cycle of the dynamic closed-loop control module 107 and the thruster allocation module 108 is 20Hz to 200Hz, and the monitoring and intervention cycle of the safety supervision module 106 is not less than the control operation cycle, so as to achieve real-time safety coverage of upper-level decisions.

[0072] Furthermore, the safety rollback command includes at least one of emergency braking, hovering, controlled levitation, and low-speed reversal. The rollback unit 1064 selects and outputs commands according to the priority order of emergency braking over hovering, hovering over controlled levitation, and controlled levitation over low-speed reversal. The system state switches between at least the normal tracking mode, the conservative degradation mode, the rollback coverage mode, and the task recovery mode. When no violation of the hard safety constraint set is triggered within a preset duration, the system switches from the rollback coverage mode to the task recovery mode, resumes the execution of the remaining action primitives, or triggers replanning.

[0073] Unlike end-to-end models that directly output untraceable control quantities, this invention uses a vision-language reasoning module 103 to transform natural language instructions and visual observations into a structured task description file, which includes action primitive types arranged in sequence, target parameters, and execution constraint fields. The task parsing module 104 performs syntax and parameter boundary checks on the structured description before task execution, enabling operators to review the task's rationality before issuing instructions and trace the decision-making basis after anomalies occur, providing a data foundation for underwater operation review, fault location, and system iteration.

[0074] A safety supervision module 106 is independently set up in the control link. This module does not rely on the output of the vision-language reasoning module 103, but independently performs rolling verification of the reference trajectory and real-time control quantities based on the hard safety constraint set constructed by the constraint set generation unit 1061. When the risk prediction unit 1062 determines that the current command will cause collision with obstacles or exceed the depth / attitude / velocity limits, the online projection correction unit 1063 first attempts to make minimal modifications to the trajectory or control quantities; if the correction still cannot pull back to the safe domain, the overwrite backoff unit 1064 directly cuts off the upper-level input with the highest priority and forcibly outputs safety backoff commands such as emergency braking, hovering, or levitation to the dynamic closed-loop control module 107. This mechanism ensures that even if the upper-level vision-language reasoning produces a completely wrong semantic understanding, the lower-level execution is still constrained by hard constraints.

[0075] The dynamic closed-loop control module 107 calculates the desired control force and torque based on a six-degree-of-freedom hydrodynamic model. Then, the thruster allocation module 108, under the condition of explicitly including upper and lower limits of thrust, upper limit of rate of change, and dead zone compensation constraints, maps the desired control quantity to thrust commands for each thruster through constraint optimization, ensuring that the issued commands always fall within the physical capability boundaries of the thrusters. At the same time, when the thruster is detected to be continuously saturated beyond the preset threshold, the safety monitoring module 106 directly triggers a safety backoff to prevent the system from operating at the edge of loss of controllability, so that the tasks planned at the upper level have true executability at the physical level.

[0076] Figure 1 This invention provides a structural schematic diagram of a hierarchical visual language action decision-making system for AUVs in underwater confined environments, as shown below. Figure 1 As shown, the system of the present invention includes:

[0077] The module includes a command receiving module 101, a perception data collection module 102, a vision-language reasoning module 103, a task parsing module 104, a task compilation module 105, a safety monitoring module 106, a dynamic closed-loop control module 107, a thruster allocation module 108, and an actuator / thruster module 109.

[0078] The instruction receiving module 101 is used to receive user natural language instructions (which may come from a surface station, a shipboard control console, or a preset task).

[0079] The perception data collection module 102 is used to collect underwater visual images and can optionally collect data from sonar, IMU, depth gauge, DVL, etc., perform time synchronization, filtering and noise reduction and feature extraction, and output state estimation x and environmental distance information d.

[0080] The visual-language reasoning module 103 is used to run a lightweight visual-language model on the local onboard computing platform and generate a structured task description file TaskJSON based on "visual images + natural language instructions".

[0081] The task parsing module 104 is used to perform syntax validation, field integrity validation, parameter out-of-bounds validation, and platform capability validation on TaskJSON; when the validation fails, a rollback strategy is triggered and a rollback instruction is sent to the security supervision module 106.

[0082] The task compilation module 105 is used to compile the sequence of action primitives into a reference trajectory r(t), which includes a sequence of track points, a velocity profile, a depth / heading profile, and a timestamp and phase termination condition for each trajectory point.

[0083] The safety monitoring module 106 is used to construct the hard safety constraint set Ω and perform online verification and projection correction on the reference trajectory or control quantity; when the hard safety condition is triggered, the reference trajectory output backoff primitive is forcibly overridden.

[0084] The dynamic closed-loop control module 107 is used to calculate the desired resultant force / resultant moment τ or intermediate control quantity u based on the AUV hydrodynamic model;

[0085] The thruster allocation module 108 is used to map τ into a multi-thruster thrust command T, and after performing thrust saturation and rate of change limiting processing, outputs the command to the actuator / thruster module 109 to drive the AUV motion.

[0086] In one embodiment, the modules interact with each other via a message bus or shared memory; the vision-language inference module 103 is deployed on an embedded high-performance computing platform, such as NVIDIA Jetson Orin NX or Xavier series, to meet the computing power requirements of large model inference; the dynamic closed-loop control module 107 and the thruster allocation module 108 run on a real-time microcontroller, such as STM32H7 series or Pixhawk flight control hardware, to ensure millisecond-level control response. The instruction receiving module 101 outputs instruction text and task trigger signals; the perception data collection module 102 outputs visual image frames, sonar range array, IMU attitude and angular velocity, depth gauge depth, DVL velocity, etc., and completes time synchronization and coordinate system unification to obtain state estimation x and environmental distance information d.

[0087] In one alternative implementation, the sensing data collection module 102 may employ extended Kalman filtering (EKF) or factor graph optimization to perform multi-sensor fusion, outputting a state vector containing position, velocity, attitude, and angular velocity; and may perform dehazing enhancement, color correction, and noise reduction on the image to improve observability under low-light / turbid conditions underwater.

[0088] In one embodiment, the visual-language reasoning module 103 has a reasoning frequency of 0.5Hz to 5Hz and is used to generate or update TaskJSON; the underlying dynamics closed-loop control module 107 and the thruster allocation module 108 have a control frequency of 20Hz to 200Hz and are used to perform trajectory tracking and thrust allocation; the safety supervision module 106 can operate at the same frequency or higher than the controller to ensure that the hard safety coverage has real-time performance.

[0089] The "Platform Capability Verification" in the task parsing module 104 includes at least: whether the action primitive type is within the primitive library, whether the parameter units and ranges match, whether the required degrees of freedom for the task are supported by the current thruster layout, and whether the preconditions for task execution are met (e.g., the "approaching target" primitive requires available distance information or target detection results). When the platform capability verification fails, the task parsing module 104 will reject the task and trigger a rollback strategy, while simultaneously writing the reason for the failure to the log for auditing.

[0090] like Figure 2 As shown, the TaskJSON output by the visual-language reasoning module 103 includes at least:

[0091] primitive field: Action primitive type;

[0092] The `target` field contains target parameters (such as target direction, target distance, target type, desired depth, desired heading, etc.).

[0093] The constraints field contains constraint parameters (such as minimum safe distance, depth boundary, speed limit, etc.).

[0094] The terminate field: Termination conditions (such as distance reached, time reached, target detected, etc.);

[0095] The logic field indicates the logical relationship between actions (sequence / condition / loop).

[0096] In one embodiment, the action primitive library includes at least one or more of the following primitive types:

[0097] Navigation primitives: Straight / Constant speed navigation, Constant depth navigation, Directional turn, Heading maintenance, Waypoint tracking;

[0098] Interactive primitives: approach the target, observe around the target, take a picture / scan the target, maintain relative position to the target;

[0099] Safety-related primitives: emergency braking, hovering, levitation, low-speed reverse, stationary rotation search;

[0100] Composite primitives: Conditional sequences or loop sequences formed by combining multiple navigation / interaction / safety primitives according to their logical fields.

[0101] For each action primitive, its required inputs (e.g., target category, desired distance, desired depth, upper velocity limit, minimum safe distance), output reference quantities (reference pose / velocity / acceleration profile), and termination conditions (distance threshold, time threshold, detection event trigger, etc.) are defined.

[0102] In one embodiment, JSONSchema, in addition to restricting the existence and type of fields, also constrains the enumeration set and numerical range: for example, the `primitive` field is restricted to enumeration values ​​in the primitive library; parameters such as distance, depth, and heading angle in the `target` field are restricted to specific units and have upper and lower bounds set; the `constraints` field restricts... Speed ​​limit The attitude limit and other parameters must be non-negative and not exceed the maximum allowed by the platform; the terminate field requires at least one termination condition to occur, and the time-based termination condition must be greater than zero. If any field violates the above constraints, the task parsing module 104 will reject and roll back.

[0103] In one alternative implementation, constrained decoding is achieved through a "token mask": at each decoding step, the currently generated JSON path (e.g., ...) is used as the basis for the decoding process. The system dynamically determines the next set of optional tokens, blocking any tokens that could lead to invalid JSON structures or out-of-bounds field values. This ensures that the model output always falls within the valid set defined in the schema, improving the determinism and parsability of the structured output.

[0104] In the specific generation process of the visual-language reasoning module 103, to ensure that the output strictly conforms to the above JSON Schema, this embodiment adopts a dynamic token masking algorithm based on a trie. The specific execution steps of this algorithm include:

[0105] Step 1: Constructing a Constraint State Machine. Before inference begins, the system first parses a state machine or prefix tree based on a predefined schema (such as JSONSchema). Each node in this tree represents a parsing state of the current JSON structure (e.g., "parsing key names", "parsing numeric values", "parsing array terminator").

[0106] Step 2: Mask Calculation. Before the VLM model generates the t-th token, the decoder queries the position of the currently generated token sequence in the state machine. Based on this position, the decoder calculates the next valid token set. For example, when the current state is "expected input key name" and the schema specifies that the key name must be "primitive" or "target", the probability of all other words in the vocabulary (such as "apple", "run") will be forced to be negative infinity. ).

[0107] Step 3: Sampling and Update. The model performs probability sampling only within the valid set to select the next Token. Subsequently, the state machine transitions to the next state based on this Token, preparing for the next round of mask calculation.

[0108] This mechanism fundamentally eliminates the "illusion" problem common in large language models (such as generating non-existent field names or malformed brackets), ensuring the syntactic accuracy of the generated task description file. This eliminates the need for the downstream task parsing module 104 to handle complex exception parsing logic, greatly improving the robustness of the system.

[0109] In this embodiment of the invention, a constrained decoding mechanism is used to ensure that the output satisfies JSONSchema:

[0110] During the decoding process, the set of output tokens is masked according to the schema to ensure that the output always lies in the valid syntax space;

[0111] Perform strong validation on required fields, numerical ranges, and enumeration sets;

[0112] When a parsing failure, missing field, or type mismatch occurs, the task parsing module 104 rejects the task and triggers a fallback strategy (such as hovering or floating), while recording the reason for the error for auditing.

[0113] This mechanism makes the upper-level semantic output interpretable, parsable, and auditable, and provides deterministic input for downstream compilation and security oversight.

[0114] like Figure 3 As shown, the safety supervision module 106 includes: a constraint set generation unit 1061, a risk prediction unit 1062, an online projection correction unit 1063, and a coverage rollback unit 1064.

[0115] Constraint set generation unit 1061 constructs hard safety constraint sets:

[0116]

[0117] in Minimum distance to obstacles can be estimated by visual depth, sonar range, or a fusion of both with confidence. For depth, and These are the lower and upper limits of depth, respectively. These are roll and pitch angles, respectively. For velocity vectors, This represents the maximum speed.

[0118] The risk prediction unit 1062 predicts the future state within the prediction window using a simplified discrete model or the prediction model inside the controller; if it predicts that Ω will be violated at any future moment, it sends a correction command to the online projection correction unit 1063.

[0119] The online projection correction unit 1063 can perform projection correction on the trajectory or control quantity:

[0120] Track projection: Spatial projection / cropping / retiming of reference trajectory points, insertion of deceleration segments, hovering segments, or obstacle-avoidance transition segments to obtain the corrected trajectory. );

[0121] Control projection: Perform minimal modification projection on the underlying control output u, and solve:

[0122]

[0123] in Constraints on the feasible region of the thruster (thrust upper and lower limits, rate of change upper limit, dead zone compensation, etc.).

[0124] Coverage rollback unit 1064 monitors hard trigger conditions: when When the depth exceeds the limit, attitude exceeds the limit, speed exceeds the limit, thruster malfunctions, or thruster saturation exceeds the threshold, the upper reference trajectory is immediately overwritten and a rollback primitive is output. The rollback primitive includes at least emergency braking, surfacing, hovering, or low-speed rollback, and is executed according to the preset priority.

[0125] In this embodiment of the invention, when the perceived confidence level decreases (e.g., due to turbidity leading to increased visual error), the safety monitoring module 106 will automatically increase... or reduce This puts the system into a conservative mode, thereby reducing the risk of collision.

[0126] like Figure 4 As shown, the dynamic closed-loop control module 107 calculates the control variables based on the AUV six-degree-of-freedom hydrodynamic model; the hydrodynamic model can be expressed as:

[0127] Where M includes the rigid body mass and the additional mass, C is the Coriolis and centrifugal term, D is the damping term, g is the gravity and buoyancy term, and τ is the desired resultant force / resultant torque.

[0128] In this embodiment of the invention, the dynamic closed-loop control module 107 may employ:

[0129] (1) Specific implementation of nonlinear model predictive control (NMPC): In this embodiment, the controller calculates the control sequence within each control cycle k based on the current state estimate x(k) by solving an open-loop optimal control problem in a finite time domain. The optimized objective function (CostFunction) J is designed as a weighted sum of the state tracking error term and the control energy consumption term, and its mathematical expression is as follows:

[0130]

[0131]

[0132] in, For the prediction horizon, the value range in this embodiment is 10 to 50 steps; This represents the predicted state at time k+i based on the state at time k. The reference trajectory generated by the task compilation module 105; u is the control input vector (i.e., the desired resultant force / resultant moment); To control the increment and suppress severe fluctuations in the control input, thus protecting the thruster motor, matrices Q, R, and S are semi-definite weighted matrices, specifically designed for underwater confined environments in this invention.

[0133] 1) State weight matrix Q: Used to adjust the degree of emphasis on tracking accuracy for different degrees of freedom. In this embodiment, depth (z) and heading (z) are weighted. The weighting coefficients for ) are set to a large value (e.g. This ensures that the AUV will not experience vertical or directional collisions in confined spaces; while the weight of the planar position (x,y) can be appropriately reduced during the cruise phase to allow for some path deviation in exchange for smoothness.

[0134] 2) Input weight matrix R: used to penalize excessive thrust output, thereby optimizing energy consumption and extending the underwater endurance of the AUV.

[0135] 3) Incremental weight matrix S: Used to limit the rate of change of thrust. In underwater environments, excessively rapid thrust changes can not only cause cavitation effects in the thruster but also induce fuselage oscillations. By increasing the diagonal elements of the S matrix, a smooth control curve can be forced out.

[0136] Furthermore, the solution to the above optimization problem is subject to the following strict physical constraints:

[0137] Dynamic constraints: , where f( The equations are discretized six-degree-of-freedom hydrodynamic equations, which include nonlinear hydrodynamic damping terms and Coriolis force terms.

[0138] Actuator saturation constraints: This corresponds to the maximum resultant force and torque that the AUV thruster configuration can provide;

[0139] Safety state constraints: The constraint set is dynamically updated by the safety supervision module 106. For example, when an obstacle is detected ahead, the state constraint set is compressed to prevent the predicted trajectory from entering the obstacle expansion region. The solver can use Sequential Quadratic Programming (SQP) or the Interior Point Method for iterative solution. Considering the computing power limitations of the embedded platform, if the optimal solution is not converged within a preset time (e.g., 50ms), the next step of the optimal solution sequence from the previous moment is directly output as the suboptimal solution, or a safety backoff strategy is triggered to ensure the real-time performance of the system.

[0140] (2) PID control: used for depth, heading or attitude channel stabilization;

[0141] (3) Combined control: MPC is responsible for planar maneuvering, and PID is responsible for depth / attitude stabilization.

[0142] The thruster allocation module 108 maps the desired τ to the thruster thrust T:

[0143] Where B is the thruster layout matrix; when the thruster is saturated or faulty, the faulty thruster can be removed from the constraints and the solution can be resolved; at the same time, the thrust change rate is limited so that the thrust command meets the dynamic capability of the actuator.

[0144] like Figure 5 As shown, the system states include at least: normal tracking state, conservative degradation state, rollback coverage state, and recovery / replanning state.

[0145] When perceived confidence decreases or risk prediction increases, the system transitions from normal tracking to a conservative degradation, lowering the speed limit and increasing the safety distance threshold; when a hard trigger condition is met, it enters a rollback overlay state, outputting a rollback primitive and freezing upper-layer trajectory updates; when continuous... Once the time meets the safety boundary conditions, the system enters the recovery state. After recovery, the remaining tasks can be executed or the visual-language reasoning module 103 can be called again to generate a new task description and compile the trajectory.

[0146] In one embodiment, "freezing upper-level trajectory updates" in the rollback overwrite state includes: pausing the incremental updates of the current TaskJSON by the task compilation module 105, freezing the reference trajectory pointer and allowing only the safety supervision module 106 to output rollback primitives; and continuously monitoring whether the hard safety constraints are restored to being met.

[0147] In one embodiment, the recovery condition employs a hysteresis mechanism to avoid frequent jitter: when continuous Time is up Furthermore, if the risk prediction unit 1062 predicts no violations within the window, it allows transition from rollback coverage to recovery / replanning state; if the recovery phase triggers again... If the thruster remains saturated, immediately return to the retraction and coverage state.

[0148] In one embodiment, the recovery / replanning state can be selected as follows: (1) continue to execute the remaining primitive sequence (regenerate the remaining reference trajectory from the current primitive); (2) call the visual-language reasoning module 103 again to generate a new TaskJSON (e.g., when environmental changes cause the original path to be unreachable); (3) directly enter safe hover and wait for manual intervention when the preset number of replanning times exceeds the threshold.

[0149] Optional implementation: Log auditing and interpretable output

[0150] In this embodiment of the invention, the system can record: original natural language instructions, visual observation summaries, TaskJSON, reference trajectory, comparison of trajectories before and after online correction, triggered rollback primitives and their triggering reasons; the above logs are used for task review, fault location and security audit.

[0151] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hierarchical visual language action decision-making method for AUVs in underwater confined environments, characterized in that, Includes the following steps: Step S1: Receive user's natural language control commands, synchronously acquire underwater visual images and fuse multi-sensor data to obtain AUV current state estimation and environmental obstacle distance information; Step S2: Input the natural language control command and the visual image into the airborne vision-language model. After semantic understanding and constrained decoding, output a structured task description file. The structured task description file contains a sequence of action primitives arranged in time sequence and the target parameters and execution constraints corresponding to each action primitive. Step S3: Perform syntax compliance verification and parameter boundary verification on the structured task description file. After the verification is passed, compile the action primitive sequence into a reference trajectory containing timestamps, track points and velocity profiles according to the preset primitive library. Step S4: During the execution of the reference trajectory, the safety supervisor performs rolling time-domain violation prediction and online correction of the reference trajectory or real-time control quantity based on a pre-built set of hard safety constraints at a period no less than the underlying control frequency; when any state quantity is predicted or detected to break through the boundary of the hard safety constraint set, the safety supervisor immediately interrupts the upper-layer trajectory input and forcibly outputs a safety rollback command to replace the reference trajectory. Step S5: The trajectory command corrected by the safety monitor or the safety reversal command is sent to the underlying dynamic closed-loop controller. The desired control force and torque are calculated based on the AUV's six-degree-of-freedom hydrodynamic model. After the thruster control allocation and saturation limiting processing, the AUV is driven to perform the corresponding action.

2. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 1, characterized in that, The set of hard safety constraints in step S4 includes at least: minimum safe distance constraint, depth boundary constraint, attitude angle boundary constraint and upper limit constraint of sailing speed; when the safety supervisor perceives the confidence level as lower than a preset threshold, it automatically increases the threshold of the minimum safe distance constraint and simultaneously decreases the threshold of the upper limit constraint of sailing speed, so that the system switches to a conservative operation mode.

3. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 1, characterized in that, The constrained decoding described in step S2 maintains a prefix tree based on structured data constraint rules. During the word-by-word decoding process of the visual-language model, the position of the currently generated sequence in the prefix tree is queried, and a mask set for the next legal token is dynamically generated. The probability value of the illegal token is set to negative infinity, forcing the model output path to always fall within the legal space defined by the structured data constraint rules.

4. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 1, characterized in that, The structured task description file includes at least a primitive type field, a target parameter field, and a constraint field; the parameter boundary verification in step S3 includes at least: verifying whether the primitive type field belongs to the preset primitive library enumeration range, verifying whether the target parameter field is within the numerical range allowed by the platform, and verifying whether the constraint field is not lower than the platform's minimum safety margin. If any verification fails, the current task is rejected and a hover or floating back is triggered.

5. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 1, characterized in that, The violation prediction in the rolling time domain in step S4 includes: starting from the current state estimate, recursively predicting the future state sequence based on the AUV discrete motion model within a preset prediction window; if the predicted value at any moment in the future state sequence falls into the violation region of the hard safety constraint set, the online correction is triggered; the online correction includes spatial pruning, velocity retiming, or local obstacle bypass reconstruction of the reference trajectory.

6. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 1, characterized in that, In step S4, the safety monitor performs a correction based on safety filtering on the real-time control quantity: with the original control quantity as the center, it solves for the nearest Euclidean projection point in the convex constraint space formed by the feasible region of the thruster and the safety invariant set, and replaces the original control quantity output with the corrected control quantity corresponding to the nearest point; when the thruster's continuous saturation time exceeds a preset threshold, the safety monitor directly triggers the safety rollback command.

7. A hierarchical visual-language action decision-making system for AUVs in underwater confined environments, used to execute the method described in any one of claims 1-6, characterized in that, include: The instruction receiving module (101) is configured to receive user natural language control instructions; The perception data collection module (102) is configured to collect underwater visual images and fuse data from the inertial measurement unit, depth gauge and Doppler velocimeter, and output AUV state estimation and environmental perception information; The visual-language reasoning module (103) is configured to run an onboard visual-language model to map the natural language control commands and the visual images into a structured task description file; The task parsing module (104) is configured to perform syntax and boundary checks on the structured task description file; The task compilation module (105) is configured to compile the validated sequence of action primitives into a timestamped reference trajectory; The safety supervision module (106) is configured to perform rolling verification and projection correction on the reference trajectory or real-time control quantity based on the hard safety constraint set, and to force the reference trajectory to be covered by a safety rollback command when a violation occurs; The dynamic closed-loop control module (107) is configured to calculate the desired control force and torque based on the AUV hydrodynamic model; The thruster allocation module (108) is configured to map the desired control force and torque into thrust commands for each thruster through constraint optimization.

8. The hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 7, characterized in that, The safety monitoring module (106) includes: The constraint set generation unit (1061) is used to dynamically maintain and update the hard safety constraint set, which includes the lower limit of obstacle distance, the upper and lower limits of depth, the upper limit of attitude angle, and the upper limit of velocity. Risk prediction unit (1062) is used to recursively predict future state sequences in the prediction time domain and determine whether the hard safety constraint set boundary is touched. The online projection correction unit (1063) is used to perform minimum norm projection on the non-compliant trajectory points or control quantities, pulling them back into the safety domain defined by the hard safety constraint set; The overlay rollback unit (1064) has a higher output priority than the task compilation module (105) and is used to directly cut off the input channel of the reference trajectory when the hard safety condition is triggered, and output the safety rollback instruction to the dynamic closed-loop control module (107).

9. A hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 7, characterized in that, The reasoning output cycle of the visual-language reasoning module (103) is 0.5Hz to 5Hz, the control operation cycle of the dynamic closed-loop control module (107) and the thruster allocation module (108) is 20Hz to 200Hz, and the monitoring and intervention cycle of the safety supervision module (106) is not less than the control operation cycle, so as to achieve real-time safety coverage of upper-level decisions.

10. A hierarchical visual language action decision-making method for AUVs in underwater confined environments according to claim 7, characterized in that, The safety rollback command includes at least one of emergency braking, hovering, controlled levitation, and low-speed rollback. The rollback coverage unit (1064) selects output according to the priority order of emergency braking over hovering, hovering over controlled levitation, and controlled levitation over low-speed rollback. The system state switches between at least normal tracking mode, conservative degradation mode, rollback coverage mode, and task recovery mode. When no violation of the hard safety constraint set is triggered within a continuous preset time, the system switches from the rollback coverage mode to the task recovery mode to resume the execution of the remaining action primitives or trigger replanning.

Images