An end-side vision closed-loop control method and system for a resource-constrained underwater vehicle
By employing a phased control and state management strategy, the problems of perception delay and stability in the visual closed-loop control of underwater vehicles are solved, thereby improving the stability and robustness of the control process. This approach is suitable for underwater vehicles with limited resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-14
AI Technical Summary
Resource-constrained underwater vehicles suffer from problems such as large perception delay, severe control coupling, and insufficient stability in end-side vision closed-loop control, which are difficult to effectively solve using existing methods.
A phased control mechanism is adopted, which only performs yaw control when the target is not aligned, and activates propulsion and heave control after the target is aligned. The controller state is reset when the control channel is not enabled or the target is lost. Combined with a lightweight target detection network and image enhancement module, coupled motion coupling and transient shocks are reduced.
It improves the control stability and robustness of underwater vehicles, reduces steady-state tracking error and overshoot, increases the target locking success rate, and meets the real-time requirements under resource-constrained conditions.
Smart Images
Figure CN122387084A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an end-side vision closed-loop control method and system for resource-constrained underwater vehicles, belonging to the field of autonomous control technology for underwater unmanned vehicles. Background Technology
[0002] With the increasing application of underwater unmanned vehicles (UAVs) in tasks such as ocean observation, structural inspection, and close-range operations, vision-based target following and closed-loop control methods have gradually become a research hotspot. Compared with sensing methods such as sonar, visual sensors have advantages such as low cost and rich information, thus showing broad application prospects in miniaturized, low-cost underwater vehicles. However, in practical underwater applications, underwater vehicles are usually limited by size and power consumption, and the computing power of their embedded computing platforms is limited, making it difficult to support the real-time operation of highly complex visual algorithms. When performing perception tasks such as target detection at the edge, there is a common problem of large inference delay and fluctuation. This type of delay is equivalent to feedback lag in the closed-loop control system, which leads to a reduction in the system's phase margin, thereby causing overshoot, oscillation, or even instability in the control process. Existing vision-based closed-loop control methods usually adopt a continuous full-channel control strategy, that is, simultaneously adjusting multiple degrees of freedom of the vehicle throughout the entire control process. This approach is prone to introducing coupled motion before the target is aligned, such as prematurely executing propulsion or heave control when the horizontal deviation is large, resulting in trajectory divergence or decreased control efficiency. Furthermore, in cases of short-term target loss or unstable detection, the controller integral term in traditional methods continues to accumulate, which can easily generate large transient shocks when the target reappears.
[0003] To address the aforementioned issues, existing methods primarily compensate for perception delays through filtering or prediction. However, these methods typically rely on relatively accurate motion models and struggle to handle the instability of inference time in deep learning models. Furthermore, existing visual closed-loop control methods often employ a "continuous full-channel control" strategy in their control structure, simultaneously adjusting multiple degrees of freedom such as yaw, propulsion, and heave even when the target is not aligned, leading to coupling motion and trajectory divergence. Simultaneously, when the target is briefly lost or detection is unstable, the integral term of the traditional PID controller continues to accumulate, easily generating transient shocks when the target reappears, and in severe cases, even causing system oscillations. These problems are particularly pronounced on resource-constrained end-side platforms, and existing methods lack targeted solutions at the control structure level. Therefore, it is necessary to design a visual closed-loop control method suitable for resource-constrained conditions to improve the system's stability and robustness at the control structure level. Summary of the Invention
[0004] To address the problems of large perception delay, severe control coupling, and insufficient stability in end-side visual closed-loop control of resource-constrained underwater vehicles, this invention provides an end-side visual closed-loop control method and system for resource-constrained underwater vehicles. The technical solution is as follows: In a first aspect, the present invention provides an end-side visual closed-loop control method for resource-constrained underwater vehicles, comprising the following steps: Step 1: Acquire real-time images and output target bounding boxes through a lightweight object detection network, prioritizing the selection of control targets based on the largest area; Step 2: Calculate the position error of the target in the image coordinate system with the image center as the reference point, and normalize the error; Step 3: Determine whether the preset alignment conditions are met based on the normalized error. If the error is less than the threshold, it is determined to be in a locked state; otherwise, it is determined to be in an unlocked state. Step 4: Implement a phased control mechanism based on the locking criterion: In the unlocked state, only the yaw channel of the underwater vehicle is controlled; in the locked state, the propulsion channel and the heave channel are controlled simultaneously. Step 5: Execute the controller state management strategy for control state switching: when the control channel is not enabled, clear the integral term of the controller corresponding to the channel; when the target is lost or no target is detected for a period of time, reset the controller state of all control channels. Step 6: Combine the control quantities of each channel into a control vector and output it to the actuator of the underwater vehicle to complete the visual servo closed-loop control.
[0005] Optionally, the phased control mechanism in step 4 includes: In the unlocked state, only the yaw control channel is enabled, and the yaw control law adopts discrete position PID control. Let the current control period be... k :
[0006] Set the control values for the propulsion channel and heave channel to zero:
[0007] In the locked state, all three control channels are enabled simultaneously, and the control laws for each channel are as follows:
[0008] in , , These are the proportional, integral, and differential coefficients of the yaw channel, respectively. , , These are the proportional, integral, and differential coefficients of the heave channel, respectively. , , These are the proportional, integral, and differential coefficients of the propulsion channel, respectively. , and These represent horizontal error, vertical error, and area error, respectively.
[0009] Optionally, the lightweight object detection network replaces the Bottleneck module in the C2f module of the baseline network with a FasterBlock module built based on partial convolutions.
[0010] Optionally, a differentiable physics-driven image enhancement module can be embedded in the front end of the lightweight target detection network to suppress image degradation based on the Jaffe-McGlamey underwater imaging model.
[0011] Optionally, the partial convolution is performed only on a portion of the input feature map, while the remaining channels are passed through identically.
[0012] Optionally, the FasterBlock module has the following structure in sequence: partial convolution, batch normalization, activation function, 1×1 convolution, and batch normalization.
[0013] Optionally, the parameters of the yaw channel are set to... , , .
[0014] Secondly, the present invention provides an end-side vision closed-loop control system for resource-constrained underwater vehicles, comprising: The target acquisition module is configured to acquire real-time images and output target bounding boxes through a lightweight target detection network, and select control targets based on the largest area. The error calculation module is configured to calculate the position error of the target in the image coordinate system with the image center as the reference point, and to normalize the error. The status judgment module is configured to determine whether the preset alignment conditions are met based on the normalized error. When the error is less than the threshold, it is judged to be in a locked state; otherwise, it is judged to be in an unlocked state. The phased control module is configured to control only the yaw channel of the underwater vehicle in the unlocked state; and to control both the propulsion channel and the heave channel in the locked state. The status management module is configured to clear the integral term of the controller corresponding to the control channel when the control channel is not enabled; and to reset the controller status of all control channels when the target is lost or no target is detected for a continuous period of time. The execution module is configured to combine the control quantities of each channel into a control vector and output it to the underwater vehicle's actuator to complete visual servo closed-loop control.
[0015] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the end-side visual closed-loop control method for resource-constrained underwater vehicles as described in any of the preceding claims.
[0016] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the end-side visual closed-loop control method for resource-constrained underwater vehicles as described in any of the preceding claims.
[0017] The beneficial effects of this invention are: This invention constructs a phased enabling mechanism for the control channel based on a target locking criterion. During the target misalignment phase, only yaw control is executed; propulsion and heave control are enabled only after alignment conditions are met. This reduces coupled motion during the misalignment phase and improves the stability of the control process. Furthermore, a controller state reset strategy resets the controller integral term when the control channel is disabled or the target is lost, avoiding transient shocks caused by integral accumulation and improving the system's robustness in target loss and re-detection scenarios. Simultaneously, this method is implemented on an embedded end-side platform, eliminating the need for high-performance computing resources and meeting the dual requirements of real-time performance and stability for resource-constrained underwater vehicles, thus possessing significant engineering application value.
[0018] Experiments with controlled variables show that, based on the same YOLOv8s detection network as the benchmark scheme, this invention can reduce steady-state tracking error by about 28%, overshoot by about 25%, and lock success rate by about 30% by introducing a phased control and state management strategy (i.e., the core control method of this invention), thus verifying the effectiveness of the core control structure established by this invention.
[0019] In one embodiment of the present invention, based on the introduction of a phased control and state management strategy, and further by introducing an improved lightweight network and image enhancement module, the method of the present invention reduces the steady-state tracking error (RMSE) from 0.2241 to 0.1351 (a reduction of 39.7%), the overshoot from 0.4777 to 0.2992 (a reduction of 37.4%), and the target lock success rate from 67.99% to 95.22% (an increase of 40.0%), under the condition of an edge-side inference frame rate of 75.6 FPS. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is an overall flowchart of the end-side vision closed-loop control method of the present invention.
[0022] Figure 2 This is a schematic diagram of the visual servo closed-loop system structure in this invention.
[0023] Figure 3 This is a schematic diagram of the control enable state machine in this invention.
[0024] Figure 4 This is a schematic diagram of the actual underwater vehicle and its experimental platform.
[0025] Figure 5 This is a schematic diagram of the error curve for a fixed-point hovering experiment.
[0026] Figure 6 This is a schematic diagram of the step response error curve.
[0027] Figure 7 This diagram illustrates the dynamic tracking of experimental system state changes and the success rate of locking. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0029] Example 1: This embodiment uses a small underwater vehicle with limited resources as a carrier to illustrate the end-side vision closed-loop control method and system proposed in this invention. The vehicle is equipped with an embedded computing platform (NVIDIA Jetson Xavier NX in this embodiment, with a power consumption limit of 15W), a front-facing vision sensor, and a propulsion actuator including a yaw channel, a propulsion channel, and a heave channel. All perception and control calculations are completed on the end side.
[0030] like Figure 1 As shown, the overall process of this embodiment includes image acquisition, target detection, error calculation, state determination, phased control and control output, etc., and each step constitutes a complete visual servo closed loop. Figure 2 The corresponding system software module structure is given, including an image acquisition module, a target detection module, an error calculation module, a state determination module, a control calculation module, and a control output module. Each module is executed sequentially on an embedded computing platform.
[0031] Step 1: Image acquisition and target detection.
[0032] Real-time image data is acquired by a vision sensor installed at the front of the aircraft. The image resolution is adjusted to a preset size (640×640 pixels in this embodiment) and then input into an embedded computing platform. Target detection adopts a lightweight convolutional neural network model and outputs a set of bounding boxes of candidate targets.
[0033] For the k There are 1 candidate targets, whose bounding boxes are represented as follows:
[0034] in The coordinates of the target center are and These represent the target width and height, respectively.
[0035] To ensure the temporal consistency of the controlled objects, a "maximum area priority" strategy is adopted to select the controlled object from the candidate objects: The center coordinates of the selected target are: The target area is denoted as .
[0036] Step 2: Error calculation and normalization.
[0037] Taking the image center as the reference point, let the image width be... Height is Then the coordinates of the image center are .
[0038] Define normalized horizontal error and vertical error:
[0039] Simultaneously, area error is defined for distance control:
[0040] in The desired target area is defined by this parameter, which corresponds to the aircraft's desired following distance and can be preset according to specific mission requirements.
[0041] Step 3: Locking criteria and state machine determination.
[0042] like Figure 3 As shown, this invention constructs a control enable state machine based on the target alignment degree. The system is in an unlocked state by default when it starts up, and a horizontal error threshold is set. (In this embodiment, we take) (This threshold can be adjusted according to actual task requirements; this implementation is only an example.) The locking criterion is defined as follows: When satisfied If a valid target is present, the system is locked; otherwise, the system is unlocked.
[0043] The state transition rules are as follows: When the target is detected and When the time comes, the device switches from the unlocked state to the locked state; When the target is lost or When the lock state is reached, the lock state is switched to the unlock state.
[0044] Step 4: Implement phased control strategies.
[0045] This phased control strategy differs from the simple sequential control in existing technologies. Its core lies in using the visual perception result of "target alignment" as the closed-loop criterion for state switching, rather than relying on preset time or open-loop commands. This achieves a precise match between the control degrees of freedom and the degree of target alignment. The phased control strategy is executed based on the state machine output. Its core principle is that during the target misalignment phase, only yaw control is executed to avoid multi-degree-of-freedom coupled motion; after the target is aligned, all control channels are activated.
[0046] (1) In the unlocked state, only the yaw control channel is enabled, while the propulsion and heave channels are closed. The yaw control law adopts discrete position PID control, and the current control period is set to k:
[0047] Set the control values for the propulsion channel and heave channel to zero:
[0048] (2) In the locked state, all three control channels are enabled simultaneously, and the control laws for each channel are as follows:
[0049] in , , These are the proportional, integral, and differential coefficients for each channel, which can be individually tuned according to the vehicle's dynamic characteristics. In this embodiment, the yaw channel parameters are taken as follows: , , The parameters of the heave and propulsion channels are adjusted accordingly based on the hydrodynamic model.
[0050] Step 5: Controller status management.
[0051] The controller state management strategy in this step differs from the integral anti-windup mechanism in conventional PID control. Integral anti-windup typically only prevents the integral term from accumulating excessively when the output is limited. However, the state management strategy of this invention targets a higher-level system state change—such as the control channel being disabled or the target being lost. It actively intervenes in the controller state at the moment of state transition, thereby fundamentally eliminating the risk of control quantity jumps caused by state transitions. To avoid adverse effects from the controller during periods when it is not involved in control, this invention introduces a state management mechanism: When a control channel is disabled (e.g., the propulsion and heave channels in an unlocked state), the integral and derivative terms of the corresponding PID are immediately cleared to zero to prevent error accumulation from causing abrupt changes in the control input.
[0052] When the system detects that the target is lost, or for a continuous period of time (In this embodiment, we take) If no valid target is detected within (seconds), the PID states of all control channels are globally reset (both integral and derivative terms are cleared to zero), and a zero control command is output as a safety protection strategy.
[0053] Step 6: Control command generation and output.
[0054] Combine the control values from each channel into a control vector:
[0055] The visual servo closed-loop control is achieved by sending signals to the propulsion actuator of the aircraft via a communication interface (MAVLink protocol is used in this embodiment).
[0056] To further reduce edge-side inference latency, in this embodiment, the object detection network adopts a lightweight structure based on partial convolution. Partial convolution only performs convolution operations on a subset of channels of the input feature map, while the remaining channels are passed through identically. Its calculation form is as follows:
[0057] in Input the number of channels. The channel division ratio (taken in this embodiment) This design reduces computational load while maintaining a regular memory access pattern, which is beneficial for achieving low-latency inference on embedded GPU platforms.
[0058] Based on the aforementioned operators, a FasterBlock module is constructed, whose structure consists of partial convolution, batch normalization, activation function, 1×1 convolution, and batch normalization. The Bottleneck within the C2f module of the baseline network (such as YOLOv8s) is replaced with FasterBlock to obtain the C2f_Faster module, thereby reducing edge-side inference latency. This replacement is performed end-to-end along the backbone network and the detector head path, achieving end-to-end complexity and latency benefits.
[0059] To further improve detection stability in degraded underwater environments, this embodiment embeds a differentiable physical enhancement (DPE) module at the front end of the target detection network to suppress underwater image degradation. This module is based on the Jaffe-McGlamey underwater imaging model.
[0060] in To observe the image, To restore the image, The attenuation coefficient is... For depth, As background light. and Set as a learnable parameter, depth Estimation via a single depthwise convolution and the Sigmoid function:
[0061] The recovery term is obtained by inversion from the imaging model:
[0062] The module output uses a residual connection:
[0063] This module consists entirely of differentiable operators, and the gradient can be backpropagated to the physical parameters through the detection loss. and This enables data-driven adaptive enhancement. Among other things... express Convolution operations are used to achieve cross-channel fusion of original features and physically recovered features, with a computational cost of 1% less additional GFLOPs.
[0064] Example 2: This embodiment provides an end-side vision closed-loop control system for resource-constrained underwater vehicles, including: The target acquisition module is configured to acquire real-time images and output target bounding boxes through a lightweight target detection network, and select control targets based on the largest area. The error calculation module is configured to calculate the position error of the target in the image coordinate system with the image center as the reference point, and to normalize the error. The status judgment module is configured to determine whether the preset alignment conditions are met based on the normalized error. When the error is less than the threshold, it is judged to be in a locked state; otherwise, it is judged to be in an unlocked state. The phased control module is configured to control only the yaw channel of the underwater vehicle in the unlocked state; and to control both the propulsion channel and the heave channel in the locked state. The status management module is configured to clear the integral term of the controller corresponding to the control channel when the control channel is not enabled; and to reset the controller status of all control channels when the target is lost or no target is detected for a continuous period of time. The execution module is configured to combine the control quantities of each channel into a control vector and output it to the underwater vehicle's actuator to complete visual servo closed-loop control.
[0065] To further verify the technical effectiveness of the method and system of the present invention, experimental verification was conducted and the results are explained: Figure 4 This is the physical platform of the underwater vehicle used in this embodiment.
[0066] Figure 5 The curve of yaw error changing with time in a fixed-point hovering experiment is presented. In this experiment, the target is placed directly in front of the camera and kept stationary, and the error time series is recorded. The root mean square error (RMSE) of the baseline scheme (YOLOv8s) is 0.2241. After adopting the method of this invention, the error fluctuation is significantly reduced, and the root mean square error (RMSE) is 0.1351, which is 39.7% lower than the baseline scheme.
[0067] Figure 6 The error variation curve in the step response experiment is presented. In this experiment, the target moves rapidly and then stops suddenly. The overshoot characteristics of the system are examined. The overshoot of the method in this invention is 0.2992, which is 37.4% lower than the baseline scheme (0.4777). The analysis shows that the baseline scheme has occasional missed detections when the target changes abruptly, which leads to control state switching and command jumps. The method in this invention achieves a smooth deceleration process due to the improved detection stability and state management mechanism.
[0068] Figure 7 The curves showing the change of system state over time in a dynamic tracking experiment are presented. The target in this experiment moves irregularly, and the success rate of tracking is taken as the percentage of frames in which the state machine is in a locked state. The success rate of the method in this invention is 95.22%, which is 40.0% higher than the benchmark scheme (67.99%).
[0069] The above experimental results verify the improvement effect of the method described in this invention on the stability and robustness of visual closed-loop control under resource-constrained conditions. To further verify the independent contribution of the core control method (phased control + state management) of this invention, this embodiment also sets up a control variable experiment. Under the condition of maintaining the same YOLOv8s detection network as the benchmark scheme, only the phased control and state management strategy described in this invention is superimposed. The experimental results show that the steady-state tracking error (RMSE) under this setting is 0.1612, the overshoot is 0.3581, and the lock success rate is 82.5%, which are significantly improved compared with the benchmark scheme (0.2241, 0.4777, 67.99%). This result shows that the improvement of the control method described in this invention itself has a substantial contribution to the system performance, and the further optimization of the network structure has a superimposed enhancement effect.
[0070] It should be noted that the specific parameters in the above embodiments (such as image resolution, threshold) Timeout period The settings (such as PID coefficients, etc.) are preferred settings for this implementation platform and mission scenario. In actual applications, they can be adjusted according to the dynamic characteristics of the vehicle, sensor performance, and operating environment. These adjustments do not depart from the protection scope of this invention.
[0071] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A closed-loop control method for end-side vision of resource-constrained underwater vehicles, characterized in that, Includes the following steps: Step 1: Acquire real-time images and output target bounding boxes through a lightweight object detection network, prioritizing the selection of control targets based on the largest area; Step 2: Calculate the position error of the target in the image coordinate system with the image center as the reference point, and normalize the error; Step 3: Determine whether the preset alignment conditions are met based on the normalized error. If the error is less than the threshold, it is determined to be in a locked state; otherwise, it is determined to be in an unlocked state. Step 4: Implement a phased control mechanism based on the locking criterion: In the unlocked state, only the yaw channel of the underwater vehicle is controlled; in the locked state, the propulsion channel and the heave channel are controlled simultaneously. Step 5: Execute the controller state management strategy for control state switching: when the control channel is not enabled, clear the integral term of the controller corresponding to the channel; When a target is lost or no target is detected for a continuous period of time, the controller status of all control channels is reset; Step 6: Combine the control quantities of each channel into a control vector and output it to the actuator of the underwater vehicle to complete the visual servo closed-loop control.
2. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 1, characterized in that, The phased control mechanism in step 4 includes: In the unlocked state, only the yaw control channel is enabled, and the yaw control law adopts discrete position PID control. Let the current control period be... k : Set the control values for the propulsion channel and heave channel to zero: In the locked state, all three control channels are enabled simultaneously, and the control laws for each channel are as follows: in , , These are the proportional, integral, and differential coefficients of the yaw channel, respectively. , , These are the proportional, integral, and differential coefficients of the heave channel, respectively. , , These are the proportional, integral, and differential coefficients of the propulsion channel, respectively. , and These represent horizontal error, vertical error, and area error, respectively.
3. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 1, characterized in that, The lightweight target detection network replaces the Bottleneck module in the C2f module of the baseline network with the FasterBlock module, which is built based on partial convolutions.
4. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 1, characterized in that, A differentiable physics-driven image enhancement module is embedded in the front end of the lightweight target detection network to suppress image degradation based on the Jaffe-McGlamey underwater imaging model.
5. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 3, characterized in that, The partial convolution only performs convolution operations on a portion of the input feature map, while the remaining channels are passed through identically.
6. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 3, characterized in that, The FasterBlock module consists of a partial convolution, batch normalization, activation function, 1×1 convolution, and batch normalization.
7. The end-side visual closed-loop control method for resource-constrained underwater vehicles according to claim 2, characterized in that, The parameters of the yaw channel are set as follows: , , .
8. An end-side vision closed-loop control system for resource-constrained underwater vehicles, characterized in that, The system includes: The target acquisition module is configured to acquire real-time images and output target bounding boxes through a lightweight target detection network, prioritizing the selection of control targets based on the largest area. The error calculation module is configured to calculate the position error of the target in the image coordinate system with the image center as the reference point, and to normalize the error. The status judgment module is configured to determine whether the preset alignment conditions are met based on the normalized error. When the error is less than the threshold, it is judged to be in a locked state; otherwise, it is judged to be in an unlocked state. The phased control module is configured to control only the yaw channel of the underwater vehicle in the unlocked state; and to control both the propulsion channel and the heave channel in the locked state. The status management module is configured to clear the integral term of the controller corresponding to the control channel when the control channel is not enabled; and to reset the controller status of all control channels when the target is lost or no target is detected for a continuous period of time. The execution module is configured to combine the control quantities of each channel into a control vector and output it to the underwater vehicle's actuator to complete visual servo closed-loop control.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the end-side visual closed-loop control method for resource-constrained underwater vehicles as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the end-side visual closed-loop control method for resource-constrained underwater vehicles as described in any one of claims 1 to 7.