Underwater robot wall-attached control method and system based on multi-vision servo

Abstract

This invention provides a method, apparatus, medium, and computer equipment for underwater robot wall-hugging control based on multi-view vision servoing, applicable to underwater robots comprising multiple cameras, inertial measurement units, and multiple thrusters. The method acquires wall images simultaneously captured by multiple cameras, estimates the wall normal vector and the distance between the robot and the wall, and filters and fuses these with inertial measurement data to obtain relative pose information. Based on the wall normal vector, target wall-hugging distance, and relative pose information, it generates normal wall-hugging control quantities and attitude correction control quantities acting in the opposite direction of the wall normal vector. Then, based on the thruster configuration relationship, it distributes thrust to obtain control commands for each thruster, enabling the robot to automatically adjust its wall-hugging thrust and attitude under visual feedback, stably maintaining a contact state with a rigid or flexible wall, reducing reliance on manual operation.

Description

Technical Field

[0001] This invention relates to the field of underwater robot control technology, and more specifically, to a method and system for controlling underwater robots to adhere to walls based on multi-view vision servoing. Background Technology

[0002] Underwater robots are commonly used for cleaning, inspection, and maintenance of underwater surfaces such as ship hulls, marine engineering structures, and aquaculture cage netting. When performing these tasks, underwater robots typically need to maintain a stable close or attached position to the target surface to ensure that cleaning tools, detection sensors, or operating mechanisms can continuously work on the surface at a predetermined distance and attitude.

[0003] Existing methods for underwater robots to adhere to surfaces include magnetic adsorption, negative pressure adsorption, and thruster-assisted attachment. Magnetic adsorption typically uses magnetic wheels, magnetic tracks, or magnetic adsorption mechanisms to attach the robot to ferromagnetic surfaces such as steel hulls. While it provides strong adhesion, its applicability is limited by the surface material, making it unsuitable for non-ferromagnetic or flexible surfaces such as aluminum alloy hulls, fiberglass hulls, composite material surfaces, and aquaculture netting. Furthermore, the mechanical contact between the magnetic adsorption mechanism and the surface can easily cause wear and tear on the antifouling coating of the hull during movement or turning.

[0004] Negative pressure adsorption typically uses a vacuum pump or negative pressure chamber to create a pressure difference between the robot and the wall surface, allowing the robot to adhere to the wall. This method requires a high degree of flatness and sealing of the wall surface. When the wall surface has abrupt changes in curvature, welds, protrusions, dirt deposits, or a porous mesh structure, it is difficult for the negative pressure chamber to form a stable seal, easily leading to problems such as insufficient adsorption force, leakage, or desorption. In addition, underwater negative pressure adsorption mechanisms have complex structures and high maintenance costs.

[0005] Thruster-driven attachment is independent of wall material and sealing conditions. It typically uses thrusters to generate a force pointing towards the wall, allowing the robot to approach or adhere to the target surface. Therefore, it is adaptable to non-ferromagnetic walls, curved walls, and flexible mesh surfaces. However, existing thruster-driven attachment methods mostly rely on manual remote operation or simple semi-closed-loop control. Operators usually need to rely on experience to judge the relative angle, distance, and attachment status between the robot and the wall based on monitoring images from multiple cameras, and manually adjust the speed or thrust of each thruster. This method suffers from problems such as response lag, low control precision, and high operational burden. Especially when the hull is curved, the mesh fluctuates with water flow, or the robot is subjected to the reaction force of the cleaning jet or water flow disturbance, it is prone to causing the robot to locally lift, slip, collide with the wall, or have unstable adhesion distance.

[0006] Furthermore, while existing underwater robots are typically equipped with cameras, these cameras are mostly used as monitoring tools for remote operation or as image inspection tools after operations. The image information they acquire is not fully involved in the estimation of the robot's contact with the wall and closed-loop control. In other words, there is a lack of effective coupling between visual perception and thruster control in existing solutions, making it impossible to automatically adjust the thruster output based on real-time changes in the wall's normal, distance, and robot attitude. Therefore, existing underwater robots still struggle to achieve stable, automatic, and reliable contact control in complex, dynamic, and multi-type wall environments. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an underwater robot wall-hugging control method and system based on multi-view vision servoing, so as to overcome the shortcomings of the existing technology.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution: Firstly, a multi-view vision servo-based underwater robot wall-hugging control method is applied to an underwater robot, the underwater robot comprising multiple cameras, an inertial measurement unit, and multiple thrusters, the method comprising: The system acquires wall images simultaneously captured by multiple cameras and estimates the wall's geometric parameters in the robot's coordinate system based on these images. The wall's geometric parameters include the wall's normal vector and the distance between the robot and the wall. The wall's normal vector is a unit vector pointing from the wall towards the underwater robot. The wall geometry parameters are filtered and fused with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall, the robot's attitude state relative to the wall, and the robot's motion speed along the normal direction of the wall. Based on the wall normal vector, the target wall-attaching distance, and the relative pose information, a wall-attaching control quantity is generated to enable the underwater robot to attach to the wall. The wall-attaching control quantity includes a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. Based on the wall-hugging control amount and the pre-calibrated thruster configuration relationship, the thruster thrust is distributed to obtain thruster control commands for each thruster; the thruster control commands are used to control the operation of each thruster so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

[0009] In one embodiment, estimating the wall geometry parameters in the robot coordinate system based on the wall image includes: The wall image is subjected to underwater image enhancement processing to obtain an enhanced wall image; The wall visual features are extracted from the enhanced wall image, and the wall visual features are converted into wall geometric observations in the robot coordinate system based on the camera intrinsic and extrinsic parameters between multiple cameras. The wall normal vector and the distance between the robot and the wall are determined based on the wall geometry observations.

[0010] In one embodiment, when the wall is a rigid wall, the step of extracting wall visual features from the enhanced wall image and converting the wall visual features into wall geometric observations in the robot coordinate system based on camera intrinsic and extrinsic parameters among multiple cameras includes: Feature points are extracted from the enhanced wall images captured by at least two cameras to obtain multiple image feature points; Feature matching is performed on image feature points corresponding to different cameras, and the matching results are filtered based on epipolar constraints to obtain matching feature points; Multi-view triangulation is performed based on the matched feature points and the camera intrinsic and extrinsic parameters between multiple cameras to obtain a local 3D point cloud of the wall in the robot coordinate system. Local plane fitting is performed on the local 3D point cloud of the wall to obtain the wall normal vector and the distance between the robot and the wall.

[0011] In one embodiment, performing local planar fitting on the local 3D point cloud of the wall to obtain the wall normal vector and the distance between the robot and the wall includes: Based on the local three-dimensional point cloud of the wall, a plane equation is fitted: ; Determine the wall normal vector based on the plane equation: ; The sign of the wall normal vector is checked according to the direction of the robot's wall-attaching axis so that the wall normal vector points to the side of the underwater robot; The distance between the robot and the wall is determined based on the plane equation: ; in, Let be the wall normal vector. The distance between the robot and the wall. , , , These are the parameters of the plane equation.

[0012] In one embodiment, when the wall surface is a flexible mesh wall surface, the step of extracting wall visual features from the enhanced wall image and converting the wall visual features into wall geometric observations in the robot coordinate system based on camera intrinsic and extrinsic parameters among multiple cameras includes: Edge detection and line segment extraction are performed on the enhanced wall image to obtain a set of grid line segments; The set of grid segments is filtered based on segment length, segment angle, and segment parallelism to obtain valid grid segments; The effective grid line segments are divided into two groups of grid line clusters; the vanishing points are calculated according to the two groups of grid line clusters respectively. Based on the vanishing point and the camera intrinsic parameters, the two wall tangential direction vectors in the camera coordinate system are obtained by back projection. The wall normal vector in the camera coordinate system is determined by the cross product of the two wall tangential direction vectors. The wall normal vector in the camera coordinate system is then converted into the wall normal vector in the robot coordinate system based on the camera extrinsic parameters. The distance between the robot and the wall is determined based on the physical size of the mesh, the pixel size of the mesh cell in the image, and the camera focal length, or the distance between the robot and the wall is determined based on the distance measurement value collected by the ultrasonic ranging sensor.

[0013] In one embodiment, obtaining the two wall tangential direction vectors in the camera coordinate system based on the vanishing point and camera intrinsic parameters through back projection includes: Based on the camera intrinsic parameter matrix For the two vanishing points respectively and By performing back projection, we obtain two tangential direction vectors of the walls: ; ; Determine the wall normal vector in the camera coordinate system based on the two wall tangential direction vectors: ; Based on the rotation matrix of the camera relative to the robot's coordinate system Determine the wall normal vector in the robot coordinate system: ; in, and These are the tangential direction vectors of the two walls in the camera coordinate system. This is the wall normal vector in the camera coordinate system. This is the wall normal vector in the robot coordinate system.

[0014] In one embodiment, generating the wall-attaching control variables for attaching the underwater robot to the wall based on the wall normal vector, the target wall-attaching distance, and the relative pose information includes: The distance error is determined based on the current distance between the robot and the wall and the target wall-attaching distance; The magnitude of the normal force against the wall is determined based on the distance error and the robot's speed of movement along the wall's normal direction. The direction of the normal force adhering to the wall is determined based on the opposite direction of the wall normal vector; The normal adhesion control quantity is generated based on the magnitude and direction of the normal adhesion force.

[0015] In one embodiment, the normal wall-attaching force satisfies: ; The normal wall-attaching force vector corresponding to the normal wall-attaching control variable satisfies: ; in, The magnitude of the normal force adhering to the wall. Based on the wall adhesion force, For distance scaling gain, For distance differential gain, The current distance between the robot and the wall. The target wall-hugging distance, Let $\frac{ ... This is a compensation item for the reaction force during operation. Let be the wall normal vector. This is the normal force vector along the wall.

[0016] In one embodiment, generating the wall-attaching control variables for enabling the underwater robot to adhere to the wall based on the wall normal vector, the target wall-attaching distance, and the relative pose information further includes: Determine the current direction of the robot's wall-hugging axis; The attitude error is determined based on the current direction of the robot's wall-attaching axis and the opposite direction of the wall normal vector; The attitude correction control quantity is generated based on the attitude error and the robot's angular velocity.

[0017] Secondly, a multi-view vision servo-based underwater robot wall-following control system is applied to an underwater robot, which includes multiple cameras, an inertial measurement unit, and multiple thrusters. The system includes: The wall geometry estimation unit is used to acquire wall images simultaneously captured by multiple cameras, and estimate the wall geometry parameters in the robot coordinate system based on the wall images. The wall geometry parameters include the wall normal vector and the distance between the robot and the wall. The state fusion unit is used to filter and fuse the wall geometry parameters with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall and the posture state of the robot relative to the wall. The wall-attaching control quantity generation unit is used to generate wall-attaching control quantities for the underwater robot to attach to the wall based on the wall normal vector, the target wall-attaching distance and the relative pose information. The wall-attaching control quantities include a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. The thruster control execution unit is used to distribute thruster thrust according to the wall-adhering control amount and the pre-calibrated thruster configuration relationship, and obtain thruster control commands for each thruster; and control each thruster to work according to the thruster control commands, so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

[0018] In summary, the present invention has the following beneficial effects: The present invention simultaneously acquires wall images using multiple cameras and automatically estimates the wall normal vector and the distance between the robot and the wall, transforming the cameras from manually monitored components into closed-loop control sensors; by filtering and fusing the wall geometric parameters with inertial measurement data, the stability of relative pose estimation is improved under conditions such as underwater disturbance and image degradation; furthermore, based on the wall normal vector, normal-direction wall-attaching control and attitude correction control quantities are generated and converted into control commands for each thruster through thruster thrust distribution, enabling the robot to automatically adjust the wall-attaching thrust direction and attitude, thereby maintaining stable adhesion on walls of different materials, curvatures, or flexible surfaces, reducing the intensity of manual operation and the risk of slippage. Attached Figure Description

[0019] Figure 1 This is a flowchart of the underwater robot wall-hugging control method based on multi-view vision servoing of the present invention; Figure 2 This is a structural diagram of the underwater robot wall-following control device based on multi-view vision servoing in an embodiment of the present invention; Figure 3 This is an internal structural diagram of the computer device in an embodiment of the present invention; Figure 4 This is a schematic diagram of the robot structure and coordinate transformation according to an embodiment of the present invention; Figure 5 This is a flowchart of the algorithm in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0021] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0022] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0023] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0024] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0025] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

[0026] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Example 1 To address the aforementioned problems, this invention provides a method for controlling underwater robots to adhere to walls based on multi-view vision servoing, such as... Figure 4 As shown, this is applied to an underwater robot, which includes multiple cameras, an inertial measurement unit, and multiple thrusters.

[0028] In one embodiment, to achieve multi-view vision servo control, a wall coordinate system, a robot coordinate system, and a camera coordinate system can be pre-established. Geometric quantities such as the wall normal vector, the distance between the robot and the wall, the camera pose, and the robot's wall-hugging posture can be uniformly described within these coordinate systems. By unifying the coordinate systems, the wall geometric parameters obtained from camera images can be transformed into the robot coordinate system and further used for subsequent state estimation, wall-hugging control quantity generation, and thruster thrust distribution.

[0029] Specifically, a wall coordinate system can be established. The wall coordinate system It is a local moving coordinate system, with its origin at... The wall coordinate system is located at the projection point of the underwater robot's current foot on the wall surface and can be locally updated as the underwater robot moves along the wall surface. of The shaft is partially tangentially arranged along the wall surface and can be approximately parallel to the underwater robot's direction of travel; The shaft is partially tangentially arranged along the wall surface and is related to the... The axes are orthogonal; The axis is set along the local normal of the wall surface and points outward from the wall surface, that is, from the wall surface towards the underwater robot. Since the hull surface may be curved, and the flexible netting may also undergo local deformation with the water flow, therefore, the wall coordinate system... It can be defined only within a local neighborhood of the underwater robot's current wall-attached position, and this local neighborhood can be approximated as a local plane. Under this definition, the wall normal vector is in the wall coordinate system. The middle can be represented as: ;in, The wall normal vector in the wall coordinate system The following is an indication.

[0030] Furthermore, a robot coordinate system can be established. The robot coordinate system The origin It can be located at the geometric center of the underwater robot body, or at the reference center where the thrust of multiple thrusters is combined. The robot coordinate system... of The axis is set along the longitudinal axis of symmetry of the underwater robot and points in the direction of the underwater robot's movement; the The axis is perpendicular to the The axis points to the left side of the underwater robot; the The axis is perpendicular to the bottom surface of the underwater robot and points towards one side of the wall. Therefore, the... The axis can serve as the wall-adhering axis of an underwater robot, representing the direction in which the underwater robot presses against the wall. When the underwater robot is in an ideal wall-adhering state, the... The direction of the axis is opposite to that of the wall normal vector, and the bottom surface of the underwater robot is basically parallel to the local plane of the wall.

[0031] Furthermore, a camera coordinate system can be established. The camera coordinate system The origin The camera coordinate system is located at the optical center of the camera. of The axis is set along the horizontal direction of the image, the The axis is set along the vertical direction of the image, the The axis is set along the optical axis of the camera and points towards the wall. Each camera can be calibrated to obtain its coordinate system relative to the robot. The extrinsic parameters enable the wall image information captured by different cameras to be uniformly converted to the robot coordinate system. Down.

[0032] In one specific implementation, the camera coordinate system To the robot coordinate system The transformation relationship can be expressed as: ;in, Let be the pose transformation matrix from the camera coordinate system to the robot coordinate system. Let be the rotation matrix from the camera coordinate system to the robot coordinate system. The optical center of the camera in the robot coordinate system The position vector below. It can be obtained in advance through the underwater calibration process and used as a fixed parameter during the operation of the underwater robot.

[0033] Wall coordinate system To the robot coordinate system The transformation relationship can be determined in real time by the state estimation process and expressed as: ;in, Let be the pose transformation matrix from the wall coordinate system to the robot coordinate system. Let be the rotation matrix from the wall coordinate system to the robot coordinate system. Let be the position vector of the origin of the wall coordinate system in the robot coordinate system. Since the wall coordinate system... of The axis is set along the wall normal; therefore, the wall normal vector in the robot coordinate system... The following can be represented by The third column is determined, that is: ;in, The wall normal vector in the robot coordinate system The following is a representation, and the... Pointing to the side of the underwater robot, The vertical distance between the robot and the wall is denoted as . ,because Let be the unit vector pointing from the wall to one side of the underwater robot, therefore, when When representing the position vector of the origin of the wall coordinate system in the robot coordinate system, it can be expressed as: For example, the translation relationship between the origin of the wall coordinate system and the origin of the robot coordinate system can be expressed as... The relevant vector form is used for consistent use in subsequent state estimation and control processes.

[0034] like Figure 1 As shown, the method includes: S1. Acquire wall images simultaneously captured by multiple cameras, and estimate the wall geometric parameters in the robot coordinate system based on the wall images. The wall geometric parameters include the wall normal vector and the distance between the robot and the wall. The wall normal vector is a unit vector pointing from the wall to the side of the underwater robot.

[0035] Specifically, in this step, multiple cameras can be positioned in the area where the underwater robot faces the wall, and they can synchronously acquire wall images at the same or nearly simultaneous moments. Synchronous acquisition reduces the time inconsistency between multiple images caused by underwater robot movement, wall deformation, or water flow disturbances. Subsequently, the control system estimates the current geometric state of the local wall in the robot's coordinate system based on the wall images acquired by the multiple cameras. This geometric state includes at least the wall normal vector and the distance between the robot and the wall.

[0036] The wall normal vector represents the local orientation of the wall; it is a unit vector and is defined as the direction from the wall towards the underwater robot. The distance between the robot and the wall represents the degree of closeness of the underwater robot relative to the wall. Since both the wall normal vector and the distance are expressed in the robot's coordinate system, subsequent control steps can directly determine the direction in which the underwater robot should generate the wall-attaching force and what kind of correction should be made to the current wall-attaching distance based on these geometric parameters.

[0037] This step involves simultaneously acquiring wall images using multiple cameras and estimating the wall normal and distance. This transforms the cameras from mere visual observation components into geometric perception sources within the wall-hugging control closed loop. This reduces reliance on manual visual judgment of wall attitude and distance, providing real-time, unified geometric input for the subsequent automatic generation of wall-hugging control quantities.

[0038] S2. The geometric parameters of the wall are filtered and fused with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance between the robot and the wall, the posture of the robot relative to the wall, and the movement speed of the robot along the normal direction of the wall.

[0039] In this step, the wall geometry parameters serve as visual observation information, and the inertial measurement unit (IMU) collects inertial measurement data as motion state observation information. The IMU data can include the underwater robot's angular velocity, acceleration, or attitude change information derived from these parameters during its motion. By filtering and fusing the visually estimated wall normal vector and distance with the IMU data, more stable relative pose information can be obtained. This relative pose information includes at least the distance state between the robot and the wall, the robot's attitude state relative to the wall, and the robot's velocity along the wall's normal direction. Specifically, the distance state characterizes the actual gap between the robot and the wall; the attitude state characterizes the robot's tilt or wall-hugging posture relative to the wall; and the robot's velocity along the wall's normal direction characterizes the robot's trend of approaching or moving away from the wall. This relative pose information provides a more stable state basis for the subsequent generation of wall-hugging control quantities.

[0040] This step filters and fuses the visually estimated wall geometry parameters with inertial measurement data, reducing estimation fluctuations caused by water turbidity, image jitter, or local occlusion when relying solely on vision. Furthermore, the fused result includes not only distance and attitude but also normal velocity information, which facilitates subsequent dynamic adjustment of the wall-attaching force, preventing the robot from excessively colliding with or veering away from the wall.

[0041] S3. Based on the wall normal vector, the target wall-attaching distance, and the relative pose information, generate wall-attaching control quantities for making the underwater robot attach to the wall. The wall-attaching control quantities include a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture.

[0042] In this step, the target wall-attaching distance is a pre-set desired distance, representing the distance the underwater robot should maintain between itself and the wall when performing cleaning, inspection, or other wall-attaching operations. The control system determines whether the underwater robot needs to move closer to the wall or reduce its pressure on the wall based on the difference between the current distance and the target wall-attaching distance. Since the wall normal vector is defined as the direction pointing from the wall towards the underwater robot, the control direction for attaching the robot to the wall is the opposite direction of the wall normal vector.

[0043] Specifically, the normal-direction wall-attaching control variable is used to induce the underwater robot to adhere to the wall in the opposite direction of the wall's normal vector, thus causing the robot to press against the wall. The attitude correction control variable is used to adjust the robot's attitude relative to the wall, ensuring that the robot maintains a predetermined wall-attaching posture during the wall-attaching process and avoiding situations such as one side tilting, excessive tilting, or attitude deviation. The normal-direction wall-attaching control variable and the attitude correction control variable can together constitute the wall-attaching control variable, which serves as the input for subsequent thruster thrust distribution.

[0044] This step uses the wall normal vector, target wall-hugging distance, and relative pose information to generate wall-hugging control variables, eliminating reliance on fixed-direction thrust or manual adjustment. The normal wall-hugging control variable adjusts its direction of action according to changes in the wall normal, while the attitude correction control variable suppresses robot wall-hugging attitude deviations, thereby improving wall-hugging stability in curved or dynamic wall scenarios.

[0045] S4. Based on the wall-hugging control amount and the pre-calibrated thruster configuration relationship, distribute the thruster thrust to obtain thruster control commands for each thruster; control each thruster to operate according to the thruster control commands, so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

[0046] In this step, the thruster configuration relationship is used to represent the installation position of each thruster on the robot body, the direction of thrust, and the correspondence between the output of each thruster and the force state of the underwater robot. Since multiple thrusters are usually distributed in different positions on the underwater robot, and the thrust direction and effect generated by each thruster are not the same, it is necessary to allocate the wall-attachment control quantity into control commands that can be executed by each thruster according to the thruster configuration relationship.

[0047] Specifically, the control system determines the thruster control commands for each thruster based on the wall-attaching control parameters and the thruster configuration. These thruster control commands can be used to control the thruster's rotational speed, thrust magnitude, or thrust output state. After each thruster operates according to its corresponding control command, the underwater robot experiences fluid thrust generated by the thrusters, causing changes in its distance from the wall and its attitude. Subsequently, multiple cameras continue to acquire new wall images, and the above steps are repeated, thus forming a visual feedback-based wall-attaching control closed loop, ensuring the underwater robot maintains continuous contact with the wall.

[0048] This step, by allocating thrust based on the thruster configuration, transforms the abstract wall-attaching control variable into specific control commands that each thruster can execute, enabling multiple thrusters to work together to produce normal wall-attaching action and attitude adjustment. This closed-loop execution method allows the robot to continuously correct its wall-attaching state based on new visual feedback, improving the automation and stability of underwater wall-attaching operations.

[0049] In summary, this embodiment estimates the geometric parameters of the wall using multi-view vision, obtains stable relative pose information by combining it with inertial measurement data, generates wall-hugging control variables based on the wall normal vector and the target wall-hugging distance, and implements these variables through thruster allocation. This allows the underwater robot to automatically maintain its contact with the wall under the influence of visual feedback. This method connects visual perception, state fusion, control variable generation, and thruster execution into a complete closed loop, reducing the burden of manual remote operation and improving the reliability of wall-hugging control in complex underwater wall environments.

[0050] In one embodiment, when performing step S1, estimating the wall geometry parameters in the robot coordinate system based on the wall image may further include the following process.

[0051] First, underwater image enhancement processing is performed on the wall images simultaneously acquired by multiple cameras to obtain enhanced wall images. Due to the underwater environment's susceptibility to light attenuation, color shifts, water turbidity, and reduced image contrast, directly using the original wall images for geometric estimation may result in unclear wall boundaries, textures, or structural information. Therefore, after obtaining the wall images, enhancement processing can be applied to improve the recognizability of the wall regions within the image. This underwater image enhancement processing can improve the clarity of effective visual information in the wall images, making subsequent extraction of wall visual features more stable.

[0052] Then, visual features of the wall are extracted from the enhanced wall image. These visual features can be understood as image information reflecting the local geometric state of the wall, such as textures, edges, structural lines, local feature points, or other visual information that can characterize the spatial relationships of the wall. By extracting visual features from the enhanced wall image, the wall information in the two-dimensional image can be transformed into visual basis data usable for subsequent geometric estimation.

[0053] Furthermore, based on the camera intrinsic and extrinsic parameters among multiple cameras, the visual features of the wall are converted into geometric observations of the wall in the robot coordinate system. The camera intrinsic parameters characterize the imaging parameters of the cameras, while the camera extrinsic parameters characterize the position and orientation relationship between the cameras and the robot coordinate system. By using the camera intrinsic and extrinsic parameters, the visual features of the wall acquired by different cameras can be unified and expressed in the robot coordinate system, allowing subsequent control processes to directly utilize the geometric information in the robot coordinate system.

[0054] In this embodiment, the wall geometry observations are used to reflect the spatial relationship between the wall and the underwater robot. The control system can determine the wall normal vector and the distance between the robot and the wall based on the wall geometry observations. The wall normal vector is a unit vector pointing from the wall towards the underwater robot, representing the local orientation of the wall; the distance between the robot and the wall represents the degree of closeness of the robot relative to the wall. Since both the wall normal vector and the distance are determined in the robot's coordinate system, subsequent steps can be based on these wall normal vectors and distances for further filtering and fusion, wall-hugging control quantity generation, and thruster control.

[0055] Through the above implementation method, before performing wall-hugging control, the underwater wall image is enhanced and visual features are extracted. Then, combined with camera intrinsic and extrinsic parameters, the image information is converted into geometric observations in the robot coordinate system, providing a unified coordinate basis for estimating the wall normal vector and distance. This method can reduce the impact of underwater image degradation on geometric estimation and provide more stable and directly usable wall space information for subsequent wall-hugging control. In one embodiment, when the wall is a rigid wall, the local spatial morphology of the wall can be geometrically estimated based on enhanced wall images captured by multiple cameras. The rigid wall can be a ship's hull, the surface of a marine engineering structure, or other walls that can be approximated as a local plane within the robot's current observation range. Since the morphological changes of a rigid wall are relatively small within the robot's current observation range, the spatial positional relationship of the wall relative to the underwater robot can be recovered by analyzing the differences in imaging the same wall area from different cameras.

[0056] Specifically, firstly, feature points are extracted from the enhanced wall images captured by at least two cameras to obtain multiple image feature points. In one specific implementation, the enhanced wall image captured by the first camera can be used as a reference image, and feature points are extracted from the reference image to obtain a set of feature points in the reference image. ;in, The feature point set corresponding to the first camera. For the first The pixel coordinates of each image feature point in the reference image. This represents the number of feature points extracted from the reference image. Correspondingly, a feature descriptor can be generated for each image feature point, and the set of feature descriptors corresponding to the reference image can be represented as: ;in, For the first Feature descriptors corresponding to each image feature point Let be the dimension of the feature descriptor. For the ... The enhanced wall image captured by each camera can also be used to extract the corresponding feature point set. and feature descriptor subsets ,in, , The number of cameras is specified. The image feature points can be locations in the wall image where grayscale, texture, edge, or local structural changes are significant, used to characterize identifiable visual information in the wall image.

[0057] Then, feature matching is performed on the image feature points corresponding to different cameras, and the matching results are filtered based on epipolar constraints to obtain the matched feature points. In one specific implementation, the feature descriptors in the reference image and the first... Nearest neighbor matching is performed on the feature descriptors in the road image to obtain preliminary matching results. To eliminate false matches, the preliminary matching results can be filtered based on the descriptor distance ratio. For example, for the first feature descriptor in the reference image... Feature descriptors In the Determining the nearest neighbor descriptor in the road image and next nearest neighbor descriptors The corresponding matching relationship is retained when the following conditions are met: ;in, This is a preset distance ratio threshold. Through the above filtering, matching points with insufficient descriptor similarity or ambiguity can be reduced.

[0058] Furthermore, the matching results after descriptor distance ratio filtering can be further filtered based on epipolar constraints. Specifically, the matching results between the first camera and the second camera can be determined based on the calibrated camera extrinsic parameters. Relative rotation matrix between road cameras and relative translation vector And based on the camera intrinsic parameter matrix Constructing the essential matrix: ;in, For the relative translation vector The constructed antisymmetric matrix, For the first camera and the second The essential matrix between the road cameras. Based on the essential matrix, it can be determined whether the matching feature points satisfy the geometric relationship of multi-camera imaging, thereby eliminating mismatched points that do not satisfy the epipolar constraint and obtaining a reliable set of matching feature points.

[0059] Furthermore, multi-view triangulation is performed based on the matched feature points and the camera intrinsic and extrinsic parameters between multiple cameras to obtain a local 3D point cloud of the wall in the robot coordinate system. The camera intrinsic parameters characterize the imaging parameters of the camera itself, while the camera extrinsic parameters characterize the position and orientation of each camera relative to the robot coordinate system. For a set of matched feature points obtained through filtering, the matched points in the first camera path can be denoted as homogeneous image coordinates. , will the The matching points in the road camera are recorded as homogeneous image coordinates. Correspondingly, the first camera and the second... The projection matrix of the road camera can be represented as follows: ; ;in, The extrinsic parameter matrix from the first camera to the robot's coordinate system. For the first The extrinsic parameter matrix from the road camera to the robot's coordinate system.

[0060] During triangulation, the following linear equation can be constructed: ;in, Let be the homogeneous coordinates of the spatial point. By solving the above linear equation, the three-dimensional spatial point of the corresponding matching feature point in the robot coordinate system can be obtained. If the homogeneous coordinates are expressed as... Then the corresponding three-dimensional point can be represented as: Repeating the above triangulation process for multiple matching feature points and multiple camera combinations can yield a local 3D point cloud of the wall in the robot coordinate system. ;in, For the first A three-dimensional point, This represents the number of three-dimensional points. The local three-dimensional point cloud of the wall is used to characterize the spatial distribution of the rigid wall within the robot's current observation range.

[0061] Subsequently, a local plane fitting is performed on the local 3D point cloud of the wall to obtain the wall normal vector and the distance between the robot and the wall. Since a rigid wall can be approximated as a local plane within the robot's current observation range, the local plane of the wall can be fitted based on the distribution of spatial points in the 3D point cloud. The orientation of this local plane is used to determine the wall normal vector, and the interval between this local plane and the origin of the robot's coordinate system is used to determine the distance between the robot and the wall. Thus, the wall normal vector and distance in the robot's coordinate system can be obtained, providing a geometric basis for subsequent filtering fusion and wall-attaching control variable generation.

[0062] In one embodiment, after obtaining the local 3D point cloud of the wall in the robot coordinate system, a local plane of the wall within the current observation range can be determined based on the local 3D point cloud. Specifically, the following plane equation can be fitted based on the local 3D point cloud of the wall: ;in, , , These are the coordinate components of a 3D point in the robot's coordinate system. , , , These are the parameters for the fitted plane equation. This plane equation represents the position and orientation of the current local wall surface in the robot coordinate system.

[0063] In one specific implementation, multiple three-dimensional points can be selected from the local three-dimensional point cloud of the wall, and a planar fitting matrix can be constructed based on the three-dimensional points. For the first... Three-dimensional points It should satisfy: Combining the equations corresponding to multiple three-dimensional points yields a matrix form: ;in, It is a matrix composed of the coordinates of multiple three-dimensional points. Let be a planar parameter vector, and: The plane parameter vector can be obtained by solving the matrix equation. In an alternative implementation, a random sampling consistency method can be used to remove outliers from the local 3D point cloud of the wall, making the fitted local plane more stable.

[0064] After obtaining the plane equation, the wall normal vector can be determined based on the normal parameters in the plane equation. Specifically, since the plane equation contains... , , These correspond to the three coordinate components of the plane's normal direction, therefore, we can... , , The resulting vector is normalized to obtain the wall normal vector: ;in, This is the wall normal vector in the robot coordinate system. Through normalization, this wall normal vector can be made a unit vector, thus facilitating the determination of the direction of the wall-hugging control variable during subsequent wall-hugging control.

[0065] Since the normal vector corresponding to the same plane equation may have two opposite directions, and this application defines the wall normal vector as a unit vector pointing from the wall towards the underwater robot, after obtaining the initial wall normal vector, it is necessary to perform a sign check on the wall normal vector according to the direction of the robot's wall-attaching axis. The robot's wall-attaching axis is a preset axis direction on the robot body used to characterize the direction the robot faces towards the wall. By comparing the relative relationship between the wall normal vector and the robot's wall-attaching axis direction, it can be determined whether the currently obtained wall normal vector points towards the underwater robot; if the direction of the current wall normal vector does not conform to the preset pointing relationship, the wall normal vector is reversed so that the final obtained wall normal vector points towards the underwater robot.

[0066] After determining the wall normal vector after sign verification, the distance between the robot and the wall can be further determined based on the plane equation. Specifically, in the robot coordinate system, the origin of the robot coordinate system can be used as the reference point of the robot body, and the distance between the robot and the wall can be determined based on the perpendicular distance from this reference point to the fitted plane. ;in, The distance between the robot and the wall. This is the constant term in the plane equation. , , This represents the normal parameter in the plane equation. Therefore, the wall normal vector in the robot coordinate system can be obtained simultaneously. and the distance between the robot and the wall This is then used as the input for subsequent filtering fusion and wall-hugging control quantity generation.

[0067] This step addresses the stable local morphology of rigid walls by converting 2D image information from multiple cameras into 3D point clouds in the robot's coordinate system through feature point extraction, feature matching, epipolar constraint filtering, and multi-view triangulation. Then, the wall normal vector and distance are obtained through local plane fitting, enabling the underwater robot to directly obtain the geometric parameters required for wall-hugging control based on its own coordinate system. At the same time, sign verification can avoid the problem of uncertain normal vector direction leading to opposite wall-hugging control direction, thereby improving the accuracy and safety of rigid wall-hugging control.

[0068] In one embodiment, when the wall surface is a flexible mesh wall, the periodic grid structure of the flexible mesh itself can be utilized to extract grid line features from the enhanced wall image, and the wall normal vector and the distance between the robot and the wall can be estimated based on the perspective geometry of the grid lines in the image. The flexible mesh wall can be aquaculture cage mesh or other flexible underwater walls with a grid-like structure. Since flexible mesh surfaces typically lack stable, non-repeating textures, and the grid cells have periodic repetitive features, directly using ordinary feature point matching methods can easily lead to mismatches. Therefore, this embodiment obtains the geometric observations of the wall surface through calculations of grid line segments, line clusters, and vanishing points.

[0069] Specifically, firstly, edge detection and line segment extraction are performed on the enhanced wall image to obtain a set of grid line segments. Optionally, edge detection can be performed on the enhanced wall image to obtain an edge image: ;in, Used to represent image coordinates The edge response at the location. Subsequently, straight line segments can be extracted from the edge image to obtain an initial set of line segments: ;in, For the initial set of line segments, For the first line segment This represents the number of line segments extracted. Each line segment... It can be represented by its two endpoints as follows: ;in, and The first The coordinates of the two endpoints of the line segment. The set of grid line segments includes multiple line segments in the image that can reflect the edge or orientation of the mesh. Since the enhanced wall image has improved the contrast and recognizability of the underwater image, candidate line segments related to the mesh structure can be obtained from the image through edge detection and line segment extraction, providing a basis for subsequent mesh line screening and vanishing point calculation.

[0070] Then, the set of grid segments is filtered based on segment length, segment angle, and segment parallelism to obtain valid grid segments. Specifically, the length of each segment can be calculated first: And retain line segments that satisfy the following length conditions: ;in, and These are the preset minimum and maximum line segment lengths, respectively. By filtering by length, excessively short noisy line segments or excessively long non-mesh boundary line segments can be removed, making the remaining line segments more likely to correspond to the grid lines in the flexible mesh.

[0071] Furthermore, the angle of each line segment can be calculated: ;in, For the first The orientation angle of a line segment relative to the image coordinate system. Since flexible mesh is typically composed of grid lines in two principal directions, the angles of the effective grid line segments should be concentrated near these two principal directions. The angles of the two principal directions can be determined based on the distribution of the line segment angles, and are denoted as follows: and And retain only line segments that satisfy the following conditions: ;in, To allow for a preset angle tolerance, this embodiment preferably uses 10°~15°. By filtering by angle, line segments whose direction deviates significantly from the main direction of the mesh can be removed, reducing the impact of algal patch edges, light streaks, or irregular boundaries of dirt on subsequent calculations.

[0072] Furthermore, line segments can be filtered based on their parallelism. For line segments that pass the length and angle filtering, their direction vectors can be calculated and normalized. For example, the first... The direction vector of a line segment can be represented as: For line segments belonging to the same candidate direction, parallelism can be determined based on the inner product of their direction vectors. When the inner product between the direction vector of a line segment and the main direction of the corresponding line cluster is less than a preset threshold, the line segment can be removed as an outlier. The resulting effective grid line segments can more accurately reflect the grid structure of the flexible mesh.

[0073] Furthermore, the effective mesh segments are divided into two groups of mesh clusters. Since flexible mesh typically includes two main weaving directions or two main mesh directions, it can be divided into two groups of mesh clusters based on the directional relationship of the effective mesh segments. Let the two groups of mesh clusters be: The corresponding directional centers are: In one specific implementation, the line cluster division result can be verified based on the included angle between the centers of the two sets of directions. For example, it can be determined whether the included angle between the two sets of grid line clusters satisfies: When the mesh of the flexible mesh is not perfectly orthogonal due to stress or deformation, the range of the included angle constraint can be relaxed according to the actual situation. Each set of mesh line clusters corresponds to a tangential direction on the mesh wall, and the two tangential directions together define the tangential plane of the local mesh wall.

[0074] Optionally, the confidence weight of a line cluster can be determined based on the number of line segments and the degree of angle concentration in each grid cluster. For example, the first... The confidence weights of a group of grid line clusters can be expressed as: ;in, For the first Confidence weights for grid line clusters For the first The number of line segments within a group of grid lines. The total number of valid grid segments. For the first The standard deviation of the angles of line segments within a mesh cluster. The more numerous and concentrated the line segments within a cluster, the more stably the cluster can represent a principal direction of the mesh, and the higher its confidence level.

[0075] Subsequently, the vanishing points are calculated based on the two sets of grid line clusters. For each set of grid line clusters, the grid line segments can be considered as line structures extending along the same tangential direction of the wall in three-dimensional space; under image perspective projection, the extensions of multiple grid line segments in the same direction tend to intersect at a single vanishing point. Therefore, the corresponding vanishing points can be calculated based on the line segments in each set of grid line clusters, thus obtaining two image geometric points related to the tangential direction of the wall.

[0076] Specifically, for the first Group of grid lines The first in A line segment can be represented as the equation of a straight line in homogeneous coordinates: ;in: ; ; Combine multiple straight lines from the same grid cluster into a matrix: ;in, For the first The number of line segments in a group of mesh line clusters. For the matrix... Perform singular value decomposition: Take the right singular vector corresponding to the minimum singular value as the homogeneous coordinates of the vanishing point of the grid line cluster, denoted as: Therefore, the vanishing points corresponding to the two sets of grid line clusters can be obtained respectively. and .

[0077] After obtaining the two vanishing points, the two wall tangential direction vectors in the camera coordinate system are obtained by back-projection based on the vanishing points and camera intrinsic parameters. Specifically, this can be achieved based on the camera intrinsic parameter matrix. By back-projecting the two vanishing points, we obtain: ; ;in, and These are the tangential direction vectors of the two walls in the camera coordinate system. For the camera intrinsic parameter matrix, and These are the vanishing points corresponding to the two sets of mesh line clusters. Since both tangential direction vectors lie within the local tangential plane of the flexible mesh, the wall normal vector in the camera coordinate system can be determined based on their cross product. ;in, This is the wall normal vector in the camera coordinate system.

[0078] Furthermore, the wall normal vector in the camera coordinate system is converted to the wall normal vector in the robot coordinate system based on the camera's extrinsic parameters. For example, coordinate transformation can be performed based on the rotation relationship of the camera relative to the robot coordinate system to obtain: ;in, This is the wall normal vector in the robot coordinate system. Let be the rotation matrix from the camera coordinate system to the robot coordinate system. Therefore, the local normal vectors of the flexible mesh wall can be uniformly expressed in the robot coordinate system, facilitating the subsequent generation of wall-attaching control variables that act in the opposite direction of the wall normal vectors.

[0079] When multiple cameras obtain corresponding wall normal vector estimation results, the results can be fused based on the line cluster confidence weights or vanishing point validity of each image to obtain the final wall normal vector. For example, if the first... The wall normal vector obtained by the road camera is Its corresponding weight is Then the merged wall normal vector can be expressed as: ;in, This represents the wall normal vector in the fused robot coordinate system. This method leverages redundant observations from multiple cameras to improve the stability of the normal vector estimation.

[0080] When determining the distance between the robot and the wall, it can be based on the physical dimensions of the mesh, the pixel size of the mesh cells in the image, and the camera's focal length. For example, when the physical dimensions of the mesh are known, let the physical dimensions of the mesh be... The pixel size of the corresponding grid cell in the image is The camera focal length is Then the distance between the robot and the wall can be expressed as: ;in, This is a correction parameter corresponding to the angle between the camera's optical axis and the wall's normal vector. Using this calculation method, the distance between the robot and the flexible mesh wall can be estimated based on the relationship between the actual size of the mesh and the image imaging scale. In other embodiments, the distance between the robot and the wall can also be determined based on distance measurements collected by an ultrasonic ranging sensor. Thus, in the flexible mesh scenario, the wall normal vector can be obtained from the geometric relationship of the grid line vanishing points, and the distance can be obtained from the image scale relationship or the ultrasonic ranging results, thereby forming a complete geometric observation of the wall.

[0081] Through the above implementation methods, in the case of a flexible mesh wall lacking stable non-repeating textures, the tangential direction of the wall can be extracted by utilizing the directional consistency of the mesh line segments and the perspective vanishing point relationship. The wall normal vector can be determined by the cross product of the two tangential directions, which can reduce the risk of feature point mismatch caused by the periodic mesh structure. At the same time, by using weighted fusion of multiple cameras and the relationship between the physical size of the mesh and the image scale or by obtaining the distance through ultrasonic ranging, the wall normal vector and distance required for wall-adhering control can still be obtained in the flexible mesh scene, thereby improving the robot's adaptability to flexible walls.

[0082] In one embodiment, after obtaining the wall normal vector, the target wall-attaching distance, and the relative pose of the underwater robot with respect to the wall, wall-attaching control variables can be generated based on the above information to enable the underwater robot to adhere to the wall. These wall-attaching control variables characterize the control actions required by the underwater robot to maintain its wall-attached state, and include at least a normal wall-attaching control variable. The normal wall-attaching control variable is used to cause the underwater robot to adhere to the wall towards one side.

[0083] Specifically, the distance error can be determined based on the current distance between the robot and the wall and the target distance to the wall. The current distance between the robot and the wall is denoted as... The target wall-hugging distance is recorded as Then the distance error can be expressed as .when Greater than When this occurs, it indicates that the actual distance between the underwater robot and the wall is greater than the expected distance, requiring an increase in the wall-attaching action; when Less than If the underwater robot is too close to the wall, it indicates that the contact force between the robot and the wall needs to be reduced accordingly to lower the risk of excessive contact or collision between the robot and the wall.

[0084] Furthermore, the magnitude of the normal force adhering to the wall is determined based on the distance error and the robot's velocity along the wall's normal direction. The robot's velocity along the wall's normal direction is denoted as... This is used to characterize the motion tendency of an underwater robot in the normal direction of the wall. By introducing... This allows the normal wall-adhering force to be related not only to the current distance deviation but also to the dynamic changes in the robot's approach to or away from the wall, thus giving the wall-adhering control a certain degree of dynamic adjustment capability.

[0085] In one specific embodiment, the normal wall-attaching force satisfies: ;in, The magnitude of the normal force adhering to the wall. Based on the wall adhesion force, For distance scaling gain, For distance differential gain, The current distance between the robot and the wall. The target wall-hugging distance, Let $\frac{ ... This is a compensation item for the reaction force of the operation.

[0086] Among them, the basic wall-attaching force Used to provide the force required to maintain the underwater robot's basic adhesion to the wall; distance proportional gain Used to adjust the normal wall-attaching force based on the deviation between the current distance and the target wall-attaching distance; distance differential gain Used to correct the normal adhesion force based on the robot's velocity along the wall's normal direction; operational reaction force compensation item. This is used to compensate for the reaction forces experienced by the underwater robot during wall-hugging operations. Due to the wall normal vector... Defined as a unit vector pointing from the wall towards the underwater robot, and since the underwater robot needs to exert a force towards the wall when attached to it, the direction of the normal force is the opposite direction of the wall normal vector. Based on this, the normal force vector corresponding to the normal attachment control quantity satisfies: ;in, Let be the wall normal vector. This is the normal force vector along the wall. From the above formula, we can see that... Used to determine the magnitude of the normal wall-attachment force. Used to determine the direction of the normal adhesion force. The control system can generate a normal adhesion control quantity based on the magnitude and direction of the normal adhesion force, and use the normal adhesion control quantity as the input for subsequent thrust distribution of the thruster.

[0087] Through the above implementation method, the underwater robot can dynamically determine the magnitude of the normal wall-attaching force based on the current wall-attaching distance, the target wall-attaching distance, and the normal motion velocity, and determine the wall-attaching action direction based on the opposite direction of the wall normal vector. This allows the wall-attaching control quantity to be adjusted according to changes in the wall geometry and the robot's motion state, thereby reducing the risk of slippage due to insufficient wall-attaching force and the risk of collision due to excessive wall-attaching force.

[0088] In one embodiment, when generating the wall-attaching control variables for attaching the underwater robot to the wall, in addition to generating the normal wall-attaching control variables, attitude correction control variables can also be generated. These attitude correction control variables are used to adjust the attitude of the underwater robot body relative to the wall, ensuring that the underwater robot maintains a suitable attitude for wall-attaching operations during the process of approaching the wall, preventing the robot from only attaching to the wall in localized areas while tilting or deviating in other areas.

[0089] Specifically, the current direction of the robot's wall-attaching axis can be determined first. This wall-attaching axis can be a pre-defined axis on the underwater robot body, representing the side of the robot facing the wall. For example, when the robot is attached to the wall in a preset posture, the wall-attaching axis should face the wall. The current direction of the robot's wall-attaching axis can be determined based on the posture state in the relative pose information, or it can be determined based on the relationship between the underwater robot's current posture and the robot's coordinate system.

[0090] After determining the current direction of the robot's wall-hugging axis, the attitude error can be determined based on the current direction of the robot's wall-hugging axis and the opposite direction of the wall normal vector. Since the wall normal vector... Defined as a unit vector pointing from the wall towards the side of the underwater robot, therefore, the desired wall-hugging direction of the underwater robot is... When the robot's current direction of its wall-hugging axis is... If there is an inconsistency, it indicates that there is an attitude deviation between the robot body and the wall, which needs to be adjusted through attitude correction control.

[0091] The attitude error can be used to characterize the degree of deviation between the current direction of the robot's wall-hugging axis and the desired wall-hugging direction. A larger deviation indicates a more pronounced tilt of the robot relative to the wall, requiring greater attitude correction; a smaller deviation indicates that the robot is closer to the predetermined wall-hugging posture, requiring less attitude correction. By using the opposite direction of the wall's normal vector as the desired wall-hugging direction, the attitude adjustment target can change with the local orientation of the wall, thus adapting to curved walls or wall environments with changing attitudes.

[0092] Furthermore, the attitude correction control quantity is generated based on the attitude error and the robot angular velocity. The robot angular velocity characterizes the speed state of the robot's attitude change. By combining the attitude error and the robot angular velocity, the trend of attitude change can be suppressed or buffered while adjusting the robot's wall-hugging attitude, preventing the robot from swaying due to excessively rapid attitude correction during wall-hugging. The generated attitude correction control quantity can be used together with the normal wall-hugging control quantity as the wall-hugging control quantity and transmitted to the subsequent thruster thrust distribution process.

[0093] Through the above implementation methods, the underwater robot can not only generate a normal wall-adhering action by pressing against the wall in the opposite direction of the wall normal vector, but also generate attitude correction control quantities based on the deviation between the robot's wall-adhering axis and the desired wall-adhering direction, so that the robot's body attitude is adjusted according to the wall orientation, thereby reducing the risk of local lifting, attitude deviation or uneven adhesion during the wall-adhering process, and improving the stability and continuity of the wall-adhering operation.

[0094] In one embodiment, after generating the wall-attaching control quantities for the underwater robot to adhere to the wall surface, thrust can be further distributed according to the wall-attaching control quantities and the pre-calibrated thruster configuration relationship to obtain thruster control commands for each thruster. The thruster thrust distribution is used to convert the aforementioned wall-attaching control quantities into specific control quantities that multiple thrusters can execute, enabling the multiple thrusters to collaboratively generate the force and torque required for the underwater robot to adhere to the wall.

[0095] Specifically, the input to this step may include the desired generalized force / torque formed by the wall-attachment control quantity. and the pre-calibrated thruster configuration matrix Wherein, the desired generalized force / torque This can include forces to press the robot against the wall, torques to adjust the robot's posture, and other control actions related to wall-hugging motion. The thruster configuration matrix... It is used to characterize the correspondence between the output of each thruster and the overall force state of the underwater robot.

[0096] In one specific embodiment, the underwater robot includes If there are one thruster, then the thruster configuration matrix is... It can be Matrix. The thruster configuration matrix. The Columns are used to represent the first The mapping relationship between the three-dimensional force and three-dimensional torque generated by each thruster on the underwater robot under unit rotational speed or unit normalized control input. This mapping relationship can be derived from the first thruster... Installation position of each thruster in the robot coordinate system and thrust direction unit vector It was jointly determined that, Indicates the first The position of each thruster relative to the origin of the robot's coordinate system Indicates the first The direction in which the thruster generates thrust.

[0097] Thruster configuration matrix Once determined, a constrained optimization problem can be established to solve for the rotational speed command vector of each thruster. .in, for A dimensional vector can be represented as: ;in, Indicates the first The speed command or normalized control command for each thruster.

[0098] In one specific implementation, the constrained optimization problem can be expressed as: The constraints are: ;in, This is the energy consumption weight matrix, used to characterize the energy consumption weights corresponding to different thruster output control quantities; These are force / torque tracking error weighting coefficients used to adjust the thruster output results relative to the desired generalized force / torque. The degree of tracking; and These represent the minimum and maximum control commands allowed for each thruster. By applying these constraints, the resulting thruster control commands can be made to satisfy the thruster's own physical output limitations.

[0099] In the above optimization problem, Indicates the current thruster control command Below, the actual generalized force / torque generated by multiple thrusters; Used to represent the deviation between the actual generalized force / torque and the expected generalized force / torque; This is used to represent the energy consumption or control cost corresponding to the thruster control output. Therefore, by solving this optimization problem, it is possible to track the desired wall-hugging control quantity as closely as possible while taking into account the thruster energy consumption and output constraints.

[0100] In real-time control, the above-mentioned constrained optimization problem can be solved using quadratic programming to obtain the thruster control command vector that satisfies the constraints. Subsequently, the thruster control command vector The commands are sent to the thruster drive unit, causing each thruster to output thrust according to the corresponding control instructions. The thrust output by each thruster works together on the underwater robot body, causing the underwater robot to produce motion and attitude adjustments corresponding to the wall-hugging control amount, thereby maintaining a state of contact with the wall under visual feedback.

[0101] Through the above implementation method, the wall-attaching control quantity can be allocated into specific control commands for multiple thrusters according to the pre-calibrated thruster configuration relationship, enabling each thruster to collaboratively generate the desired force and torque within its own output constraint range. This method avoids the problem of fixed ratio allocation or manual adjustment being difficult to adapt to different force states, and is beneficial to improving the execution accuracy, energy consumption coordination, and stability of wall-attaching control.

[0102] Finally, the rotation speed command output from the above steps is sent to the motor drive unit of each thruster. The thrusters generate physical thrust, which acts on the robot body, causing changes in the robot's pose and force state. The relative geometric relationship between the robot and the wall changes accordingly. The camera continuously acquires new wall images and feeds the data back to step S1, forming a complete visual servoing closed loop.

[0103] Example 2 Please see Figure 2 A multi-view vision servo-based underwater robot wall-hugging control device is applied to an underwater robot. The underwater robot includes multiple cameras, an inertial measurement unit, and multiple thrusters. The system includes: The wall geometry estimation unit 1 is used to acquire wall images simultaneously captured by multiple cameras, and estimate the wall geometry parameters in the robot coordinate system based on the wall images. The wall geometry parameters include the wall normal vector and the distance between the robot and the wall. The state fusion unit 2 is used to filter and fuse the wall geometry parameters with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall and the posture state of the robot relative to the wall. The wall-attaching control quantity generation unit 3 is used to generate wall-attaching control quantities for the underwater robot to attach to the wall based on the wall normal vector, the target wall-attaching distance and the relative pose information. The wall-attaching control quantities include a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. The thruster control execution unit 4 is used to distribute the thruster thrust according to the wall-adhering control amount and the pre-calibrated thruster configuration relationship, and obtain the thruster control command for each thruster; and control each thruster to work according to the thruster control command, so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

[0104] Specific limitations regarding the multi-view vision servoing-based underwater robot wall-hugging control device can be found in the above-mentioned limitations on the multi-view vision servoing-based underwater robot wall-hugging control method, and will not be repeated here. Each module in the aforementioned multi-view vision servoing-based underwater robot wall-hugging control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0105] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the present application. The specific underwater robot wall-hugging control device based on multi-view vision servoing may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] Example 3 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the underwater robot wall-hugging control method based on multi-view vision servoing as described in Embodiment 1.

[0107] Example 4 In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. When the computer program is executed by the processor, it implements a wall-following control method for an underwater robot based on multi-view vision servoing.

[0108] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: including: The system acquires wall images simultaneously captured by multiple cameras and estimates the wall's geometric parameters in the robot's coordinate system based on these images. The wall's geometric parameters include the wall's normal vector and the distance between the robot and the wall. The wall's normal vector is a unit vector pointing from the wall towards the underwater robot. The wall geometry parameters are filtered and fused with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall, the robot's attitude state relative to the wall, and the robot's motion speed along the normal direction of the wall. Based on the wall normal vector, the target wall-attaching distance, and the relative pose information, a wall-attaching control quantity is generated to enable the underwater robot to attach to the wall. The wall-attaching control quantity includes a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. Based on the wall-hugging control amount and the pre-calibrated thruster configuration relationship, the thruster thrust is distributed to obtain thruster control commands for each thruster; the thruster control commands are used to control the operation of each thruster so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for controlling underwater robot wall adhesion based on multi-view vision servoing, characterized in that, Applied to an underwater robot, the underwater robot including multiple cameras, an inertial measurement unit, and multiple thrusters, the method includes: The system acquires wall images simultaneously captured by multiple cameras and estimates the wall's geometric parameters in the robot's coordinate system based on these images. The wall's geometric parameters include the wall's normal vector and the distance between the robot and the wall. The wall's normal vector is a unit vector pointing from the wall towards the underwater robot. The wall geometry parameters are filtered and fused with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall, the robot's attitude state relative to the wall, and the robot's motion speed along the normal direction of the wall. Based on the wall normal vector, the target wall-attaching distance, and the relative pose information, a wall-attaching control quantity is generated to enable the underwater robot to attach to the wall. The wall-attaching control quantity includes a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. Based on the wall-hugging control amount and the pre-calibrated thruster configuration relationship, the thruster thrust is distributed to obtain thruster control commands for each thruster; the thruster control commands are used to control the operation of each thruster so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

2. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 1, characterized in that, The step of estimating the wall geometry parameters in the robot coordinate system based on the wall image includes: The wall image is subjected to underwater image enhancement processing to obtain an enhanced wall image; The wall visual features are extracted from the enhanced wall image, and the wall visual features are converted into wall geometric observations in the robot coordinate system based on the camera intrinsic and extrinsic parameters between multiple cameras. The wall normal vector and the distance between the robot and the wall are determined based on the wall geometry observations.

3. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 2, characterized in that, When the wall surface is a rigid wall surface, the wall surface visual features are extracted based on the enhanced wall surface image; Based on the camera intrinsic and extrinsic parameters between multiple cameras, the visual features of the wall are converted into geometric observations of the wall in the robot coordinate system, including: Feature points are extracted from the enhanced wall images captured by at least two cameras to obtain multiple image feature points; Feature matching is performed on image feature points corresponding to different cameras, and the matching results are filtered based on epipolar constraints to obtain matching feature points; Multi-view triangulation is performed based on the matched feature points and the camera intrinsic and extrinsic parameters between multiple cameras to obtain a local 3D point cloud of the wall in the robot coordinate system. Local plane fitting is performed on the local 3D point cloud of the wall to obtain the wall normal vector and the distance between the robot and the wall.

4. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 3, characterized in that, The step of performing local planar fitting on the local 3D point cloud of the wall to obtain the wall normal vector and the distance between the robot and the wall includes: Based on the local three-dimensional point cloud of the wall, a plane equation is fitted: ; Determine the wall normal vector based on the plane equation: ; The sign of the wall normal vector is checked according to the direction of the robot's wall-attaching axis so that the wall normal vector points to the side of the underwater robot; The distance between the robot and the wall is determined based on the plane equation: ; in, Let be the wall normal vector. The distance between the robot and the wall. , , , These are the parameters of the plane equation.

5. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 2, characterized in that, When the wall surface is a flexible mesh wall surface, the wall surface visual features are extracted based on the enhanced wall surface image; Based on the camera intrinsic and extrinsic parameters between multiple cameras, the visual features of the wall are converted into geometric observations of the wall in the robot coordinate system, including: Edge detection and line segment extraction are performed on the enhanced wall image to obtain a set of grid line segments; The set of grid segments is filtered based on segment length, segment angle, and segment parallelism to obtain valid grid segments; The effective grid line segments are divided into two groups of grid line clusters; the vanishing points are calculated according to the two groups of grid line clusters respectively. Based on the vanishing point and the camera intrinsic parameters, the two wall tangential direction vectors in the camera coordinate system are obtained by back projection. The wall normal vector in the camera coordinate system is determined by the cross product of the two wall tangential direction vectors. The wall normal vector in the camera coordinate system is then converted into the wall normal vector in the robot coordinate system based on the camera extrinsic parameters. The distance between the robot and the wall is determined based on the physical size of the mesh, the pixel size of the mesh cell in the image, and the camera focal length, or the distance between the robot and the wall is determined based on the distance measurement value collected by the ultrasonic ranging sensor.

6. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 5, characterized in that, The step of obtaining the two wall tangential direction vectors in the camera coordinate system based on the vanishing point and camera intrinsic parameters through back projection includes: Based on the camera intrinsic parameter matrix For the two vanishing points respectively and By performing back projection, we obtain two tangential direction vectors of the walls: ; ; Determine the wall normal vector in the camera coordinate system based on the two wall tangential direction vectors: ; Based on the rotation matrix of the camera relative to the robot's coordinate system Determine the wall normal vector in the robot coordinate system: ; in, and These are the tangential direction vectors of the two walls in the camera coordinate system. This is the wall normal vector in the camera coordinate system. This is the wall normal vector in the robot coordinate system.

7. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 1, characterized in that, The step of generating wall-adhesion control variables for attaching the underwater robot to the wall based on the wall normal vector, the target wall-adhesion distance, and the relative pose information includes: The distance error is determined based on the current distance between the robot and the wall and the target distance to the wall; The magnitude of the normal force against the wall is determined based on the distance error and the robot's speed of movement along the wall's normal direction. The direction of the normal force adhering to the wall is determined based on the opposite direction of the wall normal vector; The normal wall-attachment control quantity is generated based on the magnitude and direction of the normal wall-attachment force.

8. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 7, characterized in that, The normal wall-attaching force satisfies: ; The normal wall-attaching force vector corresponding to the normal wall-attaching control quantity satisfies: ; in, The magnitude of the normal force adhering to the wall. Based on the wall adhesion force, For distance scaling gain, For distance differential gain, The current distance between the robot and the wall. The target wall-hugging distance, Let $\frac{ ... This is a compensation item for the reaction force during operation. Let be the wall normal vector. This is the normal force vector along the wall.

9. The underwater robot wall-hugging control method based on multi-view vision servoing according to claim 1, characterized in that, The step of generating wall-adhesion control variables for attaching the underwater robot to the wall based on the wall normal vector, the target wall-adhesion distance, and the relative pose information further includes: Determine the current direction of the robot's wall-hugging axis; The attitude error is determined based on the current direction of the robot's wall-attaching axis and the opposite direction of the wall normal vector; The attitude correction control quantity is generated based on the attitude error and the robot's angular velocity.

10. A multi-view vision servo-based underwater robot wall-following control system, characterized in that, An application in underwater robots, the underwater robot comprising multiple cameras, an inertial measurement unit, and multiple thrusters, the system comprising: The wall geometry estimation unit is used to acquire wall images simultaneously captured by multiple cameras, and estimate the wall geometry parameters in the robot coordinate system based on the wall images. The wall geometry parameters include the wall normal vector and the distance between the robot and the wall. The state fusion unit is used to filter and fuse the wall geometry parameters with the inertial measurement data collected by the inertial measurement unit to obtain the relative pose information of the underwater robot relative to the wall. The relative pose information includes the distance state between the robot and the wall and the posture state of the robot relative to the wall. The wall-attaching control quantity generation unit is used to generate wall-attaching control quantities for the underwater robot to attach to the wall based on the wall normal vector, the target wall-attaching distance and the relative pose information. The wall-attaching control quantities include a normal wall-attaching control quantity that acts in the opposite direction of the wall normal vector and an attitude correction control quantity for adjusting the robot's wall-attaching posture. The thruster control execution unit is used to distribute thruster thrust according to the wall-adhering control amount and the pre-calibrated thruster configuration relationship, and obtain thruster control commands for each thruster; and control each thruster to work according to the thruster control commands, so that the underwater robot maintains a state of contact with the wall surface under visual feedback.

Images