A visual positioning method, device, equipment, and storage medium for underwater robots.

Abstract

This application discloses an underwater robot visual localization method, apparatus, device, and storage medium, relating to the field of computer technology. The method includes: acquiring an original image captured by a camera; preprocessing the original image to obtain a processed image; determining a target depth map corresponding to the processed image using depth estimation; locating and removing moving targets from the processed image using a dynamic object detection model to obtain a static scene image; the dynamic object detection model is a neural network model pre-trained using underwater images; and estimating the underwater robot's pose in real time based on the target depth map and the static scene image using a simultaneous localization and mapping algorithm, constructing a dense point cloud map, and updating the underwater robot's pose map. This method enables accurate underwater visual localization at low cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a visual positioning method, apparatus, device, and storage medium for underwater robots. Background Technology

[0002] With the ever-increasing demand for ocean exploration and underwater operations, underwater robot technology is developing rapidly. Current technologies primarily rely on sonar for positioning and tracking, but sonar-based positioning systems are expensive, have sparse data, and low update frequency. Visual positioning systems can achieve precise short-range positioning at a high cost. However, given the complex underwater environment, how to achieve visual positioning adapted to specific underwater scenarios is a pressing problem that needs to be solved. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an underwater robot visual positioning method, apparatus, device, and storage medium, which can achieve accurate underwater visual positioning at low cost. The specific solution is as follows:

[0004] In a first aspect, this application discloses a visual localization method for underwater robots, including:

[0005] Acquire the raw image captured by the camera, and preprocess the raw image to obtain the processed image;

[0006] The target depth map corresponding to the processed image is determined using depth estimation.

[0007] A static scene image is obtained by locating and removing moving targets from the processed image using a dynamic object detection model; the dynamic object detection model is a neural network model pre-trained using underwater images.

[0008] Based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a synchronous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot.

[0009] Optionally, the camera is a binocular camera, and the original image includes a binocular image;

[0010] Determining the target depth map corresponding to the processed image using depth estimation includes:

[0011] Input any monocular image from the original image into the monocular depth estimation system to obtain the monocular depth map corresponding to the original image;

[0012] The stereo image is input into the stereo depth estimation model to obtain the stereo depth map corresponding to the original image; both the monocular depth estimation system and the stereo depth estimation model are pre-trained using an underwater image dataset.

[0013] The target depth map is determined by comparing the monocular depth map and the binocular depth map.

[0014] Optionally, the monocular depth estimation system includes a monocular relative depth estimation model and a monocular absolute depth estimation model in sequence, wherein the input of the monocular absolute depth estimation model is the output of the monocular relative depth estimation model;

[0015] The monocular relative depth estimation model is used to output a corresponding relative depth map based on the input monocular image, and the monocular absolute depth estimation model is used to output a corresponding absolute depth map based on the input relative depth map.

[0016] Optionally, the target depth map is determined by comparing the monocular depth map and the binocular depth map, including:

[0017] The monocular depth map and the binocular depth map are analyzed for quality based on depth map evaluation metrics, and the depth map with higher quality is selected as the target depth map.

[0018] Optionally, the preprocessing includes image resizing, image enhancement, and image correction.

[0019] Optionally, the image correction process for the original image includes:

[0020] The original image is then distorted using the radial and tangential distortion coefficients obtained from camera calibration.

[0021] Optionally, based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot, including:

[0022] Based on the target depth map and the pose of the previous frame static scene image, feature points are extracted from the current static scene image and pose estimation is performed to obtain the pose of the current static scene image.

[0023] A single-frame point cloud is generated based on the target depth map, and the single-frame point cloud is filtered and then fused into a dense point cloud map.

[0024] Determine whether the current static scene image is a keyframe; if so, perform local binding adjustment and optimization based on the current static scene image.

[0025] If a loop is detected in the current static scene image, then pose graph optimization is performed based on the current static scene image.

[0026] Optionally, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping (SLT / Map) algorithm based on the target depth map and the static scene image, including:

[0027] Based on the target depth map and the static scene image, as well as the sensing data collected by the target sensor, the pose of the underwater robot is estimated in real time through a synchronous positioning and mapping algorithm; the target sensor includes an inertial measurement unit and / or sonar.

[0028] In another aspect, this application discloses an underwater robot visual positioning device, comprising:

[0029] The image preprocessing module is used to acquire the original image captured by the camera and preprocess the original image to obtain the processed image;

[0030] The depth estimation module is used to determine the target depth map corresponding to the processed image using depth estimation;

[0031] The dynamic object detection module is used to locate and remove moving targets from the processed image using a dynamic object detection model to obtain a static scene image; the dynamic object detection model is a neural network model pre-trained using underwater images.

[0032] The visual positioning module is used to estimate the pose of the underwater robot in real time based on the target depth map and the static scene image through a synchronous positioning and mapping algorithm, construct a dense point cloud map, and update the pose map of the underwater robot.

[0033] In another aspect, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned underwater robot visual positioning method.

[0034] In another aspect, this application discloses a computer program product, including a computer program that, when executed by a processor, implements the aforementioned underwater robot visual positioning method.

[0035] In another aspect, this application discloses an underwater robot visual positioning system, comprising:

[0036] A binocular digital camera used to acquire raw images;

[0037] The computing unit, which is communicatively connected to the binocular digital camera, is used to implement the aforementioned underwater robot visual positioning method.

[0038] The main unit of the unmanned remotely operated vehicle is communicatively connected to the computing unit.

[0039] Optionally, the computing unit includes: an AI acceleration processor, an image processing unit, and an underwater power supply;

[0040] The AI ​​acceleration processor is connected to the host of the unmanned remotely operated vehicle via Ethernet. The AI ​​acceleration processor is also connected to the left and right cameras of the binocular digital camera to receive images captured by the left and right cameras.

[0041] The image processing unit is connected to the AI ​​acceleration processor, the left eye camera, and the right eye camera respectively, and is used to receive the shooting instructions issued by the AI ​​acceleration processor, and synchronously trigger the left eye camera and the right eye camera through IO hard trigger according to the shooting instructions;

[0042] The underwater power source is connected to the AI ​​accelerator processor and is used to supply power to the AI ​​accelerator processor.

[0043] In another aspect, this application discloses an underwater robot, including a sealed pressure-bearing cavity and the aforementioned underwater robot visual positioning system; the underwater robot visual positioning system includes a binocular digital camera and a computing unit communicatively connected to the binocular digital camera;

[0044] The underwater robot's visual positioning system is located inside the sealed pressure chamber.

[0045] Optionally, the left eye camera of the binocular digital camera is located inside the first circular sealed pressure chamber, the right eye camera of the binocular digital camera is located inside the second circular sealed pressure chamber, and the computing unit is located inside the third circular sealed pressure chamber.

[0046] In this application, raw images captured by a camera are acquired, and preprocessed to obtain processed images. Depth estimation is used to determine the target depth map corresponding to the processed images. Moving targets are located and removed from the processed images using a dynamic object detection model to obtain a static scene image. The dynamic object detection model is a neural network model pre-trained using underwater images. Based on the target depth map and the static scene image, a simultaneous localization and mapping (SLAM) algorithm is used to estimate the underwater robot's pose in real time, construct a dense point cloud map, and update the underwater robot's pose map. It is evident that visual localization using SLAM achieves accurate close-range underwater localization at low cost. Furthermore, preprocessing the acquired raw images before subsequent calculations improves image quality and lays the foundation for subsequent visual localization. Additionally, considering the unique underwater environment, removing moving targets from the images before visual localization avoids interference from underwater fish and other dynamic objects, thus improving the accuracy of visual localization. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart of an underwater robot visual positioning method provided in this application;

[0049] Figure 2 This application provides a schematic diagram of a specific underwater robot depth estimation method.

[0050] Figure 3 This application provides a specific flowchart of underwater robot visual positioning.

[0051] Figure 4 A schematic diagram of a specific underwater robot visual positioning system provided in this application;

[0052] Figure 5 A schematic diagram of a circular sealed pressure-bearing cavity corresponding to a specific binocular camera provided in this application;

[0053] Figure 6 This application provides a schematic diagram of the interior of a specific circular sealed pressure-bearing cavity;

[0054] Figure 7 This application provides a schematic diagram of a specific circular sealed pressure-bearing cavity corresponding to a computer.

[0055] Figure 8 This is a schematic diagram of the structure of an underwater robot visual positioning device provided in this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Existing technologies primarily rely on sonar for positioning and tracking, but sonar-based positioning systems are expensive, have sparse data, and low update frequency. Visual positioning systems can achieve precise positioning at close range and are cost-effective. However, given the complex underwater environment, how to achieve visual positioning adapted to specific underwater scenarios is a pressing issue. To overcome these technical problems, this application proposes an underwater robot visual positioning method that can achieve accurate underwater visual positioning at low cost.

[0058] This application discloses a visual positioning method for underwater robots. (See also...) Figure 1 As shown, the method may include the following steps:

[0059] Step S11: Acquire the original image captured by the camera, and preprocess the original image to obtain the processed image.

[0060] The underwater robot's camera acquires raw underwater images. These raw images are then preprocessed to obtain processed images. Preprocessing includes image resizing, image enhancement, and image correction. Understandably, in the underwater environment, low contrast makes feature extraction difficult, necessitating image preprocessing, which primarily involves image enhancement, image correction, and underwater dynamic target detection. Image enhancement mainly improves contrast and brightness, enhancing image quality for subsequent processing. Image correction corrects distortions, such as camera distortion, to improve reconstruction accuracy.

[0061] In a specific implementation, the image correction process for the original image may include: using the radial distortion coefficient and tangential distortion coefficient obtained from camera calibration to correct the distortion of the original image, i.e., the distortion caused by the lens in actual camera imaging.

[0062] To remove distortion, both radial and tangential distortion are used to correct camera distortion. Radial distortion is caused by the lens shape and manifests as distortion at the image edges (such as barrel distortion and pincushion distortion). Tangential distortion is caused by the lens not being parallel to the imaging plane and manifests as tilting distortion of the image. The distortion model formula is:

[0063] ;

[0064] in, These are the x and y coordinates of the distortion-free image (original image). These are the x and y coordinates of the distorted image, expressed in polar coordinates as follows: Where k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients. Distortion parameters are calculated by taking images using a checkerboard calibration board, and then solving for the distortion model parameters based on the known positions of geometric feature points on the checkerboard calibration board in the image and their coordinate relationships in the real world, using the known distortion model and parameters. With the known distortion model and parameters, the correct position of each pixel in the corresponding camera-captured image after distortion removal can be calculated.

[0065] Step S12: Determine the target depth map corresponding to the processed image using depth estimation.

[0066] After obtaining the processed image, depth estimation is performed on the processed image to obtain the target depth map corresponding to the processed image.

[0067] In some embodiments, the camera is a binocular camera, and the original image includes a binocular image. Determining the target depth map corresponding to the processed image using depth estimation may include: inputting any monocular image from the original image into a monocular depth estimation system to obtain a monocular depth map corresponding to the original image; inputting the binocular image into a binocular depth estimation model to obtain a binocular depth map corresponding to the original image; both the monocular depth estimation system and the binocular depth estimation model are pre-trained using an underwater image dataset; comparing the monocular depth map and the binocular depth map to determine the target depth map. Therefore, this embodiment simultaneously performs monocular and binocular depth estimation to obtain both a monocular depth map and a binocular depth map, and then selects the higher-quality depth map as the target depth map by comparing the monocular and binocular depth maps. Alternatively, the monocular depth map can be calculated from any monocular image in the original image, or the two monocular images in the original image can be calculated simultaneously, and then the left, right, and binocular depth maps can be compared to select the target depth map.

[0068] In some embodiments, the monocular depth estimation system described above sequentially includes a monocular relative depth estimation model and a monocular absolute depth estimation model. The input of the monocular absolute depth estimation model is the output of the monocular relative depth estimation model. The monocular relative depth estimation model is used to output a corresponding relative depth map based on the input monocular image, and the monocular absolute depth estimation model is used to output a corresponding absolute depth map based on the input relative depth map. For example... Figure 2As shown, the entire depth estimation process is divided into monocular depth estimation and binocular depth estimation. Both methods first train models using underwater image datasets to obtain corresponding offline depth estimation models, and then use these models to perform online depth estimation on real-time camera images. The monocular depth estimation process involves training a monocular relative depth estimation model using the underwater image dataset to obtain a relative depth map; then, a monocular absolute model is trained to obtain a monocular depth map. Both monocular relative and absolute depth estimation models are obtained through model training. The binocular depth estimation process involves training a binocular depth estimation model using the underwater image dataset to obtain a depth map. This process of obtaining the binocular depth estimation model is performed offline. The real-time depth of a single camera is obtained by using both monocular relative and absolute depth estimation models, and the real-time depth of the binocular camera is obtained by using the binocular depth estimation model to estimate the depth of the real-time camera images.

[0069] Specifically, the target depth map is determined by comparing the monocular depth map and the binocular depth map. This includes: performing quality analysis on the monocular and binocular depth maps based on depth map evaluation metrics, and selecting the high-quality depth map as the target depth map. Depth map evaluation metrics include, but are not limited to, geometric accuracy metrics, edge / boundary accuracy metrics, robustness to occlusion and weakly textured regions, edge alignment, noise and artifacts, etc. In other words, the depth map effects of the two depth estimation methods are compared to determine the output camera's real-time depth map. This process of obtaining real-time depth is performed online in real time.

[0070] Step S13: Remove moving targets from the processed image by dynamic object detection to obtain a static scene image.

[0071] While estimating depth, a dynamic object detection model is used to locate and remove moving targets from the processed image to obtain a static scene image. The dynamic object detection model is a neural network model trained in advance using underwater images. Dynamic object detection mainly identifies and removes dynamic targets in the image, such as underwater organisms, to reduce their impact on localization and reconstruction.

[0072] Dynamic object detection identifies and segments moving objects from video or image sequences while eliminating interference from static backgrounds. It achieves accurate detection by leveraging temporal continuity analysis of pixel or region changes, combined with motion features and semantic information. Furthermore, it can integrate deep learning-based object detection with motion cues and incorporate camera motion compensation techniques to eliminate the effects of device movement.

[0073] The training process of the dynamic object detection model mentioned above includes: collecting underwater video sequences of different types of water and different lighting conditions; labeling the images in the underwater video sequences with object types; performing data augmentation processing on the labeled images to obtain an underwater image training set; and using the underwater image training set to iteratively train the initial neural network until the conditions are met to obtain the dynamic object detection model.

[0074] Specifically, video sequences of different types of water bodies and under varying lighting conditions can be collected using tethered remotely operated underwater vehicles (ROVs), autonomous underwater vehicles (AUVs), or fixed cameras to obtain an underwater image training set. Different water body types include, but are not limited to, turbid, clear, deep, and shallow water; different lighting conditions include natural light and artificial light. Object types are labeled in the underwater image training set to distinguish between dynamic and static objects; for example, static labels are added to corals, and dynamic labels are added to moving targets such as fish, divers, and underwater robots. Semantic segmentation and labeling can also be performed on static backgrounds for subsequent data augmentation or background suppression. To improve data diversity, data augmentation processing is performed, such as adding scattering noise, color shifts, and simulating different turbidity levels to simulate diverse underwater environments; linear or nonlinear motion blur is applied to dynamic targets to simulate rapid swimming effects; and optical flow can be used to generate inter-frame interpolation data to improve the model's understanding of motion continuity. The neural network model can use a lightweight model as the backbone to balance the computational limitations of underwater equipment, and combine it with a multi-scale feature fusion module to solve the small target detection problem.

[0075] Step S14: Based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a synchronous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot.

[0076] Finally, based on the obtained target depth map and static scene image, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping (SLT / Map) algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot. For example... Figure 3 As shown, left and right eye images are acquired through a binocular camera, and image size adjustment, image correction, and image enhancement are performed. Then, depth estimation is performed to obtain a depth map, dynamic object detection is performed, and then visual SLAM (Simultaneous Localization and Mapping) is performed to generate an image pose map and a dense point cloud map.

[0077] Specifically, based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a synchronous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot. This includes: extracting feature points from the current static scene image and estimating the pose based on the target depth map and the pose of the previous frame of the static scene image to obtain the pose of the current static scene image; generating a single-frame point cloud based on the target depth map, filtering the single-frame point cloud, and fusing it into the dense point cloud map; determining whether the current static scene image is a keyframe, and if so, performing local binding adjustment and optimization based on the current static scene image; detecting whether there is a loop closure based on the current static scene image, and if so, performing pose map optimization based on the current static scene image.

[0078] The tracking thread is responsible for real-time tracking and localization. It extracts feature points from the current image, estimates the initial pose of the current image using pose information from the previous image, performs depth estimation on keyframes using a depth estimation model, generates single-frame point clouds, filters the point clouds, and publishes the data. A local mapping thread constructs and optimizes local maps, selecting representative keyframes from the acquired images and performing local bundle adjustment (BA) optimization to minimize reprojection errors, thus significantly improving map accuracy. A loop closure and map fusion thread constructs and optimizes the global map, ensuring its integrity and consistency. Specifically, it identifies whether the robot has revisited previously visited areas, reducing redundant information and improving map accuracy. Upon detecting a loop closure, the loop closure and map fusion thread further identifies the current location and matches it with known maps. Finally, to further improve the accuracy of the global map, the loop closure and map fusion thread optimizes the global map, significantly enhancing its accuracy and enabling it to more accurately reflect the true structure of the environment.

[0079] In some embodiments, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping (SLT / Map) algorithm based on the target depth map and the static scene image. This includes: estimating the pose of the underwater robot in real time using a SLT / Map algorithm based on the target depth map, the static scene image, and sensor data collected by the target sensor; the target sensor includes an inertial measurement unit (IMU) and / or sonar. That is, in addition to visual localization based on the target depth map and the static scene image, sensors such as IMUs and sonar can also be used.

[0080] As can be seen from the above, this embodiment acquires the original image captured by the camera, preprocesses the original image to obtain a processed image, uses depth estimation to determine the target depth map corresponding to the processed image, locates and removes moving targets from the processed image using a dynamic object detection model, and obtains a static scene image. The dynamic object detection model is a neural network model pre-trained using underwater images. Based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot. It is evident that visual localization using a simultaneous localization and mapping algorithm achieves accurate close-range underwater localization at low cost. Furthermore, preprocessing the acquired original image before subsequent calculations improves image quality and lays the foundation for subsequent visual localization. In addition, considering the special underwater environment, removing moving targets from the image before visual localization avoids interference from underwater fish and other dynamic objects, thus improving the accuracy of visual localization.

[0081] Based on the above embodiments, this application also discloses an underwater robot visual positioning system, which includes: a binocular digital camera for acquiring raw images; a computing unit communicatively connected to the binocular digital camera for implementing the aforementioned underwater robot visual positioning method; and an unmanned remotely operated vehicle host communicatively connected to the computing unit.

[0082] The underwater robot visual positioning system consists of camera hardware and software. The hardware system comprises a binocular camera and a computing unit. After acquiring image data through the camera, the computing unit performs image enhancement, correction, and dynamic target detection to build the foundation for visual positioning and 3D reconstruction.

[0083] The aforementioned computing unit includes: an AI accelerator processor, an image processing unit, and an underwater power supply. The AI ​​accelerator processor is connected to the main unit of the unmanned remotely operated vehicle (UAV) via Ethernet. The AI ​​accelerator processor is also connected to the left and right cameras of the binocular digital camera to receive images captured by the left and right cameras. The image processing unit is connected to the AI ​​accelerator processor, the left camera, and the right camera to receive shooting commands issued by the AI ​​accelerator processor and synchronously trigger the left and right cameras via I / O hard triggering according to the shooting commands. The underwater power supply is connected to the AI ​​accelerator processor to provide power to it.

[0084] For example Figure 4As shown, the system uses digital cameras for its binoculars. Digital cameras offer stable signals and high resolution, and can transmit digital image information to a computer via USB or Gigabit Ethernet, making the system simpler than analog cameras. The binocular camera data interface uses a USB 3.0 interface for transmitting high-resolution images. It connects to the AI ​​accelerator processor (embedded AI application platform) via USB, further simplifying the system; this system uses a USB interface. To ensure simultaneous shooting by both cameras, cameras with hard triggering capabilities are required. Hard triggering is achieved through the same image processing unit, triggering both cameras. An external trigger source triggers the cameras to capture images, achieving higher synchronization accuracy. The I / O ports of Camera 1 and Camera 2 are connected to the I / O ports of the image processing unit, enabling synchronous triggering of the two cameras through I / O hard triggering, ensuring high synchronization accuracy. Camera 1 and Camera 2 are connected to the embedded AI application platform via USB 3.0 (the embedded AI application platform can have 16GB of memory, a 1TSSD card, and up to 100 TOPS of AI computing power). The embedded AI platform connects to an Ethernet switch on the ROV (remotely underwater operated vehicle) via Ethernet to enable communication between the binocular camera and the ROV main unit. The AI ​​application platform is powered by an underwater power supply (DCDC). The binocular camera system is connected to the ROV body via cables.

[0085] Based on the above embodiments, this application also discloses an underwater robot, including a sealed pressure-bearing cavity and the aforementioned underwater robot visual positioning system; the underwater robot visual positioning system includes a binocular digital camera and a computing unit communicatively connected to the binocular digital camera; the underwater robot visual positioning system is located inside the sealed pressure-bearing cavity.

[0086] The left eye camera of the binocular digital camera is located inside the first circular sealed pressure chamber, the right eye camera of the binocular digital camera is located inside the second circular sealed pressure chamber, and the computing unit is located inside the third circular sealed pressure chamber.

[0087] Understandably, since binocular cameras operate in underwater environments, pressure resistance and sealing issues need to be addressed. This application designs a circular cavity to evenly withstand high water pressure, housing the camera lens, control platform, and other components within this sealed circular cavity, adapting to the high-pressure working environment at a depth of 6000 meters. For example... Figure 5 The diagram shows the sealed pressure chamber of a binocular camera, including the aforementioned first circular sealed pressure chamber and second circular sealed pressure chamber. For example... Figure 6 The diagram shows an internal schematic of the first / second circular sealed pressure-bearing cavity, as well as a side view of the circular sealed pressure-bearing cavity. For example... Figure 7 The diagram shows the calculation unit inside the third circular sealed pressure chamber.

[0088] Accordingly, this application also discloses an underwater robot visual positioning device, see [link to relevant documentation]. Figure 8 As shown, the device includes:

[0089] Image preprocessing module 11 is used to acquire the original image captured by the camera and preprocess the original image to obtain the processed image;

[0090] Depth estimation module 12 is used to determine the target depth map corresponding to the processed image using depth estimation;

[0091] The dynamic object detection module 13 is used to locate and remove moving targets from the processed image using a dynamic object detection model to obtain a static scene image; the dynamic object detection model is a neural network model pre-trained using underwater images.

[0092] The visual positioning module 14 is used to estimate the pose of the underwater robot in real time based on the target depth map and the static scene image through a synchronous positioning and mapping algorithm, and to construct a dense point cloud map and update the pose map of the underwater robot.

[0093] As can be seen from the above, this embodiment uses simultaneous positioning and mapping algorithms for visual positioning, achieving accurate underwater close-range positioning at low cost. Furthermore, the acquired raw images are preprocessed before subsequent calculations to improve image quality and lay the foundation for subsequent visual positioning. In addition, for the special underwater environment, moving targets in the images are removed before visual positioning to avoid interference from underwater fish and other dynamic objects that may affect the positioning effect and improve the accuracy of visual positioning.

[0094] In some specific embodiments, the camera is a binocular camera, and the original image includes a binocular image; the depth estimation module 12 includes:

[0095] The monocular depth map determination unit is used to input any monocular image from the original image into the monocular depth estimation system to obtain the monocular depth map corresponding to the original image.

[0096] A binocular depth map determination unit is used to input the binocular image into a binocular depth estimation model to obtain a binocular depth map corresponding to the original image; both the monocular depth estimation system and the binocular depth estimation model are pre-trained using an underwater image dataset.

[0097] The comparison unit is used to compare the monocular depth map and the binocular depth map to determine the target depth map.

[0098] In some specific embodiments, the monocular depth estimation system includes, in sequence, a monocular relative depth estimation model and a monocular absolute depth estimation model, wherein the input of the monocular absolute depth estimation model is the output of the monocular relative depth estimation model;

[0099] The monocular relative depth estimation model is used to output a corresponding relative depth map based on the input monocular image, and the monocular absolute depth estimation model is used to output a corresponding absolute depth map based on the input relative depth map.

[0100] In some specific embodiments, the comparison unit can be used to perform quality analysis on the monocular depth map and the binocular depth map according to the depth map evaluation index, and to use the depth map with higher quality as the target depth map.

[0101] In some specific embodiments, the preprocessing includes image resizing, image enhancement, and image correction.

[0102] In some specific embodiments, the image preprocessing module 11 may specifically include:

[0103] The image correction unit is used to correct the distortion of the original image using the radial and tangential distortion coefficients obtained from camera calibration.

[0104] In some specific embodiments, the visual positioning module 14 may specifically include:

[0105] The pose estimation unit is used to extract feature points from the current static scene image and perform pose estimation based on the target depth map and the pose of the previous frame static scene image to obtain the pose of the current static scene image.

[0106] The point cloud fusion unit is used to generate a single-frame point cloud based on the target depth map, and to filter and fuse the single-frame point cloud into a dense point cloud map.

[0107] The local binding adjustment and optimization unit is used to determine whether the current static scene image is a keyframe. If so, local binding adjustment and optimization is performed based on the current static scene image.

[0108] The pose graph optimization unit is used to detect whether there is a loop closure based on the current static scene image. If so, pose graph optimization is performed based on the current static scene image.

[0109] In some specific embodiments, the visual positioning module 14 may specifically include:

[0110] A visual positioning unit is used to estimate the pose of the underwater robot in real time based on the target depth map, the static scene image, and the sensing data collected by the target sensor, using a synchronous positioning and mapping algorithm; the target sensor includes an inertial measurement unit and / or sonar.

[0111] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the underwater robot visual positioning method steps disclosed in any of the foregoing embodiments.

[0112] Furthermore, this application also discloses a computer program product, including a computer program that, when executed by a processor, implements the underwater robot visual positioning method steps disclosed in any of the foregoing embodiments.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0115] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above provides a detailed description of an underwater robot visual positioning method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A visual positioning method for underwater robots, characterized in that, include: Acquire the raw image captured by the camera, and preprocess the raw image to obtain the processed image; The target depth map corresponding to the processed image is determined using depth estimation. A static scene image is obtained by locating and removing moving targets from the processed image using a dynamic object detection model. The dynamic object detection model is a neural network model that has been trained in advance using underwater images; Based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a synchronous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot. Wherein, the camera is a binocular camera, and the original image includes a binocular image; Determining the target depth map corresponding to the processed image using depth estimation includes: Input any monocular image from the original image into the monocular depth estimation system to obtain the monocular depth map corresponding to the original image; The stereo image is input into the stereo depth estimation model to obtain the stereo depth map corresponding to the original image; both the monocular depth estimation system and the stereo depth estimation model are pre-trained using an underwater image dataset. The target depth map is determined by comparing the monocular depth map and the binocular depth map; Specifically, based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping algorithm, and a dense point cloud map is constructed to update the pose map of the underwater robot, including: Based on the target depth map and the pose of the previous frame static scene image, feature points are extracted from the current static scene image and pose estimation is performed to obtain the pose of the current static scene image. A single-frame point cloud is generated based on the target depth map, and the single-frame point cloud is filtered and then fused into a dense point cloud map. Determine whether the current static scene image is a keyframe; if so, perform local binding adjustment and optimization based on the current static scene image. If a loop is detected in the current static scene image, then pose graph optimization is performed based on the current static scene image.

2. The underwater robot visual positioning method according to claim 1, characterized in that, The monocular depth estimation system includes, in sequence, a monocular relative depth estimation model and a monocular absolute depth estimation model, wherein the input of the monocular absolute depth estimation model is the output of the monocular relative depth estimation model. The monocular relative depth estimation model is used to output a corresponding relative depth map based on the input monocular image, and the monocular absolute depth estimation model is used to output a corresponding absolute depth map based on the input relative depth map.

3. The underwater robot visual positioning method according to claim 1, characterized in that, By comparing the monocular depth map and the binocular depth map, the target depth map is determined, including: The monocular depth map and the binocular depth map are analyzed for quality based on depth map evaluation metrics, and the depth map with higher quality is selected as the target depth map.

4. The underwater robot visual positioning method according to claim 1, characterized in that, The preprocessing includes image resizing, image enhancement, and image correction.

5. The underwater robot visual positioning method according to claim 4, characterized in that, The image correction process for the original image includes: The original image is then subjected to distortion correction using the radial and tangential distortion coefficients obtained from camera calibration.

6. The underwater robot visual positioning method according to any one of claims 1 to 5, characterized in that, Based on the target depth map and the static scene image, the pose of the underwater robot is estimated in real time using a simultaneous localization and mapping algorithm, including: Based on the target depth map and the static scene image, as well as the sensing data collected by the target sensor, the pose of the underwater robot is estimated in real time through a synchronous positioning and mapping algorithm; the target sensor includes an inertial measurement unit and / or sonar.

7. A visual positioning device for an underwater robot, characterized in that, include: The image preprocessing module is used to acquire the original image captured by the camera and preprocess the original image to obtain the processed image; The depth estimation module is used to determine the target depth map corresponding to the processed image using depth estimation; The dynamic object detection module is used to locate and remove moving targets from the processed image using a dynamic object detection model to obtain a static scene image. The dynamic object detection model is a neural network model that has been trained in advance using underwater images; The visual positioning module is used to estimate the pose of the underwater robot in real time based on the target depth map and the static scene image through a synchronous positioning and mapping algorithm, and to construct a dense point cloud map and update the pose map of the underwater robot. Wherein, the camera is a binocular camera, and the original image includes a binocular image; The depth estimation module is used to input any monocular image from the original image into the monocular depth estimation system to obtain a monocular depth map corresponding to the original image; input the binocular image into the binocular depth estimation model to obtain a binocular depth map corresponding to the original image; both the monocular depth estimation system and the binocular depth estimation model are pre-trained using an underwater image dataset; by comparing the monocular depth map and the binocular depth map, the target depth map is determined; The visual positioning module is configured to: extract feature points from the current static scene image and estimate the pose of the current static scene image based on the target depth map and the pose of the previous frame static scene image; generate a single-frame point cloud based on the target depth map; filter the single-frame point cloud and fuse it into a dense point cloud map; determine whether the current static scene image is a keyframe, and if so, perform local binding adjustment optimization based on the current static scene image; detect whether there is a loop closure based on the current static scene image, and if so, perform pose map optimization based on the current static scene image.

8. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the underwater robot visual positioning method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the underwater robot visual positioning method according to any one of claims 1 to 6.

10. An underwater robot visual positioning system, characterized in that, include: A binocular digital camera used to acquire raw images; A computing unit communicatively connected to the binocular digital camera is used to implement the underwater robot visual positioning method as described in any one of claims 1 to 6; The main unit of the unmanned remotely operated vehicle is communicatively connected to the computing unit.

11. The underwater robot visual positioning system according to claim 10, characterized in that, The computing unit includes: an AI acceleration processor, an image processing unit, and an underwater power supply; The AI ​​acceleration processor is connected to the host of the unmanned remotely operated vehicle via Ethernet. The AI ​​acceleration processor is also connected to the left and right cameras of the binocular digital camera to receive images captured by the left and right cameras. The image processing unit is connected to the AI ​​acceleration processor, the left eye camera, and the right eye camera respectively, and is used to receive the shooting instructions issued by the AI ​​acceleration processor, and synchronously trigger the left eye camera and the right eye camera through IO hard trigger according to the shooting instructions; The underwater power source is connected to the AI ​​accelerator processor and is used to supply power to the AI ​​accelerator processor.

12. An underwater robot, characterized in that, It includes a sealed pressure-bearing cavity, and an underwater robot visual positioning system as described in claim 10 or 11; the underwater robot visual positioning system includes a binocular digital camera and a computing unit communicatively connected to the binocular digital camera; The underwater robot's visual positioning system is located inside the sealed pressure chamber.

13. The underwater robot according to claim 12, characterized in that, The left eye camera of the binocular digital camera is located inside the first circular sealed pressure chamber, the right eye camera of the binocular digital camera is located inside the second circular sealed pressure chamber, and the computing unit is located inside the third circular sealed pressure chamber.

Images