Monocular pose estimation method and device for AUV near-end docking based on dense point reconstruction

Abstract

This invention discloses a method and apparatus for AUV near-end docking monocular pose estimation based on dense point reconstruction, comprising: acquiring underwater images and performing image preprocessing; inputting the preprocessed images into a deep learning model for near-end docking pose estimation, the deep learning model including a target detection module for detecting targets in the preprocessed images and generating target image patches; a target dense point reconstruction module inside the dock using an encoder-decoder network to extract features from the target image patches and reconstruct the dense point coordinates of the target; a normal vector supervision module for recovering the surface normal vector map of the object from the target image patches; and an Edge-PnP pose estimation module using a graph convolutional network to learn pose features from the dense point coordinates of the target and the surface normal vector map of the object and estimate the final 6D pose. This solves the problem of key point loss and inability to perform calculations in traditional methods.

Description

Technical Field

[0001] This application relates to the field of AUV retrieval and docking technology, and in particular to a monocular pose estimation method and apparatus for near-end docking of AUVs based on dense point reconstruction. Background Technology

[0002] An Autonomous Underwater Vehicle (AUV) is a self-operating underwater vehicle. Over the past few decades, with continuous technological advancements, AUVs have become crucial tools for underwater surveying, exploration, and monitoring. The emergence of underwater support facilities such as fixed underwater docks and large mobile underwater docks has provided more comprehensive support for AUV operations. Docking with a fixed underwater dock provides basic services to the AUV, such as charging, data transmission, and mission commands. During long-duration or large-scale data acquisition missions, the AUV can periodically dock with a fixed underwater dock to ensure sufficient and secure energy and data. Docking with large mobile underwater docks allows for faster and safer deployment and recovery of AUVs, improving mission efficiency and success rates.

[0003] Currently, underwater AUV visual guidance methods typically employ a passive optical guidance scheme using light arrays. This scheme uses an array of underwater guide lights as the underwater target and extracts the pixel positions of the guide lights in the image using visual methods, combining this with prior 3D coordinate information to calculate the relative pose between the AUV and the light array. However, existing light array guidance schemes have the following shortcomings: 1. Interference from underwater floating objects and water scattering can easily lead to errors in the visual method's extraction of the guide light's center; 2. The guide lights themselves have limited features and can only be used in dark deep-sea environments. When other light sources are present, it can easily lead to errors in identifying the key points at the center of the guide lights; 3. When some guide lights in the array are outside the camera's field of view, it can easily lead to a reduction or even loss of key points, making it impossible to calculate the relative pose. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for AUV near-end docking monocular pose estimation based on dense point reconstruction.

[0005] According to a first aspect of the present invention, a method for AUV near-end docking monocular pose estimation based on dense point reconstruction is provided, comprising:

[0006] S11: Acquire underwater images containing dock station targets using the AUV's built-in monocular camera;

[0007] S12: Perform image preprocessing on the underwater image;

[0008] S13: The preprocessed image is input into a deep learning model for near-end docking pose estimation for pose estimation. This deep learning model includes a target detection module, a docking station internal target dense point reconstruction module, a normal vector supervision module, and a graph convolution-based Edge-PnP pose estimation module. The target detection module detects targets in the preprocessed image and generates target image patches. The docking station internal target dense point reconstruction module uses an encoder-decoder network to extract features from the target image patches and reconstruct the dense point coordinates of the targets. The normal vector supervision module recovers the surface normal vector map of the object from the target image patches. The Edge-PnP pose estimation module uses a graph convolutional network to learn pose features from the dense point coordinates of the targets and the surface normal vector map of the objects, and estimates the final 6D pose.

[0009] Optionally, the underwater image may undergo image preprocessing, including:

[0010] Distortion correction is performed on the underwater image to make it present the correct shape, and then noise reduction is performed to obtain a clear underwater image.

[0011] Optionally, the target object detection module is used to detect target objects in the preprocessed image and generate target object image patches, including:

[0012] The YOLO target detector is used to detect the preprocessed underwater image and crop out target image blocks of a fixed size.

[0013] Optionally, the target dense point reconstruction module inside the dock is used to extract features from the target image patch using an encoder-decoder network and reconstruct the dense point coordinates of the target, including:

[0014] The target object image patch is extracted using an encoding network, the extracted features are decoded, and the coordinates of dense points are supplemented by multi-scale intermediate internal spatial features generated during the decoding and reconstruction process.

[0015] Optionally, the normal vector supervision module is used to recover the surface normal vector map of the object from the target object image patch, including:

[0016] Features are extracted from the target image patch using an encoding network, and the surface normal vector of the object is recovered using the extracted features. During the recovery process, the mask of the target image patch is also recovered.

[0017] Optionally, the Edge-PnP pose estimation module utilizes a graph convolutional network to learn pose features from the dense point coordinates of the target object and the surface normal vector map of the object, and estimates the final 6D pose, including:

[0018] The dense 3D point coordinates of the reconstructed target and the surface normal vector map of the object are combined with the 2D pixel coordinates and the multi-scale intermediate internal space features and input into the Edge-PnP pose estimation module to extract features.

[0019] Point-by-point features are extracted using point-edge convolution, and finally, a fully connected layer is used to map the output to a 6D dimension, thus estimating the 6D pose information.

[0020] The 6D pose information refers to the position and orientation of the target object relative to the monocular camera. It is represented as a rotation matrix and a translation vector. By using a fixed difference between the camera and the AUV's own coordinate system, the position and orientation are converted into the pose result of the target object relative to the AUV.

[0021] According to a first aspect of the present invention, an AUV near-end docking monocular pose estimation device based on dense point reconstruction is provided, comprising:

[0022] The acquisition unit is used to acquire underwater images containing dock station targets using the built-in monocular camera of the AUV;

[0023] The preprocessing unit is used to perform image preprocessing on the underwater image;

[0024] The pose estimation unit is used to input the preprocessed image into a deep learning model for near-end docking pose estimation. This deep learning model includes a target detection module, a docking station interior target dense point reconstruction module, a normal vector supervision module, and a graph convolution-based Edge-PnP pose estimation module. The target detection module detects targets in the preprocessed image and generates target image patches. The docking station interior target dense point reconstruction module uses an encoder-decoder network to extract features from the target image patches and reconstruct the dense point coordinates of the targets. The normal vector supervision module recovers the surface normal vector map of the object from the target image patches. The Edge-PnP pose estimation module uses a graph convolutional network to learn pose features from the dense point coordinates of the targets and the surface normal vector map of the objects, and estimates the final 6D pose.

[0025] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0026] One or more processors;

[0027] Memory, used to store one or more programs;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0029] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0030] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0031] The preprocessed image is input into a deep learning model for near-end docking pose estimation, achieving end-to-end estimation, which greatly improves estimation efficiency and significantly enhances timeliness. By constructing a dense point reconstruction module for targets inside the dock, the dense texture information of the input image is fully utilized to overcome the problem of key point loss leading to the inability to solve in traditional methods. The Edge-PnP pose estimation module based on graph convolution fully utilizes the point-by-point feature regression of dense points, surface normal vector maps, and intermediate internal spatial features to achieve 6D pose, which is information-rich and highly accurate.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] Figure 1 This is a flowchart illustrating an AUV near-end docking pose estimation method based on dense point reconstruction, according to an exemplary embodiment.

[0035] Figure 2 This is a schematic diagram of a deep learning model for near-end docking pose estimation according to an exemplary embodiment.

[0036] Figure 3 This is a schematic diagram of a target density point reconstruction module inside a docking station, according to an exemplary embodiment.

[0037] Figure 4 This is a schematic diagram of an Edge-PnP pose estimation module according to an exemplary embodiment.

[0038] Figure 5 This is a schematic diagram of a point-edge convolutional graph convolution block according to an exemplary embodiment.

[0039] Figure 6 This is a block diagram illustrating an AUV near-end docking pose estimation device based on dense point reconstruction according to an exemplary embodiment. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0041] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0043] Please see Figure 1 , Figure 1 This is a flowchart illustrating an AUV near-end docking pose estimation method based on dense point reconstruction according to an exemplary embodiment. Specifically, it may include the following steps:

[0044] Step S11: Acquire underwater images containing dock station targets using the AUV's built-in monocular camera.

[0045] The sample images can be real images or synthetic images. In some application scenarios, the sample images may include a portion of real images and a portion of synthetic images. When the sample images are real images, the method of obtaining sample images containing the underwater docking station target with the pose to be estimated is to use an AUV to capture images of the underwater docking station target with the pose to be estimated.

[0046] Step S12: Perform image preprocessing on the underwater image.

[0047] It mainly includes two parts: image distortion correction and image denoising.

[0048] Image distortion correction aims to eliminate geometric distortions in digital images caused by cameras or other factors, restoring the image to a normal state. Image distortion can be mainly divided into radial distortion and tangential distortion. Radial distortion is distortion distributed along the radius of the camera lens, caused by light rays bending more far from the center of the lens than near the center. It can be described by the first few terms of the Taylor series expansion around the principal point:

[0049]

[0050]

[0051] in Indicates the pixel position on the distorted original image. Indicates the corrected pixel position. This is the distortion coefficient, and r is the normalized distance from the point coordinate to the principal point. Tangential distortion, caused by the lens itself not being parallel to the camera sensor plane, is usually described by the following mathematical expression:

[0052]

[0053]

[0054] in These are the distortion coefficients. A checkerboard calibration image was used as the calibration object, and the camera's distortion parameters were estimated by detecting the corner positions in the image. These distortion parameters were then used to perform anti-distortion processing on each pixel, resulting in the corrected image.

[0055] The purpose of underwater image denoising is to improve the clarity and visibility of underwater images, facilitating subsequent analysis and understanding by the system. This method uses median filtering for visually guided image denoising. For each pixel in the image, the median value of its neighboring pixels is used to replace the original pixel value. Assuming the target pixel value before filtering is... After filtering, it becomes By calculating the neighborhood The median pixel replacing the original target pixel can be expressed by the following formula:

[0056]

[0057] The image that has undergone the above image preprocessing is input into the deep learning model for near-end docking pose estimation and then enters the pose estimation part guided by near-end vision.

[0058] Underwater image preprocessing aims to improve the quality and clarity of underwater images, thereby enhancing the accuracy and robustness of underwater 6D pose estimation. It can eliminate or reduce the impact of factors such as light attenuation, scattering, and color distortion in the underwater environment on the image, making feature points in the image more prominent and easier to identify.

[0059] Step S13: Input the preprocessed image into the deep learning model for near-end docking pose estimation for pose estimation. The deep learning model for near-end docking pose estimation includes a target detection module, a docking station internal target dense point reconstruction module, a normal vector supervision module, and a graph convolution-based Edge-PnP pose estimation module. The target detection module is used to detect targets in the preprocessed image and generate target image patches. The docking station internal target dense point reconstruction module is used to extract features from the target image patches and reconstruct the dense point coordinates of the target using an encoder-decoder network. The normal vector supervision module is used to recover the surface normal vector map of the object from the target image patches. The Edge-PnP pose estimation module uses a graph convolution network to learn pose features from the dense point coordinates of the target and the surface normal vector map of the object and estimate the final 6D pose.

[0060] Specifically, the various modules in the deep learning model for near-end docking pose estimation are elaborated here.

[0061] The target object detection module is used to detect target objects in the preprocessed image and generate target object image patches, including:

[0062] The YOLO target detector is used to detect the preprocessed underwater image and crop out target image blocks of a fixed size.

[0063] Specifically, please refer to Figure 2 , Figure 2 This is a schematic diagram of a deep learning model for near-end docking pose estimation according to an exemplary embodiment. The first step of the deep learning model for near-end docking pose estimation is to use the YOLO object detector to detect the preprocessed image and crop out a target image patch of a fixed size.

[0064] After obtaining the preprocessed clear underwater image, the target image patch of the target region is cropped by the target object detection module. The image size has changed. The position of this image patch in the original image is calculated based on the global image information of the original image. and image patch size .

[0065] The following three variables are used in the regression of 3D offset in 6D pose information: , This represents the offset from the center of the detection box of the target object to the center of the real target object. It is not an absolute offset, but rather a relative offset that the network is trained to predict. It is the depth of the scaling.

[0066]

[0067] in and It is the center of the target object in the target object image block and the center of the target object image block. It is the size of the clear image after preprocessing. It's the scaling ratio.

[0068] The dense point reconstruction module for targets inside the dock is used to extract features from the target image patch using an encoder-decoder network and reconstruct the coordinates of the dense points of the target, including:

[0069] The target object image patch is extracted using an encoding network, the extracted features are decoded, and the coordinates of dense points are supplemented by multi-scale intermediate internal spatial features generated during the decoding and reconstruction process.

[0070] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of a dense point reconstruction module for a target object inside a docking station, according to an exemplary embodiment. A coding network is used to extract hierarchical features from the target object image block. The extracted features are then decoded to obtain multi-scale intermediate internal spatial features generated during the decoding process. These multi-scale intermediate internal spatial features are used to construct feature blocks, which are then used to collectively supervise the generation of the dense point coordinates of the target object.

[0071] The module for reconstructing dense points of targets inside the docking station uses implicit features extracted from the encoder backbone network: , and As input, the output is the reconstructed dense 3D coordinate points. Mask image and surface patch diagram Output 3D coordinate points in multi-scale reconstruction. During the process, intermediate feature maps of different resolutions and scales, ranging from blurry to clear, were generated during decoding. , The multi-scale reconstruction and fusion module merges and generates a clearer and more accurate dense coordinate point map of the target object. .

[0072] Real object surface patch diagram It is derived from a dense coordinate point map of the real target object by using farthest point sampling. The dense coordinate point map of the real target object is its coordinate representation in a normalized object coordinate space (NOCS). A normalized object coordinate space (NOCS) allows different objects to have a common reference frame. It is a way of describing objects in a world coordinate system. NOCS means that an object is defined in a 3D space containing unit cubes, and the object space coordinates... The dense coordinates of the actual target object are its NOCS (Noise, Color, and Dimensions).

[0073] Atrous Spatial Pyramid Pooling (ASPP) is used as a context extractor. and As input, an intermediate latent representation is generated. In addition, by get As another intermediate representation.

[0074] This is followed by a multi-layered framework consisting of four branches. The outputs of the top three branches are used to guide the final target object's dense point coordinate map. The internal multi-scale intermediate spatial features output by these three branches can be used as jump connections to build the relationship between these internal features and the reconstructed dense graph during the decoding process.

[0075] Branch 1 contains seven convolutional layers, which restrict the output dimension to 4, generating a low-resolution feature layer. To approximate the object's mask and dense 3D point coordinates, a low-resolution layer... Upsampled as multi-scale intermediate interior spatial features .

[0076] The second branch consists of six convolutional layers and nonlinear layers, which... , , The combined inputs are used to create a medium-resolution feature layer. Similarly, this stage outputs multi-scale intermediate interior spatial features. From Export from [source].

[0077] The third branch consists of an upsampling layer and two convolutional layers, which generate a high-resolution feature layer. . The first four of the 69 channels are used to construct another multi-scale intermediate interior spatial feature (spatial supervision). The remaining 65 channels constitute the surface patch diagram. .

[0078] Branch 4 is constructed directly from upsampling layers and convolutional layers to obtain global feature cues. .

[0079] These multi-scale intermediate interior spatial features (or so-called multi-scale spatial supervision) and the final reconstructed dense point coordinates of the target object With identical spatial dimensions, these multi-scale intermediate interior spatial features of different scales can characterize object details from coarse outlines to detailed surface features. The mask map, reconstructed dense point coordinates, and surface patch map are jointly generated in the following manner:

[0080]

[0081] in This represents the corresponding convolution transformation. Characterize the activation function, Representation of the characteristic matrix The Channels. Regarding the design of the loss function, These are the coordinates of the reconstructed dense points. and These are the estimated mask image and the estimated surface patch image.

[0082] The above scheme utilizes multi-scale intermediate internal spatial features to form feature blocks during the decoding process, which jointly supervise the generation of dense point coordinates of the target object, thus obtaining dense point coordinates of the 3D target object with higher accuracy.

[0083] The normal vector supervision module is used to recover the surface normal vector map of the object from the target object image block, including:

[0084] Features are extracted from the target image patch using an encoding network, and the surface normal vector of the object is recovered using the extracted features. During the recovery process, the mask of the target image patch is also recovered.

[0085] Specifically, a normal vector supervision module is used to recover the surface normal vectors of the object, because it is difficult to recover rich surface information from images of textureless objects. An encoding network is used to extract features from the target object image patch, and in the decoding network, the extracted features are used to recover the object surface normal vector map. During the recovery process, the mask of the target image patch is also recovered.

[0086] Among them, the generated surface normal map of the visible object can represent finer-grained details of the object's surface. This is achieved by utilizing the surface normal map of the target object. Encode the geometric details of the object's frontal visible area.

[0087] Obtain the true 2D surface normal vector map from the Normalized Object Coordinate Space (NOCS). The directional derivatives of the NOCS plot along the x-axis and y-axis are represented using the Sobel operator: , Pixels The normal vector at point is formulated as follows:

[0088]

[0089] This represents the cross product. This operation produces singular values ​​at the edges of objects. An erosion algorithm is used to filter out these boundary outliers and generate an erosion mask. To divide objects.

[0090] The Edge-PnP pose estimation module uses a graph convolutional network to learn pose features from the dense point coordinates of the target object and the surface normal vector map of the object, and estimates the final 6D pose, including:

[0091] The dense 3D point coordinates of the reconstructed target and the surface normal vector map of the object are combined with the 2D pixel coordinates and the multi-scale intermediate internal space features and input into the Edge-PnP pose estimation module to extract features.

[0092] Point-by-point features are extracted using point-edge convolution, and finally, a fully connected layer is used to map the output to a 6D dimension, thus estimating the 6D pose information.

[0093] The 6D pose information refers to the position and orientation of the target object relative to the monocular camera. It is represented as a rotation matrix and a translation vector. By using a fixed difference between the camera and the AUV's own coordinate system, the position and orientation are converted into the pose result of the target object relative to the AUV.

[0094] Specifically, please refer to Figure 4 , Figure 4 This is a schematic diagram of an Edge-PnP pose estimation module according to an exemplary embodiment.

[0095] A point-edge convolutional map convolutional block is used to extract point-by-point feature information from the dense 3D point coordinates of the reconstructed target object and the surface normal vector map of the object, combined with 2D pixel coordinates; another point-edge convolutional map convolutional block is used to extract point-by-point feature information from the multi-scale intermediate internal spatial features combined with 2D pixel coordinates; finally, a fully connected layer is used to map to a 6D dimension output, that is, to estimate the 6D pose information.

[0096] Among these features, the multi-scale intermediate interior spatial features can not only serve as spatial guides for generating dense 3D point coordinates of the target object, but also simplify the pose estimation process together with the generated dense 3D point coordinates of the target object. Since these multi-scale features characterize the details and state of the object from coarse shape to precise coordinates, they help to roughly identify the pose before fine pose estimation, forming a pre-estimation process.

[0097] Please see Figure 5 , Figure 5 This is a schematic diagram of a point-edge convolutional graph convolution block according to an exemplary embodiment.

[0098] For the input multi-scale geometrically supervised feature map , utilizing F = 78-dimensional points Characteristic point map.

[0099] Compute a directed graph representing the local point cloud structure. ,in and These are the vertex set and the edge set, respectively. G is constructed as... k nearest neighbors picture.

[0100] The graph includes a self-loop, meaning that each node also points to itself.

[0101] Edge features are defined as ,in It is a set of learnable parameters Nonlinear functions.

[0102] The edge convolution operation is defined by applying channel-symmetric aggregation to the edge features associated with all edges originating from each vertex (using summation or maximum maximization):

[0103]

[0104] Will Treat it as the center pixel, Consider it as a nearest neighbor block surrounding it. Given a set of n points... A 3D point cloud is generated by edge convolution, which produces a matrix with the same number of points. Dimensional point cloud.

[0105] Represents image pixels on a regular grid, and the image... It has connectivity representing a fixed-size image patch around each pixel.

[0106] h is defined as

[0107]

[0108] This will be explicitly determined by the coordinates of the center of the image patch. The captured global shape structure and by , The captured local neighborhood information is combined. The operator is defined as follows:

[0109]

[0110] Implemented through a shared MLP, at the same time

[0111]

[0112] here .

[0113] Calculate a pairwise distance matrix in the feature space, and then select the k closest points for each point.

[0114] The entire Edge-PnP pose estimation module is defined and updated through the following implementation, and finally a fully connected (FC) network is used to estimate the 6D pose result.

[0115] Define symbols: Let represent a point cloud, where n represents the number of points, F represents the dimension of the feature vector, and k represents the number of neighbors. For each point... Find its k nearest neighbors and construct a kNN graph G.

[0116] For each edge Calculate its eigenvectors ,in and It is a feature vector of two endpoints. This feature vector can capture the relative position and direction between the two points, as well as information about the two points themselves.

[0117] For each point Concatenate it with the eigenvectors of all its neighbors to obtain a matrix. ,in It is all with The eigenvectors of connected edges.

[0118] For each point For its matrix By applying an h function and implementing it through a multilayer perceptron (MLP), a new feature vector is obtained. MLP is a fully connected neural network that can learn nonlinear transformations.

[0119] For each point Max-pooling is performed on the new feature vectors of all its neighbors to obtain an aggregated feature vector. Max pooling is a dimensionality reduction operation that can extract the most salient features.

[0120] For each point It concatenates its own new feature vector with the aggregated feature vector to obtain a final feature vector. This allows us to retain some information about ourselves and our neighborhood.

[0121] Therefore, by using point-edge convolutional graph convolutional blocks, which can efficiently process point data, to process the aforementioned features, the method's perception of features can be greatly improved, thereby increasing prediction accuracy.

[0122] The preprocessed image is then used in the end-to-end pose estimation network to directly regress 6D pose information. To train this end-to-end network, the following loss function is used:

[0123]

[0124] in It is a 6D pose loss. The ground truth 6D poses during training are obtained using a self-made dataset.

[0125] The following three variables are used in the regression of 3D offset in 6D pose information: , This represents the offset from the center of the detection box of the target object to the center of the real target object. It is not an absolute offset, but rather a relative offset that the network is trained to predict. It is the scaling depth.

[0126]

[0127] The 3D offset in the predicted 6D pose can be recovered using the following method:

[0128]

[0129] in It is the focal length of the monocular camera.

[0130] Reconstruction losses include: using Loss to monitor the coordinates of dense points of the target object in the reconstruction and segmentation mask and using cross-entropy loss To supervise the surface patch map .

[0131]

[0132] Only the pixel coordinates of the target object that are visible within the target object image block are considered. The pixels representing the dense coordinates of the real target object, while The pixels representing the dense coordinate map of the reconstructed target object, and Represents the geometry of all pixels of the target object. These represent the pixels of the real mask and the pixels of the predicted mask, respectively.

[0133] For the surface normal map, cosine distance is used to measure the difference between the predicted surface normal map and the corresponding true target surface normal map, and the erosion mask map is used... loss.

[0134]

[0135] in Do not represent coordinates The predicted surface normal vector map and the actual surface normal vector map of the target object.

[0136] Ultimately, what is returned is the 6D pose information. The 6D pose information refers to the position and orientation of the target object relative to the camera, represented as a rotation matrix and a translation vector. By using a fixed difference between the camera's coordinate system and the AUV's own coordinate system, the position and orientation are converted into the pose result of the target object relative to the AUV.

[0137] Corresponding to the aforementioned embodiment of an AUV near-end docking pose estimation method based on dense point reconstruction, this application also provides an embodiment of an AUV near-end docking pose estimation device based on dense point reconstruction.

[0138] Figure 6 This is a block diagram of an AUV near-end docking pose estimation method based on dense point reconstruction, according to an exemplary embodiment. (Refer to...) Figure 6 The device includes

[0139] Acquisition unit 11 is used to acquire underwater images containing dock station targets through the built-in monocular camera of the AUV;

[0140] Preprocessing unit 12 is used to perform image preprocessing on the underwater image;

[0141] The pose estimation unit 13 is used to input the preprocessed image into a deep learning model for near-end docking pose estimation for pose estimation. The deep learning model for near-end docking pose estimation includes a target detection module, a docking station internal target dense point reconstruction module, a normal vector supervision module, and a graph convolution-based Edge-PnP pose estimation module. The target detection module detects targets in the preprocessed image and generates target image patches. The docking station internal target dense point reconstruction module uses an encoder-decoder network to extract features from the target image patches and reconstruct the dense point coordinates of the targets. The normal vector supervision module recovers the surface normal vector map of the object from the target image patches. The Edge-PnP pose estimation module uses a graph convolution network to learn pose features from the dense point coordinates of the targets and the surface normal vector map of the objects, and estimates the final 6D pose.

[0142] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0143] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the AUV near-end docking pose estimation method based on dense point reconstruction as described above.

[0145] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method for AUV near-end docking pose estimation based on dense point reconstruction.

[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. An AUV near-end docking monocular pose estimation method based on dense point reconstruction, characterized in that, The method comprises the following steps: obtaining an underwater image containing a target object of a docking station through a built-in monocular camera of an AUV; performing image preprocessing on the underwater image; inputting the preprocessed image into a deep learning model for pose estimation of a near-end docking station, wherein the deep learning model for pose estimation of the near-end docking station comprises a target object detection module, a target object dense point reconstruction module inside the docking station, a normal vector supervision module, and an Edge-PnP pose estimation module based on graph convolution, the target object detection module is used for detecting a target object in the preprocessed image and generating a target object image block, the target object dense point reconstruction module is used for extracting features of the target object image block and reconstructing dense point coordinates of the target object by using an encoding-decoding network, the normal vector supervision module is used for recovering a surface normal vector map of the object from the target object image block, and the Edge-PnP pose estimation module learns pose features from the dense point coordinates of the target object and the surface normal vector map of the object by using a graph convolution network and estimates a final 6D pose. The image preprocessing on the underwater image comprises image distortion correction and image denoising. The image distortion is divided into radial distortion and tangential distortion, wherein the radial distortion is described by the following Taylor series expansion formula around the principal point: ； ； wherein denotes the pixel position on the original image representing the distortion, denotes the corrected pixel position, is the distortion coefficient and r is the distance of the normalized point coordinate to the principal point; The tangential distortion is described by the following mathematical expression: ； ； wherein is the distortion coefficient; the chessboard calibration image is used as the calibration object, the corner point position in the image is detected, and the distortion parameters of the camera are estimated; the inverse distortion processing is performed on each pixel point by using the distortion parameters, and thus the corrected image is obtained; The first step of the deep learning model for pose estimation of the near-end docking station is to detect the preprocessed image by using a target detector YOLO and cut out a target object image block with a fixed size. After obtaining the pre-processed clear underwater image, the target object image block of the target region is cut out by the target object detection module, the image size is changed, and the position of the image block in the global image information of the original image, i.e. in the original image, is calculated and the image block size ; In the 6D pose information, the 3D offset regression is about the following three variables: , represents the offset from the center of the detection box of the target object to the center of the real target object, which is not an absolute offset, but the network is trained to predict a relative offset; is the scaled depth; ； wherein and is the center of the target object in the target object image block and the center of the target object image block; is the size of the clear image after the preprocessing, is the scaling ratio.

2. The method of claim 1, wherein, The image preprocessing on the underwater image comprises: The underwater image is corrected for distortion to present a correct shape and then is denoised to obtain a clear underwater image.

3. The method of claim 1, wherein, The target object detection module is used for detecting a target object in the preprocessed image and generating a target object image block, comprising: The preprocessed underwater image is detected by using a target detector YOLO, and a target object image block with a fixed size is cut out.

4. The method of claim 1, wherein, The target object dense point reconstruction module extracts features of the target object image block by using an encoding-decoding network and reconstructs dense point coordinates of the target object, comprising: The features of the target object image block are extracted by using an encoding network, and the extracted features are decoded, and the dense point coordinates are generated by using the multi-scale intermediate internal space features generated in the decoding reconstruction process.

5. The method of claim 4, wherein, The normal vector supervision module is used for recovering a surface normal vector map of the object from the target object image block, comprising: The features of the target object image block are extracted by using an encoding network, and the surface normal vector of the object is recovered by using the extracted features, and the mask of the target image block is recovered in the recovery process.

6. The method of claim 5, wherein, The Edge-PnP pose estimation module learns pose features from the dense point coordinates of the target object and the surface normal vector map of the object by using a graph convolution network and estimates a final 6D pose, comprising: For the reconstructed target object dense 3D point coordinates and the object surface normal vector map, combined with 2D pixel point coordinates and the multi-scale intermediate internal space features, are input into an Edge-PnP pose estimation module to extract features; Point edge convolution is used to extract point-by-point features, and finally a fully connected layer is used to map to a 6D dimension output, i.e. to estimate 6D pose information; The 6D pose information is the position and attitude of the target object relative to the monocular camera, and the representation form is a rotation matrix and a translation vector. Through the fixed difference value of the camera and the AUV itself coordinate system, the position and attitude are converted into the pose result of the target object relative to the AUV.

7. An AUV near-end docking monocular pose estimation device based on dense point reconstruction, characterized in that, Comprise: An acquisition unit configured to acquire an underwater image containing a target object of a docking station by a built-in monocular camera of an AUV; A preprocessing unit configured to perform image preprocessing on the underwater image; A pose estimation unit configured to input the preprocessed image into a deep learning model for near-end docking pose estimation, wherein the deep learning model for near-end docking pose estimation comprises a target object detection module, a docking station internal target object dense point reconstruction module, a normal vector supervision module, and an Edge-PnP pose estimation module based on graph convolution, the target object detection module is configured to detect the target object in the preprocessed image and generate a target object image block; the docking station internal target object dense point reconstruction module is configured to extract features from the target object image block using an encoding-decoding network and reconstruct the dense point coordinates of the target object; the normal vector supervision module is configured to recover the surface normal vector map of the object from the target object image block; the Edge-PnP pose estimation module learns pose features from the dense point coordinates of the target object and the surface normal vector map of the object using a graph convolution network and estimates the final 6D pose; Wherein, the image preprocessing on the underwater image comprises image distortion correction and image denoising; The image distortion is divided into radial distortion and tangential distortion, wherein the radial distortion is described by the first few terms of the Taylor series expansion around the principal point as follows: ； ； wherein denotes the pixel position on the original image representing the distortion, denotes the corrected pixel position, is the distortion coefficient and r is the distance of the normalized point coordinate to the principal point; The tangential distortion is described by the following mathematical expression: ； ； wherein is the distortion coefficient; the chessboard calibration image is used as the calibration object, the corner point position in the image is detected, and the distortion parameters of the camera are estimated; the inverse distortion processing is performed on each pixel point by using the distortion parameters, and thus the corrected image is obtained; The first step of the deep learning model for near-end docking pose estimation is to detect the preprocessed image using a target detector YOLO and to crop a target object image block of a fixed size; After obtaining the pre-processed clear underwater image, the target object image block of the target region is cut out by the target object detection module, the image size is changed, and the position of the image block in the global image information of the original image, i.e. in the original image, is calculated and the image block size ; In the 6D pose information, the 3D offset regression is about the following three variables: , represents the offset from the center of the detection box of the target object to the center of the real target object, which is not to the absolute offset, but to train the network to predict the relative offset; is the scaled depth; ； wherein and is the center of the target object in the target object image block and the center of the target object image block; is the size of the clear image after the preprocessing, is the scaling ratio.

8. An electronic device, comprising: Comprise: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1-6.

Images