A monocular vision guidance method for underwater robot docking based on key features

Abstract

The present invention discloses a real-time, accurate, and monocular visual guidance method for underwater robot docking based on active identification of key features. The method consists of two parts: far-end visual guidance and near-end visual guidance. When the vision acts at the far end, the key points of the external mechanical device of the docking station are detected. When the vision acts at the near end, the position of the key points is inferred by predicting the key features of the target object inside the docking station. The present invention designs a far-end visual guidance scheme, adopts a target detection network to detect the pixel positions of the key points of the docking station mechanical device, and uses a two-step pose estimation method to improve the accuracy of pose solution; designs a near-end visual guidance scheme, adopts the dense vector of the target area as the key feature, and uses a two-step key point position estimation method based on vector voting and vector projection to improve the accuracy of the key point pixel position, and further uses a two-step pose estimation method to improve the pose solution accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater robot recovery and docking, and in particular relates to a monocular vision guidance method for underwater robot docking based on key features. Background Art

[0002] When performing underwater exploration missions, large underwater mobile platforms are often restricted by terrain due to their bulky structures and complex motion characteristics. Underwater robots (UAVs) on mobile platforms often replace mobile platforms for underwater operations due to their superior concealment and high flexibility. Offshore fixed platforms also often use small UAVs for underwater exploration and mining operations in nearby waters. These platforms are equipped with fixed static recovery devices to house the UAVs. Whether docking with a mobile platform or a fixed device, UAVs rely on their own batteries for power. Underwater recovery docking technology enables robots to autonomously enter the recovery platform after extended underwater operations to complete battery charging and information exchange. Visual guidance is a key stage in recovery docking. Visual methods are often used to obtain the relative position and posture of the robot and the target. Autonomous underwater docking is achieved through navigation, path planning, and control modules.

[0003] Current robot vision guidance methods typically employ passive optical guidance using a light array. This approach uses an array of underwater guide lights as the underwater target. Visual methods are used to extract the pixel positions of the guide lights in the image, and the relative position between the robot and the light array is calculated using a priori 3D coordinate information. In developing the present invention, the inventors discovered that the prior art suffers from at least the following problems:

[0004] 1. Interference from underwater floating objects and scattering from the water easily lead to errors in visual methods for extracting the center of the guide light. 2. The guide light itself has few characteristics and can only be used in deep, dark environments. When there is interference from other light sources, it is easy to cause errors in the identification of the key point at the center of the guide light. 3. When the guide light is out of the camera's field of view, it is easy to lose the key point and cannot solve the relative pose. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the embodiments of the present application is to provide a monocular vision guidance method for underwater robot docking based on key features.

[0006] According to a first aspect of an embodiment of the present application, a monocular vision guidance method for underwater robot docking based on key features is provided, comprising:

[0007] Acquire underwater images and perform preprocessing;

[0008] The pre-processed underwater image is input into the proximal vision-guided deep learning model PVNet to detect the total number of image pixels occupied by the target objects inside the dock;

[0009] If the total value is less than or equal to a predetermined threshold, the current stage is remote vision guidance. The preprocessed underwater image is input into the trained key point detection network to obtain the position deviation of the external key point of the docking station from the docking station center, and the pixel 2D position of the external key point of the docking station is calculated; the pixel 2D position of the external key point of the docking station is combined with its prior world 3D coordinate to form a point pair mapping; based on the point pair mapping, a two-step pose estimation method is used to obtain the posture and position of the docking station center relative to the robot;

[0010] If the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, and the preprocessed underwater image is input into the trained key feature detection network to obtain a regional mask map and a unit vector feature map of the target object, and the unit vector key features of the target area in the unit vector feature map are extracted through the regional mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the way of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation;

[0011] According to the position of the docking station center relative to the robot, the path of the underwater robot is planned and control instructions are generated according to the path to guide the underwater robot.

[0012] Furthermore, the training process of the key point detection network includes:

[0013] Obtaining a first training set, wherein the first training set includes a plurality of images with external key points and a center point identification of the docking station and a true value of the key point position corresponding to each image;

[0014] According to the first training set, the key point detection network is trained using a first loss function, wherein the first loss function Loss far for

[0015] Loss far =λ cls Loss cls +λ size Loss size

[0016]

[0017]

[0018] Among them, Loss cls Refers to the category confidence loss, with the pixel 2D coordinate of the center point of the real docking station as the positive sample Y xyc ,and Loss is the category confidence value predicted by the key point detection network at the (x, y, c) position; size is the key point deviation loss; N is the total number of positive samples; α and β are temperature coefficients used to adjust the ratio of training difficulty; is the predicted position deviation; is the actual position deviation.

[0019] Furthermore, the process of obtaining the first training set includes:

[0020] Labelme software was used to manually mark the external corners and center points of the docking station in the collected underwater images;

[0021] The marked underwater image is represented by the category point with the center point as the docking station, and the deviation of the corner point relative to the center point is calculated as the true value of the key point position of the docking station.

[0022] Furthermore, the pre-processed underwater image is input into the trained key point detection network to obtain the position deviation of the key points outside the docking station from the docking station center, including:

[0023] The preprocessed underwater image is input into the trained key point detection network to obtain the category confidence map and key point deviation map;

[0024] The point with the maximum score is selected in the category confidence map, which is the pixel 2D coordinate of the docking station center point. The pixel 2D coordinate is used to extract the deviation of each key point at the same position in the key point deviation map. The pixel 2D coordinate of each key point is calculated by the center point coordinate and the key point deviation.

[0025] Furthermore, the training process of the key point detection network includes:

[0026] Acquire a second training set, wherein the second training set comprises a plurality of images with true values ​​of the posture and position of the target object relative to the robot;

[0027] According to the second training set, the key point detection network is trained using a second loss function, wherein the second loss function Loss is

[0028] Loss=Loss vertex+Loss mask

[0029]

[0030]

[0031]

[0032] Where, Loss vertex represents the unit vector graph loss, p i ∈O means that only the values ​​in the target area are calculated; Loss mask Represents the target area mask loss, which is calculated in the form of cross entropy loss, where Indicates the probability that the category predicted by the i-th point belongs to the target and background, y i Represents the true label value of the point; D is the total number of dimensions of the vector graph tensor; It is a real unit vector diagram; is the predicted unit vector graph; x i is the difference between the true vector map and the predicted vector map at the i-th pixel.

[0033] Furthermore, the process of obtaining the second training set includes:

[0034] Use the opencv aruco library to build the aruco calibration plate for pose calibration and make it in kind;

[0035] Place the target inside the docking station on the calibration plate, and capture images of the calibration plate with the target in different postures and positions;

[0036] For each captured image, the OpenCV Aruco library is used to detect the QR code on the calibration plate. The position of the calibration plate origin relative to the camera is calculated based on the QR code and expressed as a rotation matrix and a translation vector.

[0037] The rotation matrix and translation vector obtained by calibration are combined with the known target object point cloud model to construct the target area mask;

[0038] For the pose set obtained by calibrating the captured image, a new rotation matrix and a new translation vector for rendering are sampled;

[0039] Using images from other public datasets as background images, using blender software and the target point cloud model, and rendering according to the new rotation matrix and the new translation vector to obtain an enhanced dataset image;

[0040] The original acquired data set with the real pose results is divided into a second training set and a test set, and the rendered enhanced data set is added to the second training set for network training.

[0041] Furthermore, based on the unit vector key features of the target area, the candidate pixel 2D coordinates of the key points and their internal point sets are selected using the random sampling consistency criterion, including:

[0042] (1) Select a number of pairs of corresponding points from the point set of the unit vector key features of the target area, and for the corresponding points According to its unit vector key features Compute the line where the two unit vector key features lie:

[0043]

[0044] The intersection point of the two unit vector key features is taken as the i-th hypothetical point h of the k-th key point k,i , thus obtaining the hypothetical point set B, where the hypothetical point h k,i According to the following formula

[0045]

[0046] (2) Use vector voting to count the number of RANSAC internal points for all hypothesis points:

[0047]

[0048] where w k,i The number of votes for the i-th hypothesis point of the k-th key point; is the inner product expression; I is the indicator function. When the inequality in the function brackets is satisfied, the indicator function output is 1, otherwise it is 0; ξ is the threshold

[0049] (3) Count the number of inliers of all hypothetical points of the current k-th key point, and select the hypothetical point with the largest number of inliers as the candidate pixel 2D coordinate of the key point;

[0050] (4) Repeat steps (1)-(3) until the candidate pixel 2D coordinates of all key points are obtained.

[0051] Furthermore, the pose of the target object inside the dock relative to the robot is obtained using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation, including:

[0052] (1) The geometric pose estimation method EPnP is used to solve the pose iteration initial value:

[0053] 1) The key point set p in the world coordinate systemw Use PCA principal component analysis method to obtain the control points in the world coordinate system and the control coefficient a for each point ij :

[0054]

[0055]

[0056] 2) Construct the following equation through the camera imaging model;

[0057]

[0058] where f x ,f y ,u0,v0 are the camera internal parameters, u i ,v i , (i=1,...,N) is the pixel 2D coordinate of the i-th optimized key point, a ij ,j=1,...,4 represents the jth control coefficient of the i-th key point;

[0059] Transform the equation into the matrix Mc=0 form, by T The eigenvalue decomposition of M yields the expression where w i is the i-th eigenvector, and the weighting coefficient β is obtained using the following objective function i , and thus further obtain the control point c:

[0060]

[0061] The following residual formula is established:

[0062] E=[e 12 e 13 e 14 e 23 e 24 e 34 ] T ,

[0063]

[0064] in Let be the kth eigenvector of the i-th control point, The Jacobian matrix is ​​J = [J 12 J 13 J 14 J 23 J 24 J 34 ] T, then we get the Jacobian matrix:

[0065]

[0066] The following Gauss-Newton iteration formula is used to iteratively update and obtain β:

[0067]

[0068] Further obtain the coordinates of the control points in the camera coordinate system

[0069] 3) Use the control point coefficients and control point coordinates in the camera coordinate system to reconstruct the key point camera coordinates, and the control point coefficients and control point coordinates in the world coordinate system to reconstruct the key point direct coordinates, and perform the following formula to decentralize

[0070]

[0071] in The reconstructed key point camera coordinates, is the average coordinate of the key point set in the camera coordinate system; is the world coordinate of the reconstructed key points, is the average coordinate of the key point set in the world coordinate system;

[0072] 4) For W = (p c ) T p w Perform SVD decomposition to get [U∑V] = SVD(W), then the initial values ​​of the rotation matrix and the initial values ​​of the translation vector are R0 = UV T , Use the initial value of the rotation matrix and the initial value of the translation vector as the initial value of the pose iteration;

[0073] (2) Taking the initial value of the pose iteration as the search starting point, the pose is iteratively optimized using the objective function based on the minimum reprojection error to obtain the relative pose of the target object inside the dock relative to the robot:

[0074] The following residual matrix is ​​constructed based on the objective function based on the minimum reprojection error:

[0075] E=[e1,e2,...,e N ] T

[0076]

[0077] The objective function based on the minimum reprojection error is

[0078]

[0079] Where K is the camera intrinsic parameter matrix is the prior 3D coordinate of the kth key point in the world coordinate system, is the pixel 2D coordinate of the kth key point in the pixel coordinate system;

[0080] The Gauss-Newton method is used to solve the objective function and obtain the Jacobian matrix of the rotation R:

[0081]

[0082] Where vector() represents the row vectorization operation, then the matrix J R The size is N×9, while the size of the residual matrix E is N×1. The Gauss-Newton iteration formula is used to update the vector R after the row vectorization of the matrix R v :

[0083]

[0084] R v =R v +ΔR v

[0085] The vector R v After solving the row vectorization, the rotation matrix R can be obtained;

[0086] For the translation vector t, its Jacobian matrix is:

[0087]

[0088] Matrix J t The size is N×3, and the Gauss-Newton iteration formula is used to update the vector t':

[0089]

[0090] t'=t'+Δt'

[0091] The above Gauss-Newton formula is used to iterate continuously until the error is less than a certain threshold or the number of iterations reaches a certain upper limit, and finally the relative position of the target object inside the dock relative to the robot is obtained. The relative position includes the rotation matrix R and the translation vector t'.

[0092] (3) Transformed into the position of the docking station center relative to the robot through rigid body transformation

[0093] The following rigid body transformation is further transformed into the position of the docking station center relative to the camera:

[0094] t=t′+RΔt

[0095] Where t′ represents the translation vector of the internal target object of the dock relative to the robot, t′ represents the translation vector of the dock center relative to the robot, and Δt represents the fixed three-axis deviation between the internal target object and the dock center.

[0096] According to the following formula, the rotation matrix R is transformed into a three-axis Euler angle vector e, which is used to represent the posture deviation between the docking station center and the robot:

[0097]

[0098] The translation vector t and the three-axis Euler angle vector e are used to plan the path of the underwater robot and generate control instructions according to the path.

[0099] According to a second aspect of an embodiment of the present application, a monocular vision guidance device for underwater robot docking based on key features is provided, comprising:

[0100] An acquisition module, used to acquire underwater images and perform preprocessing;

[0101] A detection module, configured to input the pre-processed underwater image into a proximal vision-guided deep learning model to detect the total number of image pixels occupied by the target object inside the docking station;

[0102] The first pose calculation module is used to input the preprocessed underwater image into the trained key point detection network if the total value is less than a predetermined threshold, obtain the position deviation of the external key point of the docking station from the docking station center, and calculate the pixel 2D position of the external key point of the docking station; combine the pixel 2D position of the external key point of the docking station with its prior world 3D coordinate to form a point pair mapping; and use the two-step pose estimation method based on the point pair mapping to obtain the pose and position of the docking station center relative to the robot;

[0103] The second pose calculation module is used to, if the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, input the preprocessed underwater image into the trained key feature detection network, obtain the region mask map and unit vector feature map of the target object, extract the unit vector key features of the target area in the unit vector feature map through the region mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the way of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation;

[0104] The guidance module is used to plan the path of the underwater robot according to the position of the docking station center relative to the robot and generate control instructions according to the path to guide the underwater robot.

[0105] According to a third aspect of an embodiment of the present application, an underwater robot is provided, which is recovered and docked using the underwater robot docking monocular vision guidance method based on key features as described in the first aspect.

[0106] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0107] (1) This paper proposes a monocular vision guidance method for underwater robot docking based on key features. Compared with existing guidance methods, this method has higher initiative, interference resistance, stability and accuracy;

[0108] (2) The far-end visual guidance in the present invention can actively identify the key points of the external mechanical structure of the underwater docking station, and the near-end visual guidance can actively identify the key features of the target object inside the docking station and infer the key point positions, and further obtain the position information of the docking station relative to the robot. Compared with the traditional light array guidance method, it has higher initiative and is not easily affected by environmental interference. When the camera is out of the field of view, it can also solve the position result from the target area, and has higher robustness.

[0109] (3) The pose solution part of the present invention proposes a two-step pose estimation method, which can obtain pose results with higher accuracy compared to the ordinary PnP algorithm.

[0110] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0112] Figure 1 The present invention is a flowchart of a monocular vision guidance method for underwater robot docking based on key features according to an exemplary embodiment.

[0113] Figure 2 This is a data set acquisition flow chart based on an exemplary example; wherein (a) is a far-end vision-guided key point detection network data set acquisition process, and (b) is a near-end vision-guided key feature detection network data set acquisition process.

[0114] Figure 3The present invention is a method for detecting key features of a network dataset using near-end vision guidance according to an exemplary embodiment; wherein (a) is a flowchart of a dataset mask generation method, and (b) is a flowchart of a dataset enhancement method.

[0115] Figure 4 is a schematic diagram of a sphere sampling method according to an exemplary embodiment; wherein (a) is a schematic diagram of a sphere, and (b) is a schematic diagram of a sphere sampling process.

[0116] Figure 5 The figure is a schematic diagram of proximal vision-guided inlier fitting according to an exemplary embodiment.

[0117] Figure 6 The present invention is a block diagram of a monocular vision guidance device for docking an underwater robot based on key features according to an exemplary embodiment. DETAILED DESCRIPTION

[0118] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0119] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0120] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0121] It should be noted that the “posture relative to the robot” involved in this application refers to the posture relative to the camera on the robot.

[0122] Figure 1 FIG. 1 is a flow chart showing a monocular vision guidance method for underwater robot docking based on key features according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0123] Step S11: Acquire underwater images and perform preprocessing;

[0124] Step S12: Input the pre-processed underwater image into the proximal vision-guided deep learning model PVNet to detect the total number of image pixels occupied by the target object inside the docking station;

[0125] Step S13: If the total value is less than or equal to a predetermined threshold, the current stage is remote vision guidance. The preprocessed underwater image is input into the trained key point detection network to obtain the position deviation of the external key point of the docking station from the docking station center, and the pixel 2D position of the external key point of the docking station is calculated; the pixel 2D position of the external key point of the docking station is combined with its prior world 3D coordinate to form a point pair mapping; based on the point pair mapping, a two-step pose estimation method is used to obtain the posture and position of the docking station center relative to the robot;

[0126] Step S14: If the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, and the preprocessed underwater image is input into the trained key feature detection network to obtain a region mask map and a unit vector feature map of the target object, and the unit vector key features of the target area in the unit vector feature map are extracted through the region mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the manner of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation;

[0127] Step S15: planning a path of the underwater robot according to the position of the center of the docking station relative to the robot and generating control instructions according to the path to guide the underwater robot.

[0128] It can be seen from the above embodiments that the present application proposes a monocular vision guidance method for underwater robot docking based on key features. Compared with existing guidance methods, the present method has higher initiative, interference resistance, stability and accuracy; the far-end vision guidance in the present application can actively identify the key point positions of the external mechanical structure of the underwater dock, and the near-end vision guidance can actively identify the key features of the target object inside the dock and infer the key point positions, and further obtain the posture information of the dock relative to the robot. Compared with the traditional light array guidance method, it has higher initiative and is not easily affected by environmental interference. When the camera is out of the field of view, it can also solve the posture results from the target part area, and has higher robustness; the posture solution part in the present application proposes a two-step posture estimation method, which can solve the posture results with higher accuracy than the ordinary PnP algorithm.

[0129] In the specific implementation of step S11, an underwater image is acquired and preprocessed;

[0130] Specifically, the robot first acquires underwater images through its built-in camera and performs image preprocessing on the obtained underwater images, which mainly includes three parts: image distortion correction, image denoising, and image enhancement based on dark channel priors. Image distortion correction aims to reduce the image distortion caused by the camera's mechanical structure. Image distortion can be mainly divided into radial distortion and tangential distortion. Radial distortion is the distortion distributed along the radius of the camera lens. It is caused by the fact that light is more bent away from the center of the lens than near the center. It can be described by the first few terms of the Taylor series expansion around the principal point below:

[0131]

[0132]

[0133] in The pixel position on the original image represents the distortion, and (u, v) represents the pixel position after correction. The tangential distortion is caused by the non-parallelism between the lens and the camera sensor plane, and is usually described by the following mathematical expression:

[0134]

[0135]

[0136] A large number of calibration images are used to calibrate the camera's distortion coefficients during camera intrinsic parameter calibration, and the distortion coefficients are used to inversely solve the distortion-corrected image during vision guidance.

[0137] Image denoising aims to reduce the interference of underwater floating objects. This method uses the median filtering method to visually guide image denoising. For each pixel in the image, the original pixel value is replaced by the median value of the pixels in the pixel neighborhood. Assuming that the target pixel value in the image is I before filtering, pre (x,y), after filtering is I post (x, y), the original target pixel is replaced by calculating the median pixel of the neighborhood s, which can be expressed as follows:

[0138] I post (x,y)=mid{I pre (x+Δx,y+Δy),(Δx,Δy)∈s}

[0139] Image enhancement based on dark channel prior aims to enhance the clarity of underwater images and highlight the structural features of the target in the image. Dark channel prior refers to the phenomenon that the value of a certain channel in the RGB channel of the original image pixel is low. For the actual image, the process of obtaining the dark channel prior map is a local filtering process, that is, for a certain pixel x on the enhanced image J, define J dark (x) is the dark channel value required for the pixel, J c (x) is the pixel value of the c channel of the pixel, and the dark channel can be calculated using the following expression:

[0140]

[0141] in Indicates finding the minimum value of the RGB three channels of the pixel point, and y∈Ω(x) indicates searching within a certain neighborhood window of the pixel point x. Then, for the pixel x, the above formula uses the minimum channel value of all pixels in a certain window around the pixel point as the dark channel value of the pixel. The following normalized weighted model is used to define the underwater image:

[0142] I c (x) = J c (x)t(x)+A c (1-t(x))

[0143] Among them I c (x) represents the c-channel value of the input original image pixel x, J c (x) represents the c-channel value of the enhanced image pixel x to be obtained, A c represents the water light component value on the c channel of the image, and t(x) represents the transmittance function. Assuming that the transmittance within a certain neighborhood window of pixel x is constant, taking the minimum on both sides of the above equation yields:

[0144]

[0145] According to the following dark channel definition:

[0146]

[0147] Substitute the obtained transmittance expression into the equation. In order to prevent the enhanced image from being too drastic and causing the restored image scene to be unnatural, the weight w is introduced to obtain the following final transmittance expression:

[0148]

[0149] Bringing it into the normalized weighted model and taking the lower limit of the transmittance, we get the final expression for image enhancement:

[0150]

[0151] To obtain the light component A of the water body, we first extract the top 0.1% of pixels based on the dark channel size. Then, we find the RGB channel value with the highest brightness of these corresponding pixels in the input original image to serve as A. The above is the image enhancement part based on dark channel prior.

[0152] In the specific implementation of step S12, the pre-processed underwater image is input into the proximal vision-guided deep learning model PVNet to detect the total number of image pixels occupied by the target object inside the docking station;

[0153] Specifically, the predetermined threshold is approximately 1 / 64 of the total pixels of the image. This is a hyperparameter and is adjusted in practice based on the size of the target object and the image size.

[0154] In the specific implementation of step S13, if the total value is less than or equal to a predetermined threshold, the current stage is remote vision guidance, and the preprocessed underwater image is input into the trained key point detection network to obtain the position deviation of the external key point of the docking station from the docking station center, and the pixel 2D position of the external key point of the docking station is calculated; the pixel 2D position of the external key point of the docking station is combined with its prior world 3D coordinate to form a point pair mapping; based on the point pair mapping, the posture and position of the docking station center relative to the robot are obtained using a two-step pose estimation method;

[0155] Specifically, the training process of the key point detection network includes:

[0156] (1) Obtain a first training set, wherein the first training set includes several images with external key points and center point labels of the docking station and the ground truth values ​​of the key point positions corresponding to each image. This design is because both the far-end vision-guided key point detection network and the near-end vision-guided key feature detection network are deep learning supervised networks. As a supervised learning method, a large amount of datasets with real labels are required to train the network to achieve convergence. Datasets are collected for the above two networks respectively.

[0157] (2) According to the first training set, the key point detection network is trained using a first loss function, wherein the first loss function Loss far for

[0158] Loss far =λ cls Loss cls +λ size Loss size

[0159]

[0160]

[0161] Among them, Loss cls Refers to the category confidence loss, with the pixel 2D coordinate of the center point of the real docking station as the positive sample Y xyc ,and is the category confidence value predicted by the key point detection network at the position (x, y, c). For the positive sample Y xyc =1, when the predicted value When it is larger, the loss value weight Smaller, otherwise larger, so that the prediction degree of difficult and easy samples can be balanced during training; Loss size is the key point deviation loss, and the deviation of the 2D coordinates of the key point pixel relative to the center point coordinates is calculated using the L1 loss; N is the total number of positive samples; α and β are temperature coefficients used to adjust the ratio of training difficulty; is the predicted position deviation; is the true position deviation; the final loss Loss is obtained by weighting the category confidence loss and the key point deviation loss, and the network is trained with the total loss.

[0162] Among them, Figure 2 As shown in (a), the process of obtaining the first training set includes:

[0163] Labelme software was used to manually mark the external corner points and center points of the docking station in the collected underwater images; the marked underwater images were represented as the category points of the docking station with the center point as the center point, and the deviation of the corner point relative to the center point was calculated as the true value of the key point position of the docking station.

[0164] Considering that it is impossible to obtain relevant information about the target objects inside the recovery dock when the robot is far away from the dock, this method uses the target detection network to detect the minimum circumscribed rectangular corner points of the dock's external mechanical structure as key points during remote visual guidance, and combines them with their prior world 3D coordinates to calculate the position and posture of the dock relative to the robot.

[0165] Specifically, the preprocessed underwater image is input into the trained key point detection network to obtain the position deviation of the key points outside the docking station from the docking station center, including:

[0166] The preprocessed underwater image is input into the trained key point detection network to obtain the category confidence map and key point deviation map;

[0167] The point with the maximum score is selected in the category confidence map, which is the pixel 2D coordinate of the docking station center point. The pixel 2D coordinate is used to extract the deviation of each key point at the same position in the key point deviation map. The pixel 2D coordinate of each key point is calculated by the center point coordinate and the key point deviation.

[0168] The two-part pose estimation method is described in step S14 and will not be described here in detail.

[0169] In the specific implementation of step S14, if the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, and the preprocessed underwater image is input into the trained key feature detection network to obtain a region mask map and a unit vector feature map of the target object, and the unit vector key features of the target area in the unit vector feature map are extracted through the region mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the manner of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation;

[0170] When the robot is relatively close to the dock and the target object inside the dock occupies a certain image area, it enters the proximal vision guidance with higher solution accuracy. The proximal vision guidance mainly uses the deep network to predict the vector key features related to the key points, and then infers the 2D position of the key point pixels based on random sampling consistency RANSAC and vector projection. Finally, the same two-step pose estimation method as the far-end vision guidance is used to obtain the position and pose of the target object inside the dock relative to the robot, and then converted into the pose result of the dock relative to the robot through rigid body transformation. The process of proximal vision guidance is as follows: Figure 6 shown.

[0171] Specifically, the training process of the key point detection network includes:

[0172] (1) Obtain a second training set, wherein the second training set is a number of images with the true values ​​of the target's posture and position relative to the camera; this design is because the far-end vision-guided key point detection network and the near-end vision-guided key feature detection network are both deep learning supervised networks. As a supervised learning method, a large amount of data sets with real labels are required to train the network to achieve convergence.

[0173] (2) According to the second training set, the key point detection network is trained using a second loss function, wherein the second loss function Loss is

[0174] Loss=Loss vertex +Loss mask

[0175]

[0176]

[0177]

[0178] Where, Loss vertex represents the unit vector graph loss, p i ∈O means that only the values ​​in the target area are calculated for loss. The smooth L1 loss is used to calculate the real and predicted unit vectors. When the absolute error between the two is greater than 1, the smooth L1 loss is in the form of L1 loss, which can provide a stable descent gradient. When the absolute error between the two is less than 1, the smooth L1 loss is in the form of L2 loss, which can provide a descent gradient that adapts to the error change. mask Represents the target area mask loss, which is calculated in the form of cross entropy loss, where Indicates the probability that the category predicted by the i-th point belongs to the target and background, y i Represents the true label value of the point; D is the total number of dimensions of the vector graph tensor; It is a real unit vector diagram; is the predicted unit vector graph; x i is the difference between the true vector map and the predicted vector map at the i-th pixel. The final loss is a 1:1 weighted sum of the two, and the network is trained using this loss until convergence. After obtaining the trained key feature detection network, the pre-processed underwater image is fed into the network to generate a unit vector feature map and a target region mask. These two results are used to extract the unit vector feature set of the target region, which is used to estimate the 2D position of the keypoint pixels.

[0179] Among them, Figure 2 As shown in (b), the process of obtaining the second training set includes:

[0180] The opencv aruco library is used to build an aruco calibration plate for pose calibration and the actual object is produced; the target object inside the dock is placed on the calibration plate, and images of the calibration plate with the target object are collected in different postures and positions; for each collected image, the opencv aruco library is used to detect the QR code on the calibration plate, and the pose result of the calibration plate origin relative to the camera is calculated based on the QR code, which is expressed as a rotation matrix and translation vector; the rotation matrix and translation vector obtained by calibration are used to combine with the known target object point cloud model to construct a target area mask; for the pose set obtained by the calibration of the collected image, a new rotation matrix and new translation vector for rendering are sampled; with images of other public datasets as background images, blender software and the target point cloud model are used to render the dataset images for enhancement based on the new rotation matrix and new translation vector; the original collected dataset with the real pose results is divided into a second training set and a test set, and the rendered enhanced dataset is added to the second training set for network training.

[0181] The target area mask construction process is as follows Figure 3 As shown in (a): First, the point cloud model under the standard frame is transformed using the rotation matrix and translation vector obtained by calibration, and interpolation processing is performed to make the points in the point cloud model denser; all the above points are projected into the pixel coordinate system according to the camera imaging model, and white dots with appropriate radius are constructed with these points as the center under the black background image; the above constructed image is binarized, and then the graphics opening and closing processing is used to make the target area form a closed whole, and the target area mask is obtained.

[0182] The sampling process of new rotation matrix and new translation vector in the near-end vision-guided key feature network dataset collection is as follows: Figure 3 As shown in (b): First, the calibrated pose results are formed into a pose set, where the rotation matrix is ​​transformed into an Euler angle vector; the Gaussian kernel density estimation is performed on the translation vector in the pose set using the following expression:

[0183]

[0184] Where K0() is the kernel density function, h is the bandwidth, and after the probability density function f(x) of the translation vector is obtained according to the above formula, the probability density function is used for resampling to obtain a new translation vector for rendering.

[0185] The Euler angle posture vector in this pose set is the three-axis angle deviation, namely the pitch angle deviation, the heading angle deviation and the roll angle deviation. Since the pitch angle and the heading angle vary greatly during the actual navigation of the underwater robot, the spherical sampling method is used to obtain a larger range of sampling values ​​for these two angles. The spherical sampling method obtains the Euler angle vector by sampling on a sphere with a radius of 1 and converting it into a relative angle relative to the center of the sphere. The coordinates of the sampling point can be expressed as follows:

[0186]

[0187] The above expression divides the z-axis of the sphere from bottom to top into N segments, each of which intersects with the plane of the sphere to obtain an XoY plane, such as Figure 4 As shown in (a), there is only one sampling point on each latitude of the sphere, and according to the above formula x n with y n It can be seen that the sampling points on the upper and lower adjacent XoY planes differ by φ circles in the longitude direction. The sampling process is as follows: Figure 4 As shown in the dotted line in (b), for the roll angle, Gaussian kernel density estimation is used to estimate the probability density function of the original set, and resampling is performed to obtain the sampled values. Finally, a new Euler angle vector is constructed and converted into a rotation matrix.

[0188] Finally, the new rotation matrix and translation vector are combined with the target point cloud model and other dataset background images, and a rendered image is obtained using blender. Then, an enhanced dataset with pose results is obtained. After obtaining the training set with the real pose results and the real target area mask map, the target point cloud model is used Figure 4 The FPS farthest point sampling algorithm shown in the figure obtains the prior world 3D coordinates of the target key points. During the preprocessing of network training, the camera model and the real pose are used to reproject the key points into the pixel coordinate system. The pixel 2D coordinates and the real mask map are used to construct a unit vector map pointing to the 2D coordinates of the key points in the target area. The mask map and unit vector map are used as the real labels for network training.

[0189] Specifically, based on the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected using the random sampling consistency criterion, including:

[0190] (1) Select several pairs of corresponding points from the point set A of the unit vector key features of the target area, and for the corresponding points According to its unit vector key features Compute the line where the two unit vector key features lie:

[0191]

[0192] The intersection point of the two unit vector key features is taken as the i-th hypothetical point h of the k-th key point k,i , thus obtaining the hypothetical point set B, where the hypothetical point h k,i According to the following formula

[0193]

[0194] (2) For each hypothetical point, if the point is closer to the real key point position, the unit vector of other pixels in the target area pointing to the real key point is and the unit vector pointing to the hypothetical point The angle between the two unit vectors should be small, and the angle between the unit vectors can be indirectly expressed by the inner product of the vectors. Therefore, for the convenience of calculation, when the inner product of the two unit vectors is When it is greater than a certain threshold, the pixel is regarded as an inlier of the current hypothesis point, and vector voting is used to perform RANSAC inlier statistics on all hypothesis points:

[0195]

[0196] where w k,i The number of votes for the i-th hypothesis point of the k-th key point; is the inner product expression; I is the indicator function. When the inequality in the function brackets is satisfied, the indicator function output is 1, otherwise it is 0; ξ is the threshold

[0197] (3) Count the number of inliers of all hypothetical points of the current k-th key point, and select the hypothetical point with the largest number of inliers as the candidate pixel 2D coordinate of the key point;

[0198] (4) Repeat steps (1)-(3) until the candidate pixel 2D coordinates of all key points are obtained.

[0199] After obtaining the initial position of the key points as described above, an interior point fitting method based on vector projection is used to optimize the 2D position of the pixels. The internal fitting diagram is shown in the figure below: Figure 5 As shown, the real key point is H, P is one of the internal points of the key point in the above RANSAC method, and the vector Where O is the origin of the image. Since the vector The unit vector It can be obtained from the vector diagram predicted by the above network. Assuming that the unit vector is 100% accurate, the vector In the case of , we can get the value perpendicular to the vector The unit vector Then for the real key point H, the vector The lengths of the projections PA and HB on the unit vector n are equal, that is, However, the unit vector predicted by the network has a certain deviation. We can then fit multiple inliers similar to point P according to the above geometric relationship to obtain a more accurate key point position. Assume that the i-th inlier is p i =[p ix ,p iy ] T , and the unit vector n of the inner point pointing to the key point is obtained according to the predicted vector diagram i =[d iy ,-d ix ] T , due to n i is a unit vector, so the projection PA of the interior point position vector on it is available Inner product form, the final key point position required is s, then the projection HB can be expressed as s T n i , the position fitting of the final key point becomes the solution of the following objective function:

[0200]

[0201] The goal of the above expression is to minimize the sum of the projection distances of all interior point vectors. Considering that the above objective function is a typical least squares form, the optimal key point position after its transformation must satisfy the following expression:

[0202]

[0203] Transposing the above equation left to right yields And use b to represent get:

[0204]

[0205] Then the optimal key point fitting position can be obtained as:

[0206]

[0207] All m internal points are combined and expressed as a matrix A = [n1...n m ] T , the above formula can be converted into the fitting form of all interior points:

[0208] s=(A T A) -1 A T b

[0209] Finally, s is the optimized key point position obtained based on the vector projection inner point fitting. The above optimization method is used for all preliminary key points to obtain the key point pixel 2D coordinate point set for pose solution.

[0210] Specifically, the two-step pose estimation method refers to first using EPnP to obtain the initial value of the pose iteration, and then iteratively optimizing the pose based on the minimum reprojection distance, specifically including:

[0211] (1) The geometric pose estimation method EPnP is used to solve the pose iteration initial value:

[0212] 1) The key point set p in the world coordinate system w Use PCA principal component analysis method to obtain the control points in the world coordinate system and the control coefficient a for each point ij :

[0213]

[0214]

[0215] 2) Construct the following equation through the camera imaging model;

[0216]

[0217] where f x ,f y ,u0,v0 are the camera internal parameters, u i ,v i , (i=1,...,N) is the pixel 2D position of the external key point of the i-th dock or the pixel 2D coordinate of the i-th optimized key point, a ij ,j=1,...,4 represents the jth control coefficient of the i-th key point;

[0218] Transform the equation into the matrix Mc=0 form, by T The eigenvalue decomposition of M yields the expression where w i is the i-th eigenvector, and the weighting coefficient β is obtained using the following objective function i , and thus further obtain the control point c:

[0219]

[0220] The following residual formula is established:

[0221] E=[e 12 e 13 e 14 e 23 e 24 e 34] T ,

[0222]

[0223] in Let be the kth eigenvector of the ith control point, The Jacobian matrix is ​​J = [J 12 J 13 J 14 J 23 J 24 J 34 ] T , then we get the Jacobian matrix:

[0224]

[0225] The following Gauss-Newton iteration formula is used to iteratively update and obtain β:

[0226]

[0227] Further obtain the coordinates of the control points in the camera coordinate system

[0228] 3) Use the control point coefficients and control point coordinates in the camera coordinate system to reconstruct the key point camera coordinates, and the control point coefficients and control point coordinates in the world coordinate system to reconstruct the key point direct coordinates, and perform the following formula to decentralize

[0229]

[0230] in The reconstructed key point camera coordinates, is the average coordinate of the key point set in the camera coordinate system; is the world coordinate of the reconstructed key points, is the average coordinate of the key point set in the world coordinate system;

[0231] 4) For W = (p c ) T p w Perform SVD decomposition to get [U∑V] = SVD(W), then the initial values ​​of the rotation matrix and the initial values ​​of the translation vector are R0 = UV T , Use the initial value of the rotation matrix and the initial value of the translation vector as the initial value of the pose iteration;

[0232] (2) Taking the initial value of the pose iteration as the search starting point, the pose is iteratively optimized using the objective function based on the minimum reprojection error to obtain the relative pose of the target object inside the dock relative to the robot:

[0233] The following residual matrix is ​​constructed based on the objective function based on the minimum reprojection error:

[0234] E=[e1,e2,...,e N ] T

[0235]

[0236] The objective function based on the minimum reprojection error is

[0237]

[0238] Where K is the camera intrinsic parameter matrix is the prior 3D coordinate of the kth key point in the world coordinate system, is the pixel 2D coordinate of the kth key point in the pixel coordinate system;

[0239] The Gauss-Newton method is used to solve the objective function and obtain the Jacobian matrix of the rotation R:

[0240]

[0241] Where vector() represents the row vectorization operation, then the matrix J R The size is N×9, while the size of the residual matrix E is N×1. The Gauss-Newton iteration formula is used to update the vector R after the row vectorization of the matrix R v :

[0242]

[0243] R v =R v +ΔR v

[0244] The vector R v After solving the row vectorization, the rotation matrix R can be obtained;

[0245] For the translation vector t, its Jacobian matrix is:

[0246]

[0247] Matrix J t The size is N×3, and the vector t' is updated using the Gauss-Newton iteration formula:

[0248]

[0249] t'=t'+Δt'

[0250] The above Gauss-Newton formula is used to iterate continuously until the error is less than a certain threshold or the number of iterations reaches a certain upper limit, and finally the relative position of the target object inside the docking station or the docking station center relative to the robot is obtained. The relative position includes the rotation matrix R and the translation vector t'.

[0251] For proximal visual guidance, it is also necessary to convert the position of the docking station center relative to the robot through rigid body transformation:

[0252] The following rigid body transformation is further transformed into the position of the docking station center relative to the camera:

[0253] t=t′+RΔt

[0254] Where t′ represents the translation vector of the internal target object of the dock relative to the robot, t′ represents the translation vector of the dock center relative to the robot, and Δt represents the fixed three-axis deviation between the internal target object and the dock center.

[0255] According to the following formula, the rotation matrix R is transformed into a three-axis Euler angle vector e, which is used to represent the posture deviation between the docking station center and the robot:

[0256]

[0257] According to the translation vector t and the three-axis Euler angle vector e, the path of the underwater robot is planned and a control instruction is generated according to the path.

[0258] In the specific implementation of step S15, the path of the underwater robot is planned according to the position of the docking station center relative to the robot, and control instructions are generated according to the path to guide the underwater robot.

[0259] Specifically, the navigation module calculates the world coordinates of the docking station based on the position of the docking station center relative to the robot and its own current world coordinates. The planning module calculates the trajectory based on the world coordinates of the docking station and the world coordinates of the robot, and obtains the direction and size of the current movement of the robot based on the points on the trajectory. The control module controls the robot motion device according to the direction and size to provide guidance.

[0260] Corresponding to the aforementioned embodiment of the monocular vision guidance method for underwater robot docking based on key features, the present application also provides an embodiment of the monocular vision guidance device for underwater robot docking based on key features.

[0261] Figure 6 This is a block diagram of a monocular vision guidance device for underwater robot docking based on key features according to an exemplary embodiment. Figure 6 , the apparatus may include:

[0262] An acquisition module 21 is used to acquire underwater images and perform preprocessing;

[0263] A detection module 22 is configured to input the pre-processed underwater image into a proximal vision-guided deep learning model to detect the total number of image pixels occupied by the target object inside the docking station;

[0264] The first pose calculation module 23 is configured to, if the total value is less than a predetermined threshold, determine that the current stage is remote vision guidance, input the preprocessed underwater image into a trained key point detection network, obtain the position deviation of the docking station's external key points from the docking station's center, and calculate the pixel 2D position of the docking station's external key points; combine the pixel 2D positions of the docking station's external key points with their prior world 3D coordinates to form a point pair mapping; and, based on the point pair mapping, use a two-step pose estimation method to obtain the pose and position of the docking station's center relative to the robot;

[0265] The second pose calculation module 24 is used to, if the total value is greater than a predetermined threshold, determine that the current stage is proximal vision guidance, input the preprocessed underwater image into the trained key feature detection network, obtain a region mask map and a unit vector feature map of the target object, extract the unit vector key features of the target area in the unit vector feature map through the region mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; based on the unit vector key features of the target area, select the candidate pixel 2D coordinates of the key point and its internal point set using a random sampling consistency criterion, perform internal point fitting on the internal point set using a vector projection method to obtain the optimized pixel 2D coordinates of the key point; form a point pair mapping with the optimized pixel 2D coordinates and the prior 3D coordinates; based on the point pair mapping, use the two-step pose estimation method to obtain the pose of the target object inside the dock relative to the robot, and convert it into the pose of the dock center relative to the robot through a rigid body transformation;

[0266] The guidance module 25 is used to plan the path of the underwater robot according to the position of the docking station center relative to the robot and generate control instructions according to the path to guide the underwater robot.

[0267] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0268] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0269] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the monocular vision guidance method for docking an underwater robot based on key features as described above.

[0270] Correspondingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, characterized in that when the instructions are executed by a processor, the monocular vision guidance method for docking an underwater robot based on key features as described above is implemented.

[0271] An embodiment of the present application also provides an underwater robot, which is recovered and docked using the above-mentioned underwater robot docking monocular vision guidance method based on key features.

[0272] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0273] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A monocular vision guidance method for underwater robot docking based on key features, characterized in that: include: Acquire underwater images and perform preprocessing; The pre-processed underwater image is input into the proximal vision-guided deep learning model PVNet to detect the total number of image pixels occupied by the target objects inside the dock; If the total value is less than or equal to a predetermined threshold, the current stage is remote vision guidance. The preprocessed underwater image is input into the trained key point detection network to obtain the position deviation of the external key point of the docking station from the docking station center, and the pixel 2D position of the external key point of the docking station is calculated; the pixel 2D position of the external key point of the docking station is combined with its prior world 3D coordinate to form a point pair mapping; based on the point pair mapping, a two-step pose estimation method is used to obtain the posture and position of the docking station center relative to the robot; If the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, and the preprocessed underwater image is input into the trained key feature detection network to obtain a regional mask map and a unit vector feature map of the target object, and the unit vector key features of the target area in the unit vector feature map are extracted through the regional mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the way of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation; According to the position of the docking station center relative to the robot, the path of the underwater robot is planned and control instructions are generated according to the path to guide the underwater robot.

2. The method according to claim 1, characterized in that The training process of the key point detection network includes: Obtaining a first training set, wherein the first training set includes a plurality of images with external key points and a center point identification of the docking station and a true value of the key point position corresponding to each image; According to the first training set, the key point detection network is trained using a first loss function, wherein the first loss function Loss far for Loss far =λ cls Loss cls +λ size Loss size Among them, Loss cls Refers to the category confidence loss, with the pixel 2D coordinate of the center point of the real docking station as the positive sample Y xyc ,and Loss is the category confidence value predicted by the key point detection network at the (x, y, c) position; size is the key point deviation loss; N is the total number of positive samples; α and β are temperature coefficients used to adjust the ratio of training difficulty; is the predicted position deviation; S pk is the actual position deviation.

3. The method according to claim 2, characterized in that The process of obtaining the first training set includes: Labelme software was used to manually mark the external corners and center points of the docking station in the collected underwater images; The marked underwater image is represented by the category point with the center point as the docking station, and the deviation of the corner point relative to the center point is calculated as the true value of the key point position of the docking station.

4. The method according to claim 1, wherein The pre-processed underwater image is input into the trained key point detection network to obtain the position deviation of the key points outside the docking station from the docking station center, including: The preprocessed underwater image is input into the trained key point detection network to obtain the category confidence map and key point deviation map; The point with the maximum score is selected in the category confidence map, which is the pixel 2D coordinate of the docking station center point. The pixel 2D coordinate is used to extract the deviation of each key point at the same position in the key point deviation map. The pixel 2D coordinate of each key point is calculated by the center point coordinate and the key point deviation.

5. The method according to claim 1, characterized in that The training process of the key point detection network includes: Acquire a second training set, wherein the second training set comprises a plurality of images with true values ​​of the posture and position of the target object relative to the robot; According to the second training set, the key point detection network is trained using a second loss function, wherein the second loss function Loss is Loss=Loss vertex +Loss mask Where, Loss vertex represents the unit vector graph loss, p i ∈O means that only the values ​​in the target area are calculated; Loss mask Represents the target area mask loss, which is calculated in the form of cross entropy loss, where Indicates the probability that the category predicted by the i-th point belongs to the target and background, y i Represents the true label value of the point; D is the total number of dimensions of the vector graph tensor; It is a real unit vector diagram; is the predicted unit vector graph; x i is the difference between the true vector map and the predicted vector map at the i-th pixel.

6. The method according to claim 5, characterized in that The process of obtaining the second training set includes: Use the opencv aruco library to build the aruco calibration plate for pose calibration and make it in kind; Place the target inside the docking station on the calibration plate, and capture images of the calibration plate with the target in different postures and positions; For each captured image, the OpenCV Aruco library is used to detect the QR code on the calibration plate. The position of the calibration plate origin relative to the camera is calculated based on the QR code, expressed as a rotation matrix and a translation vector. The rotation matrix and translation vector obtained by calibration are combined with the known target object point cloud model to construct the target area mask; For the pose set obtained by calibrating the captured image, a new rotation matrix and a new translation vector for rendering are sampled; Using images from other public datasets as background images, using blender software and the target point cloud model, and rendering according to the new rotation matrix and the new translation vector to obtain an enhanced dataset image; The original acquired data set with the real pose results is divided into a second training set and a test set, and the rendered enhanced data set is added to the second training set for network training.

7. The method according to claim 1, characterized in that According to the key unit vector features of the target area, the candidate pixel 2D coordinates of the key points and their internal point sets are selected using the random sampling consistency criterion, including: (1) Select a number of pairs of corresponding points from the point set of the unit vector key features of the target area, and for the corresponding points According to its unit vector key features Compute the line where the two unit vector key features lie: The intersection point of the two unit vector key features is taken as the i-th hypothetical point h of the k-th key point k,i , thus obtaining the hypothetical point set B, where the hypothetical point h k,i According to the following formula (2) Use vector voting to count the number of RANSAC internal points for all hypothesis points: where w k,i The number of votes for the i-th hypothesis point of the k-th key point; is the inner product expression; I is the indicator function. When the inequality in the function brackets is satisfied, the indicator function output is 1, otherwise it is 0; ξ is the threshold; (3) Count the number of inliers of all hypothetical points of the current k-th key point, and select the hypothetical point with the largest number of inliers as the candidate pixel 2D coordinate of the key point; (4) Repeat steps (1)-(3) until the candidate pixel 2D coordinates of all key points are obtained.

8. The method according to claim 1, characterized in that The pose of the target object inside the dock relative to the robot is obtained using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation, including: (1) The geometric pose estimation method EPnP is used to solve the pose iteration initial value: 1) The key point set p in the world coordinate system w Use PCA principal component analysis method to obtain the control points in the world coordinate system and the control coefficient a for each point ij : 2) Construct the following equation through the camera imaging model; where f x ,f y ,u0,v0 are the camera internal parameters, u i ,v i , (i=1,...,N) is the pixel 2D coordinate of the i-th optimized key point, a ij ,j=1,...,4 represents the jth control coefficient of the i-th key point; Transform the equation into the matrix Mc=0 form, by T The eigenvalue decomposition of M yields the expression where w i is the i-th eigenvector, and the weighting coefficient β is obtained using the following objective function i , and thus further obtain the control point c: The following residual formula is established: in Let be the kth eigenvector of the i-th control point, The Jacobian matrix is ​​J = [J 12 J 13 J 14 J 23 J 24 J 34 ] T , then we get the Jacobian matrix: The following Gauss-Newton iteration formula is used to iteratively update and obtain β: Further obtain the coordinates of the control points in the camera coordinate system 3) Use the control point coefficients and control point coordinates in the camera coordinate system to reconstruct the key point camera coordinates, and the control point coefficients and control point coordinates in the world coordinate system to reconstruct the key point direct coordinates, and perform the following formula to decentralize in The reconstructed key point camera coordinates, is the average coordinate of the key point set in the camera coordinate system; is the world coordinate of the reconstructed key points, is the average coordinate of the key point set in the world coordinate system; 4) For W = (p c ) T p w Perform SVD decomposition to get [U ∑ V] = SVD(W), then the initial values ​​of the rotation matrix and the translation vector are R0 = UV T , Use the initial value of the rotation matrix and the initial value of the translation vector as the initial value of the pose iteration; (2) Taking the initial value of the pose iteration as the search starting point, the pose is iteratively optimized using the objective function based on the minimum reprojection error to obtain the relative pose of the target object inside the dock relative to the robot: The following residual matrix is ​​constructed based on the objective function based on the minimum reprojection error: E\[e1,e2,...,e N ] T The objective function based on the minimum reprojection error is Where K is the camera intrinsic parameter matrix is the prior 3D coordinate of the kth key point in the world coordinate system, is the pixel 2D coordinate of the kth key point in the pixel coordinate system; The Gauss-Newton method is used to solve the objective function and obtain the Jacobian matrix of the rotation R: Where vector() represents the row vectorization operation, then the matrix J R The size is N×9, while the size of the residual matrix E is N×1. The Gauss-Newton iteration formula is used to update the vector R after the row vectorization of the matrix R v : R v =R v +ΔR v The vector R v After solving the row vectorization, the rotation matrix R can be obtained; For the translation vector t', its Jacobian matrix is: Matrix J t The size is N×3, and the Gauss-Newton iteration formula is used to update the vector t': t'=t'+Δt' The Gauss-Newton formula is used to iterate continuously until the error is less than a certain threshold or the number of iterations reaches a certain upper limit, and finally the relative position of the target object inside the dock relative to the robot is obtained. The relative position includes the rotation matrix R and the translation vector t'; (3) Transformed into the position of the docking station center relative to the robot through rigid body transformation The following rigid body transformation is further transformed into the position of the docking station center relative to the camera: t=t′+RΔt Where t′ represents the translation vector of the internal target object relative to the robot, t′ represents the translation vector of the docking station center relative to the robot, and Δt represents the fixed three-axis deviation between the internal target object and the docking station center. According to the following formula, the rotation matrix R is transformed into a three-axis Euler angle vector e, which is used to represent the posture deviation between the docking station center and the robot: The translation vector t and the three-axis Euler angle vector e are used to plan the path of the underwater robot and generate control instructions according to the path.

9. A monocular vision guidance device for underwater robot docking based on key features, characterized in that: include: An acquisition module, used to acquire underwater images and perform preprocessing; A detection module, configured to input the pre-processed underwater image into a proximal vision-guided deep learning model to detect the total number of image pixels occupied by the target object inside the docking station; The first pose calculation module is used to input the preprocessed underwater image into the trained key point detection network if the total value is less than a predetermined threshold, obtain the position deviation of the external key point of the docking station from the docking station center, and calculate the pixel 2D position of the external key point of the docking station; combine the pixel 2D position of the external key point of the docking station with its prior world 3D coordinate to form a point pair mapping; and use the two-step pose estimation method based on the point pair mapping to obtain the pose and position of the docking station center relative to the robot; The second pose calculation module is used to, if the total value is greater than a predetermined threshold, the current stage is proximal vision guidance, input the preprocessed underwater image into the trained key feature detection network, obtain the region mask map and unit vector feature map of the target object, extract the unit vector key features of the target area in the unit vector feature map through the region mask map, wherein the target area is the area occupied by the target object in the preprocessed underwater image; according to the unit vector key features of the target area, the candidate pixel 2D coordinates of the key point and its internal point set are selected by adopting the random sampling consistency criterion, and the internal point set is fitted by the internal point fitting in the way of vector projection to obtain the optimized pixel 2D coordinates of the key point; the optimized pixel 2D coordinates and the prior 3D coordinates are formed into a point pair mapping; according to the point pair mapping, the pose of the target object inside the dock relative to the robot is obtained by using the two-step pose estimation method, and converted into the pose of the dock center relative to the robot through rigid body transformation; The guidance module is used to plan the path of the underwater robot according to the position of the docking station center relative to the robot and generate control instructions according to the path to guide the underwater robot.

10. An underwater robot, characterized in that: The underwater robot is recovered and docked by the underwater robot docking monocular vision guidance method based on key features according to any one of claims 1 to 8.

Images