A self-supervised depth visual odometry based on geometric constraints

Abstract

This disclosure provides a self-supervised depth visual odometry acquisition method based on geometric constraints, comprising: training a depth estimation network until the loss function satisfies a preset value, causing the depth estimation network to converge; acquiring image depth information through the depth estimation network; and inputting the left and right image sequences of a stereo image into a pose estimation network to obtain the pose, wherein the left and right image information of the stereo image includes image depth information, and the image depth information is used to recover the scale of the pose. This disclosure also provides a self-supervised depth visual odometry acquisition device, electronic device, and readable storage medium based on geometric constraints.

Description

Technical Field

[0001] This disclosure relates to the field of pose estimation in computer vision SLAM, and more particularly to a method, apparatus, electronic device, and readable storage medium for self-supervised depth vision odometry acquisition based on geometric constraints. Background Technology

[0002] Pose estimation plays a crucial role in robotics and unmanned aerial vehicle (UAV) research. Traditional pose estimation methods commonly use wheel encoders, inertial navigation devices, and GPS satellite positioning systems. Although these methods are widely used and have achieved significant results in certain specific situations, they still have considerable limitations. For example, in practical applications, wheel encoders can generate large errors due to slippage, inertial navigation devices suffer from severe cumulative errors, and GPS positioning systems cannot be used in enclosed indoor environments.

[0003] With the development of computer vision technology, methods for pose estimation using visual sensors can avoid the problems existing in traditional pose estimation methods. The process of estimating a robot's relative pose (position and orientation) by analyzing motion changes in visual sensor images is called visual odometry. Visual odometry can estimate the pose of the camera at every moment in real time and send these estimates to a backend optimization module to obtain more accurate values. This disclosure belongs to the field of pose estimation based on computer vision SLAM.

[0004] Visual odometry is a method that uses visual sensors to obtain the relative pose of a camera. Traditional visual odometry recovers camera motion using geometric principles, and its basic framework has been established over many years. In recent years, various techniques utilizing deep learning to reveal the relationship between image and camera motion have emerged and achieved significant results. Existing visual odometry technologies can be broadly categorized into two types: traditional methods and deep learning-based visual odometry.

[0005] (1) Traditional methods

[0006] Before introducing specific visual odometry techniques, we will first briefly explain the number of cameras used in odometry calculations and their characteristics. Visual odometry is generally divided into monocular visual odometry and binocular visual odometry. Monocular visual odometry has advantages such as low cost and portability, but its biggest drawback is that monocular cameras cannot recover the depth information of images. Therefore, when estimating camera pose using geometric methods, there is a scale ambiguity problem. Traditional visual odometry techniques generally incorporate certain constraints and assumptions, which limits the application scope of monocular visual odometry to some extent. Binocular cameras, on the other hand, can recover the depth information of images through the geometric model of the binocular camera. Therefore, the estimated relative pose does not have the problem of scale ambiguity. However, binocular cameras have higher requirements for equipment and require more complex calibration. At the same time, the cost of binocular cameras is higher than that of monocular cameras. Therefore, the choice between monocular and binocular visual odometry needs to consider multiple factors such as budget and application scenario.

[0007] The following section introduces traditional visual odometry techniques. Traditional visual odometry techniques are divided into two types: the feature point method and the direct method.

[0008] Moravec pioneered the technique of estimating camera motion from continuous image sequences. He designed the Moravec corner detection technique to acquire image features and used it for indoor robot navigation. Although this technique has some obvious drawbacks and limited practicality, it still forms the basis for many subsequent visual odometry techniques. In 2004, Nister et al. designed a complete visual odometry system for navigation, capable of estimating the pose of monocular or binocular cameras. The system performs a series of tasks, including feature detection, feature tracking, feature matching, and estimation of camera pose using the geometric relationships between feature points. It can operate in real-time with low latency in unknown scenarios, truly realizing outdoor robot navigation.

[0009] Through continuous and in-depth research, visual odometry based on feature points has developed a mature framework. First, feature detection is performed according to the designed feature modules. Then, feature points in adjacent images are matched, optimized using an outlier removal algorithm, and motion estimation is performed using the relationships between feature points. Furthermore, the relationships between feature points, landmark points, and camera pose can be used to achieve local optimization of visual odometry. In this framework, feature modules in feature detection serve as a crucial foundation. Currently, classic feature detection algorithms include Moravec, Forstner, Harris, ShiTomasi, SUSAN, FAST, SIFT, SURF, MSER, and Centure, among which Harris and SIFT are the most widely used. Harris corner detection is an improvement on Moravec mentioned above; it is stable to image rotation, brightness, viewpoint changes, and noise effects, and is computationally simple. However, it is overly sensitive to scale and prone to mismatches in scenes with repetitive texture features. SIFT feature points are more suitable for application in visual odometry systems. SIFT is insensitive to image rotation, viewpoint changes, brightness changes, and scale changes. Furthermore, after optimization of SIFT features using TLBBA, it can meet the real-time requirements of visual odometry. Therefore, many visual odometry systems adopt SIFT features. Besides improvements to the feature module, researchers have also made progress in improving other parts of visual odometry. Min Haigen et al. proposed a feature point matching framework and an improved random sampling consistency outlier removal technique, which has better noise resistance and improves the performance of visual odometry in dynamic environments. Niu Wenyu et al. proposed a visual odometry based on dynamic object detection. By removing feature points from detected moving object regions, the influence of dynamic objects in the image on the estimation results is reduced, effectively improving the performance of visual odometry in dynamic scenes.

[0010] The above section introduced the development of visual odometry techniques based on feature points. Feature-point-based visual odometry is insensitive to noise, can operate stably, and has low cost, leading to its widespread use. For example, the open-source visual odometry method LIBVISO is one such technique. However, feature-point-based visual odometry relies on rich scene textures to extract a large number of effective features. In scenes with simple textures, this technique suffers a significant performance drop due to a lack of features. In reality, an image contains a wealth of information, but the feature-point method only utilizes the feature point information, ignoring most of the other information. Another technique in visual odometry, the direct method, directly utilizes the pixel information in the image, thus avoiding the aforementioned drawbacks of the feature-point method. The following section introduces visual odometry using the direct method.

[0011] Visual odometry using the direct method calculates camera motion by utilizing the brightness information of pixels in an image. Therefore, as long as there are changes in brightness within the scene, this method can estimate camera motion without requiring the extraction of feature points from the image. However, the direct method relies on the assumption that pixel grayscale values ​​remain constant, a condition easily violated by factors such as changes in lighting, leading to method failure. In contrast, the feature point method is robust to changes in lighting, a factor difficult to avoid in pose estimation. Therefore, despite the advantages, disadvantages, and applicable scenarios of both methods, the feature point method remains the mainstream technique in visual odometry.

[0012] With continuous improvements by researchers, traditional visual odometry has achieved good results in some scenarios. However, it usually has very high requirements for the application scenario, and different technologies may be required for different scenarios. In addition, it generally requires tedious and complex calculations, which are problems that traditional technologies have not yet solved.

[0013] (2) Deep learning method

[0014] With the development of deep learning, deep neural networks have been widely applied in various fields, and using deep learning techniques to study visual odometry is an inevitable trend. Compared with traditional feature-based visual odometry, deep learning-based techniques no longer use manually designed feature points, nor do they require feature extraction, feature matching, and complex geometric calculations, providing a new solution for visual odometry research.

[0015] In 2015, Konda first proposed an end-to-end, deep learning-based visual odometry. This odometry model receives a sequence of stereo images and uses two identical convolutional neural networks to predict changes in camera velocity and orientation in subsequences of every five frames. These predictions are then combined to estimate the complete camera motion path. Testing showed that this technique can accurately predict most changes in orientation and velocity. However, the authors also pointed out that due to the cumulative drift problem in visual odometry, the error in predicting the global camera motion trajectory gradually increases over time.

[0016] In 2016, Kendall proposed a real-time robust six-DOF localization system that uses a convolutional neural network to estimate the camera's six-DOF pose information from a monocular RGB image. It can be used in real time both indoors and outdoors and is robust to changes in lighting, motion blur, and different camera internal characteristics.

[0017] In 2017, Wang et al. established a novel end-to-end monocular visual odometry framework using recurrent convolutional neural networks (RNNs). This framework leverages convolutional neural networks to extract image features, while recurrent neural networks (RNNs) model the temporal information of the images, thereby improving the model's pose estimation capabilities. Their experiments showed that this technique was comparable to the state-of-the-art techniques at the time.

[0018] However, the deep learning-based visual odometry training described above requires ground truth camera poses, making it a supervised deep learning method. Typically, the accuracy of ground truth camera poses significantly impacts the performance of the trained odometry, thus requiring very strict control over its acquisition. Obtaining accurate ground truth usually necessitates the combined acquisition and processing of camera poses using multiple techniques, a costly process. Unsupervised learning-based visual odometry, on the other hand, eliminates the need for ground truth camera poses, reducing the training cost.

[0019] In 2017, Zhou proposed an unsupervised learning-based technique for estimating depth and motion in monocular images. This system consists of two parts: estimating depth in a single view and pose in multiple views. Photometric loss is used as the loss function to estimate camera motion. Experimental results show that the model can predict relative depth and pose relatively accurately. However, like other monocular frames, it suffers from scale ambiguity. Li also proposed unsupervised learning-based monocular visual odometry in 2017. This method can estimate the six degrees of freedom pose of a monocular camera without using ground truth camera pose data for training, and experiments show high accuracy. However, the pose estimation network model in this technique only uses convolutional neural networks to extract features, ignoring the correlation between relative camera poses and failing to utilize the sequence information of the images. Summary of the Invention

[0020] To address at least one of the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, and readable storage medium for self-supervised depth visual odometry acquisition based on geometric constraints.

[0021] According to one aspect of this disclosure, a self-supervised depth visual odometry acquisition method based on geometric constraints is provided, comprising:

[0022] Train the depth estimation network until the loss function satisfies the preset value, so that the depth estimation network converges;

[0023] Image depth information is obtained through a depth estimation network;

[0024] The left and right image sequences of the binocular images are respectively input into the pose estimation network to obtain the pose. The left and right image information of the binocular images includes the image depth information, which is used to recover the scale of the pose.

[0025] Preferably, the pose is a six-degree-of-freedom relative pose.

[0026] According to at least one embodiment of the geometrically constrained self-supervised depth visual odometry acquisition method of the present disclosure, the pose estimation network is formed by sequentially connecting a convolutional neural network, an attention mechanism module, a convolutional recurrent neural network, and a fully connected layer.

[0027] Preferably, the fully connected layer of the pose estimation network includes a Dropout layer, which discards some neurons with a preset probability during training, thus playing an averaging role and reducing complex co-adaptation among neurons.

[0028] According to at least one embodiment of the geometrically constrained self-supervised depth visual odometry acquisition method of this disclosure, the left and right image sequences of binocular images are respectively input into a pose estimation network to obtain the pose, including:

[0029] The left and right images of the stereo image are input into a convolutional neural network, and the image features are obtained through the convolutional neural network.

[0030] The image features are input into the attention model, and the image channel refinement features are obtained through the attention model.

[0031] The image channel refinement features are input into the convolutional neural network, and the relative pose is obtained through the convolutional neural network.

[0032] According to at least one embodiment of the geometrically constrained self-supervised depth visual odometry acquisition method of this disclosure, the image features are input into an attention model, and image channel refinement features are obtained through the attention model, including:

[0033] Spatial information is gathered using average and max pooling, with the pooling direction corresponding to the height and width dimensions of the image. Average and max spatial descriptors are obtained from the pooling layer.

[0034] These two descriptors are passed to the fully connected layer, which has two layers. The first fully connected layer has c×r neural network units, and the second fully connected layer has c neural network units. c is the channel dimension of the feature map, and r is a reduction factor of the number of units in the fully connected layer relative to the channel dimension.

[0035] The fully connected layer outputs a one-dimensional tensor as a channel attention mask. The attention mask and the original feature map are multiplied by channel to generate channel-refined features.

[0036] The calculation process can be expressed by the formula:

[0037] M c(F)=σ(FC(AvgPool(F))+FC(MaxPool(F))) (1)

[0038]

[0039] The meanings of each expression are as follows:

[0040] F: Image features input to the attention model;

[0041] F': Channel refinement feature;

[0042] σ: Represents the sigmoid activation function;

[0043] FC(·): Indicates fully connected layer processing;

[0044] This indicates pixel-by-pixel multiplication;

[0045] The channel refinement features are processed by applying average and max pooling along the channel axis;

[0046] The average and max pooling descriptors generated by the pooling layer are connected along the channel axis to obtain the feature descriptor. After performing a convolution operation on the feature descriptor, a spatial attention map can be obtained. The convolution kernel size is 7×7. The spatial attention map and the channel thinning feature are multiplied to obtain the final spatial thinning feature.

[0047] The calculation process can be expressed by the formula:

[0048] M c (F')=σ(f 7×7 (AvgPool(F');MaxPool(F'))) (3)

[0049]

[0050] The meanings of each expression are as follows:

[0051] σ: Represents the sigmoid activation function;

[0052] f 7×7 : This indicates a convolution operation with a kernel size of 7×7.

[0053] According to at least one embodiment of the present disclosure, a geometrically constrained self-supervised depth visual odometry acquisition method is provided, wherein the depth estimation network consists of an encoder, a decoder, and a skip connection structure. The skip connection structure merges and stitches feature maps of the same size from different layers along the channel dimension and then inputs them into the lower layer network. The skip connection structure enables the deconvolution layer to use shallower convolutional layer features when restoring the size, thereby enabling the decoder to restore the image resolution details.

[0054] Preferably, the decoder is implemented through deconvolution.

[0055] According to at least one embodiment of the geometrically constrained self-supervised depth visual odometry acquisition method of this disclosure, the loss function is:

[0056]

[0057] The meanings of each expression are as follows:

[0058] and These represent the spatial reconstruction losses in the left and right images, respectively.

[0059] and These represent the time reconstruction losses at time t and t+1, respectively.

[0060] L dis : Represents the disparity map reconstruction loss;

[0061] L pos : Indicates a loss of consistency;

[0062] λ ap : Indicates the percentage of space loss;

[0063] λ pic : Indicates the percentage of time lost;

[0064] λ dis : Indicates the weight of each term in the loss function;

[0065] λ pos : Indicates the percentage of consistency loss;

[0066] Among them, space loss and The spatial loss function is calculated using the following method:

[0067]

[0068]

[0069] The meanings of each expression are as follows:

[0070] The index representing the structural similarity of the right image in a binocular image;

[0071] The left image of a binocular image is represented by a structural similarity index.

[0072] This represents the mean absolute value of the pixel difference between the left image constructed using the L1 norm and the reconstructed image of the left image;

[0073] This represents the mean absolute value of the pixel difference between the right image constructed using the L1 norm and the reconstructed image of the right image;

[0074] N: Represents the number of pixels in the image;

[0075] α: represents the weight of structural similarity, and 1-α represents the weight of the mean absolute value of pixel difference;

[0076] The Structural Similarity Index (SSIM) is calculated as follows:

[0077] SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ (7)

[0078]

[0079]

[0080]

[0081] The meanings of each expression in the SSIM calculation formula are as follows:

[0082] x, y: represent the two images used for comparison;

[0083] l(x,y), c(x,y), s(x,y): represent the brightness, contrast, and structural information of the image, respectively;

[0084] α, β, γ: all greater than 0;

[0085] u x : Represents the average pixel value of image x;

[0086] u y : Represents the average pixel value of the image y-axis;

[0087] This represents the variance of the x-pixel values ​​in the image;

[0088] This represents the variance of the y-pixel values ​​in the image;

[0089] δ xy : Represents the covariance of x and y;

[0090] c1, c2, c3: These represent constants used to maintain stability;

[0091] Preferably, α = β = γ = 1, c3 = 0.5c2;

[0092] The method for calculating the mean absolute value of pixel differences constructed using the L1 norm is expressed as follows:

[0093]

[0094] The meanings of each expression are as follows:

[0095] I i,j and Let (i, j) represent the pixel values ​​of pixel (i, j) in the original image and the reconstructed image, respectively.

[0096] Among them, the time reconstruction loss at times t and t+1 and The loss function is calculated using the time reconstruction method, as follows:

[0097]

[0098]

[0099] The meanings of each expression are as follows:

[0100] I t : Represents the image at time t;

[0101] I t+1 : Represents the image at time t+1;

[0102] Indicates based on I t Reconstructed image;

[0103] Indicates based on I t+1 Reconstructed image;

[0104] I t i,j : Represents the value of the pixel at coordinates (i,j) in the image at time t;

[0105] This represents the value of the pixel at coordinates (i,j) in the reconstructed image at time t.

[0106] I t+1 i,j : Represents the value of the pixel at coordinates (i,j) in the image at time t+1;

[0107] This represents the value of the pixel at coordinates (i,j) in the reconstructed image at time t+1;

[0108] N: Represents the number of pixels;

[0109] Among them, the disparity map reconstruction loss L dis The disparity loss function is used for calculation, and the calculation method is expressed as follows:

[0110]

[0111] The meanings of each expression are as follows:

[0112] D l i,j : Represents the disparity value corresponding to the pixel with coordinates (i,j) in the left image;

[0113] This represents the disparity value corresponding to the pixel with coordinates (i,j) in the reconstructed image of the left image;

[0114] D r i,j : This represents the disparity value corresponding to the pixel with coordinates (i,j) in the right figure;

[0115] This represents the disparity value corresponding to the pixel with coordinates (i,j) in the reconstructed image on the right.

[0116] N: Represents the number of pixels;

[0117] Among them, the consistency loss L pos The consistency loss function is used for calculation, and the calculation method is as follows:

[0118] L pos =λ t ||t r -t l ||+λ r ||r r -r l || (14)

[0119] Preferably, α = β = γ = 1, c3 = 0.5c2, and substituting equations (8), (9), and (10) into (7), the SSIM expression is:

[0120]

[0121] According to at least one embodiment of the geometrically constrained self-supervised depth visual odometry acquisition method of this disclosure, the original image and the reconstructed image are reconstructed through corresponding pixels, including:

[0122] Feature matching is performed on two adjacent images, and the camera motion between the two frames is recovered based on the matched feature points. Let the motion between two adjacent frames be R and t, the centers of the two cameras be O1 and O2, and the coordinates of the spatial point P be:

[0123] P = [X, Y, Z] T (15)

[0124] P is the feature point p1 corresponding to image I1:

[0125] z1p1=KP (16)

[0126] The meanings of each expression are as follows:

[0127] P: The three-dimensional coordinates of the spatial point P corresponding to feature point p1;

[0128] p1: Homogeneous coordinates of feature point p1;

[0129] z1: The depth value of the camera corresponding to frame I1;

[0130] K: Camera internal parameters;

[0131] When z1 and K are known, the coordinates of spatial point P can be recovered from feature point p1 using equation (16), and the coordinate scale is determined.

[0132] The feature point p2 corresponding to P in I2:

[0133] z2p2=K(RP+t) (17)

[0134] The epipolar constraint can be derived from equations (16) and (17):

[0135]

[0136] Equation (18) contains the spatial positional relationship between two matching feature points P1 and P2. The camera pose can be recovered based on Equation (18) and multiple matching feature points.

[0137] The relationship between the feature points of P in two adjacent frames can be recovered using equations (16) and (17):

[0138] z2p2=K(z1RK -1 p1+t) (19)

[0139] The meaning of expression (19) is as follows:

[0140] On the right side of equation (19), the feature point p1 on the first frame image has a known spatial point depth z1. The position of the corresponding spatial point P in the current camera coordinates can be recovered using the camera intrinsic parameter matrix K. Then, the position of the spatial point in the camera coordinate system corresponding to the second frame image can be calculated based on the camera motion information R,t.

[0141] The left side of equation (19) establishes an equality relationship between feature point p2 on the second image and the right side of the equation, based on the camera intrinsic parameter matrix K, depth information z2, and the left side of the equation. Thus, the relationship between p1 and p2 is established.

[0142] When transforming the spatial position of a point in the camera coordinate system corresponding to the second frame image to the feature point p2, the corresponding depth information z2 has already been obtained from the spatial point coordinates. That is, when establishing the relationship between p1 and p2, it is not necessary to know the value of z2 in advance. p2 can be estimated from p1 using the relationship in equation (19). Let T represent the motion between the two frames, and write it in the form that multiplying one side of the equation by any non-zero constant will hold:

[0143]

[0144] It is an estimate of the location of feature point p2.

[0145] According to another aspect of this disclosure, a self-supervised depth visual odometry acquisition device based on geometric constraints is provided, comprising:

[0146] The depth estimation network training module trains the depth estimation network until the loss function meets the preset value, thus enabling the depth estimation network to converge.

[0147] The image depth information acquisition module obtains image depth information through a depth estimation network.

[0148] The pose acquisition module inputs the left and right image sequences of the binocular images into the pose estimation network to obtain the pose. The left and right image information of the binocular images includes the image depth information, which is used to recover the scale of the pose.

[0149] Preferably, the pose is a six-degree-of-freedom relative pose.

[0150] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0151] The memory stores execution instructions;

[0152] A processor that executes the execution instructions stored in the memory, causing the processor to perform any of the acquisition methods described above.

[0153] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the acquisition method described in any of the preceding claims. Attached Figure Description

[0154] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0155] Figure 1 This is a flowchart illustrating a geometrically constrained self-supervised depth visual odometry acquisition method according to one embodiment of the present disclosure.

[0156] Figure 2 This is a schematic diagram showing the detailed parameters of a convolutional neural network implementation according to one embodiment of the present disclosure.

[0157] Figure 3 This is a schematic diagram showing the implementation details of a depth estimation network according to one embodiment of the present disclosure.

[0158] Figure 4 This is a schematic diagram of the quantitative comparison results of pose estimation according to one embodiment of the present disclosure.

[0159] Figure 5 This is a schematic diagram of the pose estimation comparison results according to one embodiment of the present disclosure.

[0160] Figure 6 This is a schematic diagram of the quantitative comparison results of depth estimation according to one embodiment of the present disclosure.

[0161] Figure 7 This is a schematic diagram of the depth estimation comparison results according to one embodiment of the present disclosure.

[0162] Figure 8 This is a schematic diagram of a geometrically constrained self-supervised depth visual odometry acquisition device according to one embodiment of the present disclosure.

[0163] Explanation of reference numerals in the attached figures

[0164] 1000 Self-supervised depth visual odometry acquisition device based on geometric constraints

[0165] 1002 Depth Estimation Network Training Module

[0166] 1004 Image Depth Information Acquisition Module

[0167] 1006 Pose Acquisition Module

[0168] 1100 bus

[0169] 1200 processor

[0170] 1300 memory

[0171] 1400 Other circuits. Detailed Implementation

[0172] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0173] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0174] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0175] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0176] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0177] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that will be recognized by those skilled in the art.

[0178] Figure 1 This is a schematic flowchart of a geometrically constrained self-supervised depth visual odometry acquisition method according to one embodiment of the present disclosure.

[0179] like Figure 1 As shown, the self-supervised depth visual odometry acquisition method S100 based on geometric constraints includes:

[0180] S102: Train the depth estimation network until the loss function satisfies the preset value, so that the depth estimation network converges;

[0181] S104: Obtain image depth information through a depth estimation network; and,

[0182] S106: Input the left and right image sequences of the stereo images into the pose estimation network to obtain the pose. The left and right image information of the stereo images includes image depth information, which is used to recover the scale of the pose.

[0183] Preferably, the pose is a six-degree-of-freedom relative pose.

[0184] The pose estimation network is composed of a convolutional neural network, an attention mechanism module, a convolutional recurrent neural network, and a fully connected layer connected sequentially.

[0185] Preferably, the fully connected layers of the pose estimation network include Dropout layers. During training, the Dropout layers discard some neurons with a preset probability, which plays an averaging role and reduces complex co-adaptation among neurons.

[0186] Specifically, the left and right image sequences of the binocular images are input into the pose estimation network to obtain the pose, including:

[0187] The left and right images of the stereo image are input into a convolutional neural network, and the image features are obtained through the convolutional neural network.

[0188] Image features are input into an attention model, which then extracts image channel refinement features; and...

[0189] The image channel refinement features are input into a convolutional neural network, which then obtains the relative pose.

[0190] The convolutional neural network is Conv-LSTM.

[0191] The process involves inputting image features into an attention model to obtain image channel refinement features, including:

[0192] Spatial information is gathered using average and max pooling, with the pooling direction corresponding to the height and width dimensions of the image. Average and max spatial descriptors are obtained from the pooling layer.

[0193] These two descriptors are passed to the fully connected layer. There are two fully connected layers. The first fully connected layer has c×r neural network units, and the second fully connected layer has c neural network units. c is the channel dimension of the feature map, and r is a reduction factor of the number of units in the fully connected layer relative to the channel dimension.

[0194] The fully connected layer outputs a one-dimensional tensor as a channel attention mask. The attention mask and the original feature map are multiplied by channel to generate channel-refined features.

[0195] The calculation process can be expressed by the formula:

[0196] M c (F)=σ(FC(AvgPool(F))+FC(MaxPool(F))) (1)

[0197]

[0198] The meanings of each expression are as follows:

[0199] F: Image features input to the attention model;

[0200] F': Channel refinement feature;

[0201] σ: Represents the sigmoid activation function;

[0202] FC(·): Indicates fully connected layer processing;

[0203] This indicates pixel-by-pixel multiplication;

[0204] The channel refinement features are processed by applying average and max pooling along the channel axis;

[0205] The average and max pooling descriptors generated by the pooling layer are connected along the channel axis to obtain the feature descriptor. After performing a convolution operation on the feature descriptor, a spatial attention map can be obtained. The convolution kernel size is 7×7. The spatial attention map and the channel thinning feature are multiplied to obtain the final spatial thinning feature.

[0206] The calculation process can be expressed by the formula:

[0207] M c (F')=σ(f 7×7 (AvgPool(F');MaxPool(F'))) (3)

[0208]

[0209] The meanings of each expression are as follows:

[0210] σ: Represents the sigmoid activation function;

[0211] f 7×7 : This indicates a convolution operation with a kernel size of 7×7.

[0212] The depth estimation network consists of an encoder, a decoder, and a skip connection structure. The skip connection structure merges and stitches feature maps of the same size from different layers along the channel dimension before inputting them into the next layer network. The skip connection structure enables the deconvolutional layer to use shallower convolutional layer features when restoring the size, allowing the decoder to restore the image's resolvable details.

[0213] Preferably, the decoder is implemented through deconvolution.

[0214] The loss function is:

[0215]

[0216] The meanings of each expression are as follows:

[0217] and These represent the spatial reconstruction losses in the left and right images, respectively.

[0218] and These represent the time reconstruction losses at time t and t+1, respectively.

[0219] L dis : Represents the disparity map reconstruction loss;

[0220] L pos : Indicates a loss of consistency;

[0221] λ ap : Indicates the percentage of space loss;

[0222] λ pic : Indicates the percentage of time lost;

[0223] λ dis : Indicates the weight of each term in the loss function;

[0224] λ pos : Indicates the percentage of consistency loss;

[0225] Among them, space loss and The spatial loss function is calculated using the following method:

[0226]

[0227]

[0228] The meanings of each expression are as follows:

[0229] The index representing the structural similarity of the right image in a binocular image;

[0230] The left image of a binocular image is represented by a structural similarity index.

[0231] This represents the mean absolute value of the pixel difference between the left image constructed using the L1 norm and the reconstructed image of the left image;

[0232] This represents the mean absolute value of the pixel difference between the right image constructed using the L1 norm and the reconstructed image of the right image;

[0233] N: Represents the number of pixels in the image;

[0234] α: represents the weight of structural similarity, and 1-α represents the weight of the mean absolute value of pixel difference;

[0235] The Structural Similarity Index (SSIM) is calculated as follows:

[0236] SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ (7)

[0237]

[0238]

[0239]

[0240] The meanings of each expression in the SSIM calculation formula are as follows:

[0241] x, y: represent the two images used for comparison;

[0242] l(x,y), c(x,y), s(x,y): represent the brightness, contrast, and structural information of the image, respectively;

[0243] α, β, γ: all greater than 0;

[0244] u x : Represents the average pixel value of image x;

[0245] u y : Represents the average pixel value of the image y-axis;

[0246] This represents the variance of the x-pixel values ​​in the image;

[0247] This represents the variance of the y-pixel values ​​in the image;

[0248] δ xy : Represents the covariance of x and y;

[0249] c1, c2, c3: These represent constants used to maintain stability;

[0250] Preferably, α = β = γ = 1, c3 = 0.5c2;

[0251] The method for calculating the mean absolute value of pixel differences constructed using the L1 norm is expressed as follows:

[0252]

[0253] The meanings of each expression are as follows:

[0254] I i,j and Let (i, j) represent the pixel values ​​of pixel (i, j) in the original image and the reconstructed image, respectively.

[0255] Among them, the time reconstruction loss at times t and t+1 and The loss function is calculated using the time reconstruction method, as follows:

[0256]

[0257]

[0258] The meanings of each expression are as follows:

[0259] I t : Represents the image at time t;

[0260] It+1 : Represents the image at time t+1;

[0261] Indicates based on I t Reconstructed image;

[0262] Indicates based on I t+1 Reconstructed image;

[0263] I t i,j : Represents the value of the pixel at coordinates (i,j) in the image at time t;

[0264] This represents the value of the pixel at coordinates (i,j) in the reconstructed image at time t.

[0265] I t+1 i,j : Represents the value of the pixel at coordinates (i,j) in the image at time t+1;

[0266] This represents the value of the pixel at coordinates (i,j) in the reconstructed image at time t+1;

[0267] N: Represents the number of pixels;

[0268] Among them, the disparity map reconstruction loss L dis The disparity loss function is used for calculation, and the calculation method is expressed as follows:

[0269]

[0270] The meanings of each expression are as follows:

[0271] D l i,j : Represents the disparity value corresponding to the pixel with coordinates (i,j) in the left image;

[0272] This represents the disparity value corresponding to the pixel with coordinates (i,j) in the reconstructed image of the left image;

[0273] D r i,j : This represents the disparity value corresponding to the pixel with coordinates (i,j) in the right figure;

[0274] This represents the disparity value corresponding to the pixel with coordinates (i,j) in the reconstructed image on the right.

[0275] N: Represents the number of pixels;

[0276] Among them, the consistency loss Lpos The consistency loss function is used for calculation, and the calculation method is as follows:

[0277] L pos =λ t ||t r -t l ||+λ r ||r r -r l || (14)

[0278] Preferably, α = β = γ = 1, c3 = 0.5c2, and substituting equations (8), (9), and (10) into (7), the SSIM expression is:

[0279]

[0280] The reconstruction between the original image and the reconstructed image is performed using corresponding pixels, including:

[0281] Feature matching is performed on two adjacent images, and the camera motion between the two frames is recovered based on the matched feature points. Let the motion between two adjacent frames be R and t, the centers of the two cameras be O1 and O2, and the coordinates of the spatial point P be:

[0282] P = [X, Y, Z] T (15)

[0283] P is the feature point p1 corresponding to image I1:

[0284] z1p1=KP (16)

[0285] The meanings of each expression are as follows:

[0286] P: The three-dimensional coordinates of the spatial point P corresponding to feature point p1;

[0287] p1: Homogeneous coordinates of feature point p1;

[0288] z1: The depth value of the camera corresponding to frame I1;

[0289] K: Camera internal parameters;

[0290] When z1 and K are known, the coordinates of spatial point P can be recovered from feature point p1 using equation (16), and the coordinate scale is determined.

[0291] The feature point p2 corresponding to P in I2:

[0292] z2p2=K(RP+t) (17)

[0293] The epipolar constraint can be derived from equations (16) and (17):

[0294]

[0295] Equation (18) contains the spatial positional relationship between two matching feature points P1 and P2. The camera pose can be recovered based on Equation (18) and multiple matching feature points.

[0296] The relationship between the feature points of P in two adjacent frames can be recovered using equations (16) and (17):

[0297] z2p2=K(z1RK -1 p1+t) (19)

[0298] The meaning of expression (19) is as follows:

[0299] On the right side of equation (19), the feature point p1 on the first frame image has a known spatial point depth z1. The position of the corresponding spatial point P in the current camera coordinates can be recovered using the camera intrinsic parameter matrix K. Then, the position of the spatial point in the camera coordinate system corresponding to the second frame image can be calculated based on the camera motion information R,t.

[0300] The left side of equation (19) establishes an equality relationship between feature point p2 on the second image and the right side of the equation, based on the camera intrinsic parameter matrix K, depth information z2, and the left side of the equation. Thus, the relationship between p1 and p2 is established.

[0301] When transforming the spatial position of a point in the camera coordinate system corresponding to the second frame image to the feature point p2, the corresponding depth information z2 has already been obtained from the spatial point coordinates. That is, when establishing the relationship between p1 and p2, it is not necessary to know the value of z2 in advance. p2 can be estimated from p1 using the relationship in equation (19). Let T represent the motion between the two frames, and write it in the form that multiplying one side of the equation by any non-zero constant will hold:

[0302]

[0303] It is an estimate of the location of feature point p2.

[0304] This disclosure presents a geometrically constrained, self-supervised depth visual odometry acquisition method, using the KITTI VisualOdometry benchmark as the training and testing dataset for the designed visual odometry. The KITTI VisualOdometry benchmark is a publicly available outdoor vehicle driving dataset collected by Geiger et al., widely used to evaluate various VO algorithms. Data acquisition platforms include mobile platforms, binocular color cameras, GPS, and radar scanners. The KITTI VObenchmark dataset includes 22 outdoor scenes, encompassing urban, rural, and highway scenarios. Each scene's data consists of a series of consecutive images captured by a binocular camera. The first 11 sequences, in addition to binocular image data, also provide ground truth data of the vehicle's driving pose calculated from precise localization by various onboard sensors, as well as laser point cloud data. All sequences in this dataset are street scene images, with each image containing up to 15 vehicles and 30 pedestrians. There are also various degrees of occlusion and truncation. In addition, there are large variations in brightness, large changes in perspective when cars turn, large changes in relative pose caused by inconsistent movement speeds, and issues such as road surface reflection. These are all very close to the actual use cases of odometry, but they also bring great challenges to visual odometry.

[0305] The ground truth data for camera pose provided by KITTI VO is the absolute pose of the camera. Each image corresponds to an [Rt] matrix, where R is the rotation matrix of the current frame image relative to the corresponding camera pose of the first frame image, and t is its position in the coordinate system of the first frame image. This matrix is ​​stretched into a 12-d vector and written in the ground truth data.

[0306] This disclosure presents a geometrically constrained, self-supervised deep visual odometry acquisition method built on the open-source framework Tensorflow, trained using the Adam optimizer, and pre-trained using the FlowNet-Simple model. The initial learning rate is set to 1e-4, with parameters beta1 = 0.9 and beta2 = 0.99. Training is conducted for 60 epochs, with the learning rate decaying to its original value every 20 epochs. The training samples are shuffled again after each round. Supervised learning-based visual odometry uses monocular image sequences and ground truth data as labels for training. The KITTI VO benchmark sequence 0, 1, 2, 3, 4, 5, 6, 7, 8 is used for training, and the sequence 9, 10 is used for testing. The input image size is set to 416×128×3. Data augmentation operations are randomly applied to the images, including: brightness adjustment: changing the image brightness by adjusting the image pixel values. The brightness enhancement coefficient γ∈[0.7,1.3]; cropping: expanding the input image by a scaling factor s∈[1,1.2] and then randomly cropping it according to the input image size; and rotation: randomly rotating the input image r∈[-5,5] degrees. Data augmentation operations are widely used in various deep learning algorithms to process data. They can increase the amount of training data, improve the model's generalization ability, and also increase data noise, improving the model's robustness.

[0307] Meanwhile, to avoid overfitting, this disclosure adds a Dropout layer to the fully connected layers of the pose estimation network. During training, the Dropout layer discards some neurons with probability p, effectively averaging the results and reducing complex co-adaptation between neurons. In this disclosure, the probability p is set to 0.5, and a Dropout layer is added between every two fully connected layers.

[0308] During training, depth is recovered using stereo images from binocular cameras; therefore, a sequence of binocular images is used to train both the pose estimation network and the depth estimation network simultaneously. Testing uses monocular images, allowing individual testing of either the pose estimation network or the depth estimation network.

[0309] This disclosure provides a deep network model with good experimental results, constructs a robust loss function that considers multiple factors, and establishes a deep recovery model. In practical use, when the above model or loss function is changed, the purpose of pose estimation can still be achieved, but the pose estimation accuracy will be different.

[0310] This disclosure utilizes self-supervised learning to estimate pose from neighboring images, and these methods have achieved satisfactory results on corresponding public datasets. The methods provided in this disclosure can be applied to the localization and navigation of mobile devices such as robots and drones equipped with monocular vision cameras.

[0311] Figure 2 This is a detail of the neural network parameters according to one embodiment of the present disclosure.

[0312] The feature extraction part of this disclosure is based on the convolution part of FlowNetSimple, and its specific implementation details can be found in [link to implementation details]. Figure 2The kernel size was transitioned from 7×7 to 5×5, and finally maintained at 3×3. The first layer had 3 zeros, the second and third layers had 2 zeros, and the remaining layers had 1 zero. The two images whose relative pose changes to be estimated were stacked in the channel dimension and then input into the convolutional network to extract features.

[0313] LSTM's excellent performance has led to its widespread use in tasks processing sequential inputs; however, it is not optimal for image processing tasks. The LSTM network uses fully connected layers for feature extraction, requiring the input to be stretched into a one-dimensional tensor. This causes LSTM to disrupt the spatial structure of features to some extent when extracting temporal features. To address the problems of LSTM, this section improves upon it by using Conv-LSTM units. First, we introduce a variant of LSTM, FC-LSTM. The difference between FC-LSTM and LSTM is that its gate layers also accept cell state inputs. Its internal structure can be represented by the following formula:

[0314]

[0315] Conv-LSTM is based on the structure of FC-LSTM, but it replaces the fully connected layer computations within LSTM with convolutional neural network computations, where both the network input and output are three-dimensional tensors. In fact, traditional LSTM can be considered a special case of Conv-LSTM, since fully connected layers are also a type of convolution. For Conv-LSTM, the feature map is padded during convolution to maintain the same input and output size for the convolution computation. Specifically, the internal operations of Conv-LSTM are as follows:

[0316]

[0317] In the formula, * represents a convolution operation. The convolution operation replaces the fully connected layer, enabling the Conv-LSTM unit to extract spatiotemporal information. It not only adds constraints between pose information across multiple frames, but also adds constraints between image features across multiple frames, which is more in line with the idea of ​​BA optimization.

[0318] The final pose estimation network consists of a convolutional neural network, an attention mechanism block, a convolutional recurrent neural network, and a fully connected layer connected sequentially. The network input is a continuous sequence of images, and the output is the corresponding relative pose with six degrees of freedom.

[0319] First, two consecutive images are stacked together along the channel dimension. A sequence of N images can be processed into N-1 tensors using the sliding window method. The input first enters a convolutional network, where the convolutional neural network extracts the image features. These features are then input into an attention mechanism module. The features are processed sequentially by channel attention and spatial attention mechanisms, outputting refined features. The input and output sizes of the attention module are the same. Finally, the refined feature map is input into a Conv-LSTM.

[0320] This disclosure uses two Conv-LSTM layers, each containing a convolutional neural network with a kernel size of 5×5×256 and padding using SAME. The time step is set to 8. The output features are flattened into 1-D vectors and then input into fully connected layers. The purpose of the fully connected layers is to perform feature dimensionality reduction and generate the final result using the image features extracted by the preceding networks. This disclosure designs a total of 3 fully connected layers after the Conv-LSTM, with 1000, 128, and 6 neural network units respectively.

[0321] The final pose estimation network takes an image sequence as input and outputs the corresponding six-degree-of-freedom relative pose. The six-dimensional relative pose estimated by the network is as follows: (Δx, Δy, Δz, Δψ, Δχ, Δφ).

[0322] Figure 3 This is a schematic diagram showing the implementation details of a depth estimation network according to one embodiment of the present disclosure.

[0323] To recover image depth, an image depth estimation network needs to be built to estimate depth pixel by pixel. Dense prediction problems involving pixel-by-pixel prediction in computer vision are often solved using encoder-decoder structures, such as semantic segmentation. Convolutional networks, after multiple convolutional kernel pooling operations, reduce the image size, and the final output cannot correspond to every pixel of the original image. Decoder structures, on the other hand, can map the low-resolution feature image output by the encoder back to the size of the input image, enabling pixel-by-pixel prediction.

[0324] The decoder is implemented using deconvolution, which primarily uses deconvolution for upsampling, gradually restoring the feature map to its original size. Deconvolution, also known as transposed convolution, can be understood as the inverse operation of convolution, but it is not a complete inverse of forward convolution; rather, it is a special type of convolution. It first enlarges the input image size by padding with zeros according to a certain ratio, then rotates the convolution kernel to perform convolution. Deconvolution can only restore the size, not the original values ​​of the feature map before convolution.

[0325] In addition, encoder-decoder architectures often use skip connections. Skip connections merge and stitch feature maps of the same size from different layers along the channel dimension before inputting them into the next layer. This structure allows the deconvolutional layer to utilize shallower convolutional layer features when restoring dimensions. Compared to deep features, shallow features of convolutional layers have more local detail information, which can help the decoder restore higher resolution details.

[0326] In this embodiment, instead of directly constructing a depth estimation network, the image disparity map is first estimated, and then the image depth is recovered using the relationship between depth and disparity. This disclosure designs a depth estimation network based on an encoder-decoder + skip connection structure; network details can be found in [link to network details]. Figure 3 The network, as shown in the table, is based on an encoder-decoder architecture. The encoder consists of 7 convolutional modules, each containing two convolutional layers with the same kernel size and number of channels. Correspondingly, the decoder also has 7 deconvolutional modules, each consisting of one deconvolutional layer and one convolutional layer, both with the same kernel size and number of channels. The decoder utilizes skip connections to help the deconvolutional layers recover higher-resolution details. The last four deconvolutional modules of the decoder sequentially output four disparity maps at different scales through a convolutional layer output module with a sigmoid activation function. Each disparity map has twice the resolution of the previous one. The sigmoid function ensures that the output value is between 0 and 1, a value normalized relative to the width of the input image.

[0327] Figure 4 This is a quantitative comparison result of pose estimation based on one embodiment of the present disclosure.

[0328] like Figure 4 As shown, from Figure 4 As can be seen from the results, the quantitative evaluation results of visual odometry based on self-supervised learning, the traditional monocular visual odometry method Viso2-mono, and the deep learning method SfM-Learner based on self-supervised learning on the test set are as follows: Figure 4 The trajectory of the response is visible. Figure 5

[0329] The results of visual odometry based on self-supervised learning, the traditional monocular visual odometry method Viso2-mono, and the deep learning method SfM-Learner based on self-supervised learning on the test set are quantitatively evaluated as follows: Figure 4 .

[0330] Figure 5 This is a schematic diagram of pose estimation results according to one embodiment of the present disclosure.

[0331] like Figure 5 As shown, the pose estimation trajectories of different methods are presented. The method in this disclosure outperforms traditional monocular visual odometry and the self-supervised deep learning method SfM-Learner. Compared with supervised learning-based methods, when trained on the same training set until convergence, the visual odometry based on self-supervised learning slightly lags behind the visual odometry based on supervised learning. However, this may be because the visual odometry based on self-supervised learning is limited by the estimation performance of the depth estimation network and affected by errors in image reconstruction techniques. In practical applications, the advantage of visual odometry based on self-supervised learning lies in its significantly reduced difficulty in dataset collection, allowing for training on more datasets at a lower cost, thereby improving its generalization ability and final performance. Correspondingly,

[0332] Figure 6 This is a schematic diagram illustrating a quantitative comparison of depth estimation results according to one embodiment of the present disclosure.

[0333] like Figure 6 As shown in the figure, a quantitative comparison diagram of depth estimation results is presented, and the error evaluation indicators are shown in the table below.

[0334]

[0335] In the table above, the meanings of each expression are: n: total number of pixels; D i : The estimated depth value of the i-th pixel; The true depth value corresponding to the i-th pixel; T: threshold

[0336] The primary task of visual odometry based on self-supervised learning is to estimate the relative pose of the camera. The depth estimation mechanism is designed to implement the self-supervised learning mechanism, and its performance affects the performance of the pose estimation network. This disclosure evaluates the system. Figure 6 The following images randomly display a comparison of the true depth values ​​and the results of the SfM-Learner method on some images. The true depth map values ​​are obtained after processing. While KITTI provides LiDAR data containing depth information, this LiDAR point cloud data is relatively sparse. Directly mapping the LiDAR data onto the image would result in many mismatched points; therefore, interpolation is performed to ensure it corresponds to the color image. The error evaluation metrics used are those in the aforementioned error evaluation metric table, with 50m as the maximum depth threshold for quantitative comparison of the depth estimation methods. Figure 6It is known that the method disclosed herein is superior to the methods of SfM-Learner and Eigen et al., but slightly inferior to the methods of Garg et al. It is hypothesized that when the depth estimation network and pose estimation network are trained simultaneously, the training of the pose estimation network introduces noise into the training of the depth estimation network. However, the depth estimation result is actually only an intermediate value, its main function being to assist the pose estimation network in constructing the loss function. Correspondingly, Figure 7 This is a visualization of the depth estimation comparison results.

[0337] Figure 7 This is a schematic diagram of the depth estimation comparison results according to one embodiment of the present disclosure.

[0338] like Figure 7 As shown, the visualization compares the depth estimation results of different methods when inputting different images. It should be noted that, for the sake of visualization, the ground truth data is interpolated from the laser data.

[0339] This disclosure constructs a visual odometry model based on self-supervised learning. The system input is a continuous sequence of monocular images, and the output is the corresponding six-DOF relative pose. The key feature is that it no longer uses data with ground truth pose values ​​for training. Instead, it incorporates ideas from traditional visual odometry methods, utilizing the geometric relationships between adjacent images to recover the pixel correspondences between images for image reconstruction. The estimated image is then compared with the original image to construct a loss function. Since the scale of pose reconstruction requires image depth information, this disclosure also includes a depth estimation network. This network is based on an encoder-decoder structure. The encoder extracts image features, the decoder recovers the image resolution, and finally, the input image is estimated pixel-by-pixel, outputting a depth estimation map. The depth estimation network primarily utilizes the geometric relationships of stereo camera images, reconstructing the image using bilinear sampling, and then comparing the estimated map with the original image to construct a spatial loss function. The model is trained using a sequence of stereo images, and tested using a sequence of monocular images. Testing shows that this disclosure outperforms general techniques.

[0340] Figure 8 This is a schematic diagram of a geometrically constrained self-supervised depth visual odometry acquisition device according to one embodiment of the present disclosure.

[0341] like Figure 8 As shown, the self-supervised depth visual odometry acquisition device 1000 based on geometric constraints includes:

[0342] The depth estimation network training module 1002 trains the depth estimation network until the loss function meets the preset value, so that the depth estimation network converges.

[0343] Image depth information acquisition module 1004 acquires image depth information through a depth estimation network; and...

[0344] The pose acquisition module 1006 inputs the left and right image sequences of the stereo images into the pose estimation network to obtain the pose. The left and right image information of the stereo images includes image depth information, which is used to recover the scale of the pose.

[0345] Preferably, the pose is a six-degree-of-freedom relative pose.

[0346] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0347] The memory stores the instructions to be executed.

[0348] The processor executes the execution instructions stored in memory, causing the processor to perform any of the methods described above.

[0349] According to another aspect of this disclosure, a readable storage medium is provided that stores executable instructions, which, when executed by a processor, are used to implement the method described above.

[0350] Figure 7 An example diagram of an apparatus employing a hardware implementation of a processing system is shown. This apparatus may include corresponding modules that perform one or more steps of the flowchart described above. Therefore, each or more steps of the flowchart may be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0351] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0352] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0353] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0354] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0355] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0356] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0357] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware, and the program can be stored in a readable storage medium. When executed, the program includes one or a combination of the steps of the method implementation.

[0358] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0359] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0360] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0361] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A self-supervised depth visual odometry acquisition method based on geometric constraints, characterized in that, include: Train the depth estimation network until the loss function satisfies the preset value, so that the depth estimation network converges; Image depth information is obtained through a depth estimation network; as well as The left and right image sequences of the binocular images are respectively input into the pose estimation network to obtain the pose. The left and right image information of the binocular images includes the image depth information. The image depth information is used to recover the scale of the pose. The pose estimation network is formed by sequentially connecting a convolutional neural network, an attention mechanism module, a convolutional recurrent neural network, and a fully connected layer. The pose is a six-degree-of-freedom relative pose. Specifically, the left and right image sequences of the binocular images are input into the pose estimation network to obtain the pose, including: The left and right images of the binocular image are input into the convolutional neural network, and the image features are obtained through the convolutional neural network. The image features are input into an attention model, and the attention model is used to obtain image channel thinning features; and The image channel thinning features are input into the convolutional neural network, and the relative pose is obtained through the convolutional neural network; A self-supervised depth visual odometry acquisition method based on geometric constraints; The loss function is: The meanings of each expression are as follows: and : These represent the spatial reconstruction losses of the left and right images, respectively; and : These represent the time reconstruction losses at time t and t+1, respectively; : Represents the disparity map reconstruction loss; : Indicates a loss of consistency; : Indicates the percentage of space loss; : Indicates the percentage of time lost; : Indicates the weight of each term in the loss function; : Indicates the percentage of consistency loss; Among them, space loss and The spatial loss function is calculated using the following method: (18) (19) The meanings of each expression are as follows: : Represents the structural similarity index of the right image in a binocular image; : Represents the structural similarity index of the left image in a binocular image; : Represents the mean absolute value of the pixel difference between the left image constructed using the L1 norm and the reconstructed image of the left image; : Represents the mean absolute value of the pixel difference between the right image constructed using the L1 norm and the reconstructed image of the right image; N: Represents the number of pixels in the image; : Represents the weight of structural similarity, 1- This indicates the weighting of the mean absolute value of pixel differences. The Structural Similarity Index (SSIM) is calculated as follows: (14) (15) (16) (17) The meanings of each expression in the SSIM calculation formula are as follows: : These represent two images used for comparison; These represent the brightness, contrast, and structural information of the image, respectively. All are greater than 0; : Represents an image The average pixel value; : Represents an image The average pixel value; : Represents an image Variance of pixel values; : Represents an image Variance of pixel values; :express and covariance; : These represent constants used to maintain stability; in, ; The method for calculating the mean absolute value of pixel differences constructed using the L1 norm is expressed as follows: The meanings of each expression are as follows: and : Represents the pixel value of pixel (i, j) in the original image and the reconstructed image, respectively; Among them, the time reconstruction loss at times t and t+1 and The loss function is calculated using the time reconstruction method, as follows: (20) (21) The meanings of each expression are as follows: : Represents the image at time t; : Represents the image at time t+1; : indicates based on Reconstructed image; : indicates based on Reconstructed image; : This indicates the coordinates in the image at time t. The pixel value; : Represents the coordinates in the reconstructed image at time t. The pixel value; : This represents the coordinates in the image at time t+1. The pixel value; : Represents the coordinates in the reconstructed image at time t+1. The pixel value; N: Represents the number of pixels; Among them, disparity map reconstruction loss The disparity loss function is used for calculation, and the calculation method is expressed as follows: (22) The meanings of each expression are as follows: : indicates that the coordinates in the left figure are The disparity value corresponding to the pixel; : indicates that the coordinates in the reconstructed left figure are The disparity value corresponding to the pixel; : indicates that the coordinates in the right figure are The disparity value corresponding to the pixel; : indicates that the coordinates in the reconstructed graph on the right are The disparity value corresponding to the pixel; N: Represents the number of pixels; Among them, consistency loss The consistency loss function is used for calculation, and the calculation method is as follows: (23) in, Substituting equations (15), (16), and (17) into (14), the SSIM expression becomes: 。 2. The self-supervised depth visual odometry acquisition method based on geometric constraints according to claim 1, characterized in that, The fully connected layers of the pose estimation network include Dropout layers, which discard some neurons with a preset probability during training, thus playing an averaging role and reducing complex co-adaptation among neurons.

3. The self-supervised depth visual odometry acquisition method based on geometric constraints according to claim 1, characterized in that, The image features are input into an attention model, and the attention model is used to obtain image channel refinement features, including: Spatial information is gathered using average and max pooling, with the pooling direction corresponding to the height and width dimensions of the image. Average and max spatial descriptors are obtained from the pooling layer. These two descriptors are passed to a fully connected layer, which has two layers. The first fully connected layer has... There are 1 neural network unit, and the second fully connected layer has 1 One neural network unit It is the channel dimension size of the feature map. It is a reduction factor for the number of fully connected layer units relative to the channel dimension size; The fully connected layer outputs a one-dimensional tensor as a channel attention mask. The attention mask and the original feature map are multiplied by channel to generate channel-refined features. The calculation process can be expressed by the following formula: (1) (2) The meanings of each expression are as follows: Image features that are input to the attention model; Channel refinement features; : Represents the sigmoid activation function; FC(): Indicates fully connected layer processing; : indicates pixel-by-pixel multiplication; The channel refinement features are processed by applying average and max pooling along the channel axis; The average and max pooling descriptors generated by the pooling layer are concatenated along the channel axis to obtain the feature descriptor. A spatial attention map is then obtained by performing a convolution operation on the feature descriptor, with a kernel size of [size missing]. The spatial attention map and the channel refinement feature are multiplied together to obtain the final spatial refinement feature; The calculation process can be expressed by the following formula: (3) (4) The meanings of each expression are as follows: : Represents the sigmoid activation function; : indicates that the kernel size is The convolution operation.

4. The self-supervised depth visual odometry acquisition method based on geometric constraints according to claim 1, characterized in that, The depth estimation network consists of an encoder, a decoder, and a skip connection structure. The skip connection structure merges and concatenates feature maps of the same size from different layers along the channel dimension before inputting them into the next layer network. The skip connection structure enables the deconvolutional layer to use shallower convolutional layer features when restoring the size, allowing the decoder to restore the image's resolvable details. The decoder is implemented through deconvolution.

5. The self-supervised depth visual odometry acquisition method based on geometric constraints according to claim 4, characterized in that, The original image and the reconstructed image are reconstructed using corresponding pixels, including: Feature matching is performed on two adjacent images, and the camera motion between the two frames is recovered based on the matched feature points. Let the camera motion between two adjacent frames be denoted as . The centers of the two cameras are respectively spatial point The coordinates are: (7) In the picture Corresponding feature points : (8) The meanings of each expression are as follows: Feature points Corresponding spatial point 3D coordinates; Feature points The corresponding homogeneous coordinates; : in frame The corresponding depth value of the camera; Camera internal parameters; When known and At that time, from the feature points using equations (8) and (1.8) Recover the space point The coordinates are determined, and the coordinate scale is fixed; exist Corresponding feature points : (9) The epipolar constraint is derived from equations (8)(1.8) and (9)(1.9): (10) Equation (10)(1.10) contains the spatial positional relationship between two matching feature points P1 and P2. The camera pose is recovered based on Equation (10)(1.10) and multiple matching feature points. The results can be recovered from equations (8)(1.8) and (9)(1.9). The relationship between corresponding feature points in two adjacent image frames: (11) The meaning of expression (11) is as follows: The right-hand side of equation (11) shows the feature points on the first frame image. The depth of its corresponding spatial point is known. Using the camera intrinsic parameter matrix Recover its corresponding spatial point The position at the current camera coordinates, and then based on the camera motion information. Calculate the position of the spatial point in the camera coordinate system corresponding to the second frame image; The feature points on the left side of equation (11) in the second image From the camera intrinsic parameter matrix Depth information Establish an equality relationship with the right side of the equation. and The relationship was thus established; From the spatial position of the spatial point in the camera coordinate system corresponding to the second frame image to the feature point During the conversion, its corresponding depth information It has been obtained from the coordinates of the spatial point, that is, in the process of establishing and No need to know in advance about the relationship The value of is obtained from the relationship in equation (11)(1.11). Estimate ,use To represent motion between two frames, we can write it in the form that multiplying one side of the equation by any non-zero constant holds true: (12) Feature points An estimated location.

6. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the acquisition method as described in any one of claims 1 to 5.

7. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the acquisition method as described in any one of claims 1 to 5.

Images