A Deep Learning-Based Image Robust Visual Odometry Construction Method
By guiding optical flow estimation and adding confidence information in visual odometry, the accuracy problems of optical flow and pose estimation in blurred scenes are solved, achieving more robust pose estimation and improving the accuracy of visual odometry.
Patent Information
- Application Number
- CN202310983442.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-07
- Publication Date
- 2026-01-30
- Estimated Expiration
- 2043-08-07
AI Technical Summary
Existing deep learning-based optical flow and pose estimation networks have poor accuracy in blurry scenes, making it difficult to achieve accurate pose estimation.
By training a motion extraction network to extract motion information of pixels in a blurred image and guiding optical flow estimation, the confidence information of the optical flow map is increased. By combining supervised and unsupervised training methods, the pose estimation network is optimized to improve robustness.
More accurate and robust camera pose estimation was achieved in blurred scenes, improving the accuracy of visual odometry.
Smart Images
Figure CN117079072B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robust construction, and more particularly to a method for robustly constructing visual odometry based on deep learning. Background Technology
[0002] In recent years, the fields of mobile robotics and autonomous driving have developed rapidly, and autonomous localization, as a key component, has attracted increasing research and attention. Autonomous localization is mainly achieved through LiDAR and visual sensors. Among these, visual sensors are relatively inexpensive and have a wide range of applications, thus Visual SLAM based on this sensor is widely used.
[0003] Visual odometry currently falls into two categories: those based on traditional methods and those based on deep learning. Traditional methods achieve relatively good pose estimation results in clear scenes, but their accuracy degrades significantly in challenging scenarios. In contrast, deep learning-based methods have greater potential for handling challenging situations. Currently, deep learning-based methods primarily rely on optical flow estimation, but existing optical flow and pose estimation networks suffer severe degradation in blurry scenes. Therefore, the problem of achieving accurate pose estimation in blurry scenes remains unsolved. Summary of the Invention
[0004] To address the aforementioned problems, this invention proposes a method for robustly constructing visual odometry based on deep learning. This invention extracts motion information of pixels from blurred images to guide optical flow estimation and adds confidence information of the optical flow graph, thereby achieving robust and accurate pose estimation.
[0005] The technical solution of the present invention to solve the above-mentioned technical problems is:
[0006] A method for robustly constructing visual odometry based on deep learning includes the following steps:
[0007] (1) Train the motion extraction network using supervised training. Use the trained motion extraction network to output the pixel motion vectors based on the blur trails of the image to obtain the motion information of the pixels in frame i+1, where i≥0.
[0008] (2) Train the optical flow estimation network in an unsupervised training manner. The network takes the motion information of the two adjacent frames of the i-th frame and the i+1-th frame and the pixel of the i+1-th frame as input and outputs the estimated optical flow map.
[0009] (3) The pose estimation network is trained in a supervised training manner. The network takes the estimated optical flow map and its confidence information as input and outputs the estimated camera pose.
[0010] Furthermore, the training structure of the motion extraction network in step (1) includes two modules: a blur generation module and a motion extraction module. The supervision labels for its training are the zero offset of pixels in the real blurred image and the clear image. The training process is divided into two streamlines: Streamline 1 is to input a blurred image and a clear image of the same scene. The motion extraction network extracts the motion information of the blurred image. The blur generation module blurs the clear image based on the extracted motion information. Then, the generated blurred image and the real blurred image form a supervision relationship. Streamline 2 is to input a clear image. The motion extraction network extracts the motion information of the clear image. The generated motion offset and the zero offset form a supervision relationship.
[0011] Specifically, in step (1), the total loss generated by the training framework of the motion extraction network is the sum of streamline one loss and streamline two loss; wherein, the streamline one loss is expressed as... Streamline loss is expressed as The specific format is as follows:
[0012] (1) Difference loss between the blurred image generated by the blur generation module and the real blurred image
[0013]
[0014] in, and The l2 loss and SSIM (Structural Similarity Index) loss are respectively the generated blurred image B and the real blurred image B.
[0015] (2) Vector smoothing loss of motion information
[0016]
[0017] Where H and W are the height and width of the image, respectively, and V(x,y) is the unit motion vector of a pixel, x mid and y mid The center point of the convolution kernel;
[0018] (3) Loss of offset magnitude of motion information
[0019]
[0020] Where D(x,y) is the motion vector corresponding to the pixel, and α is the upper limit of the pixel offset.
[0021] Specifically, the total loss of the training framework of the motion extraction network is
[0022] Further, step (2) specifically involves: firstly, using a convolutional network to extract features from two frames of images from top to bottom or bottom to top according to the pyramid principle; then, constructing a matching cost layer using the features extracted from each layer; and using the pixel motion vectors output by the motion extraction network as prior information to guide the construction of the matching cost region; finally, inputting the extracted feature layer and the constructed matching cost layer into the optical flow estimation network to calculate the required optical flow map.
[0023] Furthermore, in step (2), the optical flow estimation network is trained in an unsupervised manner, using two adjacent frames I1 and I2 as inputs to generate the estimated optical flow map F. 12 Then, the optical flow map is applied to image I2, and backward warping is used to generate the estimated image I1; the total loss of this training structure Photometric consistency loss of images I1 and I1 and edge smoothing loss of optical flow map The sum, in the following form:
[0024]
[0025] Where, λ phot , These are the weights corresponding to the photometric uniformity loss and the edge smoothing loss, respectively. The Charbonnier loss function is used. Let β be the partial derivative along the x and y directions, and β be the weighting variable.
[0026] Furthermore, in step (3), the confidence information layer used by the pose estimation network has the same resolution as the optical flow map, and is generated from the scalar information of the motion vector output by the motion extraction network. The confidence level of each pixel is inversely proportional to the magnitude of the motion corresponding to that pixel. The pose estimation network is trained using supervised training. During training, the optical flow map and the confidence information layer are used as inputs, and the six-degree-of-freedom pose of the camera is used as the supervision label. The total loss of this training structure is... The specific format is as follows:
[0027]
[0028] in, R represents the estimated translational pose and rotational pose, respectively, while T and R represent the corresponding true poses.
[0029] Specifically, the method uses two adjacent frames as input and the camera's six-DOF pose as supervision labels for joint training of the overall network. The trained network includes an optical flow estimation network and a pose estimation network, while the motion information extraction network does not have its parameters updated during training.
[0030] The beneficial effects of this invention are as follows:
[0031] This invention utilizes the motion information contained in blurred images as prior information, increases the accuracy of optical flow estimation by constraining the distribution of the optical flow matching cost region, and adds confidence information to the optical flow map, ultimately achieving robust estimation of camera pose in blurred scenes. Attached Figure Description
[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0033] Figure 1 This is a schematic diagram of the overall system structure of the present invention;
[0034] Figure 2 This is a schematic diagram of the motion extraction network training structure of the present invention;
[0035] Figure 3 This is a schematic diagram of the unsupervised optical flow network training structure of the present invention;
[0036] Figure 4 This is a schematic diagram of the pose estimation network training structure of the present invention. Detailed Implementation
[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0038] like Figure 1 As shown, a method for robustly constructing visual odometry based on deep learning includes the following steps:
[0039] Step 1: First, train the motion retrieval network using supervised training.
[0040] like Figure 2 As shown, the training structure of the motion extraction network consists of two modules: a blur generation module and a motion extraction module. The supervision labels for training are the zero offsets of pixels in the real blurred image and the sharp image. The training process is divided into two streamlines. Streamline 1: Input a blurred image and a sharp image of the same scene. The motion extraction network extracts the motion information of the blurred image. The blur generation module blurs the sharp image based on the extracted motion information. The generated blurred image and the real blurred image form a supervision relationship. Streamline 2: Input a sharp image. The motion extraction network extracts the motion information of the sharp image. The generated motion offset and the zero offset form a supervision relationship.
[0041] The total loss of the motion extraction network training framework is the sum of streamline 1 loss and streamline 2 loss; where streamline 1 loss is expressed as... The loss of streamline 2 is expressed as The specific format is as follows:
[0042] (1) Difference loss between the blurred image generated by the blur generation module and the real blurred image
[0043]
[0044] in, and These represent the l2 loss and SSIM (Structural Similarity Index) loss for the generated blurred image B and the real blurred image B, respectively.
[0045] (2) Directional smoothing loss of extracted motion information
[0046]
[0047] Where H and W are the height and width of the image, respectively, and V(x,y) is the unit motion vector of a pixel, (x mid ,y mid ) represents the coordinates of the center point of the convolution kernel.
[0048] (3) Offset magnitude loss of extracted motion information
[0049]
[0050] Where D(x,y) is the direction vector corresponding to the pixel, and α is the upper limit of the pixel offset.
[0051] The total loss of the motion extraction network training framework can be expressed as:
[0052] Step 2: Train the optical flow estimation network using unsupervised training.
[0053] like Figure 3As shown, the optical flow estimation network first uses a convolutional network to extract features from two frames of images from top to bottom or bottom to top according to the pyramid principle. Then, a matching cost (Cost Volume) layer is constructed for the features extracted from each layer. During this process, the pixel motion vectors output by the motion extraction network are used as prior information to guide the construction of the matching cost region, thereby expanding the effective search area and reducing invalid matching points. Finally, the extracted feature layer and the constructed matching cost layer are input into the optical flow estimation network to calculate a more robust and accurate optical flow map.
[0054] The optical flow estimation network is trained unsupervised, using two adjacent frames I1 and I2 as inputs to generate the estimated optical flow map F. 12 Then, the optical flow map is applied to image I2, and backward wrapping is used to generate the estimated image I1; the total loss of this training structure Photometric consistency loss of images I1 and I1 and edge smoothing loss of optical flow map The sum, in the following form:
[0055]
[0056] Where, λ phot , These are the weights corresponding to the photometric uniformity loss and the edge smoothing loss, respectively. The Charbonnier loss function is used. Let β be the partial derivative along the x and y directions, and β be the weighting variable.
[0057] Step 3: Train the pose estimation network using supervised training.
[0058] like Figure 4 As shown, the confidence information layer used in the pose estimation network has the same resolution as the optical flow map. It is generated from the scalar information of the motion vectors output by the motion extraction network, and the confidence level of each pixel is inversely proportional to the magnitude of the motion corresponding to that pixel. The pose estimation network is trained using supervised training. During training, the optical flow map and the confidence information layer are used as inputs, and the six-DOF pose of the camera is used as the supervision label. The total loss of this training structure is... The specific format is as follows:
[0059]
[0060] in, R represents the estimated translational pose and rotational pose, respectively, while T and R represent the corresponding true poses.
[0061] Step 4: Press Figure 1As shown, the motion extraction network, optical flow estimation network, and pose estimation network are coupled together, and the overall joint training is started in a supervised manner.
[0062] The entire network is trained jointly, using two adjacent frames as input and the camera's six-DOF pose as the final supervision label, with no intermediate supervision labels. During training, only the parameters of the optical flow estimation network and the pose estimation network are updated; the motion extraction network is not updated.
[0063] After training, the fuzzy robust visual odometry takes two adjacent frames as input. The motion extraction module extracts the motion vector of the pixel in frame i+1. Then, the motion vector is used as a prior to input the optical flow estimation network. The optical flow estimation network uses this prior information to guide the construction of the matching cost region, thereby obtaining a more accurate optical flow map. Finally, the pose estimation network uses the estimated optical flow map and the established confidence information layer to realize pose estimation in the fuzzy scene.
[0064] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0065] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A method for robustly constructing a visual odometry based on deep learning, characterized in that, The method comprises the following steps: (1) training a motion extraction network using a supervised training method, using the motion extraction network after training to output pixel motion vectors according to image blur trails, to obtain motion information of pixel points of an i+1 image, wherein i≥0; the training structure of the motion extraction network comprises two modules: a blur generation module and a motion extraction module, and the training supervision label is the zero offset of pixel points in a real blur image and a clear image; The training process comprises two streams: stream one is that a blur image and a clear image of the same scene are input, the motion extraction network extracts motion information of the blur image, the blur generation module blurs the clear image based on the extracted motion information, and then the generated blur image and the real blur image form a supervision relationship; stream two is that a clear image is input, the motion extraction network extracts motion information of the clear image, and the generated motion offset and the zero offset form a supervision relationship; (2) training an optical flow estimation network in an unsupervised training manner, the network taking adjacent two images of an i frame and an i+1 frame and motion information of pixel points of the i+1 frame as input, and outputting an estimated optical flow map; specifically: first, using a convolution network, features of the two images are extracted from top to bottom or from bottom to top according to the pyramid principle, then the features extracted by each layer are used to construct a matching cost layer, and the motion vectors of each pixel output by the motion extraction network are used as prior information to guide the construction of the matching cost area, finally the extracted feature layer and the constructed matching cost layer are input into the optical flow estimation network to calculate the required optical flow map; (3) training a pose estimation network in a supervised training manner, the network taking the estimated optical flow map and its confidence information as input, and outputting an estimated camera pose.
2. The method of claim 1, wherein, In the step (1), a total loss generated by a training framework of the motion extraction network is a sum of a flow line one loss and a flow line two loss; wherein, the flow line one loss is expressed as The flow line two loss is expressed as The specific form is as follows: (1) difference loss between the blurred image generated by the blur generation module and the real blurred image wherein, and are the l2 loss and SSIM loss of the generated blurred image B and the real blurred image B, respectively. (2) vector smoothness loss of motion information where H and W are the height and width of the image respectively, V(x, y) is the unit motion vector of the pixel point, and x mid and y mid are the center points of the convolution kernel; (3) motion information offset size loss Wherein, D(x,y) is the motion vector corresponding to the pixel point, and a is the set upper limit of the pixel point offset.
3. The method of claim 2, wherein, The total loss of the training framework of the motion extraction network is 4. The method of claim 1, wherein, In step (2), the optical flow estimation network is trained unsupervised. During training, two adjacent frames, I1 and I2, are used as inputs to generate the estimated optical flow map F. 12 Then, the optical flow map is applied to image I2, and backward warping is used to generate an estimated image I1; the total loss of the training structure of this optical flow estimation network is... Photometric consistency loss of images I1 and I1 and edge smoothing loss of optical flow map The sum, in the following form: where λ phot , are the weights of the photometric consistency loss and the edge smoothness loss, respectively, is the Charbonnier loss function, is the partial derivative along the x and y directions, and β is a weight variable.
5. The method of claim 1, wherein, The confidence information layer used by the pose estimation network in the step (3) has the same resolution as the optical flow map and is generated from the scalar information of the motion vector output by the motion extraction network, and the confidence of each pixel point is inversely proportional to the motion corresponding to the pixel; the pose estimation network is trained in a supervised training manner, and during the training, the optical flow map and the confidence information layer are taken as input quantities, and the six-degree-of-freedom pose of the camera is taken as a supervision label; the total loss of the training structure of the optical flow estimation network The specific form is as follows: wherein, R are the estimated translational and rotational poses, respectively, and T, R are the corresponding ground truth poses.
6. The method of claim 1, wherein, The joint training of the overall network of the method takes adjacent two images as input and six degrees of freedom of a camera pose as a supervision label, the trained network comprises an optical flow estimation network and a pose estimation network, and the motion information extraction network does not update parameters in the training process.