A visual odometer method for inspection robots

Abstract

The present invention relates to a visual odometer method for an inspection robot, including obtaining the internal parameters of the robot camera; capturing images through a visible light camera during the operation of the robot; selecting two adjacent frames in the captured images as source frames and target frames; inputting the source frame into an unsupervised depth estimation network to obtain predicted depth; performing lightweight improvement of dense modules on the unsupervised depth estimation network; using an optical flow network to predict forward and backward optical flows for source and target frames, and screening high-quality matching feature point pairs according to the consistency principle of the forward and backward optical flows; obtaining calculated depth through triangulation; and aligning the predicted depth and calculated depth between frames according to a scale alignment method to obtain a robot positioning trajectory with consistent scale. Compared with the prior art, the present invention realizes efficient positioning of the robot during the inspection process, avoids the failure of the inspection results due to insufficient positioning accuracy, saves inspection resources, and has a high practical use value in power plant inspection tasks.

Description

Technical Field

[0001] The present invention relates to the field of robot vision, and in particular to a visual odometer method for an inspection robot. Background Art

[0002] In the power industry, intelligent inspection robots are assisting or replacing humans to complete routine inspection tasks in more and more power scenarios, but existing robot inspection and positioning technologies mostly rely on fixed tracks or markers and cannot perform mobile inspections in complex environments. Usually, robots carry visual sensors for detection, so using visual odometry for positioning not only saves costs, but also improves the autonomy and effectiveness of robot movement. Among the existing methods, there are methods that regard visual odometry as a classification problem, use CNN to process input images, and implement visual odometry; there are also methods that combine recurrent neural networks and CNNs, add time series modeling of image sequences, and implement pose estimation end-to-end. However, the cost of acquiring the true value data set of the above-mentioned supervised learning method is high, and the number of supervised training data sets is limited, and the generalization is low. Summary of the invention

[0003] The purpose of the present invention is to provide a visual odometer method for an inspection robot in order to overcome the defects of the above-mentioned prior art, realize monocular visual odometer by unsupervised learning method, and have good accuracy and reliability.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A visual odometer method for an inspection robot comprises the following steps:

[0006] S1. Calibrate the visible light camera installed on the robot to obtain the internal parameters of the camera;

[0007] S2, capturing images at set times from the video acquired by the visible light camera during the operation of the robot;

[0008] S3, select two adjacent frames in the captured image, and mark the previous frame image as source frame I s , the next frame image is marked as the target frame I t ;

[0009] S4, source frame I s Input to the unsupervised depth estimation network to get the predicted depth D s ; A dense module based on DenseNet theory is added to the unsupervised depth estimation network, and a lightweight improvement is made to the dense module. In the dense module, the output features of each processing unit are cascaded to the end of the last dense module through a jump connection;

[0010] S5. Use the optical flow network to analyze the source frame Is and target frame I t Predict the forward optical flow and the backward optical flow, and select high-quality matching feature point pairs according to the consistency principle of the forward and backward optical flows; the optical flow network adopts the LiteFlowNet network model;

[0011] S6. Obtain the calculated depth D′ through triangulation based on the high-quality matching feature point pairs s ;

[0012] S7, predict the depth D s And calculate the depth D′ s The inter-frame scale alignment is performed according to the scale alignment method to obtain the robot positioning trajectory with consistent scale.

[0013] Furthermore, in step S1, the visible light camera is calibrated with an internal reference using a checkerboard calibration method.

[0014] Furthermore, the unsupervised depth estimation network adopts an encoder-decoder form, and the encoder includes a 7×7 convolution layer with a stride of 2, a 3×3 pooling layer with a stride of 2, four dense modules, and a transition layer corresponding to each dense module, each transition layer includes a 1×1 convolution layer and a 3×3 convolution layer.

[0015] Furthermore, in the unsupervised depth estimation network, the dense module includes four processing units, each of which includes a batch normalization layer, a ReLU activation function layer, a 1×1 convolution layer and a 3×3 convolution layer arranged in sequence, and each processing unit cascades the features output by the processing unit to the end of the dense module through a jump connection.

[0016] Furthermore, a network feature measurement loss function is introduced into the unsupervised depth estimation network, and the network is trained by back propagation of the network feature measurement loss function, wherein the network feature measurement loss function includes synthesis loss, depth loss and convergence loss.

[0017] Furthermore, the synthetic loss L f The expression is:

[0018]

[0019] Where: φ(p) is the feature representation of the image, the subscript s represents the source frame, and the subscript t represents the synthesized frame.

[0020] Furthermore, the depth loss L d The expression is:

[0021]

[0022] Where: φ(p) is the feature representation of the image, represents the first-order differential operator, p represents the image pixel, e represents the exponential base, and I(p) represents the pixel of the image.

[0023] Furthermore, the convergence loss L c The expression is:

[0024]

[0025] Where: φ(p) is the feature representation of the image, represents the second-order differential operator, and p represents the image pixel.

[0026] Furthermore, in step S5, an N-best selection strategy is used to screen high-quality matching feature point pairs.

[0027] Furthermore, in step S7, the scale alignment factor S is set t Perform inter-frame matching scale alignment, the expression is:

[0028]

[0029] Where: N represents the number of feature points, represents the predicted depth of the i-th pixel in the t-th frame, represents the calculated depth of the pixel at the i-th frame, and M represents the number of image frames.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The visual odometer of the present invention is based on an unsupervised depth estimation network and a multi-view geometry algorithm, and a lightweight dense module is applied in the unsupervised depth estimation network, which can better predict the image depth. It has the advantages of high accuracy, strong generalization and a small number of network model parameters. It has better robustness for image depth estimation in large scenes and is suitable for inspection robot positioning in complex environments of power plants.

[0032] 2. Aiming at the inherent scale drift problem of monocular visual odometer, the present invention designs a scale alignment principle. This principle can effectively align the scales between image stream frames starting from the first frame image, thereby maintaining the scale consistency of the robot inspection trajectory and avoiding the robot's inability to accurately locate due to scale problems.

[0033] 3. The feature metric loss of the present invention is applied to the training of unsupervised depth estimation networks, which greatly improves the network model's ability to deeply learn images in areas with insufficient scene detail texture, unstable lighting, etc. The present invention improves the defect that the single degree loss cannot create an effective supervisory signal training network, improves the quality of network training, and integrates the network into the visual odometer method, thereby improving the accuracy of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the overall framework diagram of the present invention.

[0035] Figure 2 It is a schematic diagram of the depth estimation network model constructed by the present invention.

[0036] Figure 3 This is a schematic diagram of the structure of the improved dense module of the present invention.

[0037] Figure 4a This is the inherent scale loss problem of the monocular visual odometer that the present invention addresses.

[0038] Figure 4b This is the positioning trajectory of the monocular visual odometer after the scale problem is solved by the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] like Figure 1 As shown, this embodiment provides a visual odometer method for an inspection robot, which performs real-time positioning of an image sequence acquired by an intelligent inspection robot using a monocular visible light camera, and specifically includes the following steps:

[0041] Step 1: Calibrate the visible light camera installed on the robot and obtain the camera's intrinsic parameters using the checkerboard calibration method.

[0042] Step 2: During the operation of the robot, the video obtained by the visible light camera is intercepted at a fixed time, and the intercepted images are stored in the server, and the unsupervised depth estimation network model is deployed on the server;

[0043] Step 3: Select two adjacent frames in the captured image in the server and resize them to 640×480, and mark the previous frame as source frame I s , the next frame image is marked as the target frame I t The source frame is input into the unsupervised depth estimation network to obtain the predicted depth D s .

[0044] Step 4: Use the optical flow network to s and target frame I t Predict the forward optical flow and backward optical flow, and select high-quality matching feature point pairs based on the consistency principle of the forward and backward optical flows; the optical flow network adopts the LiteFlowNet network model.

[0045] Step 5: Obtain the calculated depth D′ through triangulation based on the high-quality matching feature point pairs s .

[0046] Step 6: Predicted depth D s And calculate the depth D′ s The inter-frame scale alignment is performed according to the scale alignment method to obtain the robot positioning trajectory with consistent scale.

[0047] Step 7: Send the scale-consistent positioning trajectory to the robot body so that the robot can obtain its own positioning information.

[0048] In this embodiment, the unsupervised depth estimation network adopts DenseNet, and a lightweight improvement is made to the dense module. In the dense module, the output features of each processing unit are cascaded to the end of the dense module through a jump connection, as follows:

[0049] like Figure 2 As shown in Figure 1, the basic structure of the unsupervised depth estimation network adopts the encoder-decoder form. The encoder consists of a 7×7 convolution layer with a stride of 2, a 3×3 pooling layer with a stride of 2, and four dense modules and their three adjacent transition layers (Tansition Layer). The transition layer contains a 1×1 convolution layer and a 3×3 convolution layer to ensure that the size of the feature map cascaded to the subsequent layer is consistent. The decoder contains a 1×1 convolution layer, a 2×2 bilinear upsampling layer, and four upsampling modules in sequence. In the upsampling module, two groups of 3×3 convolution layers and a 2×2 bilinear upsampling layer are included. A jump connection is added between the encoder and the decoder to ensure the smooth flow of feature transfer.

[0050] like Figure 3 As shown, in this embodiment, each dense block (Dense Block) is lightly improved, combined with actual scene requirements, and the dense module connection method is simplified to reduce the network computing resource consumption while ensuring the network's ability to encode scene details. The dense module includes four processing units, each of which includes a batch normalization layer (BN), a ReLU activation function layer, a 1×1 convolution layer, and a 3×3 convolution layer set in sequence. In each dense module, it is no longer necessary to cascade the output features of the current processing unit to each subsequent processing unit. It is only necessary to cascade the features output by the processing unit to the end through a jump connection, that is, only the last processing unit takes the same operation as DenseNet.

[0051] x0~x3 represent the characteristics of each processing unit; the parameter k of the dense module, i.e., the growth rate, represents the number of feature maps cascaded from each processing unit to the final one; D0~D3 are nonlinear transformation functions, defined as follows:

[0052] x l =D l-1 (x0, x1, ..., x l-1 ), l = 1, 2, 3

[0053] Where l represents the sequence number of each processing unit.

[0054] Therefore, in the dense module, the output of each processing unit is subjected to the nonlinear optimization function D l After the operation, k feature maps are generated. Therefore, the output of a dense module has k0+(l-1)k feature maps as the input of the next module, where k0 is the number of feature channels of the module input layer l0. In the visual odometer scenario, the lightweight dense module is sufficient to retain the feature information of the scene, reduce feature redundancy, and reduce model parameters. Compared with traditional models, it ensures the flexibility of the network and the continuity of features.

[0055] In the training process of the unsupervised depth estimation network, this embodiment proposes a network feature measurement loss function, and uses its back propagation training network to improve the network's learning ability of scene details and depth recovery ability. The network feature measurement loss function consists of synthesis loss, depth loss and convergence loss, as shown in the following figure:

[0056] (1) Synthetic loss

[0057] According to the view synthesis theory, let p s is the homogeneous coordinate of a pixel in the source frame, is the homogeneous coordinate of the corresponding pixel in the synthetic frame, then according to the camera pose and depth value, we can get:

[0058]

[0059] Where: K represents the intrinsic parameter matrix of the camera; D s Describes the pixel depth of the network output in the source frame; Represents the pose matrix from source frame to target frame.

[0060] From the above formula, the synthetic feature map can be obtained through the feature map of the source frame Therefore, the expression of the feature map synthesis loss function is:

[0061]

[0062] Where: φ(p) represents the feature representation of the image, the subscript s represents the source frame, and the subscript t represents the synthesized frame.

[0063] (2) Depth loss

[0064] In order to ensure that the network can still learn features with large gradient changes in weak texture areas of the scene, a depth loss function is introduced:

[0065]

[0066] Where: represents the first-order differential operator, p represents the image pixel, e represents the exponential base, and I(p) represents the pixel of the image.

[0067] (3) Convergence loss

[0068] In order to smooth the feature gradient, a penalty term for the second-order gradient is introduced, namely the convergence loss:

[0069]

[0070] Where: represents the second-order differential operator.

[0071] In summary, the feature metric loss L fm The expression is as follows:

[0072] L fm =λ f L f +λ d L d +λ c L c

[0073] Where: f is the synthetic loss weight; d is the depth loss weight; c is the convergence loss weight. Corresponding to different losses, different weights are set according to different requirements in practical applications.

[0074] In network training, the Adam optimizer is used for depth estimation network training. The exponential decay rate of the first-order moment estimation is set to β1=0.9; the exponential decay rate of the second-order moment estimation is set to β2=0.999; the learning rate is set to α=1e-4 (e represents the exponent, which means multiplying the previous number by 10 to the negative power of 4); the batch size is set to 4; and the dense module growth rate k is set to 12. The synthetic loss weight λ f Set to 1; depth loss weight λ d Set to 1e-3; convergence loss weight λ c Set to 1e-3; photometric loss weight λ ph Set to 5; depth consistency loss weight λ dc Set to 1.

[0075] In summary, the unsupervised depth estimation network of this embodiment is improved in three aspects: ① network training loss function; ② deeper network layers; ③ fewer network training parameters, so as to improve the accuracy of the depth estimation network algorithm.

[0076] The improved dense module is based on DenseNet. By improving the way of feature transfer between layers, the shallow features are retained to the greatest extent, which has a better estimation effect on image details. At the same time, the transition module, upsampling module and skip connection structure are used to improve the network feature transfer capability.

[0077] The depth estimation network adopts an encoder-decoder form in its basic structure. By reusing shallow features, the network's ability to encode scene details is improved, allowing the depth estimation network to obtain more refined image feature expressions. In the encoder part, an improved dense module structure is combined to achieve an optimal balance between network training parameters and feature reuse.

[0078] In this embodiment, the specific development of step 4 is as follows:

[0079] First, the consistency criterion of the front and back optical flows is used to select high-quality optical flow matches. The higher the consistency, the higher the matching accuracy. s , I t ), assuming the forward optical flow is The backward optical flow is Optical flow consistency is calculated as:

[0080]

[0081]

[0082] Where: p f is the correspondence between pixels in the forward and backward optical flows; ω is the parameter for performing transformation operations on pixels in the optical flow; and p is a pixel point in the image.

[0083] Then, according to the above calculation formula, the best N pairs of 2D-2D matching points are selected through the conventional N-Best selection strategy. i and j represent pixel superscript indices.

[0084] In this embodiment, the specific development of step six is ​​as follows:

[0085] First, we use nonlinear optimization to solve PnP and calculate the representation T′ of the initial camera pose on the Lie group SE(3). The main calculation formula is as follows:

[0086] Consider a spatial point selected by the consistency of optical flow, whose homogeneous coordinates are p i =(X i , Y i , Z i ) T Its normalized plane homogeneous projection point coordinates in the source frame are u i =(ui , v i ) T From the theoretical model of the monocular camera, we can see that the relationship between the two is as follows:

[0087] s i u i =KTp i

[0088] In the formula, s i is the depth of the 3D point, and K is the intrinsic parameter matrix of the monocular camera.

[0089] Since there is a certain error between camera observation and actual calculation, the 3D point is reprojected by the initial pose transformation T′, and the pixel position of the reprojection is consistent with the target frame I t There is an error in the observation position in , that is, the reprojection error. Based on this, a nonlinear optimization problem is constructed as follows

[0090]

[0091] In the formula, n represents the number of projection points, T ＊ represents the pose transformation matrix, u represents the projection of the i-th spatial point on the pixel plane, s i represents the depth of the i-th spatial point, i represents the i-th spatial point, p i represents the homogeneous coordinates of the i-th spatial point.

[0092] The unconstrained optimization is constructed using Lie algebra and solved through BA (Bundle Adjustment) to obtain the initial pose T′.

[0093] Then, the calculated depth is aligned with the network predicted depth using the scale alignment principle, the scale of the global positioning trajectory is fixed, and the scale-consistent robot positioning trajectory is obtained, which is expanded as follows:

[0094] The predicted depth of the 0th frame image at the i-th position is The corresponding calculated depth obtained by triangulation is Then, let

[0095]

[0096] As the initial scale alignment factor. Where N represents the number of feature points. Then, the same operation is performed on each frame, that is,

[0097]

[0098] Where N represents the number of feature points. represents the predicted depth of the i-th pixel in the t-th frame, represents the calculated depth of the pixel at the i-th frame, and M represents the number of image frames.

[0099] If the adjacent image sequence I s and I t The scale alignment factor between the two is consistent, that is, s s ≈s t , then with the constraint propagation of inter-frame matching scale alignment, the global scale of the pose is fixed. Finally, a globally consistent trajectory T = [R, t] is output.

[0100] The comparison between the scale-unaligned trajectory and the scale-aligned trajectory is shown in Figure 4a and Figure 4b shown.

[0101] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A visual odometer method for an inspection robot, characterized in that: The following steps are involved: S1. Calibrate the visible light camera installed on the robot to obtain the internal parameters of the camera; S2, capturing images at set times from the video acquired by the visible light camera during the operation of the robot; S3, select two adjacent frames in the captured image, and mark the previous frame image as source frame I s , the next frame image is marked as the target frame I t ; S4, source frame I s Input to the unsupervised depth estimation network to get the predicted depth D s ; A dense module based on DenseNet theory is added to the unsupervised depth estimation network, and a lightweight improvement is made to the dense module. In the dense module, the output features of each processing unit are cascaded to the end of the last dense module through a jump connection; S5. Use the optical flow network to analyze the source frame I s and target frame I t Predict the forward optical flow and the backward optical flow, and select high-quality matching feature point pairs according to the consistency principle of the forward and backward optical flows; the optical flow network adopts the LiteFlowNet network model; S6. Obtain the calculated depth D′ through triangulation based on the high-quality matching feature point pairs s ; S7, predict the depth D s And calculate the depth D′ s Perform inter-frame scale alignment according to the scale alignment method to obtain the robot positioning trajectory with consistent scale; A network feature measurement loss function is introduced into the unsupervised depth estimation network, and the network is trained by back propagation of the network feature measurement loss function, wherein the network feature measurement loss function includes synthesis loss, depth loss and convergence loss; The combined loss L f The expression is: Where: φ(p) is the feature representation of the image, the s subscript represents the source frame, and the t subscript represents the synthesized frame; The depth loss L d The expression is: Where: φ(p) is the feature representation of the image, represents the first-order differential operator, p represents the image pixel, e represents the exponential base, and I(p) represents the pixel of the image; The convergence loss L c The expression is: Where: φ(p) is the feature representation of the image, represents the second-order differential operator, and p represents the image pixel.

2. A visual odometer method for an inspection robot according to claim 1, characterized in that: In step S1, the visible light camera is calibrated with an internal reference using a checkerboard calibration method.

3. A visual odometer method for an inspection robot according to claim 1, characterized in that: The unsupervised depth estimation network adopts an encoder-decoder form, and the encoder includes a 7×7 convolution layer with a stride of 2, a 3×3 pooling layer with a stride of 2, four dense modules, and a transition layer corresponding to each dense module, each transition layer includes a 1×1 convolution layer and a 3×3 convolution layer.

4. A visual odometer method for an inspection robot according to claim 1, characterized in that: In the unsupervised depth estimation network, the dense module includes four processing units, each of which includes a batch normalization layer, a ReLU activation function layer, a 1×1 convolution layer, and a 3×3 convolution layer, which are set in sequence. The features output by each processing unit are cascaded to the end of the dense module through a jump connection.

5. The visual odometer method for an inspection robot according to claim 1, characterized in that: In step S5, the N-best selection strategy is used to screen high-quality matching feature point pairs.

6. The visual odometer method for an inspection robot according to claim 1, characterized in that: In step S7, the scale alignment factor S is set t Perform inter-frame matching scale alignment, the expression is: Where: N represents the number of feature points, represents the predicted depth of the i-th pixel in the t-th frame, represents the calculated depth of the pixel at the i-th frame, and M represents the number of image frames.

Images