Lightweight infrared depth odometer method

Abstract

The invention discloses a lightweight infrared depth odometer method. Firstly, an infrared image is processed, information carried in the image is enriched, and noise in the image is suppressed; secondly, image frames are tracked, and rough pose estimation is given through data association between the image frames and reference frames; particularly, when the infrared camera processes non-consistency correction, a motion recovery module is designed, and it is guaranteed that under the condition that the non-consistency correction problem occurs, the provided method can still provide accurate pose estimation. And finally, optimizing the pose of the key frame by adopting a sliding window strategy. The depth information enhanced infrared visual odometer method provided by the invention has the advantages of low calculation consumption and the like on the basis of ensuring relatively robust operation, and has the capability of realizing resource-limited embedded equipment deployment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight infrared depth odometer method. Background Art

[0002] In the field of discrete elevator manufacturing, accurate navigation and positioning technology is essential to improve assembly efficiency and ensure safety. The manufacturing and assembly of elevator systems are usually carried out in narrow shafts and complex workshop environments, which require automation systems to be able to operate and move accurately in limited spaces. However, these environments often have dim lighting, no lighting or complex lighting conditions, which poses a challenge to traditional vision systems. The high precision requirements and special space constraints of elevator assembly make the need for environmental perception and position confirmation very urgent. Therefore, it is of great significance to develop visual odometry systems that can adapt to these complex environments, especially systems that can operate reliably in dim lighting, no lighting and complex lighting conditions.

[0003] Depth-compensated infrared visual odometry technology can bring significant benefits in discrete elevator manufacturing. Traditional vision systems may fail in low-light or highly reflective environments, while infrared depth sensors can provide reliable environmental and depth information under these conditions. By incorporating infrared depth data, the system can not only improve the accuracy of spatial perception and environmental understanding, but also enhance the system's adaptability to environmental changes and obstacle avoidance capabilities. The application of this technology can not only improve the safety and efficiency of the elevator assembly process, but also reduce the need for manual intervention, thereby optimizing the overall production process and resource utilization efficiency. Therefore, depth-compensated infrared visual odometry technology is of great significance in discrete elevator manufacturing. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art. In the discrete manufacturing process of elevators, there are autonomous perception problems such as insufficient lighting, complex lighting and dust obstruction in environments such as factory workshops and elevator shafts. A lightweight infrared depth odometer method is provided to compensate for the deficiency that general visual odometers cannot operate robustly for high-noise, low-texture infrared images.

[0005] The object of the present invention is achieved through the following technical solutions: A lightweight infrared depth odometer method, comprising:

[0006] For the image captured by the infrared camera, estimate its pose with respect to the reference frame and use the residual to determine whether the tracking is successful;

[0007] According to the sampling timestamp size between adjacent frames, it is detected whether the infrared camera is performing inconsistency correction. If so, ORB feature extraction and matching are performed on the reference frame and the NUC post-frame, and the average optical flow is calculated for all matched points. The map points held by the reference frame are projected back to the reference frame plane, and their corresponding positions on the NUC post-frame are calculated. After obtaining the projection coordinates of the NUC post-frame, the reprojection error is minimized to obtain the position of the reference frame relative to the NUC post-frame and determine whether the tracking is successful through the residual. The NUC post-frame is the first frame captured after the inconsistency correction.

[0008] If the tracking is successful, the time between the last generated key frame and the current frame or the number of key frames in the sliding window is used to determine whether the current frame is a key frame;

[0009] For key frames, their observation points are converted into candidate map points. The observation points and depth priors of non-key frames are used to perform epipolar line search to estimate the inverse depth value of the candidate map points. If the inverse depth value of the candidate map point converges to the threshold range, it is converted into a map point. If the map point is observed by at least one non-dominant frame and the energy is bounded, the map point is activated and added to the subsequent sliding window optimization. The non-dominant frame refers to the key frame that does not hold the map point.

[0010] The motion parameters of the key frames and the inverse depth of the activated map points are optimized through a sliding window strategy, and the historical frames and their map point information are used as the prior of the sliding window; wherein the historical frames are key frames that are no longer directly involved in the optimization calculation.

[0011] Furthermore, the infrared camera captures 16-bit thermal images, which need to be converted into 8-bit images, and then use adaptive grayscale histogram equalization to improve the image contrast, and finally perform non-local mean filtering to denoise the image.

[0012] Furthermore, estimating the pose of the image captured by the infrared camera with respect to the reference frame includes:

[0013] A direct method is used to estimate the pose of the current frame: first, the relative pose initial value is set using the motion model hypothesis, and the observation points in the reference frame are back-projected to the current frame to construct a photometric error function. That is, the difference in grayscale values ​​between the observation points of the same 3D point in the two frames is directly used as the error. The Levenberg-Marquart method is used to optimize the pose and photometric parameters layer by layer from the top layer to the bottom layer of the image gradient pyramid. The error function is:

[0014]

[0015] Among them, ||·|| γ is the Huber norm; Np It is the first j Back-projection to the Ith frame i The pixel point on the frame and the set of seven pixels around it, I j Indicates the current frame, I i represents the reference frame, is the square of the second norm of the image gradient value at point p, I j [p′] and I i [p] are respectively j and I i The image grayscale value of pixel p′ and pixel p on the frame; w p is the Huber weight, c is a constant, and the calculation formula is:

[0016]

[0017] The nonlinear equation is approximated by performing Taylor expansion on the error and neglecting quadratic and higher terms:

[0018] J T WJδx=-J T W

[0019] Among them, the Jacobian matrix J is the partial derivative of r with respect to the image frame pose:

[0020]

[0021] The superscript T indicates the transpose operation, and W is w p The diagonal matrix formed by r is the residual between the corresponding observation points of the two image frames:

[0022] r=(I j [p′(Ti i ,T j ,d,K)]-I i [p])

[0023] Among them, I i [p],I j [p′(T i ,T j ,d,K)] is the observed pixel grayscale value of the same 3D point in the reference frame and the current frame; T i ,T j ,d,K are the first i Frame, I j The transformation matrix of the frame image, the inverse depth of the pixel point p and the camera intrinsic parameter, δx is the increment of the motion state parameter; iteratively solve the linear equation and update the motion state from the reference frame to the current frame x←x+δx; after several iterations, find the pose estimation that minimizes the residual from the reference frame to the current frame image as the final result of this pose estimation.

[0024] Furthermore, ORB feature extraction and matching are performed on the reference frame and the NUC post-frame, and the average optical flow is calculated for all matched points:

[0025]

[0026] Where ii is the index of the matching feature point set, n is the number of matching point pairs, OF(q ii ,q′ ii ) is the optical flow between the reference frame and the NUC post-frame matching points:

[0027] OF(q ii ,q′ ii )=||q′ ii -q ii ||2

[0028] where q ii is the feature point on the reference frame, q i ' i It is the feature point on the NUC frame;

[0029] The map points held by the reference frame are projected back to the pixel points on the reference frame plane, and their corresponding positions on the NUC post-frame are calculated:

[0030] s′=s+OF

[0031] Where s is the pixel coordinate set on the reference frame, and s′ is the pixel coordinate set corresponding to the NUC post-frame;

[0032] After obtaining the projection coordinates of the NUC post-frame, the reprojection error is minimized to obtain the position and posture of the reference frame relative to the NUC post-frame, including:

[0033] Recover the camera motion during NUC by minimizing the reprojection error:

[0034]

[0035] Where idx is the index of the pixel set on the NUC frame, s′ idc are the pixel coordinates of the 3D point observed on the NUC post-frame, are all the map points held by the reference frame, π t is the projection equation of the infrared camera; is the pose to be solved, that is, the pose of the reference frame relative to the NUC post-frame.

[0036] Furthermore, when ORB feature extraction and matching are performed on the reference frame and the NUC post-frame, in case that feature points cannot be extracted or are mismatched, the following steps are also included:

[0037] Through the projection equation of the depth camera, we get the point cloud:

[0038]

[0039] Among them, id is the index on the depth map, pc id is the projected 3D point; π d is the projection equation of the depth camera; is the pixel coordinate on the depth map, yes Depth value; downsample the point cloud PC and perform ICP solution on the downsampled point cloud of the adjacent depth map:

[0040]

[0041] Where index is the index of the point cloud collection, are the downsampled point cloud of the previous frame and the point cloud of the next frame, N is the number of points in the point cloud, R and t are the rotation and displacement components in the transformation matrix from the previous frame point cloud to the next frame point cloud, respectively; assuming that a total of m frames of depth maps are captured during NUC, the total motion T nuc for:

[0042] T nuc =T1T2…T m-1

[0043] Where T i represents the pose transformation matrix from the i-th to the i+1-th frame during NUC, 1≤i≤m-1, then the motion of the camera during NUC is restored as:

[0044]

[0045] Where T td is the external parameter from the depth camera to the infrared camera, T tnuc is the movement of the infrared camera during NUC.

[0046] Furthermore, the Gauss-Newton method is used to optimize the inverse depth value of the generated map points.

[0047] Furthermore, the sliding window is a sequence of key frames, in which all key frames and their map points are added to a linear equation:

[0048] J sw T W sw J sw δx sw =-J sw T W sw r sw

[0049] The superscript T represents the matrix transpose, and the subscript sw represents that the entire equation maintains all the state quantities to be optimized in the sliding window. sw It is the Jacobian matrix of the photometric residuals composed of all frames with co-viewing relationship and their corresponding points in the sliding window; W sw is the matrix composed of the corresponding Huber weights, δx sw are all the state increments to be optimized in the sliding window; r sw It is the photometric residual composed of all frames and their corresponding points in the sliding window.

[0050] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned lightweight infrared depth odometer method.

[0051] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned lightweight infrared depth odometer method is implemented.

[0052] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned lightweight infrared depth odometer method when executed by a processor.

[0053] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in that the infrared camera can overcome the limitations of the traditional visual system when the lighting conditions are poor or there are visual degradation factors (such as smoke or obstructions). Through the thermal radiation information of the external environment obtained by the infrared sensor, the system can stably and accurately perceive the surrounding environment, without being affected by changes in lighting, ensuring accurate navigation and positioning in dim or unlit environments. The depth information of the environment captured by the depth camera can provide accurate scale information for the infrared odometer and suppress the drift of the infrared odometer. The application of this technology not only improves the operating accuracy and efficiency during the assembly process, but also enhances the adaptability and stability of the visual odometer to environmental changes. The image processing method for infrared images can suppress image noise while improving contrast. It is a prerequisite for the stable operation of the system. Finally, the experiment shows that the method of the present invention can meet the real-time requirements of the visual odometer while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0055] Figure 1 An overall framework diagram of a lightweight infrared depth odometer method provided by an embodiment of the present invention;

[0056] Figure 2 An image processing effect provided by an embodiment of the present invention;

[0057] Figure 3 A back-projection process in photometric error optimization provided by an embodiment of the present invention;

[0058] Figure 4 A diagram for graph optimization provided by an embodiment of the present invention;

[0059] Figure 5 An operation effect diagram provided by an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, design ideas and technical solutions of the embodiments of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementation methods can be combined with each other.

[0062] Glossary:

[0063] 3D points are triangulated from two frames of 2D observations;

[0064] The depth prior refers to the pixel depth values ​​provided by the depth map to the thermal image;

[0065] A 3D map point (referred to as a map point) refers to a 3D point that can be converted into a candidate map point.

[0066] The invention discloses a lightweight infrared depth odometer method, which is applied to visual degradation environments such as insufficient light, light changes, and smoke occlusion. Figure 1 As shown, the following steps are included:

[0067] Step 1. Use the infrared camera to obtain the 16-bit thermal image of the current environment and convert it into an 8-bit image. Figure 2 The details are as follows:

[0068] Image_8u(i,j)=(Image_16u(i,j)-min) / (max-min)*255

[0069] Among them, the 16-bit thermal image needs to be normalized to transform its grayscale value into the range of [0,255]. min and max are the imaging grayscale values ​​corresponding to the maximum and minimum values ​​of the ambient temperature prior, respectively, and i and j are the indexes of the rows and columns in the image, respectively. Since the dynamic range changes in this process, the distribution of the image grayscale histogram is relatively concentrated, which is manifested in the image as an overall gray and low contrast. Therefore, it is necessary to use a grayscale histogram equalization strategy to increase the contrast and make the image texture richer. The adaptive grayscale histogram method is used to adjust the grayscale histogram distribution of the image. The adaptive grayscale histogram equalization process is as follows: first, the image is divided into several blocks, and the grayscale histogram and shearing threshold are calculated according to each image block; then histogram equalization is used, and finally the grayscale value of the pixel is reconstructed. The image processed by the adaptive grayscale histogram method has large noise. In order to suppress the noise, non-local mean denoising is used to remove high-frequency noise. The non-local mean denoising process is as follows: for a certain pixel x, search for similar pixels in the search window, calculate the non-local uniform estimate of each pixel x, and obtain the grayscale value of the weighted average of the neighborhood of x:

[0070]

[0071] Where Ω is the set of pixels in the neighborhood of x, w(x,y) is the weight of pixel y to x, is the weighted average pixel value of pixel x, I(y) is the pixel value of pixel y in the neighborhood of x, and is calculated based on grayscale similarity:

[0072]

[0073] Where N(x) and N(y) represent the search windows centered at pixel x and y respectively, Z(x) is the weight normalization factor, the purpose is to make the sum of all weights equal to 1, σ is the parameter that controls the weight distribution, and exp is the exponential function.

[0074] Step 2. Track the image processed in step 1, use the direct method to roughly estimate the image pose, and determine whether the tracking is successful by the photometric residual. If the tracking is successful, determine whether the current frame is a key frame based on the time from the last key frame generated to the current frame or the number of key frames in the sliding window.

[0075] The motion model is used to make assumptions about the pose, and the points in the reference frame are back-projected into the current frame to construct the photometric error function, that is, the difference in the grayscale values ​​of the corresponding points is directly used as the error, and the Levenberg-Marquart method is used to optimize the pose and photometric parameters from the top layer of the image gradient pyramid to the bottom layer. The error function is:

[0076]

[0077] Among them is ||·|| γ is the Huber norm; N p It is in the I j The Ith i The set of points on the frame (reference frame) and the seven points around it; w p are the Huber weights:

[0078]

[0079] is the square of the second norm of the image gradient value at point p, c is a constant; I j [p′] and I i [p] are respectively j and I i Image gray value of point p on the frame; construct a linear equation:

[0080] J T WJδx=-J T W

[0081] Where r is the residual between the pixels of two frames: r = (I j [p′(T i ,T j ,d,c)]-I i [p]), T i ,T j ,D,c are the first i Frame, I j The transformation matrix of the frame image, the inverse depth of the pixel p and the camera internal parameters. The Jacobian matrix J is the partial derivative of the residual r with respect to the motion parameters and photometric parameters:

[0082]

[0083] δ is a small change in the state quantity; W is the sum of w pThe diagonal block matrix is ​​constructed. Solve the equation to obtain the state increment δx, and update the initial value x←x+δx. If the residual obtained by the pose projection after several iterations of optimization is within an acceptable range, the optimized pose is used as a rough estimate of the relative pose from the reference frame to the current frame. If the residuals obtained after iterations of all motion model assumptions are within an unacceptable range, the frame tracking fails. If the tracking fails, return to step 1.

[0084] Step 3. Detect whether the camera is performing non-uniform correction (NUC) based on the sampling timestamp size between adjacent frames. If it is in the non-uniform correction state, associate the reference frame and the NUC post-frame. The association method is as follows: First, assume that the optical flow of all points in the image, that is, the movement on the pixel plane is similar. Based on this assumption, the optical flow of feature points is used to approximate the optical flow of all points. Perform ORB feature extraction and matching on the reference frame and the NUC post-frame and calculate the average optical flow of all matched points:

[0085]

[0086] Where ii is the index of the matching feature point pair, n is the number of matching point pairs, OF(q ii ,q′ ii ) is the optical flow between the reference frame and the NUC post-frame matching points:

[0087] OF(q ii ,q′ ii )=||q′ ii -q ii ||2

[0088] where q ii is the feature point on the reference frame, q′ ii is the feature point on the NUC post-frame. Project the map points held by the reference frame back to the reference frame plane and calculate their corresponding positions on the NUC post-frame:

[0089] s′=s+OF

[0090] Where s is the pixel coordinate set on the reference frame, and s′ is the pixel coordinate set corresponding to the NUC post-frame. After the association is established, the camera motion during NUC is restored by minimizing the reprojection error:

[0091]

[0092] Where idx is the index of the pixel set on the NUC frame; s′ idc is the pixel coordinate of the 3D point observed on the NUC post-frame, which is a pixel coordinate in s′; are all the map points held by the reference frame, π t is the projection equation of the infrared camera; is the pose to be solved, that is, the pose of the reference frame relative to the NUC post-frame.

[0093] Since there is a certain error in the association of points, after solving the equation, the obtained pose is added as the initial value to the above photometric optimization, and the photometric residual is used to determine whether the tracking is successful.

[0094] In one embodiment, since the infrared thermal image has the characteristic of low texture, there may be a situation where feature points cannot be extracted or mismatched. In view of this situation, the camera motion during NUC is solved using the information provided by the depth camera: First, the point cloud can be obtained through the projection equation of the depth camera:

[0095]

[0096] Among them, id is the index on the depth map, pc id is the 3D point obtained by projection; π d is the projection equation of the depth camera; is the pixel coordinate on the depth map, yes Since the point cloud of the depth map is dense, the calculation is very time-consuming, so the point cloud PC is downsampled and the downsampled point cloud of the adjacent depth map is solved by ICP:

[0097]

[0098] Where index is the index of the point cloud after downsampling, are the downsampled point cloud of the previous frame and the next frame respectively, N is the number of points in the point cloud, R and t are the rotation and displacement components in the transformation matrix from the previous frame point cloud to the next frame point cloud respectively. Assuming that a total of m frames of depth images are captured during NUC, the total motion T nuc for:

[0099] T nuc =T1T2…T m-1

[0100] Where T i Represents the pose transformation matrix from the i-th to the i+1-th frame during NUC, then the motion of the camera during NUC can be recovered as:

[0101]

[0102] Where T td is the external parameter from the depth camera to the infrared camera, T tnuc is the motion of the infrared camera during NUC. Then the direct method is used to calculate Ttnuc Optimize and use the residual to determine whether the tracking is successful.

[0103] Step 4. If the tracking is successful, determine whether the current frame is a key frame based on the time from the last key frame generated to the current frame or the number of key frames in the sliding window; if the tracking fails, return to step 1.

[0104] Step 5. For key frames, convert their observation points into candidate map points, use the observation points and depth priors of non-key frames to perform polar line search, and estimate the inverse depth value of the candidate map points; if the inverse depth value of the candidate map point converges to the threshold range, it is converted into a map point. If the map point is observed by at least one non-dominant frame and the energy is bounded, the map point is activated, and the activated map point will be added to the subsequent sliding window optimization; among them, for non-key frames, the optimization process of the candidate map points of the key frames is added.

[0105] The depth of the candidate map points is estimated by epipolar search. For non-keyframes, no new map points are generated, and the inverse depth parameter range of the map points of the keyframes is updated by epipolar search; for keyframes, 2D observation points with converged inverse depth parameter range are used to generate 3D map points. Epipolar search is a search method based on the geometric constraints satisfied by the camera's observations of the same 3D point at different times. Let the observation of the candidate map point on the keyframe image plane I1 be p, and the optical centers O1O2 of the two-frame camera form a plane called the epipolar plane. This plane and the non-keyframe image plane I2 intersect with a straight line k2, which is called the epipolar line. Although the observation p′ on the non-keyframe image plane I2 is unknown, it is known that the observation p′ is located on l2, so the two-dimensional search problem that should have been on the non-keyframe image plane I2 degenerates into a one-dimensional search problem on the epipolar line l2. Search along l2 for a point with a similar grayscale value to p, and consider it to be observation p′. In order to make the epipolar search more robust, pixel block matching is used, that is, the grayscale value difference between the image block centered on p and the image block centered on p′ is calculated. After searching for observation p′, the intersection of line segments O1p and O2p′ is the location of the candidate map point. Among them, the depth information provides a priori information for the epipolar search process. During the search, since the depth prior information of the candidate map point on the key frame and the pose relative to the non-key frame are known, the inverse depth of the map point is optimized by Gauss-Newton on the non-key frame with the depth prior information of the observation point as the initial value.

[0106] Step 6. Optimize the motion parameters (including posture) of the key frame and the inverse depth of the activated map point through the sliding window strategy, and use the historical frame and its map point information as the prior of the sliding window; wherein the historical frame is the key frame that no longer directly participates in the optimization calculation. Figure 4 shown.

[0107] The sliding window maintains the status information of several key frames and their held map points. Each key frame projects the map points it holds to the other key frames in the sliding window. For all map points that can be projected to the other key frames, the photometric error is calculated and the equation is solved:

[0108] J sw T W sw J sw δx sw =-J sw T W sw r sw

[0109] The superscript T represents the matrix transpose, and the subscript sw represents that the entire equation maintains all the state quantities to be optimized in the sliding window. sw It is the Jacobian matrix of the photometric residuals composed of all frames with co-viewing relationship and their corresponding points in the sliding window; W sw is the matrix composed of the corresponding Huber weights, δx sw are all the state increments to be optimized in the sliding window; r sw It is the photometric residual composed of all frames and their corresponding points.

[0110] Finally, the operating results of the present invention are as follows Figure 5 As shown, the black dotted line (GT) is the true value trajectory, the blue solid line (DepthEnhanced) is the running trajectory of the lightweight infrared-depth odometer proposed in the present invention, and the green solid line (ThermalOnly) is the running trajectory of the monocular infrared odometer. From the results, it can be seen that the lightweight infrared-depth odometer proposed in the present invention can estimate the movement of the camera more accurately, and the effect is significantly improved compared with the monocular infrared odometer without depth information.

[0111] Figure 6 This is a schematic diagram of an electronic device structure provided by an embodiment of the present invention. Figure 6 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, a lightweight infrared depth odometer method of the present invention is implemented.

[0112] It should be noted that, in addition to Figure 6 In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0113] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned lightweight infrared depth odometer method is implemented.

[0114] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above-mentioned lightweight infrared depth odometer method when executed by a processor.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0120] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A lightweight infrared depth odometer method, characterized in that: include: For the image captured by the infrared camera, estimate its pose with respect to the reference frame and use the residual to determine whether the tracking is successful; According to the sampling timestamp size between adjacent frames, it is detected whether the infrared camera is performing inconsistency correction. If so, ORB feature extraction and matching are performed on the reference frame and the NUC post-frame, and the average optical flow is calculated for all matched points. The map points held by the reference frame are projected back to the reference frame plane, and their corresponding positions on the NUC post-frame are calculated. After obtaining the projection coordinates of the NUC post-frame, the reprojection error is minimized to obtain the position of the reference frame relative to the NUC post-frame and determine whether the tracking is successful through the residual. The NUC post-frame is the first frame captured after the inconsistency correction. If the tracking is successful, the time between the last generated key frame and the current frame or the number of key frames in the sliding window is used to determine whether the current frame is a key frame; For keyframes, their observation points are converted into candidate map points, and the observation points and depth priors of non-keyframes are used to perform epipolar line search to estimate the inverse depth value of the candidate map points; If the inverse depth value of a candidate map point converges to the threshold range, it is converted into a map point. If the map point is observed by at least one non-dominant frame and the energy is bounded, the map point is activated and the activated map point is added to the subsequent sliding window optimization. Among them, non-dominant frames refer to key frames that do not hold the map point; The motion parameters of the key frames and the inverse depth of the activated map points are optimized through a sliding window strategy, and the historical frames and their map point information are used as the prior of the sliding window; wherein the historical frames are key frames that are no longer directly involved in the optimization calculation.

2. A lightweight infrared depth odometer method according to claim 1, characterized in that: The infrared camera captures 16-bit thermal images, which need to be converted into 8-bit images, and then use adaptive grayscale histogram equalization to improve the image contrast, and finally perform non-local mean filtering to denoise the image.

3. A lightweight infrared depth odometer method according to claim 1, characterized in that: For the image captured by the infrared camera, estimating its pose with respect to the reference frame includes: A direct method is used to estimate the pose of the current frame: first, the relative pose initial value is set using the motion model hypothesis, and the observation points in the reference frame are back-projected to the current frame to construct a photometric error function. That is, the difference in grayscale values ​​between the observation points of the same 3D point in the two frames is directly used as the error. The Levenberg-Marquart method is used to optimize the pose and photometric parameters layer by layer from the top layer to the bottom layer of the image gradient pyramid. The error function is: Among them, ||·|| γ is the Huber norm; N p It is the first j Back-projection to the Ith frame i The pixel point on the frame and the set of seven pixels around it, I j Indicates the current frame, I i represents the reference frame, is the square of the second norm of the image gradient value at point p, I j [p′] and I i [p] are respectively j and I i The image grayscale value of pixel p′ and pixel p on the frame; w p is the Huber weight, c is a constant, and the calculation formula is: The nonlinear equation is approximated by performing Taylor expansion on the error and neglecting quadratic and higher terms: J T WJδx=-J T Wr Among them, the Jacobian matrix J is the partial derivative of r with respect to the image frame pose: The superscript T indicates the transpose operation, and W is w p The diagonal matrix formed by r is the residual between the corresponding observation points of the two image frames: r=(I j [p′(T i ,T j ,d,K)]-I i [p]) Among them, I i [p],I j [p′(T i ,T j ,d,K)] is the observed pixel grayscale value of the same 3D point in the reference frame and the current frame; T i ,T j ,d,K are the first i Frame, I j The transformation matrix of the frame image, the inverse depth of the pixel point p and the camera intrinsic parameter, δx is the increment of the motion state parameter; iteratively solve the linear equation and update the motion state from the reference frame to the current frame x←x+δx; after several iterations, find the pose estimation that minimizes the residual from the reference frame to the current frame image as the final result of this pose estimation.

4. A lightweight infrared depth odometer method according to claim 1, characterized in that: The ORB feature extraction and matching are performed on the reference frame and the NUC post-frame, and the average optical flow is calculated for all matched points: Where ii is the index of the matching feature point set, n is the number of matching point pairs, OF(q ii ,q′ ii ) is the optical flow between the reference frame and the NUC post-frame matching points: OF(q ii ,q′ ii )=||q′ ii -q ii ||2 where q ii is the feature point on the reference frame, q′ ii It is the feature point on the NUC frame; The map points held by the reference frame are projected back to the pixel points on the reference frame plane, and their corresponding positions on the NUC post-frame are calculated: s′=s+OF Where s is the pixel coordinate set on the reference frame, and s′ is the pixel coordinate set corresponding to the NUC post-frame; After obtaining the projection coordinates of the NUC post-frame, the reprojection error is minimized to obtain the position and posture of the reference frame relative to the NUC post-frame, including: Recover the camera motion during NUC by minimizing the reprojection error: Where idx is the index of the pixel set on the NUC frame, s′ idx are the pixel coordinates of the 3D point observed on the NUC post-frame, are all the map points held by the reference frame, π t is the projection equation of the infrared camera; is the pose to be solved, that is, the pose of the reference frame relative to the NUC post-frame.

5. A lightweight infrared depth odometer method according to claim 1, characterized in that: When ORB feature extraction and matching are performed on the reference frame and the post-NUC frame, the camera motion during NUC is solved using the information provided by the depth camera in the case where feature points cannot be extracted or are mismatched, including: Through the projection equation of the depth camera, we get the point cloud: Among them, id is the index on the depth map, pc id is the 3D point obtained by projection; π d is the projection equation of the depth camera; is the pixel coordinate on the depth map, yes Depth value; downsample the point cloud PC and perform ICP solution on the downsampled point cloud of the adjacent depth map: Where index is the index of the point cloud collection, are the downsampled point cloud of the previous frame and the point cloud of the next frame, N is the number of points in the point cloud, R and t are the rotation and displacement components in the transformation matrix from the previous frame point cloud to the next frame point cloud, respectively; assuming that a total of m frames of depth maps are captured during NUC, the total motion T nuc for: T nuc =T1T2…T m-1 Where T i represents the pose transformation matrix from the i-th to the i+1-th frame during NUC, 1≤i≤m-1, then the motion of the camera during NUC is restored as: Where T td is the external parameter from the depth camera to the infrared camera, T tnuc is the movement of the infrared camera during NUC.

6. A lightweight infrared depth odometer method according to claim 1, characterized in that: The Gauss-Newton method is used to optimize the inverse depth value of the generated map points.

7. A lightweight infrared depth odometer method according to claim 1, characterized in that: The sliding window is a sequence of key frames, in which all key frames and their map points are added to a linear equation: J sw T W sw J sw δx sw =-J sw T W sw r sw The superscript T represents the matrix transpose, and the subscript sw represents that the entire equation maintains all the state quantities to be optimized in the sliding window. sw It is the Jacobian matrix of the photometric residuals composed of all frames with co-viewing relationship and their corresponding points in the sliding window; W sw is the matrix composed of the corresponding Huber weights, δx sw are all the state increments to be optimized in the sliding window; r sw It is the photometric residual composed of all frames and their corresponding points in the sliding window.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a lightweight infrared depth odometer method as described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a lightweight infrared depth odometer method as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the lightweight infrared depth odometer method described in any one of claims 1 to 7 is implemented.

Images