A visual odometer method, storage medium and device based on illumination suppression

Abstract

The present invention belongs to the technical field of synchronous positioning and map creation, and discloses a visual odometer method based on illumination suppression, comprising the following steps: step S1, collecting image information, and using a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas; step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes; step S3, using a feature extraction algorithm based on average grayscale value to supplement feature points in sparse scenes; step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating poses based on tracked feature points. The present invention can maintain a certain degree of dynamic response capability, thereby enhancing the network's processing capability for illuminated scene images and improving the tracking stability of the SLAM system in illuminated scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of simultaneous location and mapping (SLAM), and in particular relates to a visual odometer method, storage medium and device based on illumination suppression. Background Art

[0002] Simultaneous Location And Mapping (SLAM) refers to the technology of a robot building a model of the environment while moving in an unknown environment and estimating its own motion at the same time. Among them, the visual odometry uses continuous camera images to estimate the travel distance to determine the equivalent odometry information. Visual odometry allows robots or vehicles to use any type of movement on any surface to improve navigation accuracy. Traditional visual SLAM methods are divided into feature point methods and direct methods, and the two methods differ mainly in the front end. Among them, the feature point method first extracts image feature points and descriptors, then performs feature point matching, and finally uses the reprojection error to optimize the robot's posture. The direct method uses the image grayscale value to obtain the robot's initial posture information by minimizing the photometric error, and then calculates the pixel depth to achieve self-positioning. The direct method directly minimizes the error in the sensor space and avoids feature matching and extraction. However, if Fig.11 As shown in the figure, several sets of image pairs with typical illumination changes do not meet the grayscale invariance assumption. These scenes are divided into four typical categories, namely, image blur caused by light diffusion (Blur), illumination changes in sparse scenes (Sparse), overexposed images (Overexposed), and dynamic light source (Active Lighting) images. Five pairs of images with small angle differences but large illumination changes are collected for each category. Using the direct method based on the illumination invariance assumption in scenes with drastic illumination changes will cause excessive pose errors, resulting in serious trajectory drift in large-area high-light scenes and even tracking loss. Summary of the invention

[0003] The purpose of the present invention is to provide a visual odometer method based on illumination suppression, which is used to solve the technical problem in the prior art that the use of a direct method based on the assumption of illumination invariance in scenes with drastic illumination changes will result in excessive posture errors, leading to severe trajectory drift or even tracking loss in large-area high-light scenes.

[0004] The visual odometer method based on illumination suppression comprises the following steps.

[0005] Step S1, collect image information, and use a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas.

[0006] Step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes.

[0007] Step S3: Use a feature extraction algorithm based on average gray value to supplement feature points in the sparse scene.

[0008] Step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating the pose based on the tracked feature point pairs.

[0009] The neural network in step S1 adopts a GAN architecture, in which the generator network uses an encoder-decoder framework. In the encoder-decoder structure, a residual block with a highlight attention mechanism is used to restore the original color of the object. The residual block is connected in series after the encoder. The highlight attention mechanism includes channel attention and spatial attention. The channel attention uses a global maximum sampling method to compress the feature map, and the spatial attention uses a global average sampling method to compress the feature map. The original image is convolved with the feature map output by the encoder-decoder structure to obtain a highlight removal map. The highlight mask is extracted from the original image through brightness threshold processing, and the highlight mask is convolved with the highlight removal image to obtain a highlight suppression image.

[0010] Preferably, in step S1, the highlight attention mechanism compresses the feature map, and finally obtains the channel attention vector I c and the spatial attention matrix I s :

[0011]

[0012] Among them, [max] represents the maximum value vector of all channels, the subscript c represents the sequence number of the channel, and max c Indicates the maximum value of the image pixel under the cth channel; [mean] indicates the average value matrix of all channels on each pixel, the subscript xy indicates the pixel coordinate (x, y) position, mean xy Represents the average value of all channel pixels at the coordinate (x, y);

[0013] For the maximum value vector [max], this method uses the piecewise maximum sampling method to calculate the maximum value max c First, expand the image matrix row of a single channel into a vector and divide it into multiple segments. Find the maximum value in each segment, build a probability distribution based on the maximum value in the segment, and then randomly sample the maximum values ​​in these segments according to the probability distribution, and select the maximum value in a segment as the maximum value of the channel. c ;

[0014] For the mean matrix [mean], this method uses the weighted shift average sampling method to calculate the mean value mean xy, and the mean value mean xy The vector composed of all corresponding channel pixels is divided into multiple windows. The weight of each channel pixel in the window is determined by a smooth weight function, and then the weighted average value in the window is calculated as the average value of all channel pixels at the corresponding position. xy .

[0015] Preferably, in the segmented maximum sampling method, the vector obtained by expanding the row of the image matrix of a single channel is z=[z1 z2 … z n ], the vector z is divided into k non-overlapping regions. If n is not divisible by k, the vector is automatically filled to meet the needs of the block division to ensure that n is divisible by k, so that each segment of the vector contains n / k elements after being divided into multiple segments. The formula for calculating the maximum value is:

[0016]

[0017] Among them, m j is the maximum value within the jth segment, z i is the i-th element in the vector z; this method smoothes the probability distribution through an adjustable probability function, the probability function P(m j )as follows:

[0018]

[0019] Among them, m l is the maximum value within the lth segment, and f(·) is a function that maps the maximum value within the segment to probability: where γ controls the degree of preference.

[0020] Preferably, in the weighted shift average sampling method, xy The corresponding vector composed of all channel pixels is u=[u1 u2…u n ], divide the vector u into k non-overlapping windows, and the center of the j-th window is located at the j-th sample point u j At , the sample point belongs to the channel pixel in the vector u, then the weight ω of the i-th sample point in the window i The formula is:

[0021]

[0022] The weighted average v in the window j It is expressed as:

[0023]

[0024] in, This means rounding down to ensure that the index does not exceed the bounds of the vector.

[0025] Preferably, the residual block further comprises a gated convolution and an expansion layer, the gated convolution assigns different weights to each feature element, and the feature map output by the gated convolution layer is F o In the activation phase, the feature map F output by the gated convolutional layer is o Fusion of the channel attention vector I c and the spatial attention matrix I s Output, the formula is:

[0026]

[0027] Among them, σ is the sigmoid activation function, MLP(·) is a fully connected neural network, and F is the feature map output by highlight attention.

[0028] Preferably, in step S3, the feature point extraction algorithm is a concave-convex feature point extraction algorithm, which uses the unevenness of the wall surface and illumination to select concave-convex points with obvious differences as concave-convex feature points, and only performs score calculation once for each pixel point, calculates the score according to the average gray value, and normalizes the score, and lists points that are significantly greater than or less than the average gray value as candidate feature points. The score calculation formula is expressed as:

[0029]

[0030] Among them, I xy is the gray value of the pixel at coordinate (x, y), I xy (i, j) is the grayscale value of the pixel at coordinate (i, j) in the 3*3 area centered at coordinate (x, y), S is the feature point score, λ is the extraction threshold, and M is the amplitude resolution of the image; according to the calculated score, the candidate feature points with scores exceeding the set threshold are screened and extracted as concave-convex feature points.

[0031] Preferably, in step S2, the feature extraction network includes a dynamic snake convolution module, and the feature extraction link is completed by ordinary convolution and snake convolution operations, the convolution feature map obtained by ordinary convolution is decoded into a 1-channel feature map and a 3-channel feature map, the dynamic snake convolution feature is merged with the 3-channel feature map, and L2 normalization is performed to obtain a feature map for optical flow tracking; at the same time, a sigmoid function is used for the 1-channel feature map to reduce its value between [0, 1] to obtain a point probability map for feature extraction; finally, optical robustness feature points are extracted from the point probability map and the convolution feature map, and the feature maps of two consecutive frames are tracked using the optical flow method to obtain feature points in the latter frame corresponding to the previous frame.

[0032] Preferably, in step S4, the concave and convex feature points are used as the starting points of the optical flow, and the tracking link tracks the feature points using the grayscale invariant assumption. On the basis of the LK optical flow, the grayscale invariant assumption is fused with the convolution feature consistency to determine the tracked feature point pairs, and then the pose is calculated based on the tracked feature point pairs. The convolution feature consistency means that the convolution feature vectors of the same feature point position in different images are the same.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the visual odometer method based on illumination suppression as described above are implemented.

[0034] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the steps of the visual odometer method based on illumination suppression as described above are implemented.

[0035] The present invention has the following advantages:

[0036] 1. The present invention uses a generative adversarial network architecture and uses a large number of highlight data sets for model training. A residual block containing an attention mechanism is set in the encoder-decoder structure, and the attention mechanism is improved. A highlight attention method is proposed. The highlight attention method consists of two parts, which respectively obtain the attention information of the image in the channel and space. Each part includes two links: sampling and activation. In a strong light environment, the highlight pixel area has a higher pixel value in the image than the ordinary area. Therefore, the channel attention in the scheme uses the global maximum sampling method to compress the feature map, and the spatial attention uses the global average sampling method to compress the feature map. On this basis, this scheme adopts an innovative segmented maximum sampling method and a weighted displacement average sampling method to calculate the maximum value and the average value respectively. In this way, on the one hand, while maintaining a preference for large values, more flexibility is introduced; on the other hand, when processing data with local characteristics, a more delicate smoothing effect can be provided, while maintaining a certain degree of dynamic response capability, thereby enhancing the network's processing capability for illuminated scene images, improving the tracking stability of the SLAM system in illuminated scenes, and overcoming the technical problem of serious trajectory drift or even tracking loss in large-area highlight scenes.

[0037] 2. The optical flow method of the present invention also includes two links: feature extraction and tracking. In the feature extraction link, in order to improve the feature extraction capability in sparse scenes, this method adopts a new feature point extraction algorithm to extract feature points from the feature map fused with dynamic snake convolution. The algorithm mainly uses the unevenness of the wall surface and illumination to select concave and convex points with obvious differences as feature points. This method only calculates the score for each pixel once, and the extraction speed is faster. The algorithm calculates the score based on the average grayscale value and normalizes the score. Points that are significantly greater than or less than the average grayscale are listed as candidate feature points, and corner points with strong invariance (i.e., concave and convex feature points) are extracted based on the score. Based on the grayscale invariance assumption of the optical flow method, the convolution feature invariance assumption is proposed to obtain a hybrid optical flow method that is robust to light.

[0038] 3. The present invention also obtains an image generator with a highlight mask and a highlight removal map through model training, and then uses the highlight mask to perform secondary highlight removal on the highlight area. A lightweight convolutional neural network is used in the optical flow tracking link to extract convolution features that are robust to lighting. The feature extraction network fused with dynamic snake convolution can adaptively focus on slender and tortuous local structures to accurately capture the complex lighting features in the image. In order to enable the convolution kernel to focus on the complex geometric features of the target, a deformation offset is introduced, and the deformation offset is iterated to select the position to be observed for each target to be processed in turn, thereby ensuring the continuity of attention and preventing the receptive field from spreading too far due to large deformation offsets. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The following is a basic flow chart of a visual odometer method based on illumination suppression according to the present invention.

[0040] Figure 2 The flowchart of the SLAM system of the present invention is applied.

[0041] Figure 3 This is a flowchart of the neural network that integrates highlight attention in the present invention.

[0042] Figure 4 Flowchart of the highlight attention mechanism in the present invention.

[0043] Figure 5 Flow chart of the feature extraction network including the dynamic snake convolution module in the present invention.

[0044] Figure 6 It is a schematic diagram of the principle of the concave-convex feature point extraction algorithm in the present invention.

[0045] Figure 7 This is a comparison chart of the highlight suppression effect between the present invention and the existing technologies such as IAT, IBCLN, Spec-CGAN and JSHDR.

[0046] Figure 8 The figure is a comparison chart of the extraction effects obtained by performing feature extraction on images before and after highlight suppression using the present invention using existing feature extraction algorithms such as FAST, SFIT, GFTT and LET-NET.

[0047] Fig. 9 This is a comparison chart of the extracted quantities obtained by using the GFTT feature extraction algorithm before and after highlight suppression at different thresholds using the method of the present invention.

[0048] Fig.10 This is a comparison diagram of the extracted quantities obtained by using the SIFT feature extraction algorithm before and after highlight suppression at different thresholds using the method of the present invention.

[0049] Fig.11 These are four sets of images of bright light scenes with typical illumination variations that do not conform to the grayscale invariance assumption.

[0050] Fig.12 This is a comparison chart of the absolute trajectory error between the method of the present invention and three existing algorithms, OV2SLAM, SVO_Pro and LDSO.

[0051] Fig.13 This is a comparison chart of the relative pose error between the method of the present invention and three existing algorithms, OV2SLAM, SVO_Pro and LDSO.

[0052] Fig.14 A real scene layout diagram of a SLAM experiment conducted with a real scene of illumination changes and strong exposure to verify the present invention.

[0053] Fig.15 The following is a comparison chart of the trajectories obtained by the present invention and the ORB-SLAM3 and OV2SLAM algorithms in SLAM experiments. DETAILED DESCRIPTION

[0054] The specific implementation modes of the present invention will be further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0055] Embodiment 1.

[0056] like Figure 1-Figure 6 As shown, the present invention provides a visual odometer method based on illumination suppression, comprising the following steps.

[0057] Step S1, collect image information, and use a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas.

[0058] In this step, the image information is first collected by the monocular camera of the robot, and then the obtained image information is input into the neural network that integrates the highlight attention. The neural network in this step adopts the GAN (Generative Adversarial Network) architecture, in which the generator network uses the encoder-decoder framework and adopts an iterative dual-branch structure. For a given input image, the neural network extracts the feature map through the encoder-decoder structure. In the encoder-decoder structure, the residual block E4 with the highlight attention mechanism and gated convolution is used to restore the original color of the object, and the residual block E4 is connected in series after the encoder. Then the original image is convolved with the feature map output by the encoder-decoder structure to obtain a highlight removal map. Because in most cases the highlight area is usually large and it is difficult to remove it at one time, the original image is also processed by brightness thresholding to extract the highlight mask in this step, and the highlight mask is then convolved with the highlight removal image to obtain a more realistic and accurate highlight suppression image.

[0059] Specifically, the generator network consists of three layers of downsampling convolution blocks, residual blocks, and three layers of upsampling convolution blocks. The residual module contains gated convolution, dilation layer, and the key improvement of this solution - the highlight attention mechanism. Unlike ordinary convolution, gated convolution assigns different weights to each feature element, thereby avoiding feature contamination in the highlight area. In gated convolution, given the input feature map F i , output feature map F o It can be expressed as:

[0060]

[0061] Where σ is the sigmoid function, is the ReLU activation function, symbol and ⊙ are the convolution operator and element-wise product operator respectively, and W g and W f is the convolution kernel.

[0062] In a strong light environment, the highlight pixel area has a higher pixel value in the image than the ordinary area. Therefore, the highlight attention mechanism includes channel attention and spatial attention. The channel attention uses the global maximum sampling method to compress the feature map, and the spatial attention uses the global average sampling method to compress the feature map. Finally, the channel attention vector I is obtained. c and the spatial attention matrix I s :

[0063]

[0064] Among them, [max] represents the maximum value vector of all channels, the subscript c represents the sequence number of the channel, and max cIndicates the maximum value of the image pixel under the cth channel; [mean] indicates the average value matrix of all channels on each pixel, the subscript xy indicates the pixel coordinate (x, y) position, mean xy Represents the average value of all channel pixels at the coordinate (x, y) position.

[0065] For the maximum value vector [max], this method uses the piecewise maximum sampling method to calculate the maximum value max c The segmented maximum sampling method first divides the vector into multiple segments, finds the maximum value in each segment, and then samples these maximum values ​​according to certain rules. c The image matrix row of a single channel is expanded into a vector z = [z1 z2…z n ], divide the vector z into k non-overlapping regions. If n is not divisible by k, the vector is automatically filled to meet the needs of the block division to ensure that n is divisible by k, so that each segment of the vector contains n / k elements after being divided into multiple segments. The formula for calculating the maximum value is:

[0066]

[0067] Among them, m j is the maximum value within the jth segment, z i is the i-th element in the vector z.

[0068] A probability distribution is constructed based on the maximum value in the segment, and an adjustable probability function is used to smooth the probability distribution. The probability function P(m j )as follows:

[0069]

[0070] Among them, m l is the maximum value within the lth segment, and f(·) is a function that maps the maximum value within the segment to probability: Among them, γ controls the degree of preference, and selecting 80% of the image amplitude resolution can better focus on the highlight information. According to the above probability distribution, random sampling is performed to select a maximum value m within a segment. j As the maximum value of the channel c This allows for more flexibility while maintaining the preference for large values.

[0071] For the mean matrix [mean], this method uses the weighted shift average sampling method to calculate the mean value mean xyThis method not only considers the values ​​of all sample points, but also their time order or importance weight; the weight of each channel pixel is no longer simply given in advance, but is determined by a smooth weight function, which decreases monotonically with the pixel position to reflect the higher importance of the central pixel. Specifically, compared with the mean value, xy The corresponding vector composed of all channel pixels is u=[u1 u2…u n ], divide the vector u into k non-overlapping windows, and the center of the j-th window is located at the j-th sample point x j At , the sample point belongs to the channel pixel in the vector u, then the weight ω of the i-th sample point in the window i The formula is:

[0072]

[0073] The weighted average v in the window j It is expressed as:

[0074]

[0075] in, It means rounding down to ensure that the index does not exceed the vector range. The weighted average v j As the average value of all channel pixels at the coordinate (x, y) position xy This method can provide a more delicate smoothing effect when processing data with local characteristics, while maintaining a certain degree of dynamic responsiveness.

[0076] In the activation phase, the feature map F output by the gated convolutional layer is o Fusion of the above two types of attention information (i.e., channel attention vector I c and the spatial attention matrix I s ) output, the formula is expressed as:

[0077]

[0078] Among them, σ is the sigmoid activation function, MLP(·) is a fully connected neural network, and F is the feature map output by highlight attention.

[0079] The network discriminator used in the neural network of this scheme includes ten convolutional layers and two fully connected layers. The number of channels of the first convolutional layer is 64, which is doubled every two convolutional layers. The convolution step size of each odd-numbered layer is 1, and the convolution step size of each even-numbered layer is 2. Except for the first convolutional layer, each convolutional layer is followed by a group normalization layer and a leaky ReLU activation function, and the first fully connected layer is followed by a leaky ReLU activation function. Since the network of this method adds a high-light attention mechanism, the network's processing ability for illuminated scene images is enhanced, and the tracking stability of the SLAM system in illuminated scenes is improved.

[0080] Step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes.

[0081] The feature extraction network contains a dynamic snake convolution module, which uses four ordinary convolutions and one snake convolution operation to complete the feature extraction link. The decoder is used to output the point probability map and the feature map. Finally, the optical robust feature points are extracted from the point probability map and the convolution feature map. The optical flow method is used to track the feature maps of two consecutive frames to obtain the feature points in the latter frame corresponding to the previous frame.

[0082] First, the image feature encoder converts the input image I (I∈H×W×3) into an image of size H×W×16. The first convolution operation uses a 3×3 convolution kernel and expands the feature map to 8 channels. The last convolution operation uses a 1×1 convolution kernel to increase the number of channels of the feature map by 16. The 1×1 convolution feature map is then decoded into two feature maps, a 1-channel feature map and a 3-channel feature map. Finally, the dynamic snake convolution feature is fused with the 3-channel feature map and L2 normalized to obtain a feature map for optical flow tracking. At the same time, the sigmoid function is used for the 1-channel feature map to reduce its value between [0, 1] to obtain a point probability map for feature extraction. The ReLU activation function is used after each convolution in the feature extraction network, and the resolution of the original image is maintained in all convolution operations.

[0083] The method for extracting dynamic snake convolution features is as follows: First, a standard two-dimensional convolution kernel K is given in the image, and the center coordinate of the convolution kernel is K i =(x i ,y i ). In the dynamic snake convolution, the standard convolution kernel is straightened in the x-axis and y-axis directions, and the specific position of each grid in K is expressed as: K i±c =(x i±c ,y i±c ), where c represents the horizontal distance or vertical distance from the center grid. Taking the x-axis direction as an example, a convolution kernel with a side length of 9, where c = {0, 1, 2, 3, 4}, represents the horizontal distance from the center grid.

[0084] Each grid position K in the convolution kernel K i±c The selection of is a cumulative process. From the center position K i Initially, the position of the grid away from the center depends on the position of the previous grid, so the deformation offset is cumulative, ensuring that the convolution kernel can conform to the linear morphological structure. Finally, the grid position of the convolution kernel in the x-axis direction becomes:

[0085]

[0086] Among them, Δy represents the change of the convolution kernel in the y-axis direction. Due to the change in two dimensions (x-axis, y-axis), the dynamic snake convolution covers a 9×9 range during the deformation process. The snake convolution designed according to the dynamic structure can better adapt to various lighting environments, so as to better perceive key features and obtain dynamic snake convolution features.

[0087] Step S3: Use a feature extraction algorithm based on average gray value to supplement feature points in the sparse scene.

[0088] In this step, this method, like the traditional optical flow method based on the grayscale invariance assumption, includes two steps: feature extraction and tracking. In the feature extraction step, in order to improve the feature extraction capability in sparse scenes, this method adopts a new feature point extraction algorithm, the concave-convex feature point extraction algorithm, to extract feature points from the feature map fused with dynamic snake convolution features. The feature point extraction algorithm mainly uses the unevenness of the wall surface and illumination to select concave-convex points with obvious differences as feature points, namely concave-convex feature points.

[0089] The concave-convex feature point extraction algorithm only calculates the score once for each pixel point to improve the extraction speed. During the calculation, the score is calculated according to the average gray value, and the score is normalized. Points that are significantly greater or less than the average gray value are listed as candidate feature points. The score calculation formula is expressed as:

[0090]

[0091] Among them, I xy is the gray value of the pixel at coordinate (x, y), I xy (i, j) is the gray value of the pixel at coordinate (i, j) in the 3*3 region centered at coordinate (x, y), S is the feature point score, λ is the extraction threshold, and M is the amplitude resolution of the image. Based on the calculated scores, the positions with high scores are considered to have strong invariance and are easy to track, so the candidate feature points with scores exceeding the set threshold are extracted as concave-convex feature points (also called corner points).

[0092] In the above steps S2 and S3, the LET-NET optical flow tracking network is improved by the dynamic snake convolution module and the concave-convex feature point extraction algorithm, respectively, which can better adapt to the lighting environment and feature sparse scenes and enhance the tracking ability of the network.

[0093] Step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating the pose based on the tracked feature point pairs.

[0094] In this step, after obtaining the position of the concave and convex feature points, use it as the starting point of the optical flow. The tracking stage uses the grayscale invariant assumption to track feature points. On the basis of the LK optical flow, the grayscale invariant assumption is combined with the convolution feature consistency to determine the tracked feature point pairs, and then the pose is calculated based on the tracked feature point pairs. Convolution feature consistency means that the convolution feature vectors of the same feature point position in different images are the same. After convolution, the algorithm based on the grayscale invariant assumption is more accurate.

[0095] The following is an explanation of the process of the above-mentioned visual odometer method based on illumination suppression in combination with specific experiments.

[0096] like Figure 7 As shown in the figure, the experiment uses SHIQ and PSD datasets containing scenes with light gradients and sparse textures to verify the visual effect of highlight suppression and prove it through quantitative data. First, the images processed by this method are visually compared with those obtained by existing technologies such as IAT, IBCLN, Spec-CGAN and JSHDR. Among them, IAT and IBCLN only adjust the image exposure and cannot suppress the highlight area, while Spec-CGAN only has a certain highlight suppression effect. This method integrates highlight attention in a highlight environment and uses a highlight mask for secondary suppression of the highlight area. It has good highlight suppression and feature extraction capabilities in strong lighting and sparse texture scenes.

[0097] like Figure 8 As shown in the figure, in order to verify the effectiveness of the highlight suppression network of this scheme for SLAM feature point extraction, the experiment selected the feature extraction algorithms FAST, SFIT, GFTT and LET-NET to perform extraction comparison experiments on the images before and after the highlight suppression of this scheme. The extraction threshold and extraction quantity of all feature extraction algorithms are fixed, and the areas before and after the improvement are circled with red frames. The results show that the highlight suppression images generated by this scheme have improved the extraction effect of various existing feature extraction algorithms.

[0098] like Fig. 9 and Fig.10As shown in the figure, in order to further verify the effectiveness of this method, the experiment also selected GFTT and SIFT algorithms to extract 1000 feature points under different thresholds. The horizontal axis is the increasing extraction threshold, and the vertical axis represents the number of extractions. By comparing the number of extractions of this method and the comparison algorithm, it can be concluded that under different feature extraction threshold conditions, the effect of this method is more effective.

[0099] like Fig.12 and Fig.13 As shown in the figure, in order to verify the overall effect of this method, the widely used benchmark datasets KITTI and EuRoC datasets are used to evaluate the trajectory error for scenes with highlights and illumination changes, and the algorithms OV2SLAM, SVO_Pro and LDSO are selected for comparison with this method. All algorithms use sparse optical flow or direct method based on the grayscale invariance assumption, and deal with the illumination problem, and all have loop detection. The experiment uses the root mean square error (RMSE) of absolute trajectory and relative pose to measure the positioning accuracy. The absolute trajectory error (ATE) is suitable for evaluating the overall consistency between the estimated trajectory of the SLAM algorithm and the actual motion trajectory of the camera, while the relative pose error (RPE) is more suitable for evaluating the degree of trajectory drift. Therefore, the experiment can further verify the effectiveness of this method by comparing the absolute trajectory error and relative pose error of each algorithm. All algorithms are tested under monocular 10HZ conditions. The experimental results show that due to the improvement of this method for highlights and illumination changes, the final trajectory has a high degree of consistency with the actual trajectory.

[0100] like Fig.14 As shown in the figure, to verify the feasibility of this method, a SLAM experiment was conducted in a real scene with illumination changes and strong exposure. The Intel RealSense D435i sensor was fixed on the robot about 1m above the ground. In the experiment, the robot was made to bypass obstacles through the host computer to form a "9" loop. The scene scale was about 5.5m×4.8m. The experimental results are shown in the figure. Fig.15 As shown in the figure, it can be seen that the ORB-SLAM3 algorithm is affected by strong light, resulting in a reduction in feature points and an increase in initialization time, which ultimately leads to a missing track at the beginning and loop failure. Since the OV2SLAM algorithm is affected by light changes, the grayscale invariance assumption fails and the trajectory accuracy is poor. This method combines improved high light suppression and optical flow tracking networks, and is still highly robust in strong light environments.

[0101] Embodiment 2.

[0102] Corresponding to the first embodiment of the present invention, the second embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following steps are implemented according to the method of the first embodiment.

[0103] Step S1, collect image information, and use a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas.

[0104] Step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes.

[0105] Step S3: Use a feature extraction algorithm based on average gray value to supplement feature points in the sparse scene.

[0106] Step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating the pose based on the tracked feature point pairs.

[0107] The above storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), optical disk and other media that can store program codes.

[0108] The specific limitations on the steps implemented after the program in the computer-readable storage medium is executed can be found in Example 1, and will not be described in detail here.

[0109] Embodiment three.

[0110] Corresponding to the first embodiment of the present invention, the third embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the program, the following steps are implemented according to the method of the first embodiment.

[0111] Step S1, collect image information, and use a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas.

[0112] Step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes.

[0113] Step S3: Use a feature extraction algorithm based on average gray value to supplement feature points in the sparse scene.

[0114] Step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating the pose based on the tracked feature point pairs.

[0115] The specific limitations on the above steps of implementing the computer device can be found in Example 1, and will not be described in detail here.

[0116] It should be noted that each box in the block diagram and / or flow chart in the accompanying drawings of the specification of the present invention, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and machine instructions.

[0117] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A visual odometer method based on illumination suppression, characterized in that: The following steps are involved: Step S1, collecting image information, and using a neural network fused with highlight attention to extract feature information from the image information and suppress highlight areas; Step S2, using a feature extraction network to extract feature points and feature maps that are robust to illumination changes; Step S3, using a feature extraction algorithm based on average gray value to supplement feature points in the sparse scene; Step S4, using a multi-layer optical flow tracking algorithm to track feature points, and calculating the pose based on the tracked feature point pairs; The neural network in step S1 adopts a GAN architecture, in which the generator network uses an encoder-decoder framework. In the encoder-decoder structure, a residual block with a highlight attention mechanism is used to restore the original color of the object. The residual block is connected in series after the encoder. The highlight attention mechanism includes channel attention and spatial attention. The channel attention uses a global maximum sampling method to compress the feature map. The spatial attention uses a global average sampling method to compress the feature map. The original image is convolved with the feature map output by the encoder-decoder structure to obtain a highlight removal map. The original image is processed by brightness thresholding to extract a highlight mask, and the highlight mask is convolved with the highlight removal image to obtain a highlight suppression image. In step S1, the highlight attention mechanism compresses the feature map and finally obtains the channel attention vector I c and the spatial attention matrix I s : ， Among them, [max] represents the maximum value vector of all channels, and the subscript c Indicates the channel number, max c Indicates c The maximum value of image pixels under the channel; [ mean ] represents the average value matrix of all channels at each pixel, subscript xy Represents pixel coordinates ( x , y )Location, mean xy Represents coordinates ( x , y ) is the average value of all channel pixels at the position; For the maximum value vector [max], this method uses the piecewise maximum sampling method to calculate the maximum value max c First, expand the image matrix row of a single channel into a vector and divide it into multiple segments. Find the maximum value in each segment, build a probability distribution based on the maximum value in the segment, and then randomly sample the maximum values ​​in these segments according to the probability distribution, and select the maximum value in a segment as the maximum value of the channel. c ; For the mean matrix [ mean ], this method uses the weighted shift average sampling method to calculate the average value mean xy , and the average mean xy The vector composed of all corresponding channel pixels is divided into multiple windows. The weight of each channel pixel in the window is determined by a smooth weight function, and then the weighted average value in the window is calculated as the average value of all channel pixels at the corresponding position. mean xy .

2. The visual odometer method based on illumination suppression according to claim 1, characterized in that: In the segmented maximum sampling method, the vector obtained by expanding the image matrix row of a single channel is z =[ z 1 z 2 … z n ],vector z is divided into k non-overlapping regions. n Cannot be k If the value is divisible by an integer, the vector is automatically filled to accommodate the block requirements, ensuring n Can be k Divide by integers, so that the vector is divided into multiple segments, each containing n / k elements, the formula for calculating the maximum value is: ， in, m j For the j The maximum value within a segment, z i For vector z The i elements; this method smoothes the probability distribution through an adjustable probability function, the probability function P ( m j )as follows: ， in, m l For the l The maximum value within a segment, f (·) is a function that maps the maximum value within a segment to a probability: , where γ controls the degree of preference.

3. The visual odometer method based on illumination suppression according to claim 1, characterized in that: In the weighted shift average sampling method, mean xy The corresponding vector composed of all channel pixels is u =[ u 1 u 2 … u n ], the vector u Divide k non-overlapping windows, and the j The center of the window is located at j Sample points u j At, the sample point belongs to the vector u , then the channel pixels in the window i The weight of the sample points ω i The formula is: ， The weighted average within the window v j It is expressed as: ， in, This means rounding down to ensure that the index does not exceed the bounds of the vector.

4. The visual odometer method based on illumination suppression according to claim 1, characterized in that: The residual block also includes a gated convolution and a dilation layer. The gated convolution assigns different weights to each feature element. The feature map output by the gated convolution layer is F o , in the activation phase, the feature map output by the gated convolutional layer F o Fusion of the channel attention vectors I c and the spatial attention matrix I s Output, the formula is: ， in, is the sigmoid activation function, MLP (·) is a fully connected neural network, F Feature map output by highlight attention.

5. The visual odometer method based on illumination suppression according to claim 1, characterized in that: In step S3, the feature point extraction algorithm is a concave-convex feature point extraction algorithm. By using the unevenness of the wall surface and illumination, concave-convex points with obvious differences are selected as concave-convex feature points. The score is calculated only once for each pixel point. The score is calculated according to the average gray value, and the score is normalized. Points that are obviously greater than or less than the average gray value are listed as candidate feature points. The score calculation formula is expressed as: ， in, I xy is the coordinate ( x , y )’s grayscale value, I xy ( i , j ) is the coordinate ( x , y ) is the center of the 3*3 area coordinates ( i , j )’s grayscale value, S is the feature point score, λ is the extraction threshold, M is the amplitude resolution of the image; according to the calculated score, the candidate feature points with scores exceeding the set threshold are screened and extracted as concave-convex feature points.

6. The visual odometer method based on illumination suppression according to claim 1, characterized in that: In the step S2, the feature extraction network includes a dynamic snake convolution module, and the feature extraction link is completed by ordinary convolution and snake convolution operations. The convolution feature map obtained by ordinary convolution is decoded into a 1-channel feature map and a 3-channel feature map. The dynamic snake convolution feature is merged with the 3-channel feature map, and L2 normalization is performed to obtain a feature map for optical flow tracking; at the same time, a sigmoid function is used for the 1-channel feature map to reduce its value between [0, 1] to obtain a point probability map for feature extraction; finally, optical robustness feature points are extracted from the point probability map and the convolution feature map, and the feature maps of two consecutive frames are tracked using the optical flow method to obtain feature points in the latter frame corresponding to the previous frame.

7. The visual odometer method based on illumination suppression according to claim 1, characterized in that: In step S4, the concave and convex feature points are used as the starting points of the optical flow. The tracking link tracks the feature points using the grayscale invariant assumption. On the basis of the LK optical flow, the grayscale invariant assumption is combined with the convolution feature consistency to determine the tracked feature point pairs. Then, the pose is calculated based on the tracked feature point pairs. The convolution feature consistency means that the convolution feature vectors of the same feature point position in different images are the same.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a visual odometer method based on illumination suppression as described in any one of claims 1 to 7 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of a visual odometer method based on illumination suppression as described in any one of claims 1 to 7 are implemented.

Images