A Robot Vision SLAM Method, Storage Medium and Device for Large View Angles

Abstract

The present invention belongs to the technical field of SLAM, and discloses a robot vision SLAM method, a storage medium and a device for large viewing angles. The algorithm includes: Step S1, collecting image information, and using a multi-scale and multi-stage neural network to extract feature information from the image and repair blurred areas; Step S2, predicting the positions of the feature points matched with the image based on the IMU sensor, and judging whether the feature points of the next frame are within the predicted range; Step S3, dividing the image, dividing it into several regions, calculating the information entropy for each region, and performing grid division according to the size of the information entropy; Step S4, proposing a new adjacent grid domain grid statistical algorithm to further screen the feature points selected by the IMU sensor. The present invention repairs blurred image frames, so that a large number of image feature points can be accurately extracted during large viewing angle movement, and the timeliness and accuracy are improved simultaneously in precise matching through improvement.

Description

Technical Field

[0001] The present invention belongs to the technical field of Simultaneous Location And Mapping (SLAM), and particularly relates to a robot vision SLAM method, a storage medium, and a device for a large viewing angle. Background Art

[0002] Simultaneous Location And Mapping (SLAM) refers to the process of simultaneously constructing a map of the current environment and locating the position and pose of a robot itself in an unknown environment using sensors carried by the robot. However, in actual scenarios, limited by the camera frame rate and environmental factors, blurred image frames are likely to occur during large viewing angle movements. Vision SLAM systems often face problems such as blur, occlusion, dynamic objects, or light source interference. The lack of feature information in images makes it difficult for traditional feature extraction algorithms to extract a large number of image feature points. These problems seriously affect the accuracy of feature extraction and matching and the robustness of the algorithm. Therefore, a vision SLAM algorithm suitable for large viewing angles is crucial for improving the robustness of the system.

[0003] Some existing technologies use the adjacent grid domain grid statistical algorithm to further screen correct matching feature point pairs based on rough matching of feature points. However, in the existing technologies, due to the need to calculate and evaluate the matching effect of feature points in the neighborhood for each region during precise matching after gridification, it will cause the defects of large computational complexity and long feature matching time, affecting the timeliness of simultaneous location and mapping. On the other hand, if the number of adjacent grid domains participating in the calculation is insufficient, it will affect the accuracy of precise matching. Moreover, during movement, the number of feature points contained in the images collected in different regions is not the same. If this is not considered, when dividing the grid for calculation, it may either result in insufficient feature points in the adjacent grid domain due to overly dense grid division, unable to obtain an accurate precise matching effect, or result in too many feature points in each grid due to overly sparse grid division, with the calculation results between different feature points being close, unable to well reflect the actual matching situation, and also reducing the accuracy of precise matching. Summary of the Invention

[0004] The purpose of the present invention is to provide a robot vision SLAM method for a large viewing angle, which is used to solve the technical problems in the existing technologies, namely, how to accurately extract a large number of feature points from blurred images and how to screen the rough matching results to achieve a precise matching effect that takes into account both timeliness and accuracy.

[0005] The described robot vision SLAM method for a large viewing angle includes the following steps.

[0006] A robot vision SLAM method for large viewing angles, characterized in that it includes the following steps:

[0007] Step S1, collect image information, and use a multi-scale and multi-stage neural network to extract feature information from the image and repair blurred areas;

[0008] Step S2, based on the IMU sensor, perform position prediction on the feature points matched by the image, and judge whether the next-frame feature points are within the predicted range;

[0009] Step S3, divide the image, divide it into several regions, calculate the information entropy for each region, and perform grid division according to the size of the information entropy;

[0010] Step S4, propose a new adjacent grid domain grid statistical algorithm to further screen the feature points selected by the IMU sensor.

[0011] Preferably, in step S4, the angle change of the camera is sensed by the IMU sensor, and an angle threshold is set. If the camera undergoes a viewing angle transformation and the transformation angle is greater than the angle threshold, the diagonal adjacent grid domain grid algorithm is used to calculate the feature score of the grid. Conversely, if the transformation angle is less than the angle threshold, the cross adjacent grid domain grid algorithm is used to calculate the feature score of the grid; weights are assigned to each grid according to the number of correct matching pairs of feature points in each grid; the adjacent grid domain grid algorithm calculates the feature scores of the current grid and the four grids adjacent to it or diagonally symmetric to it, and the sum of the feature scores of these five grids is called the feature score of the adjacent grid domain grid; since the image may rotate when the transformation angle is greater than the angle threshold, this step uses a rotational motion kernel to perform a rotational operation on these five grids around the center, and the maximum adjacent grid domain feature score in 4 different cases is statistically determined as the feature score of the diagonal adjacent grid domain; the feature score of the neighborhood calculated by the feature points after rough matching is compared with the corresponding feature score threshold. If it is greater than the feature score threshold, it indicates that the rough matching result is correct, otherwise it indicates that the rough matching result is incorrect. Thus, the screening of the rough-matched feature points is completed, and the accurate matching of the feature points is realized.

[0012] Preferably, in step S3, the information entropy is calculated for each region, and the image is divided by comparing the local information entropy: for places with smaller information entropy, larger grids are used for division; for regions with larger information entropy, smaller grids are used for division, so that the feature points that were previously at the grid boundary are also within the grid.

[0013] Preferably, since the grid has been divided based on the information entropy by region, the score uses an adaptive feature score threshold. The following is the calculation formula for a new adaptive feature score threshold used in step S4:

[0014]

[0015] Where: τ represents an adaptive feature score threshold, k is a weight coefficient for the number of feature points, and μ i represents the mean of the number of rough matching features of the i-th grid in the neighboring grid domain, i represents the grid number, π is the basic threshold, and p(x β ) is the probability that the random gray variable x takes the value x β , and is used to represent the probability that the gray level x β appears in the image. β is the number of the random gray variable, n is the number of values of the random variable, and n is in the range of (0, 255).

[0016] Preferably, when the camera transformation angle is different, the feature score formula for the neighboring grid domain is:

[0017]

[0018] Where, S is the feature score of the neighboring grid domain, w represents the mean of the number of rough matching features in each grid in the neighboring grid domain, i represents the grid number, n represents the number of rotations, α i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during movement, and δ is the angle threshold; s i,m is the flag score of the m-th feature point of the i-th grid in the cross-shaped neighboring grid domain, represents the flag score of the m-th feature point of the i-th grid in the diagonal neighboring grid domain after the n-th rotation, and the correct matching number of each grid near the statistical matching point is obtained to get the flag score of the corresponding grid.

[0019] Preferably, in the step S1, the multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, s3. Each scale has 1, 2, and 3 stages respectively, and each stage is composed of a single lightweight encoder-decoder UNet module. Each UNet module shares the same network structure but has different weights; each UNet module is trained to be able to generate residual features for converting a blurred image into a clear image, and these residual features are added to the blurred image to generate a deblurred image; cross-stage feature fusion is performed between each stage and between each scale through the CSFF module. A feature attention module and an attention supervision module are introduced at the beginning and end of each scale respectively, and different weights are assigned to different pixel and channel features on the feature map, effectively utilizing the information on the feature map and improving the deblurring effect of the network.

[0020] Preferably, the feature attention module includes a ResNet101 network, a channel attention mechanism, and a spatial attention mechanism. The information obtained by the camera is used to extract feature information through the ResNet101 network. Then, these feature maps are further adjusted and enhanced through the channel attention mechanism and the spatial attention mechanism to improve their expression ability and attention. Subsequently, the feature maps are processed through bilinear upsampling and 1×1 convolution operations to adjust the adjusted and enhanced feature maps to the same size as the input image and transform the channel dimension of the feature maps; the input feature F of the attention supervision module in First, a residual image R is generated through a 1×1 convolutional layer S , and the obtained residual image and the degraded image I are combined to obtain a restored image X S , and then the restored image X is processed through a 1×1 convolutional layer and a sigmoid activation function S , to generate an attention map M; the input feature F processed through the 1×1 convolutional layer is recalibrated through the attention map M in , to obtain the feature guided by the attention map, that is, the attention-enhanced feature F generated by the SAM out , and the feature F out is passed to the next stage.

[0021] Preferably, in the step S2, the acceleration and angular velocity values of the camera are measured by an IMU sensor, the displacement of the camera is obtained through integration, and the rotation matrix of the camera is obtained through quaternions, so as to obtain the camera coordinates of the next frame. First, the two-dimensional coordinates of the feature points in the previous frame are used to calculate the three-dimensional coordinates P of the feature points in the previous frame through the two-dimensional to three-dimensional coordinate projection formula 0 (x 0 , y 0 , z 0 ), and then the three-dimensional coordinates P of the camera in the next frame are obtained through the rotation matrix and the translation vector 1 (x 1 , y 1 , z 1 ), and finally, they are converted back into two-dimensional coordinates through the projection formula. A circle is drawn with a radius r based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments to obtain a suitable value, so that the two-dimensional coordinate position of the feature points matching the next frame must be within the circle; the formula for calculating the camera coordinates of the next frame is as follows:

[0022]

[0023] where, [x 1 , y 1 , z 1 T represents the camera coordinates of the current frame, [x 0 , y 0 ​, z 0 T is represented as the camera coordinates of the previous frame, T represents the transpose, q w is the real part of the quaternion, q x , q y , q z are the three components of the imaginary part of the quaternion, t 0 , t f are the times at the start and end points of the interval respectively, t 2σ-1 and t 2σ represent the times of the nodes within the interval, η represents how many equally - time intervals (t 0， t f ) are divided into, generally η is taken as 4, σ represents the number of nodes within the interval; v(t 0 ) is the initial velocity, v(t f ) is the velocity of the camera at the end point of the interval, v(t 2σ-1 ) and v(t 2σ ) represent the velocities of each node within the interval.

[0024] The present invention also provides a computer - readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of a robot vision SLAM method for large viewing angles as described above are implemented.

[0025] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the computer program, the steps of a robot vision SLAM method for large viewing angles as described above are implemented.

[0026] The present invention has the following advantages:

[0027] 1. The adjacent grid - domain grid statistical algorithm is a new statistical method provided after improvement in the present invention for the accurate matching of feature points. The present invention applies the adjacent grid - domain grid statistical algorithm to achieve adaptive grid division based on image region division and information entropy, and further realizes the calculation of the adaptive feature score threshold for the grid under the corresponding division method. This improvement enables the grid division and the adaptive feature score threshold to adapt to the information entropy (reflecting the number of feature points) in the image region. Compared with the grid division technology in the prior art, it can make the feature points that were previously on the grid boundary be within the grid, and can more accurately judge which feature points need to be removed in the dense feature information area during false - match elimination, improving the accuracy of accurate matching.

[0028] ​2. The present invention calculates the grid feature scores through a new adjacent grid domain grid statistical algorithm, which selects different adjacent grid domain feature score algorithms based on the conversion angle. In the scenario of large viewing angles, this can reduce the defect of poor feature matching effect caused by viewing angle transformation. The cross adjacent grid domain and diagonal adjacent grid domain are respectively applied to different situations, which can better adapt to the large viewing angle scenario, reduce the number of false matches, increase the number of feature matches, and improve the accuracy of matching; at the same time, compared with the conventional grid algorithm, the calculation amount is significantly reduced, taking into account the timeliness and accuracy of accurate matching.

[0029] 3. Aiming at the problems that the system is prone to image blurring, resulting in fewer feature points extracted, tracking loss, and difficult feature matching in the large viewing angle scenario, the present invention proposes a new multi-scale multi-stage MSA-net algorithm. This network introduces the network RestNet101 at different scales for feature detail restoration, and through the attention mechanism and spatial attention mechanism, jointly optimizes to enhance the blurred image, improving the number of feature points extracted by the system under large viewing angles, as well as the overall positioning accuracy and robustness of the system.

[0030] 4. The present invention introduces an IMU sensor, calculates the attitude change and position deviation of the camera through integration, predicts the approximate position of the feature points in the next frame of the image. Using this method can roughly eliminate a part of the wrong matching pairs, greatly reducing the time consumption of the overall feature matching, making it more suitable for SLAM system applications and meeting the requirements of timeliness in the application of the present invention. Description of the Drawings

[0031] Figure 1 It is the basic flowchart of a robot vision SLAM method for large viewing angles according to the present invention.

[0032] Figure 2 It is the schematic flowchart of the SLAM system applying the present invention.

[0033] Figure 3 It is the flowchart of the multi-scale multi-stage deblurring network in the present invention.

[0034] Figure 4 It is the flowchart of the feature attention module in the present invention.

[0035] Figure 5 It is the schematic diagram of the principle of IMU feature prediction in the present invention.

[0036] Figure 6 It is the schematic diagram of the principle of the adjacent grid domain grid statistical algorithm in the present invention. Among them, the left figure is the schematic diagram of cross adjacent grid domain grid statistics, and the right figure is the schematic diagram of diagonal adjacent grid domain grid statistics.

[0037] Figure 7Comparison chart of the feature point extraction effects of the blurred image and the image obtained by restoring the blurred area of the present invention

[0038] Figure 8 Comparison chart of the feature matching effects of the blurred image and the image obtained by restoring the blurred area of the present invention

[0039] Figure 9 Comparison chart of the effect of eliminating false matches between the present invention and other existing technologies under different data sets. The existing technologies include SLAM3 and OV 2 _SLAM, Ours represents the SLAM system applying the present invention

[0040] Figure 10 Trajectory map obtained by the present invention under the MH01 sequence of the EUROC data set

[0041] Figure 11 Trajectory map obtained by the present invention under the MH03 sequence of the EUROC data set

[0042] Figure 12 Trajectory map obtained by the present invention under the MH04 sequence of the EUROC data set

[0043] Figure 13 Trajectory map obtained by the present invention under the V103 sequence of the EUROC data set

[0044] Figure 14 Error line chart comparing the present invention with other existing technologies under the KITTI data set. The existing technologies include SLAM3 and OV 2 _SLAM, Ours represents the SLAM system applying the present invention

[0045] Figure 15 True scene layout in another group of experiments, and comparison chart of the trajectory generated by the present invention and the trajectories of other existing algorithms. The upper image is the true scene layout, and the lower image is the comparison chart of the trajectories. The existing technologies include ORB_SLAM3 and OV 2 _SLAM, OURS is the SLAM system applying the present invention Detailed implementation manners

[0046] The following is a detailed description of the specific implementation manners of the present invention by referring to the accompanying drawings and describing the embodiments, so as to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention

[0047] Embodiment 1

[0048] As Figures 1 - 6As shown in the figure, the present invention provides a robot vision SLAM method for large viewing angles, including the following steps.

[0049] Step S1, collect image information, and use a multi-scale multi-stage neural network to extract feature information from the image and repair blurred regions.

[0050] In this step, first, the binocular camera carried by the robot is used to collect image information, and the obtained image information is input into the multi-scale multi-stage deblurring network. The multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, and s3. Each scale has 1, 2, and 3 stages respectively, and each stage is composed of a single lightweight encoder-decoder UNet module. These UNet modules share the same network structure but have different weights. Each UNet module is trained to be able to generate residual features that transform the blurred image into a clear image. These residual features are added to the blurred image to generate a deblurred image. A Cross-stage Feature Fusion (CSFF) module is introduced into the network, and cross-stage feature fusion is performed between each stage and between each scale through the CSFF module. The CSFF module makes the network less vulnerable to information loss and is beneficial for the feature information of the previous stage to enrich the features of the next stage. The multi-scale multi-stage deblurring network also adds a Feature Attention Module (FAM) connected to the UNet module at the beginning of each scale and an Attention Supervision Module (SAM) at the end.

[0051] There is a common problem in current deblurring networks: when processing blurred pictures, the network cannot better focus on the blurred regions and cannot make better use of key feature information, and the channel weights and pixel value weights of the blurred regions cannot be allocated. This results in the prior art being unable to fully distinguish the importance of channel and pixel information in the image, thus affecting the effect of image deblurring. To solve this problem, this algorithm introduces a feature attention module and an attention supervision module at the beginning and end of each scale respectively. These modules can assign different weights to different pixels and channel features on the feature map, effectively utilize the information on the feature map, and improve the deblurring effect of the network.

[0052] The feature attention module includes a ResNet101 network, a channel attention mechanism, and a spatial attention mechanism. The information obtained by the camera is used to extract more efficient feature information through the ResNet101 network. Then, these feature maps are further adjusted and enhanced through the channel attention mechanism and the spatial attention mechanism to improve their expressive ability and attention. The channel attention and spatial attention mechanisms can flexibly adjust the weights and responses of the feature maps, which helps to capture key feature information. Subsequently, the feature maps are processed through bilinear upsampling and 1×1 convolution operations to adjust the adjusted and enhanced feature maps to the same size as the input image and transform the channel dimension of the feature maps. Such a design ensures the correspondence between the output feature maps and the input image, which is beneficial to the alignment of the feature maps and the original image. To enable useful feature information to be passed to the next stage, this problem is solved by using an attention supervision module. The input feature F of the attention supervision module is obtained from the UNet module in , F in First, a residual image R is generated through a 1×1 convolutional layer S , and the obtained residual image and the degraded image I are combined to obtain a restored image X S . This can suppress the features with insufficient information in the current stage and only output useful features to the next stage. Then, the restored image X is processed through a 1×1 convolutional layer and a sigmoid activation function S , to generate an attention map M. The input feature F processed through the 1×1 convolutional layer is recalibrated through the attention map M in , to obtain the feature guided by the attention map, that is, the attention-enhanced feature F generated by the SAM out , and the feature F out is passed to the next stage.

[0053] Step S2: Based on the IMU sensor, perform position prediction on the feature points matched in the image, and determine whether the feature points in the next frame are within the predicted range.

[0054] In this step, the IMU sensor is used to measure the values of the acceleration and angular velocity of the camera, the displacement of the camera is obtained through integration, and the rotation matrix of the camera is obtained through quaternions, so as to obtain the camera coordinates of the next frame. Then, the two-dimensional coordinates of the feature points in the next frame are obtained through the projection formula. Since there are errors in the two-dimensional coordinates obtained by the foregoing method, a circle is drawn with a radius r based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments to obtain a suitable value, so that the two-dimensional coordinate position of the feature points matched in the next frame must be within the circle.

[0055] Calculate the rotation matrix of the camera, then multiply the rotation matrix by the camera coordinates of the previous frame, and add the displacement of the camera to obtain the camera coordinates of the next frame. The formula for calculating the camera coordinates of the next frame is as follows:

[0056]

[0057] Among them, [x 1 , y 1 , z 1 T represents the camera coordinates of the current frame, [x 0 , y 0 , z 0 T represents the camera coordinates of the previous frame, T represents the transpose, q w is the real part of the quaternion, q x , q y , q z are the three components of the imaginary part of the quaternion, t 0 , t f are the times of the start and end points of the interval respectively, t 2σ-1 and t 2σ represent the times of the nodes within the interval, η represents how many equally - time intervals (t 0， t f ) is divided into, generally η is taken as 4, σ represents the number of nodes within the interval; v(t 0 ) is the initial speed, v(t f ) is the speed of the camera at the end of the interval, v(t 2σ-1 ) and v(t 2σ ) represent the speeds of each node within the interval. Because the time taken for the previous and next frame images is short, the acceleration is generally defaulted to be constant, so the speed values for different time periods can be obtained respectively. Using quaternions to calculate the rotation matrix will not have problems such as gimbal lock and can obtain a more accurate rotation matrix.

[0058] To convert the three - dimensional coordinates into two - dimensional coordinates, this step projects the three - dimensional coordinates into two - dimensional coordinates to obtain the feature point positions of the next frame. The projection formula used for the calculation is as follows:

[0059]

[0060] Among them, L represents the difference in the left - and - right abscissas of the binocular camera, that is, the parallax, b represents the baseline of the binocular camera, and f represents the focal length of the camera. Due to the pinhole imaging principle of the camera, there are scaling and translation between the pixel coordinate system O 1 -M - N and the camera coordinate system O - X - Y - Z. Therefore, it is assumed that the pixel coordinates are scaled by α times on the M - axis coordinate system and by β times on the N - axis. α and β represent the magnification factors of the coordinates. c x , c y ​​It represents the displacements of the origins of two coordinate systems in the X and Y axis directions. u and e represent the coordinates in the pixel coordinate system, and x, y, and z represent the coordinates in the camera coordinate system. The three-dimensional coordinates P of the feature points in the previous frame are obtained from the two-dimensional coordinates of the feature points in the previous frame through the above formula. 0 (x 0 ,y 0 ,z 0 ). Then, the three-dimensional coordinates P of the camera in the next frame are obtained through the rotation matrix and translation vector. 1 (x 1 ,y 1 ,z 1 ). Then, the two-dimensional coordinates of the feature points in the current frame are obtained through the above formula.

[0061] Based on the positions of the feature points in the previous frame, this solution obtains the three-dimensional coordinates of the feature points in the camera coordinate system through the formula for calculating the camera coordinates in the next frame, obtains the three-dimensional coordinates of the feature points in the current frame through quaternions and camera displacements, and finally reversely calculates the coordinates of the feature points in the frame coordinate system through the projection formula. Since the above data are obtained through sensors, but there are errors in the obtained coordinates due to factors such as noise and cumulative integration errors, a circle with radius r is drawn with the obtained coordinate system as the center to roughly screen the feature points and eliminate false matches to reduce the matching time.

[0062] Step S3: Divide the image into several regions, calculate the information entropy for each region, and perform grid division based on the magnitude of the information entropy.

[0063] To further improve the matching accuracy, this method introduces a grid division method. To ensure that the algorithm is not affected by the grid size and reduce the matching time, the feature matching algorithm is improved using local entropy-based grid division. For an image, it is divided into several (four in this embodiment) regions, the information entropy is calculated for each region, and the image is divided by comparing the local information entropy: for places with smaller information entropy, larger grids are used for division, so that there will be more feature points in the grid and it is easier to eliminate false matches; for regions with larger information entropy, smaller grids are used for division, so as to achieve finer grid division. In this way, the feature points that were previously on the grid boundary are also within the grid, so that when eliminating false matches, it is possible to better judge which matching pairs in the grid need to be eliminated in regions with dense feature information.

[0064] Step S4: A new adjacent grid domain grid statistical algorithm is proposed to further screen the feature points selected by the IMU sensor, improve the positioning accuracy and map construction quality of the algorithm, and complete the accurate matching of the feature points.

[0065] To reduce the time for feature matching and enable the robot to perform better feature matching in the face of large-angle scenarios, this step uses the adjacent grid domain grid statistical algorithm to improve the performance of the algorithm and enhance the robustness of the system. Based on the principle that there must be correctly matched feature points in the neighborhood of correctly matched feature points, the number of correct matching pairs between the neighborhoods of feature points in adjacent frames is scored, and the resulting score is the flag score of the neighborhood. See Appendix Figure 6 for the division and marking of the neighborhood. In the case of a large viewing angle, rough matching of feature points is first performed, and the angle change of the camera is sensed by the IMU sensor, so that different statistical methods are used to calculate the feature scores of the grid. An angle threshold is set. If the camera undergoes a viewing angle transformation and the transformation angle is greater than the angle threshold, the diagonal adjacent grid domain grid algorithm is used to calculate the feature scores of the grid. Conversely, if the transformation angle is less than the angle threshold, the cross adjacent grid domain grid algorithm is used to calculate the feature scores of the grid. By this method, in the face of a large-angle scenario, the adjacent grid domain statistical algorithm can better capture the motion information along the diagonal direction, improve the robustness of the matching, which is especially beneficial for feature matching with a large change in the camera viewing angle. The adjacent grid domain grid algorithm only counts the feature scores of the current grid and the four grids adjacent to it or symmetric to it diagonally. The sum of the feature scores of these five grids is called the adjacent grid domain feature score.

[0066] Weights are assigned to each grid according to the number of correctly matched pairs of feature points in each grid. When the conversion angle obtained by the IMU sensor is less than the angle threshold, the cross adjacent grid domain method is used to calculate its feature score. In the face of a large-angle scenario, when the transformation angle of the camera viewing angle is greater than the angle threshold, the diagonal adjacent grid domain method is used to calculate its feature score. Since image rotation may occur at this time, the rotation motion kernel is used to rotate these five grids around the center. After 4 rotations, the diagonal adjacent grid domain returns to its initial state. Therefore, only the maximum adjacent grid domain feature score in 4 different cases needs to be counted to determine the feature score of the diagonal adjacent grid domain. The formula for the adjacent grid domain feature score in the face of different transformation angles is as follows:

[0067]

[0068] where S is the adjacent grid domain feature score, w represents the mean value of the number of rough matching features in each grid in the adjacent grid domain, i represents the grid number, n represents the number of rotations, α i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during motion, δ is the angle threshold; s i,m is the flag score of the m-th feature point in the i-th grid in the cross adjacent grid domain, It represents the flag score of the m-th feature point of the i-th grid in the diagonal neighboring grid domain after the n-th rotation. If the paired feature points of the rough match are all located in the corresponding grids, it is a correct match and 1 point is counted based on this feature point. The number of correct matches in each grid near the matching points is counted to obtain the flag score of the corresponding grid.

[0069] Regarding the calculation of the threshold τ of the feature score, since the grids have been partitioned based on information entropy by region, the threshold τ corresponding to the grid partitioning in different regions should also change accordingly. Therefore, a new adaptive formula for calculating the feature score threshold is adopted in this step as follows:

[0070]

[0071] Where: τ represents the adaptive feature score threshold, k is the weight coefficient of the number of feature points, and μ i represents the mean value of the number of rough match features of the i-th grid in the neighboring grid domain, i represents the grid number, π is the basic threshold, and p(x β ) is the probability that the random gray variable x takes the value of x β , which is used to represent the probability that the gray level is x β appears in the image. β is the number of the random gray variable, and n is the number of values of the random variable. n is within the range of (0, 255). The feature score of the neighborhood calculated from the feature points after rough matching is compared with the adaptive feature score threshold τ. If it is greater than the adaptive feature score threshold τ, it indicates that the rough matching result is correct; otherwise, it indicates that the rough matching result is incorrect. Thus, the screening of the feature points of the rough match is completed, and the accurate matching of the feature points is realized.

[0072] The following will illustrate the process of the above-mentioned robot vision SLAM method based on deep learning under large viewing angles by combining specific experiments.

[0073] Such as Figures 7 - 14As shown in the figure, to verify the deblurring experiment in a large viewing angle scenario, the TUM dataset is used to verify the deblurring algorithm proposed in this paper. To verify the deblurring effect of the algorithm in a blurred scenario, the algorithm has achieved better recovery effects in terms of brightness and details. To verify the effectiveness of the present invention in the application of the SLAM system, partial deblurred images in the publicly available indoor Euroc dataset are selected for a comparative experiment on feature matching. It can be seen that the number of enhanced matches has increased significantly and the distribution is more balanced, improving the robustness of feature matching. The accuracy and recall rate curves of this algorithm and other algorithms on the KITTI dataset show that this algorithm still maintains a high accuracy at a high recall rate, which is better than other algorithms. The running estimation and error broken line graph of this algorithm in the outdoor KITTI dataset shows that the trajectory formed by this method is similar to the real trajectory. In the broken line graph, it can be seen that after combining this algorithm, compared with other algorithms, the average absolute trajectory error and the average relative pose error have been reduced.

[0074] As Figure 15 shown, in another group of experiments, an indoor obstacle site is selected as the experimental environment, and a large viewing angle scenario is set for experimental verification. From the experimental results, it can be seen that due to the influence of the large viewing angle, the original ORB_SLAM3 algorithm has a large trajectory drift in the area where the camera angle changes greatly, while OV2-SLAM has a tracking loss phenomenon, resulting in inaccurate positioning accuracy and inability to close the loop. However, this algorithm shows good results in generating the trajectory, and through actual scene experiments, it is proved that this algorithm is more superior in the face of large viewing angle scenarios.

[0075] Embodiment 2.

[0076] Corresponding to Embodiment 1 of the present invention, Embodiment 2 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the following steps are implemented according to the method of Embodiment 1.

[0077] Step S1, collect image information, and use a multi-scale multi-stage neural network to extract feature information from the image and repair the blurred area.

[0078] Step S2, based on the IMU sensor, perform position prediction on the feature points matched by the image, and judge whether the next frame of feature points is within the predicted range.

[0079] Step S3, divide the image, divide it into several regions, calculate the information entropy for each region, and perform grid division according to the size of the information entropy.

[0080] Step S4, propose a new adjacent grid domain grid statistical algorithm to further screen the feature points screened by the IMU sensor.

[0081] The above storage medium includes various media that can store program codes, such as USB flash drives, external hard drives, read-only memories (ROMs), random access memories (RAMs), optical discs, etc.

[0082] For the specific limitations on the steps implemented after the program in the computer-readable storage medium, reference can be made to Embodiment 1, and details will not be described here again.

[0083] Embodiment 3.

[0084] Corresponding to Embodiment 1 of the present invention, Embodiment 3 of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the computer program, the following steps are implemented according to the method of Embodiment 1.

[0085] Step S1: Collect image information, and use a multi-scale and multi-stage neural network to extract feature information from the image and repair blurred areas.

[0086] Step S2: Based on the IMU sensor, perform position prediction on the feature points matched by the image, and determine whether the next-frame feature points are within the predicted range.

[0087] Step S3: Divide the image, divide it into several regions, calculate the information entropy for each region, and perform grid division based on the size of the information entropy.

[0088] Step S4: Propose a new adjacent grid domain grid statistical algorithm to further screen the feature points selected by the IMU sensor.

[0089] For the specific limitations on the steps implemented by the computer device, reference can be made to Embodiment 1, and details will not be described here again.

[0090] It should be noted that each block in the block diagrams and / or flowcharts in the accompanying drawings of the present invention, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and machine instructions obtained.

[0091] The present invention has been described exemplarily above in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited by the above methods. As long as various non-substantive improvements are made by adopting the inventive concept and technical solution of the present invention, or the inventive concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A robot vision SLAM method for a large viewing angle, characterized by: The following steps are involved: Step S1, collecting image information, and using a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area; Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range; Step S3, dividing the image into several regions, calculating the information entropy for each region, and dividing the image into grids according to the size of the information entropy; Step S4, proposing a new neighborhood grid statistical algorithm to further screen the feature points screened by the IMU sensor; The neighborhood domain grid statistical algorithm comprises: sensing the angle change of the camera through the IMU sensor, setting an angle threshold, if the camera changes the viewing angle and the change angle is greater than the angle threshold, then using the diagonal neighborhood domain grid algorithm to calculate the feature score of the grid, otherwise using the cross neighborhood domain grid algorithm to calculate the feature score of the grid if the change angle is less than the angle threshold; assigning weights to each grid according to the number of correct matching pairs of feature points in each grid; the neighborhood domain grid algorithm counts the feature scores of the current grid and four grids adjacent to it or diagonally symmetrical to it, and the sum of the feature scores of these five grids is called the feature score of the neighborhood domain grid; since the image may be rotated when the change angle is greater than the angle threshold, this step uses a rotation motion to check the five grids to rotate around the center, and the largest neighborhood domain feature score under four different circumstances can be determined as the feature score of the diagonal neighborhood domain; comparing the feature score of the neighborhood calculated by the feature point after rough matching with the corresponding feature score threshold, if it is greater than the feature score threshold, it means that the rough matching result is correct, otherwise it means that the rough matching result is wrong, thereby completing the screening of the feature points of rough matching and achieving accurate matching of the feature points.

2. A robot vision SLAM method for a large viewing angle according to claim 1, characterized in that: In step S3, information entropy is calculated for each region, and the image is divided by comparing local information entropies so that feature points previously located at the grid boundary are also located within the grid.

3. A robot vision SLAM method for a large viewing angle according to claim 2, characterized in that: Since the grid has been divided into regions based on information entropy, the score adopts an adaptive feature score threshold. The calculation formula of a new adaptive feature score threshold is adopted in step S4 as follows: Where: τ represents the adaptive feature score threshold, k is the weight coefficient of the number of feature points, μ i represents the mean value of the number of coarse matching features of the i-th grid in the neighborhood, i represents the number of the grid, π is the basic threshold, p(x β ) is a random grayscale variable x with a value of x β The probability of grayscale x β The probability of appearing in the image, β is the number of the random grayscale variable, n is the number of values ​​of the random variable, and n is in the range of (0, 255).

4. The robot vision SLAM method for a large viewing angle according to claim 1, characterized in that: The feature score formula based on the neighborhood domain at different camera conversion angles is: Among them, S is the feature score of the neighborhood domain, w is the mean of the number of coarse matching features of each grid in the neighborhood domain, i is the number of the grid, n is the number of rotations, and α is i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during the movement, and δ is the angle threshold; s i,m is the mark score of the mth feature point of the ith grid in the cross neighborhood domain, It represents the mark score of the mth feature point of the ith grid in the diagonal neighborhood domain rotated n times. The number of correct matches of each grid near the matching point is counted to obtain the mark score of the corresponding grid.

5. The robot vision SLAM method for a large viewing angle according to claim 1, characterized in that: In the step S1, the multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, and s3, each scale has 1, 2, and 3 stages respectively, and each stage is composed of a single lightweight encoder-decoder UNet module, and each UNet module shares the same network structure but has different weights; each UNet module is trained to generate residual features that can convert blurred images into clear images, and these residual features are added to the blurred image to generate a deblurred image; cross-stage feature fusion is performed between each stage and between each scale through the CSFF module, and a feature attention module and an attention supervision module are introduced at the beginning and end of each scale respectively, and different weights are assigned to different pixel and channel features on the feature map, effectively utilizing the information on the feature map and improving the deblurring effect of the network.

6. The robot vision SLAM method for a large viewing angle according to claim 5, characterized in that: The feature attention module includes the ResNet101 network, the channel attention mechanism and the spatial attention mechanism. The information obtained by the camera is extracted through the ResNet101 network, and then the feature maps are further adjusted and enhanced through the channel attention mechanism and the spatial attention mechanism to improve their expressiveness and attention. Subsequently, the feature maps are processed through bilinear upsampling and 1×1 convolution operations, and the adjusted and enhanced feature maps are adjusted to feature maps of the same size as the input image, and the channel dimension of the feature maps is transformed; the input feature F of the attention supervision module in First, the residual image R is generated through a 1×1 convolution layer S , the obtained residual image is combined with the degraded image I to obtain the restored image X S , and then restore the image X through a 1×1 convolution layer and a sigmoid activation function S , generate the attention map M; recalibrate the input feature F after the 1×1 convolution layer through the attention map M in , we get the feature guided by the attention map, that is, the attention-enhanced feature F generated by SAM out , and the feature F out Pass to the next stage.

7. The robot vision SLAM method for a large viewing angle according to claim 1, characterized in that: In the step S2, the acceleration and angular velocity of the camera are measured by the IMU sensor, the displacement of the camera is obtained by integration, and the rotation matrix of the camera is obtained by quaternion, so as to obtain the camera coordinates of the next frame; first, the two-dimensional coordinates of the feature points of the previous frame are calculated by the projection formula from two-dimensional to three-dimensional coordinates to obtain the three-dimensional coordinates P0 (x0, y0, z0) of the feature points of the previous frame, and then the three-dimensional coordinates P1 (x1, y1, z1) of the camera of the next frame are obtained by the rotation matrix and the translation vector, and finally converted into two-dimensional coordinates by the projection formula, and a circle is drawn with a radius r based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments, so that the two-dimensional coordinate position of the feature points matching the next frame must be within the circle; The formula for calculating the camera coordinates of the next frame is as follows: Where [x1, y1, z1] T Represents the camera coordinates of the current frame, [x0, y0, z0] T Represented as the camera coordinates of the previous frame, T represents transposition, q w is the real part of the quaternion, q x ,q y ,q z are the three components of the imaginary part of the quaternion, t0, t f are the starting and ending times of the interval, respectively, and t 2σ-1 and t 2σ represents the time of the node in the interval, and η represents the time of (t0, t f ) is divided into intervals of equal time, η is taken as 4, σ is the number of nodes in the interval; v(t0) is the initial speed, v(t f ) is the speed of the camera at the end of the interval, v(t 2σ-1 ) and v(t 2σ ) represents the speed of each node in the interval.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a robot vision SLAM method for a large viewing angle as described in any one of claims 1 to 7 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of a robot vision SLAM method for a large viewing angle are implemented as described in any one of claims 1 to 7.

Images