Robot vision algorithm for large visual angle, storage medium and equipment

Abstract

The invention belongs to the technical field of SLAM (Simultaneous Localization and Mapping), and discloses a robot vision algorithm for a large visual angle, a storage medium and equipment, and the algorithm comprises the steps: S1, collecting image information, extracting feature information from an image through employing a multi-scale and multi-stage neural network, and repairing a fuzzy region; s2, performing position prediction on the feature points matched with the image based on an IMU sensor, and judging whether the feature points of the next frame are in a prediction range or not; s3, dividing the image into a plurality of regions, calculating the information entropy of each region, and performing grid division according to the size of the information entropy; and S4, providing a new neighbor grid domain grid statistical algorithm to further screen the feature points screened by the IMU sensor. According to the method, the blurred image frame is repaired, so that a large number of image feature points can be accurately extracted in large-view-angle motion, and timeliness and accuracy are improved in accurate matching through improvement.

Description

Technical Field

[0001] The present invention belongs to the field of simultaneous location and mapping (SLAM) technology, and in particular relates to a robot vision algorithm, storage medium and device for a large viewing angle. Background Art

[0002] Simultaneous Location And Mapping (SLAM) refers to the process of using sensors carried by robots to simultaneously build a map of the current environment and locate their own position and posture in an unknown environment. However, in actual scenarios, due to limitations in camera frame rate and environmental factors, blurred image frames are prone to occur in wide-angle motion. Visual SLAM systems often face problems such as blur, occlusion, interference from dynamic objects or light sources. The lack of feature information in images makes it difficult for traditional feature extraction algorithms to extract a large number of image feature points. These problems seriously affect the accuracy of feature extraction and matching and the robustness of the algorithm. Therefore, a visual SLAM algorithm suitable for wide-angle motion is crucial to improving the robustness of the system.

[0003] Some existing technologies use neighborhood grid statistical algorithms to further screen correctly matched feature point pairs based on rough feature point matching. However, in the existing technologies, accurate matching after gridding requires calculation and evaluation of the matching effect of neighborhood feature points in each area, which results in large amount of calculation and long feature matching time, affecting the timeliness of synchronous positioning and map construction. On the other hand, if the number of neighborhood grids involved in the calculation is insufficient, the accuracy of accurate matching is affected. Moreover, during motion, the number of feature points contained in different areas of the captured image is not the same. If this is not taken into account, when dividing the grid for calculation, the number of feature points contained in the neighborhood grid may be insufficient due to the over-dense grid division, and accurate accurate matching effect cannot be obtained. Alternatively, the number of feature points in each grid may be too large due to the over-sparse grid division, and the calculation results between different feature points are close, which cannot reflect the actual matching situation well and also reduces the accuracy of accurate matching. Summary of the invention

[0004] The purpose of the present invention is to provide a robot vision algorithm for a large viewing angle, which is used to solve the technical problems in the prior art, that is, how to accurately extract a large number of feature points from the blurred image, and how to screen the rough matching results to achieve a precise matching effect that takes into account both timeliness and accuracy, because blurred image frames are prone to appear in large-viewing angle motion.

[0005] The robot vision algorithm for a large viewing angle includes the following steps.

[0006] A robot vision algorithm for a large viewing angle, characterized by comprising the following steps:

[0007] Step S1, collecting image information, and using a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area;

[0008] Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range;

[0009] Step S3, dividing the image into several regions, calculating the information entropy for each region, and dividing the image into grids according to the size of the information entropy;

[0010] Step S4, a new neighborhood grid statistics algorithm is proposed to further screen the feature points screened by the IMU sensor.

[0011] Preferably, in step S4, the angle change of the camera is sensed by the IMU sensor, and an angle threshold is set. If the camera changes its viewing angle and the change angle is greater than the angle threshold, the feature score of the grid is calculated by the diagonal neighborhood domain grid algorithm. Conversely, if the change angle is less than the angle threshold, the feature score of the grid is calculated by the cross neighborhood domain grid algorithm. A weight is assigned to each grid according to the number of correct matching pairs of feature points in each grid. The neighborhood domain grid algorithm counts the feature scores of the current grid and four grids adjacent to or diagonally symmetrical to it, and the sum of the feature scores of these five grids is called the feature score of the neighborhood domain grid. Since the image may rotate when the change angle is greater than the angle threshold, this step uses a rotation motion to check the five grids for rotation around the center, and the maximum neighborhood domain feature score under four different circumstances can be determined as the feature score of the diagonal neighborhood domain. The feature score of the neighborhood calculated by the feature point after coarse matching is compared with the corresponding feature score threshold. If it is greater than the feature score threshold, it means that the coarse matching result is correct, otherwise it means that the coarse matching result is wrong, thereby completing the screening of the feature points of the coarse matching and achieving accurate matching of the feature points.

[0012] Preferably, in step S3, the information entropy is calculated for each area, and the image is divided by comparing the local information entropies: areas with smaller information entropy are divided by larger grids; areas with larger information entropy are divided by smaller grids, so that the feature points previously on the grid boundary are also within the grid.

[0013] Preferably, since the grid has been divided into regions based on information entropy, the score adopts an adaptive threshold, and the step S4 adopts a new adaptive threshold calculation formula as follows:

[0014]

[0015] Where: τ represents the adaptive threshold of the feature score, k is the weight coefficient of the number of feature points, μ i represents the mean value of the number of coarse matching features of the i-th grid in the neighborhood, i represents the number of the grid, π is the basic threshold, p(x β ) is a random grayscale variable x with a value of x β The probability of grayscale x β The probability of appearing in the image, β is the number of the random grayscale variable, n is the number of values ​​of the random variable, and n is in the range of (0, 255).

[0016] Preferably, the characteristic score formula of the neighborhood domain under different camera transformation angles is:

[0017]

[0018] Among them, S is the feature score of the neighborhood domain, w is the mean of the number of coarse matching features of each grid in the neighborhood domain, i is the number of the grid, n is the number of rotations, and α is i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during the movement, and δ is the angle threshold; s i,m is the mark score of the mth feature point of the ith grid in the cross neighborhood domain, It represents the mark score of the mth feature point of the ith grid in the diagonal neighborhood domain rotated n times. The number of correct matches of each grid near the matching point is counted to obtain the mark score of the corresponding grid.

[0019] Preferably, in step S1, the multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, and s3, each scale has 1, 2, and 3 stages respectively, and each stage is composed of a single lightweight encoder-decoder UNet module, and each UNet module shares the same network structure but has different weights; each UNet module is trained to generate residual features that can convert blurred images into clear images, and these residual features are added to the blurred image to generate a deblurred image; cross-stage feature fusion is performed between each stage and between each scale through the CSFF module, and a feature attention module and an attention supervision module are introduced at the beginning and end of each scale respectively, and different weights are assigned to different pixels and channel features on the feature map, effectively utilizing the information on the feature map and improving the deblurring effect of the network.

[0020] Preferably, the feature attention module includes a ResNet101 network, a channel attention mechanism and a spatial attention mechanism. The information obtained by the camera is subjected to feature information extraction through the ResNet101 network, and then the feature maps are further adjusted and enhanced through the channel attention mechanism and the spatial attention mechanism to improve their expressiveness and attention. Subsequently, the feature maps are processed through bilinear upsampling and 1×1 convolution operations, and the adjusted and enhanced feature maps are adjusted to feature maps of the same size as the input image, and the channel dimension of the feature maps is transformed; the input feature F of the attention supervision module in First, the residual image R is generated through a 1×1 convolution layer S , the obtained residual image is combined with the degraded image I to obtain the restored image X S , and then restore the image X through a 1×1 convolution layer and a simgmoid activation function S , generate the attention map M; recalibrate the input feature F after the 1×1 convolution layer through the attention map M in , we get the feature guided by the attention map, that is, the attention-enhanced feature F generated by SAM out , and the feature F out Pass to the next stage.

[0021] Preferably, in step S2, the acceleration and angular velocity of the camera are measured by an IMU sensor, the displacement of the camera is obtained by integration, and the rotation matrix of the camera is obtained by quaternion, so as to obtain the camera coordinates of the next frame. First, the two-dimensional coordinates of the feature points of the previous frame are calculated by the projection formula from two-dimensional to three-dimensional coordinates to obtain the three-dimensional coordinates P0 (x0, y0, z0) of the feature points of the previous frame, and then the three-dimensional coordinates P1 (x1, y1, z1) of the camera of the next frame are obtained by the rotation matrix and the translation vector, and finally converted into two-dimensional coordinates by the projection formula, and a circle is drawn with a radius r based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments to obtain a suitable value, so that the two-dimensional coordinate position of the feature points matching the next frame must be within the circle; the formula for calculating the camera coordinates of the next frame is as follows:

[0022]

[0023] Where [x1, y1, z1] T Represents the camera coordinates of the current frame, [x0, y0, z0] T Represented as the camera coordinates of the previous frame, T represents transposition, q w is the real part of the quaternion, q x ,q y ,q z are the three components of the imaginary part of the quaternion, t0, t f are the starting and ending times of the interval, respectively, and t2σ-1 and t 2σ represents the time of the node in the interval, η represents the time of (t 0， t f ) is divided into intervals of equal time. Generally, η is taken as 4, and σ is the number of nodes in the interval. v(t0) is the initial velocity, v(t f ) is the speed of the camera at the end of the interval, v(t 2σ-1 ) and v(t 2σ ) represents the speed of each node in the interval.

[0024] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the robot vision algorithm for a large viewing angle as described above are implemented.

[0025] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the steps of a robot vision algorithm for a large viewing angle as described above are implemented.

[0026] The present invention has the following advantages:

[0027] 1. The neighborhood domain grid statistical algorithm is a new statistical method provided by the present invention after improvement, which is used for accurate matching of feature points. The present invention applies the neighborhood domain grid statistical algorithm, realizes adaptive grid division based on image area division and information entropy, and further realizes adaptive threshold calculation of grid feature score under the corresponding division method. This improvement allows the grid division and adaptive threshold to adapt to the information entropy (reflecting the number of feature points) in the image area. Compared with the grid division technology in the prior art, it can make the feature points that were previously on the grid boundary line specifically within the grid, and can more accurately judge which feature points need to be eliminated in the feature information-intensive area when eliminating false matches, thereby improving the accuracy of accurate matching.

[0028] 2. The present invention uses a new neighborhood domain grid statistics algorithm to select different neighborhood domain feature score algorithms based on the conversion angle to calculate the grid feature score. This can reduce the defects of poor feature matching due to perspective change in scenes with a wide viewing angle. The cross neighborhood domain and the diagonal neighborhood domain are applied to different situations respectively and can better adapt to scenes with a wide viewing angle, reduce the number of mismatches, increase the number of feature matches, and improve the matching accuracy; at the same time, compared with conventional grid algorithms, the amount of calculation is significantly reduced, taking into account the timeliness and accuracy of precise matching.

[0029] 3. Aiming at the problem that the system is prone to image blur in wide viewing angle scenes, resulting in less feature point extraction, tracking loss, and difficulty in feature matching, the present invention proposes a new multi-scale and multi-stage MSA-net algorithm. The network introduces the network RestNet101 at different scales to restore feature details, and through the attention mechanism and the spatial attention mechanism, jointly optimizes the blurred image to enhance the number of feature points extracted by the system at a wide viewing angle and the overall positioning accuracy and robustness of the system.

[0030] 4. The present invention introduces an IMU sensor, which calculates the camera's posture change and position deviation through integration, and predicts the approximate position of the feature point in the next frame of the image. This method can roughly eliminate some erroneous matching pairs, greatly reducing the time consumption of the overall feature matching, making it more suitable for SLAM system applications and meeting the timeliness requirements of the present invention in applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a basic flow chart of a robot vision algorithm for a large viewing angle according to the present invention.

[0032] Figure 2 The figure is a schematic diagram of the process of applying the SLAM system of the present invention.

[0033] Figure 3 Flow chart of the multi-scale and multi-stage deblurring network in the present invention.

[0034] Figure 4 Flowchart of the feature attention module in the present invention.

[0035] Figure 5 It is a schematic diagram of the principle of IMU feature prediction in the present invention.

[0036] Figure 6 Schematic diagram of the principle of the neighborhood domain grid statistics algorithm in the present invention, wherein the left figure is a schematic diagram of the cross neighborhood domain grid statistics, and the right figure is a schematic diagram of the diagonal neighborhood domain grid statistics.

[0037] Figure 7 A comparison of the feature point extraction effects of a blurred image and an image with the blurred area repaired by the present invention.

[0038] Figure 8 It is a comparison diagram of the feature matching effects between a blurred image and an image with the blurred area repaired by the present invention.

[0039] Fig. 9 This is a comparison chart of the effect of eliminating mismatches between the present invention and other existing technologies under different data sets, where the existing technologies include SLAM3 and OV 2 _SLAM, Ours represents the SLAM system to which the present invention is applied.

[0040] Fig.10 This is the trajectory diagram obtained by the present invention under the EUROC dataset MH01 sequence.

[0041] Fig.11 This is the trajectory diagram obtained by the present invention under the EUROC dataset MH03 sequence.

[0042] Fig.12 This is the trajectory diagram obtained by the present invention under the EUROC dataset MH04 sequence.

[0043] Fig.13 This is the trajectory diagram obtained by the present invention under the EUROC dataset V103 sequence.

[0044] Fig.14 This is an error line graph comparing the present invention with other existing technologies under the KITTI dataset, where the existing technologies include SLAM3 and OV 2 _SLAM, Ours represents the SLAM system to which the present invention is applied.

[0045] Fig.15 The above image is the layout of the real scene in another set of experiments, and the comparison diagram of the trajectory generated by the present invention and the trajectory of other existing algorithms. The upper image is the layout of the real scene, and the lower image is the comparison diagram of the trajectory. The existing technologies include ORB_SLAM3 and OV 2 _SLAM, OURS is the SLAM system to which the present invention is applied. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention will be further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0047] Embodiment 1.

[0048] like Figure 1-Figure 6 As shown, the present invention provides a robot vision algorithm for a large viewing angle, comprising the following steps.

[0049] Step S1, collect image information, and use a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area.

[0050] In this step, first, the image information is collected by the binocular camera of the robot, and the obtained image information is input into the multi-scale multi-stage deblurring network. The multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, and s3. Each scale has 1, 2, and 3 stages, respectively, and each stage is composed of a single lightweight encoder-decoder UNet module. These UNet modules share the same network structure but have different weights. Each UNet module is trained to generate residual features that can convert blurred images into clear images. These residual features are added to the blurred image to generate a deblurred image. A cross-stage feature fusion (CSFF) module is introduced into the network, and cross-stage feature fusion is performed through the CSFF module between each stage and each scale. The CSFF module makes the network less susceptible to information loss, which is beneficial to enrich the features of the next stage with the information of the previous stage. The multi-scale and multi-stage deblurring network also adds a feature attention module (FAM) at the beginning of each scale connected to the UNet module, and a supervision attention module (SAM) at the end.

[0051] There is a common problem in the current deblurring network: when processing blurred images, the network cannot better focus on the blurred area and cannot better utilize the key feature information, and the channel weights and pixel value weights of the blurred area cannot be assigned. This results in the inability of existing technologies to fully distinguish the importance of channel and pixel information in the image, thus affecting the image deblurring effect. To solve this problem, this algorithm introduces a feature attention module and an attention supervision module at the beginning and end of each scale, respectively. These modules can assign different weights to different pixel and channel features on the feature map, effectively utilize the information on the feature map, and improve the network's deblurring effect.

[0052] The feature attention module includes the ResNet101 network, the channel attention mechanism and the spatial attention mechanism. The information obtained by the camera is used to extract feature information more efficiently through the ResNet101 network, and then the channel attention mechanism and the spatial attention mechanism are used to further adjust and enhance these feature maps to improve their expressiveness and attention. The channel attention and spatial attention mechanisms can flexibly adjust the weights and responses of the feature maps, which helps to capture key feature information. Subsequently, the feature maps are processed by bilinear upsampling and 1×1 convolution operations, and the adjusted and enhanced feature maps are adjusted to feature maps of the same size as the input image, and the channel dimension of the feature maps is transformed. This design ensures the correspondence between the output feature map and the input image, which is conducive to the alignment of the feature map with the original image. In order to allow useful feature information to be passed to the next stage, this problem is solved by using the attention supervision module. The input feature F of the attention supervision module is obtained by the UNet module in , F in First, the residual image R is generated through a 1×1 convolution layer S , the obtained residual image is combined with the degraded image I to obtain the restored image X S This can suppress features that are not rich in information in the current stage and only output useful features to the next stage. Then the image X is restored through a 1×1 convolution layer and a simgmoid activation function. S , generate the attention map M. The input feature F after the 1×1 convolution layer is recalibrated through the attention map M in , we get the feature guided by the attention map, that is, the attention-enhanced feature F generated by SAM out , and the feature F out Pass to the next stage.

[0053] Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range.

[0054] In this step, the acceleration and angular velocity of the camera are measured by the IMU sensor, the displacement of the camera is obtained by integration, and the rotation matrix of the camera is obtained by quaternion, so as to obtain the camera coordinates of the next frame, and then the two-dimensional coordinates of the feature points of the next frame are obtained by the projection formula. Since the two-dimensional coordinates obtained by the above method have errors, a circle with a radius r is drawn based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments to obtain a suitable value, so that the two-dimensional coordinate position of the feature points matching the next frame must be within the circle.

[0055] Calculate the rotation matrix of the camera, then multiply the rotation matrix by the camera coordinates of the previous frame, and add the camera displacement to obtain the camera coordinates of the next frame. The formula for calculating the camera coordinates of the next frame is as follows:

[0056]

[0057] Where [x1, y1, z1] T Represents the camera coordinates of the current frame, [x0, y0, z0] T Represented as the camera coordinates of the previous frame, T represents transposition, q w is the real part of the quaternion, q x ,q y ,q z are the three components of the imaginary part of the quaternion, t0, t f are the starting and ending times of the interval, respectively, and t 2σ-1 and t 2σ represents the time of the node in the interval, η represents the time of (t 0， t f ) is divided into intervals of equal time. Generally, η is taken as 4, and σ is the number of nodes in the interval. v(t0) is the initial velocity, v(t f ) is the speed of the camera at the end of the interval, v(t 2σ-1 ) and v(t 2σ ) represents the speed of each node in the interval. Because the time taken by the previous frame and the next frame is short, the acceleration is generally assumed to be unchanged, so the speed values ​​of different time periods can be calculated respectively. Using quaternions to calculate the rotation matrix will not cause problems such as universal lock, and a more accurate rotation matrix can be obtained.

[0058] In order to convert the three-dimensional coordinates into two-dimensional coordinates, this step projects the three-dimensional coordinates into two-dimensional coordinates to obtain the feature point position of the next frame. The projection formula used for calculation is as follows:

[0059]

[0060] Among them, L represents the difference between the left and right horizontal coordinates of the binocular camera, that is, the parallax, b represents the baseline of the binocular camera, and f represents the focal length of the camera. Due to the scaling and translation between the pixel coordinate system O1-MN and the camera coordinate system OXYZ based on the pinhole imaging principle of the camera, it is assumed that the pixel coordinates are scaled by α times on the M axis coordinate system and β times on the N axis, where α and β represent the magnification of the coordinates. x ,c y Represents the displacement of the origin of the two coordinate systems in the X and Y axis directions, u and e represent the coordinates of the pixel coordinate system, and x, y and z represent the coordinates of the camera coordinate system. The two-dimensional coordinates of the feature point of the previous frame are calculated by the above formula to obtain the three-dimensional coordinates P0 (x0, y0, z0) of the feature point of the previous frame, and then the three-dimensional coordinates P1 (x1, y1, z1) of the camera of the next frame are obtained by the rotation matrix and translation vector, and then the two-dimensional coordinates of the feature point of the current frame are calculated by the above formula.

[0061] This solution obtains the three-dimensional coordinates of the feature points in the camera coordinate system based on the feature point position of the previous frame by calculating the formula for the camera coordinates of the next frame, and obtains the three-dimensional coordinates of the feature points in the current frame through quaternion and camera displacement. Finally, the coordinates of the feature points in the frame coordinate system are reversely calculated through the projection formula. Because the above data are all obtained through sensors, but the coordinates obtained are erroneous due to factors such as noise and accumulated integral errors, a circle with a radius r is drawn with the coordinate system to be calculated as the center, and the feature points are roughly screened to eliminate false matches and reduce matching time.

[0062] Step S3, divide the image into several regions, calculate the information entropy for each region, and divide the image into grids according to the size of the information entropy.

[0063] In order to further improve the matching accuracy, this method introduces a grid division method. In order to ensure that the algorithm is not affected by the grid size and reduce the matching time, the feature matching algorithm is improved by grid division based on local entropy. For a picture, it is divided into several (four in this embodiment) areas, and the information entropy is calculated for each area. The image is divided by comparing the local information entropy: for areas with smaller information entropy, a larger grid is used for division, so that the number of feature points in the grid will be more and it is easier to eliminate mismatches; for areas with larger information entropy, a smaller grid is used for division, thereby achieving finer grid division, so that the feature points that were previously on the grid boundary are also in the grid, so that when eliminating mismatches, it can better judge which matching pairs in the grid need to be eliminated in areas with dense feature information.

[0064] Step S4, a new neighborhood grid statistics algorithm is proposed to further screen the feature points screened by the IMU sensor, improve the positioning accuracy of the algorithm and the quality of map construction, and complete the precise matching of feature points.

[0065] In order to reduce the time of feature matching and enable the robot to better perform feature matching in scenes with a wide viewing angle, this step uses a neighborhood grid statistical algorithm to improve the performance of the algorithm and the robustness of the system. Based on the principle that there must be a correctly matched feature point in the neighborhood of a correctly matched feature point, the number of correctly matched pairs between the neighborhoods of feature points in adjacent frames is scored, and the resulting score is the mark score of the neighborhood. See the attached diagram for the division and marking of the neighborhood. Figure 6. In the case of a large viewing angle, a rough match of feature points is first performed, and the IMU sensor is used to sense the angle change of the camera, so as to use different statistical methods to calculate the feature score of the grid. An angle threshold is set. If the camera changes its viewing angle and the change angle is greater than the angle threshold, the feature score of the grid is calculated by the diagonal neighborhood domain grid algorithm. Otherwise, if the change angle is less than the angle threshold, the feature score of the grid is calculated by the cross neighborhood domain grid algorithm. With this method, when facing a large viewing angle scene, the neighborhood domain network statistical algorithm can better capture the motion information along the diagonal direction and improve the robustness of the matching, which is especially beneficial for feature matching with a large change in camera viewing angle. The neighborhood domain grid algorithm only counts the feature scores of the current grid and the four grids adjacent to it or diagonally symmetrical to it. The sum of the feature scores of these five grids is called the neighborhood domain feature score.

[0066] Assign weights to each grid based on the number of correctly matched pairs of feature points in each grid. When the conversion angle obtained by the IMU sensor is less than the angle threshold, use the cross-neighborhood domain method to calculate its feature score. In scenes facing a wide viewing angle, when the transformation angle of the camera's viewing angle is greater than the angle threshold, use the diagonal neighborhood domain method to calculate its feature score. Since the image may rotate at this time, the five grids are rotated around the center using a rotational motion core. After four rotations, the state of the diagonal neighborhood domain is the same as the initial state. Therefore, it is only necessary to count the largest neighborhood domain feature scores under four different conditions to determine the feature score of the diagonal neighborhood domain. The feature score formula for the neighborhood domain under different transformation angles is:

[0067]

[0068] Among them, S is the feature score of the neighborhood domain, w is the mean of the number of coarse matching features of each grid in the neighborhood domain, i is the number of the grid, n is the number of rotations, and α is i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during the movement, and δ is the angle threshold; s i,m is the mark score of the mth feature point of the ith grid in the cross neighborhood domain, It represents the mark score of the mth feature point of the ith grid in the diagonal neighborhood domain rotated n times. If the paired feature points of the rough match are all located in the corresponding grids, it is a correct match and 1 point is scored based on the feature point. The number of correct matches of each grid near the matching point is counted to obtain the mark score of the corresponding grid.

[0069] As for the calculation of the threshold τ of the feature score, since the grid has been divided by region based on information entropy, the threshold τ corresponding to the grid division in different regions should also change accordingly, so this step uses a new adaptive threshold calculation formula as follows:

[0070]

[0071] Where: τ represents the adaptive threshold of the feature score, k is the weight coefficient of the number of feature points, μ i represents the mean value of the number of coarse matching features of the i-th grid in the neighborhood, i represents the number of the grid, π is the basic threshold, p(x β ) is a random grayscale variable x with a value of x β The probability of grayscale x β The probability of appearing in the image, β is the number of random grayscale variables, n is the number of values ​​of the random variable, and n is within the range of (0, 255). The feature score of the neighborhood calculated by the feature point after rough matching is compared with the adaptive threshold τ. If it is greater than the adaptive threshold τ, it means that the rough matching result is correct, otherwise it means that the rough matching result is wrong, thereby completing the screening of feature points for rough matching and achieving accurate matching of feature points.

[0072] The following is an explanation of the process of the above-mentioned robot vision algorithm based on deep learning under a wide viewing angle by combining specific experiments.

[0073] like Figure 7-Figure 14 As shown in the figure, the experiment of deblurring in a wide viewing angle scene is verified, and the TUM dataset is used to verify the deblurring algorithm proposed in this paper. In order to verify the deblurring effect of the algorithm in a blurry scene, the algorithm has achieved better recovery effects in brightness and details. In order to verify the effectiveness of the present invention in the application of SLAM system, some deblurred images in the public indoor Euroc dataset are selected for feature matching comparison experiments. It can be seen that the number of enhanced matches has increased significantly and the distribution is more balanced, which improves the robustness of feature matching. The accuracy and recall rate curves of this algorithm and other algorithms in the KITTI dataset respectively show that this algorithm still maintains a high accuracy rate at a high recall rate, which is better than other algorithms. The running estimation and error line graph of this algorithm in the outdoor KITTI dataset show that the trajectory formed by this method is close to the true trajectory. In the line graph, it can be seen that after combining this algorithm, the average absolute trajectory error and the average relative pose error are reduced compared with other algorithms.

[0074] like Fig.15As shown in the figure, in another set of experiments, an indoor obstacle field was selected as the experimental environment, and a wide-angle scene was set for experimental verification. From the experimental results, it can be seen that due to the influence of the wide angle of view, the original ORB_SLAM3 algorithm produced a large trajectory drift in the area where the camera angle changes greatly, while OV2-SLAM experienced tracking loss, resulting in inaccurate positioning accuracy and inability to close the loop. However, this algorithm showed a better effect in generating trajectories. The actual scene experiment proved that this algorithm is more superior when facing the wide-angle scene.

[0075] Embodiment 2.

[0076] Corresponding to the first embodiment of the present invention, the second embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following steps are implemented according to the method of the first embodiment.

[0077] Step S1, collect image information, and use a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area.

[0078] Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range.

[0079] Step S3, divide the image into several regions, calculate the information entropy for each region, and divide the image into grids according to the size of the information entropy.

[0080] Step S4, a new neighborhood grid statistics algorithm is proposed to further screen the feature points screened by the IMU sensor.

[0081] The above storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), optical disk and other media that can store program codes.

[0082] The specific limitations on the steps implemented after the program in the computer-readable storage medium is executed can be found in Example 1, and will not be described in detail here.

[0083] Embodiment three.

[0084] Corresponding to the first embodiment of the present invention, the third embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the following steps are implemented according to the method of the first embodiment.

[0085] Step S1, collect image information, and use a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area.

[0086] Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range.

[0087] Step S3, divide the image into several regions, calculate the information entropy for each region, and divide the image into grids according to the size of the information entropy.

[0088] Step S4, a new neighborhood grid statistics algorithm is proposed to further screen the feature points screened by the IMU sensor.

[0089] The specific limitations on the above steps of implementing the computer device can be found in Example 1, and will not be described in detail here.

[0090] It should be noted that each box in the block diagram and / or flow chart in the accompanying drawings of the specification of the present invention, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and machine instructions.

[0091] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A robot vision algorithm for large viewing angles, characterized by: The following steps are involved: Step S1, collecting image information, and using a multi-scale and multi-stage neural network to extract feature information from the image and repair the blurred area; Step S2: Predict the position of the feature points matched by the image based on the IMU sensor, and determine whether the feature points of the next frame are within the predicted range; Step S3, dividing the image into several regions, calculating the information entropy for each region, and dividing the image into grids according to the size of the information entropy; Step S4, a new neighborhood grid statistics algorithm is proposed to further screen the feature points screened by the IMU sensor.

2. The robot vision algorithm for a large viewing angle according to claim 1, characterized in that: In step S4, the neighborhood domain grid statistical algorithm includes: sensing the angle change of the camera through the IMU sensor, setting an angle threshold, if the camera changes its viewing angle and the change angle is greater than the angle threshold, then using the diagonal neighborhood domain grid algorithm to calculate the feature score of the grid, otherwise the cross neighborhood domain grid algorithm is used to calculate the feature score of the grid; weights are assigned to each grid according to the number of correct matching pairs of feature points in each grid; the neighborhood domain grid algorithm counts the feature scores of the current grid and four grids adjacent to it or diagonally symmetrical to it, and the sum of the feature scores of these five grids is called the feature score of the neighborhood domain grid; since the image may rotate when the change angle is greater than the angle threshold, this step uses a rotation motion to check the five grids to rotate around the center, and the largest neighborhood domain feature score under four different circumstances can be determined as the feature score of the diagonal neighborhood domain; the feature score of the neighborhood calculated by the feature point after rough matching is compared with the corresponding feature score threshold, and if it is greater than the feature score threshold, it means that the rough matching result is correct, otherwise it means that the rough matching result is wrong, thereby completing the screening of the feature points of rough matching and achieving accurate matching of the feature points.

3. The robot vision algorithm for a large viewing angle according to claim 2, characterized in that: In step S3, information entropy is calculated for each region, and the image is divided by comparing local information entropies: areas with smaller information entropy are divided by larger grids; areas with larger information entropy are divided by smaller grids, so that feature points that were previously on the grid boundary are also within the grid.

4. The robot vision algorithm for a large viewing angle according to claim 3, characterized in that: Since the grid has been divided into regions based on information entropy, the score adopts an adaptive threshold. The step S4 adopts a new adaptive threshold calculation formula as follows: Where: τ represents the adaptive threshold of the feature score, k is the weight coefficient of the number of feature points, μ i represents the mean value of the number of coarse matching features of the i-th grid in the neighborhood, i represents the number of the grid, π is the basic threshold, p(x β ) is a random grayscale variable x with a value of x β The probability of grayscale x β The probability of appearing in the image, β is the number of the random grayscale variable, n is the number of values ​​of the random variable, and n is in the range of (0, 255).

5. The robot vision algorithm for a large viewing angle according to claim 2, characterized in that: The feature score formula based on the neighborhood domain at different camera conversion angles is: Among them, S is the feature score of the neighborhood domain, w is the mean of the number of coarse matching features of each grid in the neighborhood domain, i is the number of the grid, n is the number of rotations, and α is i represents the weight of the i-th grid, m represents the feature point number, and its value range is (1, w), θ is the angle transformed by the camera during the movement, and δ is the angle threshold; s i,m is the mark score of the mth feature point of the ith grid in the cross neighborhood domain, It represents the mark score of the mth feature point of the ith grid in the diagonal neighborhood domain rotated n times. The number of correct matches of each grid near the matching point is counted to obtain the mark score of the corresponding grid.

6. The robot vision algorithm for a large viewing angle according to claim 1, characterized in that: In the step S1, the multi-scale multi-stage deblurring network uses a multi-stage multi-scale architecture network and has three scales s1, s2, and s3, each scale has 1, 2, and 3 stages respectively, and each stage is composed of a single lightweight encoder-decoder UNet module, and each UNet module shares the same network structure but has different weights; each UNet module is trained to generate residual features that can convert blurred images into clear images, and these residual features are added to the blurred image to generate a deblurred image; cross-stage feature fusion is performed between each stage and between each scale through the CSFF module, and a feature attention module and an attention supervision module are introduced at the beginning and end of each scale respectively, and different weights are assigned to different pixel and channel features on the feature map, effectively utilizing the information on the feature map and improving the deblurring effect of the network.

7. The robot vision algorithm for large viewing angle according to claim 6, characterized in that: The feature attention module includes the ResNet101 network, the channel attention mechanism and the spatial attention mechanism. The information obtained by the camera is extracted through the ResNet101 network, and then the feature maps are further adjusted and enhanced through the channel attention mechanism and the spatial attention mechanism to improve their expressiveness and attention. Subsequently, the feature maps are processed through bilinear upsampling and 1×1 convolution operations, and the adjusted and enhanced feature maps are adjusted to feature maps of the same size as the input image, and the channel dimension of the feature maps is transformed; the input feature F of the attention supervision module in First, the residual image R is generated through a 1×1 convolution layer S , the obtained residual image is combined with the degraded image I to obtain the restored image X S , and then restore the image X through a 1×1 convolution layer and a simgmoid activation function S , generate the attention map M; recalibrate the input feature F after the 1×1 convolution layer through the attention map M in , we get the feature guided by the attention map, that is, the attention-enhanced feature F generated by SAM out , and the feature F out Pass to the next stage.

8. The robot vision algorithm for a large viewing angle according to claim 1, characterized in that: In the step S2, the acceleration and angular velocity of the camera are measured by the IMU sensor, the displacement of the camera is obtained by integration, and the rotation matrix of the camera is obtained by quaternion, so as to obtain the camera coordinates of the next frame; first, the two-dimensional coordinates of the feature points of the previous frame are calculated by the projection formula from two-dimensional to three-dimensional coordinates to obtain the three-dimensional coordinates P0 (x0, y0, z0) of the feature points of the previous frame, and then the three-dimensional coordinates P1 (x1, y1, z1) of the camera of the next frame are obtained by the rotation matrix and the translation vector, and finally converted into two-dimensional coordinates by the projection formula, and a circle is drawn with a radius r based on the two-dimensional coordinates of the feature points. The radius r is obtained through multiple experiments to obtain a suitable value, so that the two-dimensional coordinate position of the feature points matching the next frame must be within the circle; The formula for calculating the camera coordinates of the next frame is as follows: Where [x1, y1, z1] T Represents the camera coordinates of the current frame, [x0, y0, z0] T Represented as the camera coordinates of the previous frame, T represents transposition, q w is the real part of the quaternion, q x ,q y ,q z are the three components of the imaginary part of the quaternion, t0, t f are the starting and ending times of the interval, respectively, and t 2σ-1 and t 2σ represents the time of the node in the interval, η represents the time of (t 0， t f ) is divided into a number of equal time intervals, generally η is taken as 4, and σ is represented by the number of nodes in the interval; v(t0) is the initial velocity, v(t f ) is the speed of the camera at the end of the interval, v(t 2σ-1 ) and v(t 2σ ) represents the speed of each node in the interval.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a robot vision algorithm for a large viewing angle as described in any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of a robot vision algorithm for a large viewing angle as described in any one of claims 1-8.

Images