A visual slam method and system based on saliency prediction and a storage medium

Abstract

The application discloses a visual SLAM method and system based on saliency prediction and a storage medium, and belongs to the technical field of simultaneous localization and mapping. Firstly, saliency prediction based on multi-modal fusion is performed according to a current image frame and a depth frame to obtain a gray image of the current image frame and a saliency mask containing an effective structured area, and then feature extraction and matching are performed to obtain matching feature points; then, saliency entropy of the current image frame is calculated, a key frame is judged, and a map point is created to obtain a hierarchical local map in real time; finally, global BA weighted optimization is performed, and a global map is constructed by continuously expanding and maintaining the hierarchical local map. The saliency prediction technology based on multi-modal fusion is introduced, geometric information and depth information are fused to accurately perceive the effective structured area in the image, and salient features are captured and accurately matched, the system perception and correlation capabilities are enhanced, and the stability and overall performance of the system are comprehensively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of simultaneous localization and mapping, and particularly relates to a visual SLAM method based on saliency prediction, a system and a storage medium. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) technology is the core of autonomous navigation and environment perception technology, enabling robots to achieve real-time localization and map construction in unknown environments. As an important branch of SLAM, visual SLAM uses camera images for localization and mapping, and has the advantages of low cost and high performance. However, traditional visual SLAM methods still have deficiencies.

[0003] Current mainstream visual SLAM methods often ignore the key role of effective structured regions such as edges and corners in feature extraction, matching and pose estimation, and adopt a one-size-fits-all approach to image regions. This not only fails to fully exploit the rich information contained in effective structured regions, but also is easily disturbed by the negative effects of textureless and repetitive texture regions, resulting in waste of computing resources and restriction of the overall performance improvement of the system. Therefore, there is an urgent need for a visual SLAM method that can always pay attention to effective structured regions in the environment to improve the performance and efficiency of the system. SUMMARY

[0004] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a visual SLAM method and system based on saliency prediction, which introduces a saliency prediction technology using multi-modal fusion to guide the feature extraction and matching, key frame selection and pose optimization process, thereby improving the attention to effective structured regions in the environment and effectively addressing the performance deficiencies and low efficiency of traditional methods.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0006] On the one hand, the present application provides a visual SLAM method based on saliency prediction, comprising the following steps:

[0007] Using a depth camera mounted on a mobile platform to obtain image frames and depth frames of the current surrounding environment;

[0008] Performing saliency prediction based on multi-modal fusion according to the current image frames and depth frames to obtain a grayscale image of the current image frames and a saliency mask containing effective structured regions;

[0009] Performing saliency mask-driven feature extraction and matching on the current image frames in combination with the saliency mask to obtain matching feature points with saliency values;

[0010] Calculate the saliency entropy of the current image frame according to the matching feature points with the significant value, and determine whether the current image frame is the current key frame according to the saliency entropy and save it to the key frame set KeyFrames;

[0011] Create the current map point by using the current key frame and perform real-time grading to obtain the graded local map;

[0012] According to the current map point level and the saliency mean value of the graded local map, the camera pose is globally weighted and optimized by BA;

[0013] By continuously processing new image frames and depth frames captured by the depth camera in real time, the graded local map is expanded and maintained, thereby constructing a global map.

[0014] As a preferred technical solution, the saliency mask of the current image frame containing the effective structured area is obtained, specifically:

[0015] The current image frame is subjected to grayscale processing to obtain a corresponding grayscale image; the grayscale processing adopts a weighted average method;

[0016] The Sobel operator is used to process the grayscale image to obtain the edge image of the current image frame; the Sobel operator performs two-dimensional convolution operation on the grayscale image in the horizontal gradient direction and the vertical gradient direction, and calculates the gradient amplitude of the pixel point by using the Pythagorean theorem to obtain the edge image;

[0017] The edge image is subjected to smoothing processing using a Gaussian filter, and then the smoothed edge image is subjected to depth adjustment using the depth frame to obtain the saliency mask of the current image frame containing the effective structured area.

[0018] As a preferred technical solution, the matching feature points with the significant value are obtained, specifically:

[0019] A Gaussian image pyramid IPyramid and a Gaussian image pyramid Mpyramid of the grayscale image and the saliency mask of the current image frame are constructed, respectively;

[0020] Each layer of the IPyramid and the MPyramid is rasterized and divided; the length of each raster after rasterization is calculated based on the grid parameter and the global gradient mean value of the current image frame;

[0021] Each raster of each layer of the IPyramid is traversed, and the ORB feature point extraction threshold of each raster image in the IPyramid is calculated according to the raster image and the corresponding raster image in the MPyramid;

[0022] According to the ORB feature point extraction threshold, the ORB feature point extraction algorithm is used to obtain feature points and corresponding descriptors in the grid images of each layer of the IPyramid, and the extracted feature points and corresponding descriptors are respectively saved into the feature point set KeyPoints and the descriptor set KeyPointsDes with the layer number in the IPyramid as a label;

[0023] The feature points in the feature point set KeyPionts are uniformly quantized by a quadtree, and part of the feature points with uneven distribution are removed, and the uniformly quantized feature points and corresponding descriptors are re-saved into the feature point set KeyPionts and the descriptor set KeyPointsDes with the layer number in the IPyramid as a label, as the feature points of the current image frame;

[0024] The feature points in the feature point set KeyPoints are traversed, the gray values of the corresponding positions in the MPyramid are obtained as the saliency values of the feature points of the current image frame according to the coordinates of the feature points, and are saved into the saliency value set KeyPointsSal in the same storage structure;

[0025] The current image frame and the previous image frame are matched for feature points to obtain a matching feature point set KeyPointsMatch.

[0026] As a preferred technical solution, the calculation method of the ORB feature point extraction threshold is:

[0027] Based on the pixel number, pixel gradient value and global gradient mean value of each grid image in each layer of the IPyramid, the global gradient variance is calculated to describe the prominence of each grid image in the IPyramid in the overall image frame;

[0028] Based on the pixel number, pixel gradient value and gradient mean value of each grid image in each layer of the IPyramid, the local gradient variance is calculated to describe the structural richness within each grid image in the IPyramid;

[0029] The gray mean value of each grid image in each layer of the IPyramid at the corresponding position grid image in each layer of the MPyramid is calculated, and the saliency value of the corresponding position grid image in each layer of the MPyramid is quantified by the normalized gray mean value;

[0030] According to the ratio of the global gradient variance and the local gradient variance of each grid image in each layer of the IPyramid, and in combination with the saliency value of the corresponding position grid image in each layer of the MPyramid, the ORB feature point extraction threshold of each grid image in each layer of the IPyramid is calculated;

[0031] The feature point matching specifically is:

[0032] According to the saliency value of each feature point in the saliency value set KeyPointsSal, the saliency weight of each feature point in the current image frame is calculated;

[0033] For each feature point of the current image frame, the descriptors of all feature points in the previous image frame are traversed, the saliency weighted Hamming distance is calculated and recorded, the descriptor with the smallest distance is selected as the best matching candidate, and if the smallest distance is smaller than the preset distance threshold, the feature point corresponding to the descriptor is determined as a matching feature point. The indexes of the matching feature points of the current image frame and the previous image frame are saved to the matching feature point set KeyPointsMatch.

[0034] As a preferred technical solution, the current image frame is determined as the current key frame according to the saliency entropy, specifically:

[0035] According to the number of matching feature points in the current image frame and the saliency value probability distribution of the matching feature points, the intra-frame entropy is calculated;

[0036] According to the intra-frame entropy of the image frame, the inter-frame entropy between the current image frame and the adjacent image frame is calculated within a time window;

[0037] The intra-frame entropy and the inter-frame entropy of the current image frame are combined to obtain the saliency entropy of the current image frame;

[0038] The saliency entropy is judged, and if it is greater than a preset entropy threshold, the current image frame is the current key frame, and its frame index is saved to the key frame set KeyFrames; otherwise, the current image frame is not the current key frame.

[0039] As a preferred technical solution, the current key frame is used to create a current map point and perform real-time grading, specifically:

[0040] The matching feature points of the current key frame are triangulated, and the 3D map points corresponding to the 2D feature points in the current key frame are calculated, and the current map point is added to the graded local map;

[0041] The saliency value of the matching feature points of the current key frame is assigned to the current map point as the saliency value of the map point;

[0042] According to the saliency value of the current map point and the saliency average of all map points in the graded local map, the grade of the current map point is divided.

[0043] As a preferred technical solution, the camera pose is globally weighted and optimized according to the current map point grade and the saliency average of the graded local map, specifically:

[0044] According to the significance value and the level of the current map point and the significance average of the hierarchical local map, the position optimization weighting weight corresponding to the current map point is calculated;

[0045] The information matrix and the re-projection error in the global BA optimization process are weighted by using the position optimization weighting weight corresponding to the current map point.

[0046] As a preferred technical solution, the information matrix and the re-projection error in the global BA optimization process are weighted, specifically:

[0047] The position of the 3D map point, the rotation matrix of the camera and the translation vector are used as optimization variables to construct the objective function of the global BA optimization.

[0048] The re-projection error and the information matrix in the objective function are weighted by using the position optimization weighting weight corresponding to the current map point.

[0049] Another aspect of the application provides a visual SLAM system based on significance prediction, which is applied to the visual SLAM method based on significance prediction and includes a data acquisition unit, a significance prediction unit, a feature extraction and matching unit, a key frame determination unit, a map point creation and hierarchical unit, a global BA weighted optimization unit and a global map construction unit.

[0050] The data acquisition unit is used to acquire image frames and depth frames of the current surrounding environment by using a depth camera mounted on a mobile platform.

[0051] The significance prediction unit is used to perform multi-modal fusion-based significance prediction according to the current image frames and depth frames to obtain a gray image of the current image frames and a significance mask containing an effective structured area.

[0052] The feature extraction and matching unit is used to perform significance mask-driven feature extraction and matching on the current image frames in combination with the significance mask to obtain matching feature points with significance values.

[0053] The key frame determination unit is used to calculate the significance entropy of the current image frames according to the matching feature points with significance values, and to determine whether the current image frames are current key frames and save them to a key frame set KeyFrames according to the significance entropy.

[0054] The map point creation and hierarchical unit is used to create current map points by using the current key frames and perform real-time hierarchical processing to obtain hierarchical local maps.

[0055] The global BA weighted optimization unit is used to perform global BA weighted optimization on the camera pose according to the level of the current map point and the significance average of the hierarchical local map.

[0056] The global map construction unit is configured to expand and maintain a hierarchical local map by continuously processing new image frames and depth frames captured by the depth camera in real time, thereby constructing a global map.

[0057] The application also provides a computer readable storage medium storing a program, when the program is executed by a processor, the method for visual SLAM based on saliency prediction is realized.

[0058] Compared with the prior art, the application has the following advantages and beneficial effects:

[0059] 1. The saliency prediction technology based on multi-modal fusion is introduced in the application, which can accurately perceive the effective structured area in the image and provide guidance for subsequent visual processing tasks; the spatial distribution features of the effective structured area in the image are described from multiple levels by fusing geometric information and depth information.

[0060] 2. The feature extraction and matching method driven by the saliency mask is used in the application, which realizes efficient capture and accurate matching of the salient features of the effective structured area, significantly enhances the perception and correlation ability of the system, and effectively alleviates the problem of low matching efficiency in the texture sparse or repetitive area in the traditional method, thereby improving the stability and overall performance of the system.

[0061] 3. The key frame selection method based on saliency entropy is used in the application, which effectively realizes the priority reservation of the high saliency information frame and avoids the frequent introduction of redundant frames; the use efficiency of the key frame is significantly improved by fully expressing the key visual information.

[0062] 4. The pose optimization method based on map point grading is realized in the application, which greatly strengthens the contribution of the effective structured area map point in the pose optimization process; the accuracy of the pose estimation is significantly improved by preferentially optimizing the high-quality map point. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 The overall flowchart of the visual SLAM method based on saliency prediction in the embodiment of the present application.

[0065] Figure 2 The overall block diagram of the visual SLAM system based on saliency prediction in the embodiment of the present application.

[0066] Figure 3It is a structural schematic view of the computer readable storage medium in the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the protection scope of the present application.

[0068] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be contained in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean that it refers to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0069] As shown in Figure 1 the present embodiment, a visual SLAM method based on saliency prediction comprises the following steps:

[0070] S1, using a depth camera carried on a mobile platform to acquire an image frame and a depth frame of a current surrounding environment.

[0071] In the present embodiment, the mobile platform is a robot, and the depth camera is used to capture image data and distance data of the current surrounding environment; however, the mobile platform is not limited thereto, and other mobile platforms that need to realize autonomous positioning and environment perception also fall within the protection scope of the present application.

[0072] S2, performing saliency prediction based on multi-modal fusion according to the current image frame and the depth frame to obtain a gray image of the current image frame and a saliency mask containing an effective structured area.

[0073] Further, in step S2, first, a saliency prediction technology based on multi-modal fusion is performed according to the current image frame and the depth frame to obtain a saliency mask capable of describing an effective structured area of the image frame, specifically:

[0074] S201, performing gray processing on the current image frame to obtain a corresponding gray image I gray ; the gray processing adopts a weighted average method, and the calculation formula is: Gray = 0.299·R + 0.587·G + 0.114·B, wherein Gray represents the pixel value of the image frame after gray processing, and R, G and B respectively represent the pixel values on the red, green and blue three channels of the image frame;

[0075] S202, processing the gray-scale image using a Sobel operator to obtain an edge map I of the image frame edge The Sobel operator performs a two-dimensional convolution operation on the gray-scale image in the horizontal gradient direction and the vertical gradient direction, and calculates the gradient amplitude of the pixel point using the Pythagorean theorem to obtain the edge map.

[0076] In this embodiment, the horizontal direction G x and the vertical direction G y of the Sobel operator are defined as follows:

[0077] ,

[0078] The gradients of the image frame in the horizontal direction and the vertical direction are obtained by convolving the gray-scale image I gray with G x and G y respectively:

[0079] S x (x,y) = G x *I gray (x,y), S y (x,y) = G y *I gray (x,y),

[0080] where S x (x,y) is the horizontal direction gradient, S y (x,y) is the vertical direction gradient, and * represents a two-dimensional convolution operation.

[0081] The gradient amplitude at the pixel point is calculated using the Pythagorean theorem, and then the edge map I edge of the current image frame is obtained:

[0082] .

[0083] S203, the edge map I edge is smoothed using a Gaussian filter to enhance spatial connectivity; the formula of the Gaussian filter is:

[0084] ,

[0085] where (x,y) is the coordinate of the pixel point in the edge map, G(·) represents the Gaussian function, and σ represents the standard deviation of the Gaussian filter.

[0086] S204, finally, the smoothed edge map is depth-adjusted using the depth frame to obtain the final saliency mask I maskThe depth adjustment of the embodiment adjusts the gray value of the edge in the near place to be higher and the gray value of the edge in the far place to be lower, so that the final saliency mask containing the effective structured area can preferentially focus on the clear area in the near place. The depth adjustment formula is:

[0087] ,

[0088] wherein, I mask (x,y) represents the pixel value of the (x,y) position in the saliency mask, I gaussian (x,y) represents the pixel value of the (x,y) position in the edge map after smoothing processing, and I depth (x,y) is the pixel value of the (x,y) position in the depth frame. S3, the feature extraction and matching of the saliency mask driven current image frame are combined to obtain the matching feature points with saliency values.

[0089] Further, in step S3, the feature extraction and matching of the saliency mask driven current image frame are combined to obtain the matching feature points with saliency values, specifically:

[0090] S301, the Gaussian image pyramid IPyramid and the Gaussian image pyramid MPyramid of the current image frame gray scale map and the saliency mask are respectively constructed to realize the extraction of stable and scale invariant feature points at different scales.

[0091] S302, each layer of IPyramid and MPyramid is rasterized and divided; the length W of each grid after rasterization is calculated based on the grid parameter and the global gradient average of the current image frame, and the calculation formula is:

[0092] ,

[0093] wherein, μ is the grid parameter, is the global gradient average of the current image frame.

[0094] S303, then each grid of each layer of IPyramid is traversed, and the threshold value of the ORB feature point extraction of each grid image of each layer of IPyramid is calculated according to the grid image and the corresponding grid image in each layer of MPyramid.

[0095] Further, the calculation process of the ORB feature point extraction threshold value is:

[0096] First, the global gradient variance Var global to describe the prominence of the grid image in the whole image frame, and the formula is as follows:

[0097] ,

[0098] where N is the total number of pixels in the grid image, G(x i , y i ) is the gradient value of the i-th pixel in the grid image, is the global gradient mean value of the current image frame;

[0099] Then, the local gradient variance Var local is calculated to describe the structural richness within the grid image, and the formula is as follows:

[0100] ,

[0101] where N is the total number of pixels in the grid image, is the gradient mean value of the grid image;

[0102] Subsequently, the gray mean value of the grid image corresponding to each grid image in the IPyramid layers in the MPyramid layers is calculated, and the saliency value of the corresponding grid image in the MPyramid layers is quantified by the normalized gray mean value. Wherein, for the saliency value s i of the grid image i in the IPyramid layers corresponding to the grid image in the MPyramid layers, the calculation method is as follows:

[0103] ,

[0104] where σ i is the gray mean value of the i-th grid image in the IPyramid layers corresponding to the grid image in the MPyramid layers, is the gray mean value of all grid images in the MPyramid layers;

[0105] Finally, the ORB feature point extraction threshold of each grid image in the IPyramid layers is calculated, and the formula is as follows:

[0106] ,

[0107] where φ(i) is the ORB feature point extraction threshold of the i-th grid image in the IPyramid layers, λ is the threshold parameter, s i is the normalized saliency value of the i-th grid image in the IPyramid layers corresponding to the grid image in the MPyramid layers, and in the formula, the structural characteristics of the grid image are represented by calculating the ratio of the global gradient variance and the local gradient variance of the grid image i in the IPyramid layers.

[0108] S304, according to the ORB feature point extraction threshold φ, using the ORB feature point extraction algorithm to obtain the feature points and the corresponding descriptors in the IPyramid grid image of each layer, and saving the extracted feature points and the corresponding descriptors to the feature point set KeyPoints and the descriptor set KeyPointsDes respectively with the layer number in the IPyramid as the label.

[0109] S305, uniformly quadtreeing the feature points in the feature point set KeyPionts, eliminating part of the unevenly distributed feature points, and re-saving the uniformly quadtreeed feature points and the corresponding descriptors to KeyPionts and KeyPointsDes with the layer number in the IPyramid as the label as the feature points of the current image frame.

[0110] S306, traversing the feature points in the feature point set KeyPoints, obtaining the gray value of the corresponding position in the MPyramid as the saliency value of the current image frame feature point according to the coordinates of the feature points, and saving to the saliency value set KeyPointsSal in the same storage structure.

[0111] S307, performing feature point matching on the current image frame and the previous image frame to obtain the matching feature point set KeyPointsMatch.

[0112] Further, the feature point matching is specifically:

[0113] First, the saliency weight of each feature point in the current image frame is calculated:

[0114]

[0115] wherein ε ij represents the saliency weight of the jth feature point in the i layer image in the current image frame IPyramid, KeyPointsSal[i][j] represents the saliency value corresponding to the jth feature point in the i layer image in the current image frame IPyramid, and b is a bias.

[0116] Next, for each feature point in the current image frame, traverse the descriptors of all feature points in the previous image frame, calculate the saliency weighted Hamming distance between them and record it, select the descriptor with the smallest distance as the best matching candidate, and if the smallest distance is less than the preset distance threshold, determine that the feature point corresponding to the descriptor is a matching feature point, and save the indices of the matching current image frame feature point and the previous image frame feature point to the matching feature point set KeyPointsMatch.

[0117] ​In the embodiment, for each feature point KeyPointsSal[i][j] in the current image frame, the formula for calculating the saliency-weighted Hamming distance of the feature point in the previous image frame is:

[0118] ,

[0119] wherein, is the saliency-weighted Hamming distance between the i-th feature point in the j-th image in the i-th layer of the current image frame IPyramid and the y-th feature point in the x-th image in the previous image frame IPyramid; is an indicator function, returning 1 when the condition is true, otherwise returning 0; represents the k-th bit of the descriptor of the i-th feature point in the j-th image in the i-th layer of the current image frame IPyramid, represents the k-th bit of the descriptor of the y-th feature point in the x-th image in the previous image frame IPyramid.

[0120] S4, calculate the saliency entropy of the current image frame according to the matching feature points with saliency values, and determine whether the current image frame is the current key frame according to the saliency entropy and save it to the key frame set KeyFrames.

[0121] Further, the step S4 of determining whether the current image frame is the current key frame is:

[0122] S401, calculate the intra-frame entropy E in to evaluate the distribution of saliency values of matching feature points in the current image frame, the calculation formula is:

[0123] ,

[0124] wherein, N is the total number of matching feature points in the current image frame, represents the probability distribution of the saliency value s i of the matching feature point i; when E in is larger, it indicates that the distribution of saliency values of matching feature points is more dispersed, and the image frame contains more different effective structured regions; on the contrary, if E in is smaller, it indicates that the saliency values of most feature points in the image frame are close, and the effective structured region in the image frame is less.

[0125] S402, calculate the inter-frame entropy E out to measure the change of saliency values of feature points between adjacent image frames in a time window, the calculation formula is:

[0126] ,

[0127] wherein, M is the frame number of the time window, t is the frame index in the time window, the value range is [k-M, k], including the current image frame k, represents the probability distribution of the intra-frame entropy of the image frame t; out The greater the inter-frame entropy E out , the greater the difference in the effective structured area of each frame in the time window, and the richer the scene information; the smaller the inter-frame entropy E in , the more similar the effective structured area of each frame, and the higher the repetition degree of the scene information.

[0128] S403, combine the intra-frame entropy E in and the inter-frame entropy E out to calculate the saliency entropy ξ of the current image frame: ξ = E in + E out If the saliency entropy ξ of the current image frame is greater than a preset entropy threshold, the current image frame is determined to be a current key frame, and its frame index is saved to the key frame set KeyFrames; otherwise, it is not a current key frame.

[0129] S5, create a current map point using the current key frame and perform real-time grading to obtain a graded local map.

[0130] Further, the way to obtain the graded local map in step S5 is:

[0131] S501, triangulate the matching feature points of the current key frame, calculate the 3D map points corresponding to the 2D feature points in the current key frame, and add the current map point to the graded local map.

[0132] S502, assign the saliency value of the matching feature points of the current key frame to the current map point as the saliency value of the current map point.

[0133] S503, grade according to the saliency value of the current map point and the average saliency value of all map points in the graded local map.

[0134] The specific mechanism of the grading in this embodiment is:

[0135] ,

[0136] wherein, Level(m p ) is the level of the current map point m p , S(m p ) represents the saliency value of the current map point m p , represents the average saliency value of all map points in the graded local map; this division mechanism divides the current map point into three levels, wherein the third-level map point is the most important map point, and the first-level map point is the least important map point, and a graded local map is obtained based on this.

[0137] S6、according to the level of the current map point and the average of the significance of the hierarchical local map, globally weighted BA optimization is performed on the camera pose.

[0138] Further, in step S6, according to the level of the map point and the average of the significance of the hierarchical local map, globally weighted BA optimization is performed on the camera pose, specifically:

[0139] S601, according to the level Level(m p ), the significance value S(m p ) of the current map point and the average of the significance of the hierarchical local map , the position optimization weighting weight ω corresponding to the current map point m p is calculated:

[0140] ,

[0141] wherein λ Level represents a weight coefficient related to the level of the current map point, for balancing the influence of map points of different levels on the optimization result.

[0142] S602, using the position optimization weighting weight ω corresponding to the current map point m p , the information matrix and the re-projection error in the global BA optimization process are weighted.

[0143] Further, the global BA optimization process is specifically:

[0144] First, taking the position of the 3D map point, the rotation matrix and the translation vector of the camera as the optimization variable, the objective function of the global BA optimization is constructed:

[0145] ,

[0146] wherein {X i , R l , t l} are optimization variables, including the position X i of the 3D map point and the rotation matrix R l and the translation vector t l of the camera, P L refers to the three-dimensional coordinates of all map points that can be observed in the local map key frame, K L represents the pose of the local map key frame having co-view relationship with the current key frame, K F represents the pose of the local map key frame having no co-view relationship with the current key frame, X k is defined as the matching set between the map points in P L and the feature points in the key frame k, ρ() is a robust kernel function, and E k,jrepresents the re-projection error between the estimated pose and the ground truth.

[0147] Then the re-projection error E in the objective function is optimized using the position of the current map point k,j and the information matrix is weighted as follows:

[0148] ,

[0149] ,

[0150] where ω j is the position optimized weighting of the current map point j, is the observed position of the current map point j, π (·) is the projection function of the pinhole camera, is the information matrix, which is used to describe the uncertainty between the observation and the ground truth; by adaptively adjusting the weight, the re-projection error and the information matrix of the map point are reasonably controlled; for important map points, the re-projection error is appropriately amplified to improve the attention of the system in the optimization process; at the same time, the information matrix is appropriately increased to improve the credibility of the observation value in the optimization process; for unimportant map points, the opposite strategy is adopted.

[0151] S7, by continuously processing the new image frames and depth frames captured by the depth camera in real time, expanding and maintaining the hierarchical local map, thereby constructing the global map.

[0152] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously.

[0153] Based on the same idea as one of the above-mentioned embodiment of a saliency prediction based visual SLAM method, the present application also provides a saliency prediction based visual SLAM system, which can be used to execute the above-mentioned saliency prediction based visual SLAM method. For the convenience of description, in the structural schematic diagram of an embodiment of a saliency prediction based visual SLAM system, only the parts related to the embodiments of the present application are shown, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and can include more or fewer components than the illustrated, or combine certain components, or different component arrangements.

[0154] As Figure 2As shown, another embodiment of the present application provides a visual SLAM system based on saliency prediction, comprising a data acquisition unit, a saliency prediction unit, a feature extraction and matching unit, a key frame determination unit, a map point creation and grading unit, a global BA weighted optimization unit, and a global map construction unit.

[0155] The data acquisition unit is configured to acquire image frames and depth frames of a current surrounding environment using a depth camera mounted on a mobile platform.

[0156] The saliency prediction unit is configured to perform saliency prediction based on multi-modal fusion according to the current image frames and depth frames, to obtain a grayscale image of the current image frames and a saliency mask containing an effective structured area.

[0157] The feature extraction and matching unit is configured to perform saliency mask driven feature extraction and matching on the current image frames in combination with the saliency mask, to obtain matching feature points with saliency values.

[0158] The key frame determination unit is configured to calculate a saliency entropy of the current image frames based on the matching feature points with saliency values, and to determine whether the current image frames are current key frames and save them into a key frame set KeyFrames according to the saliency entropy.

[0159] The map point creation and grading unit is configured to create current map points using the current key frames and perform real-time grading, to obtain a graded local map.

[0160] The global BA weighted optimization unit is configured to perform global BA weighted optimization on camera poses according to the current map point grades and the saliency mean value of the graded local map.

[0161] The global map construction unit is configured to expand and maintain the graded local map by continuously processing new image frames and depth frames captured in real time by the depth camera, to construct a global map.

[0162] It should be noted that the visual SLAM system based on saliency prediction of the present application corresponds to the visual SLAM method based on saliency prediction of the present application, and the technical features and advantages described in the embodiment of the visual SLAM method based on saliency prediction are applicable to the embodiment of the visual SLAM system based on saliency prediction, and the specific content can be referred to the description in the method embodiment of the present application, which will not be repeated here.

[0163] In addition, in an implementation of the above embodiment, the logic division of each program module is only illustrative. In actual applications, the above-mentioned function allocation can be completed by different program modules according to needs, for example, for the configuration requirements of corresponding hardware or the convenience of software implementation. That is, the internal structure of the above-mentioned visual SLAM system based on saliency prediction is divided into different program modules to complete all or part of the above-mentioned functions.

[0164] As shown in Figure 3 In one embodiment, a computer readable storage medium is provided, which stores a program in the memory. When the program is executed by a processor, the visual SLAM method based on saliency prediction is implemented, specifically:

[0165] An image frame and a depth frame of a current surrounding environment are acquired using a depth camera mounted on a mobile platform;

[0166] Saliency prediction based on multi-modal fusion is performed according to the current image frame and the depth frame, to obtain a gray image of the current image frame and a saliency mask containing an effective structured area;

[0167] Feature extraction and matching of the current image frame are performed in combination with the saliency mask, to obtain matching feature points with saliency values;

[0168] The saliency entropy of the current image frame is calculated according to the matching feature points with saliency values, and it is judged whether the current image frame is a current key frame according to the saliency entropy and saved to a key frame set KeyFrames;

[0169] A current map point is created using the current key frame and real-time grading is performed, to obtain a graded local map;

[0170] The camera pose is globally BA weighted optimized according to the current map point level and the saliency mean value of the graded local map;

[0171] By continuously processing new image frames and depth frames captured in real time by the depth camera, the graded local map is expanded and maintained, thereby constructing a global map.

[0172] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0173] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A visual SLAM method based on saliency prediction, characterized in that: The steps include: Use the depth camera on the mobile platform to obtain image frames and depth frames of the current surrounding environment; Perform saliency prediction based on multimodal fusion according to the current image frame and depth frame to obtain the grayscale image of the current image frame and the saliency mask containing the effective structured area; Combined with the saliency mask, the saliency mask-driven feature extraction and matching of the current image frame are performed to obtain matching feature points with saliency values, specifically: Construct the Gaussian image pyramid IPyramid of the grayscale image of the current image frame and the Gaussian image pyramid MPyramid of the saliency mask respectively; The IPyramid and MPyramid layers are rasterized and divided. The side length of each grid after rasterization is calculated based on the grid parameters and the global gradient mean of the current image frame. Traverse each grid in each layer of IPyramid and calculate the ORB feature point extraction threshold of each raster image in each layer of IPyramid based on the raster image and the corresponding raster image in MPyramid; According to the ORB feature point extraction threshold, the ORB feature point extraction algorithm is used to obtain the feature points and corresponding descriptors in the raster images of each layer of IPyramid. The extracted feature points and corresponding descriptors are saved in the feature point set KeyPoints and the descriptor set KeyPointsDes respectively with the layer number in IPyramid as the label; Perform quadtree homogenization on the feature points in the feature point set KeyPionts, remove some unevenly distributed feature points, and save the homogenized feature points and corresponding descriptors back to the feature point set KeyPionts and descriptor set KeyPointsDes with the layer number in IPyramid as the label as the feature points of the current image frame; Traverse the feature points in the feature point set KeyPoints, obtain the grayscale value of the corresponding position in MPyramid according to the coordinates of the feature point as the saliency value of the feature point in the current image frame, and save it to the saliency value set KeyPointsSal in the same storage structure; Match the feature points of the current image frame and the previous image frame to obtain the matching feature point set KeyPointsMatch; Calculate the saliency entropy of the current image frame based on the matching feature points with saliency values, and determine whether the current image frame is the current key frame based on the saliency entropy and save it to the key frame set KeyFrames; Use the current keyframe to create the current map point and perform real-time grading to obtain the current graded local map, specifically: Triangulate the matching feature points of the current keyframe, calculate the 3D map points corresponding to the 2D feature points in the current keyframe, and add the current map points to the hierarchical local map; Assign the saliency value of the current keyframe matching feature point to the current map point as the saliency value of the map point; The current map point is graded based on the comparison between the saliency value of the current map point and the saliency mean value of all map points in the graded local map, thereby obtaining the current graded local map; Perform global BA weighted optimization on the camera pose based on the current map point level and the saliency mean of the current graded local map; The global map is constructed by continuously processing new image frames and depth frames captured by the depth camera in real time, expanding and maintaining the current hierarchical local map.

2. The visual SLAM method based on saliency prediction according to claim 1, wherein Obtain the saliency mask containing the valid structured area of ​​the current image frame, specifically: Grayscale processing is performed on the current image frame to obtain a corresponding grayscale image; the grayscale processing adopts a weighted average method; The grayscale image is processed using a Sobel operator to obtain an edge map of the current image frame; the Sobel operator performs a two-dimensional convolution operation on the grayscale image in the horizontal gradient direction and the vertical gradient direction, and calculates the gradient amplitude of the pixel point using the Pythagorean theorem to obtain the edge map; The edge map is smoothed using a Gaussian filter, and then the depth frame is used to perform depth adjustment on the smoothed edge map to obtain a saliency mask containing valid structured areas in the current image frame.

3. The visual SLAM method based on saliency prediction according to claim 1, wherein The calculation method of the ORB feature point extraction threshold is: Based on the number of pixels and pixel gradient values ​​of each raster image in each layer of IPyramid and the global gradient mean of the current image frame, the global gradient variance is calculated to describe the prominence of each raster image in each layer of IPyramid in the overall image frame; Based on the number of pixels, pixel gradient value and gradient mean of each raster image in each layer of IPyramid, the local gradient variance is calculated to describe the structural richness of each raster image in each layer of IPyramid. Calculate the grayscale mean of each raster image in each layer of IPyramid and the corresponding position raster image in each layer of MPyramid, and quantify the significance value of the raster image in the corresponding position in each layer of MPyramid by normalizing the grayscale mean; According to the ratio of the global gradient variance to the local gradient variance of each raster image in each layer of IPyramid, combined with the saliency value of the raster image at the corresponding position in each layer of MPyramid, the ORB feature point extraction threshold of each raster image in each layer of IPyramid is calculated; The feature point matching is specifically as follows: Calculate the significance weight of each feature point in the current image frame based on the significance value of each feature point in the significance value set KeyPointsSal; For each feature point in the current image frame, traverse its descriptors with all feature points in the previous image frame, calculate and record the significance weighted Hamming distance, select the descriptor with the smallest distance as the best matching candidate, and if the minimum distance is less than the preset distance threshold, the feature point corresponding to the descriptor is determined to be the matching feature point, and the indexes of the matched feature points of the current image frame and the previous image frame are saved in the matching feature point set KeyPointsMatch.

4. The visual SLAM method based on saliency prediction according to claim 1, wherein The method of judging whether the current image frame is the current key frame according to the saliency entropy is specifically as follows: Calculate the intra-frame entropy based on the number of matching feature points in the current image frame and the probability distribution of the saliency values ​​of the matching feature points; Calculating the inter-frame entropy between the current image frame and adjacent image frames according to the intra-frame entropy of the image frames within the time window; Combining the intra-frame entropy and inter-frame entropy of the current image frame to obtain the saliency entropy of the current image frame; The saliency entropy is judged. If it is greater than the preset entropy threshold, the current image frame is the current key frame, and its frame index is saved to the key frame set KeyFrames; otherwise, the current image frame is not the current key frame.

5. The visual SLAM method based on saliency prediction according to claim 1, wherein The camera pose is globally optimized based on the saliency mean of the current map point level and the current graded local map, specifically: Calculate the position optimization weight corresponding to the current map point based on the significance value and level of the current map point and the significance mean of the current graded local map; The information matrix and reprojection error in the global BA optimization process are weighted by optimizing the weighted weight corresponding to the current map point.

6. The visual SLAM method based on saliency prediction according to claim 5, wherein The information matrix and reprojection error in the global BA optimization process are weighted, specifically: The objective function of global BA optimization is constructed using the position of 3D map points, the rotation matrix and translation vector of the camera as optimization variables; The reprojection error and information matrix in the objective function are weighted using the position optimization weight corresponding to the current map point.

7. A visual SLAM system based on saliency prediction, characterized in that: The visual SLAM method based on saliency prediction as described in any one of claims 1 to 6 comprises a data acquisition unit, a saliency prediction unit, a feature extraction and matching unit, a key frame determination unit, a map point creation and classification unit, a global BA weighted optimization unit, and a global map construction unit; The data acquisition unit is used to obtain image frames and depth frames of the current surrounding environment using a depth camera mounted on the mobile platform; The saliency prediction unit is used to perform saliency prediction based on multimodal fusion according to the current image frame and the depth frame, and obtain a grayscale image of the current image frame and a saliency mask containing a valid structured area; The feature extraction and matching unit is used to perform saliency mask-driven feature extraction and matching on the current image frame in combination with the saliency mask to obtain matching feature points with saliency values; The key frame determination unit is used to calculate the significance entropy of the current image frame based on the matching feature points with significance values, and determine whether the current image frame is the current key frame based on the significance entropy and save it to the key frame set KeyFrames; The map point creation and grading unit is used to create the current map point using the current key frame and perform real-time grading to obtain the current graded local map; The global BA weighted optimization unit is used to perform global BA weighted optimization on the camera pose according to the current map point level and the saliency mean of the current hierarchical local map; The global map construction unit is used to expand and maintain the current hierarchical local map by continuously processing new image frames and depth frames captured in real time by the depth camera, thereby constructing a global map.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the visual SLAM method based on saliency prediction according to any one of claims 1 to 6 is implemented.

Images