A learning-based saliency localization method

Abstract

The present application relates to the technical field of map saliency localization, and particularly relates to a saliency localization method based on learning. The saliency localization method based on learning provided by the present application adopts a saliency prediction model to simulate the mechanism in a SLAM framework. A saliency map is predicted according to a saliency model, and the model can capture scene semantics and geometric information. The value of the saliency map is used as the weight of a feature point in a traditional bundle adjustment method. Detailed experiments conducted on the KITTI and EuRoc datasets with the most advanced algorithms show that the algorithm proposed by the present application is superior to existing algorithms in indoor and outdoor environments, and significantly improves the positioning accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of map saliency localization, and particularly relates to a learning-based saliency localization method. BACKGROUND

[0002] Visual recognition, combined with a map, can be widely used in automatic driving, machine vision and other fields, and this is based on image saliency recognition and positioning. Recently, the development of visual SLAM (Simultaneous Localization and Mapping) methods has been witnessed, from methods based on geometric models to methods based on deep learning. Deep learning-based methods have successfully demonstrated good ability to solve challenging problems such as dynamic objects and lighting changes. However, the progress of these algorithms only relies on the success of deep learning itself, without exploring the internal mechanism relationship between deep learning and SLAM. Some researchers have tried to use semantic information to improve the perception ability of SLAM, such as using semantic prior information to filter dynamic objects. However, a difficult problem is that semantic information cannot completely determine whether an object is dynamic or static, just like a car can be mobile or static. These algorithms treat all selected feature points as equally important in the localization and mapping tasks.

[0003] The way humans perform localization and mapping tasks is very different, and they focus on the most prominent objects or features, giving different weights to different features or objects. This is called the active approach, which actively seeks the most prominent features and objects in the field of view according to the specific task to meet the needs of the task. For example, when people walk in a new environment, they usually focus on prominent landmarks for localization. Therefore, the concept of saliency and attention has begun to be popular in the robotics community.

[0004] For example, Salient-DSO [Liang H J, Sanket N J, Fermüller C, et al. Salient dso: Bringing attention to direct sparse odometry [J]. IEEE Transactions on Automation Science and Engineering, 2019, 16(4): 1619-1626.] uses a saliency model to mimic human qualitative vision, which selects feature points in salient regions. The saliency model used in this framework is trained on the CVL-UMD dataset using SalGAN. However, this saliency model does not fully describe everything that the SLAM system should focus on, which makes it unable to work in outdoor environments. This is mainly due to the single focus and center bias of the human gaze dataset.

[0005] The invention patent with publication number "CN115409931A" discloses a three-dimensional reconstruction method based on image and point cloud data fusion. The technical solution described provides a three-dimensional reconstruction method based on image and point cloud data fusion, which relates to the fields of computer vision and computer graphics. The three-dimensional reconstruction method obtains panoramic point cloud of the measured object through point cloud registration and fusion. Then, according to the corresponding image data, significant feature extraction and multi-scale aggregated feature extraction are performed to obtain the significant feature vector and multi-scale aggregated feature vector of each point in the panoramic point cloud. A neural radiance field is used for point-based volume rendering to obtain a three-dimensional model with close-to-real color and texture information. However, this method lacks semantic information and has poor robustness.

[0006] The invention patent with publication number "CN115272268A" discloses an RGBD image saliency detection method based on depth estimation. The technical solution described provides an RGBD image saliency detection method based on depth estimation. The method uses depth estimation to generate an estimated depth map using RGB features, which is used as a supplement to the original depth map. The fusion of the two is used as input for the depth modality. The fused depth features provide more spatial information to the network, helping to locate the salient target. However, this method does not combine geometric and semantic information, and the positioning accuracy and robustness need to be improved. SUMMARY

[0007] To solve the above problems of the prior art, the purpose of the present application is to provide a learning-based saliency positioning method. Based on the Salient-KITTI dataset, DI-Net is used to obtain a saliency model to predict the area that the SLAM system should focus on. Finally, compared with traditional optimization methods that treat all features as equally important, salient feature points should play a more important role, just like the way humans process visual information. At the same time, the geometric and semantic information are combined according to the saliency algorithm to improve the positioning accuracy and robustness.

[0008] The technical solution adopted by the present application is as follows: a learning-based saliency positioning method uses a computer to process video maps under the algorithm framework of ORB-slam3 as follows:

[0009] S1, tracking the information of images I of the video map,

[0010] S2, obtaining key frames according to the tracked information of images I,

[0011] S3, creating a local map of key frames based on the key frames obtained in step S2, and establishing a database of key frames,

[0012] S4, performing saliency prediction on the image I of the video map in S1, and obtaining a saliency map,

[0013] S5, importing the saliency map obtained in S4 into S1 as a basis for tracking image information,

[0014] S6, importing the saliency map obtained in S4 into S3 as a basis for local mapping of the key frame,

[0015] S7, merging images, merging the saliency map obtained in S4 and the local mapping of the key frame created in S3 to obtain a saliency positioning map;

[0016] Wherein the saliency prediction in S4 includes the following steps:

[0017] A1, extracting the image I of the video map in S1 and inputting it into a graphics processor to pre-process the semantic weight Wc of I,

[0018] A2, extracting the input image I i Geometric information G i ,

[0019] A3, obtaining the semantic mask C of the image information Ii using the semantic segmentation network SDCNet: i = SDCNet(I i ),

[0020] A4, filtering the geometric information using the semantic mask:

[0021] A5, multiplying the geometric information filtered in A4 by the semantic weight Wc defined in A1 step,

[0022]

[0023]

[0024] A6, Gaussian blurring the result obtained in A4 step:

[0025]

[0026] A7, performing saliency map normalization on the result of Gaussian blurring in A6:

[0027]

[0028] ​​​It is worth mentioning that the semantic weight Wc gives different weights to different semantic categories of features, in which traffic lights, traffic signs, buildings and other absolutely static or singular objects are given the highest weight, and trees, vegetation are given the lowest weight. The specific form is that the higher the weight of the feature, the higher the gray value in the saliency map; the geometric feature G i For including point features and line features, point features include geometry-based features (ORB, SIFT) and deep learning-based features SuperPoint, and line features include road markings and some edge information extracted by Canny algorithm; semantic C i For the model obtained by training on the Cityscapes dataset using the SDC-Net algorithm, and then obtaining the semantic segmentation model to extract the semantic segmentation mask for each image; For filtering the geometric information using the mask, and removing the feature information in the area of no interest; The product of the semantic weight Wc; After normalization and Gaussian blurring, a dataset for training the saliency prediction model is obtained.

[0029] Further, the information of the image I of the tracking video map in S1 step includes the following steps:

[0030] B1, using a graphics processor to extract ORB features of the image,

[0031] B2, using the active saliency map of S4 step as the basis for feature point weighting, performing pose estimation on the image features extracted in B2,

[0032] B3, using a graphics processor to perform local map tracking on the image after pose estimation,

[0033] B4, using a graphics processor to create a new key frame according to the tracking result.

[0034] Further, the key frame obtained according to the information of the tracked image I in S2 step is the new key frame created in B4.

[0035] Further, in S3 step, creating a local map of the key frame according to the key frame obtained in S2 step includes the following steps:

[0036] C1, inserting key frame information into the graphics processor,

[0037] C2, generating a place map according to C1 step,

[0038] ​C3, the saliency map obtained according to the S4 step is taken as the basis for feature point weighting, and the place graph generated in the C2 step is optimized under the framework of the local SBA algorithm, wherein the local SBA optimization refers to optimizing a group of commonly visible key frames KL and all map points PL in the key frames. Xk is defined as the matching set between the points in PL and the key points in the key frame k, and thus the optimization method is defined as:

[0039]

[0040]

[0041] C4, according to the algorithm framework of ORB-slam 3, the place graph optimized in the C3 step is used to remove redundant key frames, and a local mapping of the key frames is created,

[0042] C5, each key frame after the removal of the redundant key frames in the C1-C4 steps is used to create a database of the key frames.

[0043] Further, the S7 step of merging the map includes two parts of loop correction and loop recognition for each local mapping of the key frames, wherein the loop recognition includes sim3 optimization formula recognition based on the ORB-slam 3 algorithm and database query for each local mapping of the key frames, and the loop correction is performed after the sim3 optimization formula recognition based on the ORB-slam 3 algorithm and the database query are satisfied; wherein the loop correction of each local mapping of the key frames includes optimization of the essential graph based on the ORB-slam 3, and after the optimization is completed, the loop fusion is performed with the saliency map in the S4 step, and finally a saliency positioning map is obtained.

[0044] Beneficial effects:

[0045] The method provided by the application is a saliency positioning method based on learning, which uses a saliency prediction model to simulate this mechanism in the SLAM framework. A saliency map is predicted according to a saliency model, which can capture scene semantics and geometric information. The value of the saliency map is taken as the weight of the feature points in the traditional bundle adjustment method. Detailed experiments conducted on the KITTI and EuRoc datasets with the most advanced algorithms show that the algorithm proposed by the application is superior to existing algorithms in indoor and outdoor environments, and significantly improves the positioning accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 It is a working schematic diagram of the saliency positioning method based on learning provided by the application;

[0047] Figure 2 It is an experimental verification of the map image according to the algorithm;

[0048] Figure 3Results of a saliency model trained on the Salient-KITTI dataset according to the algorithm are compared with results using a saliency model trained on the SALICON dataset;

[0049] wherein, Figure 2 The experiments were run on a Linux computer equipped with an Nvidia GeForce GTX 1650 GPU, an AMD Ryzen 5-2600 CPU and 8 GB of memory;

[0050] The datasets used in the evaluation include the KITTI dataset and the EuRoc dataset, as follows:

[0051] KITTI dataset: contains real-world image data collected from urban, rural and highway scenarios. Each image can contain up to 15 cars and 30 pedestrians, with varying degrees of occlusion and truncation. For odometry / SLAM, the dataset provides 22 sequences of autonomous vehicle scenarios with a total length of 39.2 kilometers, and provides 11 sequences (00-10 sequences) with ground truth;

[0052] EuRoc dataset: is a monocular, stereo and IMU dataset for indoor MAV. It contains two scenes, one is an industrial environment and the other is a common room environment. This dataset provides 11 sequences with ground truth. These sequences are divided into three difficulty levels according to the texture quality, scene brightness and motion speed. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the protection scope of the present application.

[0054] As Figure 1 A method of learning-based saliency localization shown in the flowchart uses a computer to process the video map under the ORB-slam3 algorithm framework as follows:

[0055] S1, track the information of the image I of the video video map,

[0056] S2, obtain a key frame according to the tracked information of the image I,

[0057] S3, create a local map of the key frame according to the key frame obtained in step S2, and establish a database of the key frame,

[0058] S4, performing saliency prediction on the image I of the map in S1, and obtaining a saliency map,

[0059] S5, importing the saliency map obtained in S4 into S1 as a basis for tracking image information,

[0060] S6, importing the saliency map obtained in S4 into S3 as a basis for local mapping for creating key frames,

[0061] S7, merging images, merging the saliency map obtained in S4 with the local mapping of the key frames created in S3 to obtain a saliency positioning map;

[0062] At present, there are more than 20 data sets for saliency prediction model training, but most of the data sets are related to detection tasks and a few are task-free data sets. Moreover, the scenes of these data sets are mainly concentrated in natural scenery, social activities, sports, etc., and there is little involvement in the field of transportation. The DR(eye)VE data set is the only saliency data set for the automatic driving task. After analyzing the images of this data set, it is found that the data set has the following characteristics: 1. Saliency is often related to the height of the object; 2. Saliency is related to dynamic objects or sub-regions of dynamic objects; 3. Long-term existing objects will be continuously focused on; 4. Newly appearing objects will often be more focused on; 5. When the scene is chaotic, or the scene is empty, or there is no obvious object in the scene, attention will often be focused on the center of the image.

[0063] In order to make the trained saliency model suitable for SLAM / VO tasks, a saliency map calculation method based on scene geometry and semantic information is proposed, and this method is used to make a saliency data set to train a saliency prediction model. In order to extract the region of interest of the feature, we extract different kinds of feature points from the image, analyze the distribution of the feature points, and determine which regions are the regions of interest of SLAM / VO. Through analysis, it is known that the regions with corner points, edges, rich texture, and strong contrast are the regions where feature points are concentrated, and are the regions that the algorithm should focus on. In addition, from a higher level, we want to give more attention to regions such as lane markings, traffic signs, and buildings, and try to ignore meaningless regions such as the sky. At the same time, based on the static world assumption, the algorithm should ignore dynamic objects to reduce false matches. The proposed saliency prediction model not only focuses on low-level geometric features, but also focuses on high-level semantic features, and not only focuses on local regions in front of the vehicle, but also focuses on global regions.

[0064] Among them, the saliency prediction in S4 includes the following steps:

[0065] A1. Extract image I from the video map in processing step S1 and input it into the graphics processor, pre-weighting the semantic weight Wc of I.

[0066] A2. Extract the input image I i Geometric information G i ,

[0067] A3. Obtain the semantic mask of image information Ii using the semantic segmentation network SDCNet: C i =SDCNet( I i ),

[0068] A4. Filtering geometric information using semantic masks:

[0069] A5. Geometric information after filtering the geometric information in A4. Multiply by the semantic weight Wc defined in step A1,

[0070]

[0071]

[0072] A6. Regarding the data obtained in step A4... Apply Gaussian blur:

[0073]

[0074] The result after Gaussian blurring in A7 and A6 Normalize the saliency plot:

[0075]

[0076] It's worth noting that semantic weights Wc assign different weights to features of different semantic categories. Traffic lights, traffic signs, buildings, and other absolutely static or singular objects receive the highest weights, while trees and vegetation receive the lowest. Specifically, features with higher weights have higher grayscale values ​​in the saliency map; geometric features G... i This includes point features and line features. Point features include geometry-based features (ORB, SIFT) and deep learning-based features (SuperPoint). Line features include road markings and some edge information extracted using the Canny algorithm; semantic C i The model obtained by training the SDC-Net algorithm on the Cityscapes dataset is used to extract the semantic segmentation mask for each image. To remove feature information in regions where saliency is not of interest after filtering geometric information using a mask; is filtered according to the mask and the semantic weight Wc; is normalized and Gaussian blurred, and a dataset for training the saliency prediction model is obtained.

[0077] In the embodiment, the information of the image I of the tracked video map in the S1 step comprises the following steps:

[0078] B1, using a graphics processor to extract ORB features of the image,

[0079] B2, using the saliency map in the S4 step as the basis for feature point weighting, performing pose estimation on the image features extracted in B2,

[0080] B3, using a graphics processor to perform local map tracking on the image after pose estimation,

[0081] B4, using a graphics processor to create a new key frame according to the tracking result.

[0082] In the embodiment, the key frame obtained according to the information of the tracked image I in the S2 step is the new key frame created in B4.

[0083] In the embodiment, in the S3 step, creating a local map of the key frame according to the key frame obtained in the S2 step comprises the following steps:

[0084] C1, inserting key frame information into a graphics processor,

[0085] C2, generating a place graph according to the C1 step,

[0086] C3, using the saliency map obtained in the S4 step as the basis for feature point weighting, optimizing the place graph generated in the C2 step under a local SBA algorithm framework, wherein local SBA optimization refers to optimizing a group of commonly visible key frames KL and all map points PL in these key frames. Define Xk as the matching set between the points in PL and the key points in the key frame k, and thus the optimization method is defined as:

[0087]

[0088]

[0089] C4, according to the algorithm framework of ORB-slam3, removing redundant key frames from the optimized place graph in the C3 step, and creating a local map of the key frame,

[0090] C5, using each key frame after removing the redundant key frames in the C1-C4 steps to create a database of key frames.

[0091] In the embodiment, the step S7 of merging the map comprises two parts of loop correction and loop recognition for the local map of each key frame, wherein the loop recognition comprises sim3 optimization type recognition based on the ORB-slam3 algorithm and database query for the local map of each key frame, and the loop correction is performed after the sim3 optimization type recognition based on the ORB-slam3 algorithm and the database query are satisfied; wherein the loop correction of the local map of each key frame comprises essential graph optimization based on ORB-slam3, and after the optimization, the loop fusion is performed with the saliency map of the step S4, and finally the saliency positioning map is obtained.

[0092] As shown in the experimental verification of the map image, Figure 2 It can be known that when there is no important object in the image, the attention is concentrated in the center of the image, and other important information is ignored for the model trained on the SALICON dataset. In contrast, the model trained on the Salient KITTI dataset can successfully capture these important information. In addition, these models can also reduce the influence of dynamic objects, so that the points with high saliency are usually stable and robust points, as shown in the rectangular box in Figure 2 .

[0093] As shown in the experimental verification of the map image, Figure 3 , it can be known that in the monocular and binocular cases, the system is more accurate than ORB-SLAM3, because SBA makes the prominent feature points fully play their value. Therefore, using the saliency map can make the algorithm of the system have more advantages in pose estimation. The visual saliency contains high-level semantic information, which itself makes the feature more robust. The SBA method proposed by the system further enhances the role of these salient points. In addition, the saliency model of the system only needs 8 milliseconds to infer an image, and the whole system can still run in real time on the platform of the system. Through experiments, the algorithm proposed by the system has achieved accurate results in various environments. Therefore, compared with Salient-DSO which only works in indoor and static environment, the algorithm proposed by the system is more practical in application. Even in indoor environment, the algorithm proposed by the system has super performance compared with most advanced technologies.

[0094] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of learning-based saliency localization, characterized by: The computer is used to process the video map under the algorithm framework of ORB-slam3 as follows: S1, tracking the image I of the video map, S2, obtaining a key frame according to the tracked image I, S3, creating a local map of the key frame according to the key frame obtained in S2, and establishing a database of the key frame, S4, performing saliency prediction on the image I of the video map in S1, and obtaining a saliency map, S5, importing the saliency map obtained in S4 into S1 as a basis for tracking the image information, S6, importing the saliency map obtained in S4 into S3 as a basis for creating a local map of the key frame, S7, merging the saliency map obtained in S4 and the local map of the key frame created in S3 to obtain a saliency positioning map. The saliency prediction in S4 includes the following steps: A1, extracting the image I of the video map in S1 and inputting it into a graphics processor to pre-process the semantic weight Wc of I, A2, extracting the input image I i geometric information G i , A3, obtaining a semantic mask of the image information Ii using a semantic segmentation network SDCNet: C i = SDCNet(I i ), A4, filtering geometric information with semantic masks: A5, geometry information after filtering the geometry information of A4 multiplying the semantic weight Wc defined in the step of A1, A6. To the pair obtained in step A4 Gaussian blur is performed: A7, result after Gaussian blur in A6 normalizing the saliency map:

2. The method of claim 1, wherein: The tracking of the image I of the video map in S1 includes the following steps: B1, using a graphics processor to extract ORB features of the image, B2, using the saliency map in S4 as a basis for feature point weighting to perform pose estimation on the image features extracted in B2, B3, using a graphics processor to track the image after pose estimation, B4, using a graphics processor to create a new key frame according to the tracking result.

3. The method of claim 2, wherein: The key frame obtained in S2 according to the tracked image I is the new key frame created in B4.

4. The method of claim 1, wherein: In S3, the creation of a local map of the key frame according to the key frame obtained in S2 includes the following steps: C1, inserting key frame information into a graphics processor, C2, generating a place graph according to C1, C3, using the saliency map obtained in S4 as a basis for feature point weighting to optimize the place graph generated in C2 under the local SBA algorithm framework, C4, removing redundant key frames from the optimized place graph in C3 according to the algorithm framework of ORB-slam3 to create a local map of the key frame, C5, creating a database of the key frame from each key frame after removing the redundant key frames in C1-C4.

5. The method of claim 1, wherein: The merging of the map in S7 includes two parts of loop closure correction and loop closure recognition for each local map of the key frame, wherein the loop closure recognition includes sim3 optimization formula recognition based on the ORB-slam3 algorithm and database query for each local map of the key frame, and the loop closure correction is performed after meeting the sim3 optimization formula recognition based on the ORB-slam3 algorithm and the database query; wherein the loop closure correction of each local map of the key frame includes essential graph optimization based on ORB-slam3, and after the optimization, the loop is fused with the saliency map in S4 to finally obtain a saliency positioning map.

Images