Visual SLAM method and system based on fully convolutional neural network in dynamic scenes

Abstract

The present invention provides a visual SLAM method and system based on a full convolutional neural network in a dynamic scene, the method comprising: acquiring an image data set; constructing a full convolutional neural network model according to the image data set; using the full convolutional neural network model to perform semantic segmentation on a monocular real-time image currently collected by a camera to obtain a semantic label image; removing dynamic feature points of the monocular real-time image according to the semantic label image to obtain static feature points of the monocular real-time image; estimating the position and posture of the camera according to the static feature points. Through the method provided by the present invention, dynamic targets can be accurately identified and semantic segmentation can be completed, the accuracy and robustness of camera tracking can be effectively improved, and the positioning and mapping accuracy of visual SLAM in dynamic scenes can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual SLAM method based on a full convolutional neural network in a dynamic scene and a visual SLAM system based on a full convolutional neural network in a dynamic scene. Background Art

[0002] Simultaneous Localization And Mapping (SLAM) refers to the process of a robot estimating its own position and building an environmental map in an unfamiliar environment through its own sensors. It is a prerequisite for many robot application scenarios, such as path planning, collision-free navigation, and environmental perception. Visual SLAM refers to the perception of using visual information to estimate the camera's own position and build a three-dimensional map of the environment.

[0003] In the prior art, the relative displacement of two adjacent frames of the input image can be estimated based on the matching of feature points between the two frames, thereby calculating the actual displacement of the camera. However, slowly moving dynamic objects in the scene can cause deviations in the camera pose calculation, resulting in inaccurate positioning of the entire visual SLAM system. Summary of the invention

[0004] In view of the technical problem in the prior art that slowly moving dynamic objects cause deviations in camera pose calculation, thereby resulting in inaccurate positioning of the entire visual SLAM system, the present invention provides a visual SLAM method based on a fully convolutional neural network in a dynamic scene and a visual SLAM system based on a fully convolutional neural network in a dynamic scene. The method can effectively improve the accuracy and robustness of camera tracking, and improve the positioning and mapping accuracy of visual SLAM in dynamic scenes.

[0005] To achieve the above-mentioned purpose, the present invention provides, on one hand, a visual SLAM method based on a fully convolutional neural network in a dynamic scene, the method comprising the following steps: acquiring an image data set; constructing a fully convolutional neural network model according to the image data set; using the fully convolutional neural network model to perform semantic segmentation on the monocular real-time image currently collected by the camera to obtain a semantic label image; according to the semantic label image, dynamic feature points of the monocular real-time image are eliminated to obtain static feature points of the monocular real-time image; and estimating the position and posture of the camera according to the static feature points.

[0006] Furthermore, the use of the fully convolutional neural network model to perform semantic segmentation on the monocular real-time image currently captured by the camera to obtain a semantically labeled image includes: using a convolution kernel with the same input data size as the fully connected layer of the VGG16 network in the fully convolutional neural network model to convert the fully connected layer of the VGG16 network into a convolutional layer to obtain an FCN-VGG16 network; optimizing the FCN-VGG16 network; and using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantically labeled image.

[0007] Furthermore, the method also includes: after each convolution calculation in the FCN-VGG16 network, using a linear rectification function as an excitation function to perform sparsity processing on the previous convolution layer; and performing a pooling operation on each pooling layer in the FCN-VGG16 network.

[0008] Furthermore, the optimizing the FCN-VGG16 network includes: introducing a skip structure to perform upsampling and fusion operations on each pooling layer in the FCN-VGG16 network to obtain the optimized FCN-VGG16 network.

[0009] Furthermore, the method of using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantic label image includes: determining the number of categories of the heat map of the optimized FCN-VGG16 network; determining the predicted probability that the monocular real-time image belongs to the target category; identifying the dynamic target of the monocular real-time image according to the number of categories of the heat map and the predicted probability; and performing semantic segmentation on the monocular real-time image according to the identified dynamic target to obtain the semantic label image.

[0010] Furthermore, the dynamic target of the monocular real-time image is identified according to the number of categories of the heat map and the predicted probability in the following manner:

[0011]

[0012] Wherein, M is the number of categories of the heat map, C is the target category of the monocular real-time image, and y ic is a dummy parameter, p ic is the predicted probability that the monocular real-time image belongs to the target category C, and L is the loss value; the dynamic target of the monocular real-time image is identified according to the size of the loss value.

[0013] Further, the method of removing the dynamic feature points of the monocular real-time image according to the semantic segmentation result to obtain the static feature points of the monocular real-time image includes: performing image pyramid stratification on the monocular real-time image with a first scaling factor to obtain a multi-layer first scaled image; performing image pyramid stratification on the semantic label image with a second scaling factor to obtain a multi-layer second scaled image; sequentially removing the dynamic feature points in each first scaled image: performing ORB feature extraction on each first scaled image to obtain the ORB feature points of each first scaled image; outputting image coordinate values ​​of pixel points in the dynamic area of ​​each second scaled image to form a set U i , where i is the number of layers of the second zoomed image, and the dynamic area is the area where the dynamic target is located; the coordinate value of each ORB feature point of the first zoomed image is compared with the corresponding set U i in each first scaled image, remove the coordinate values ​​that are different from the set U i ORB feature points whose coordinate values ​​match those in the first zoomed image; forming a static feature point set M with the ORB feature points retained in each first zoomed image; estimating the position and posture of the camera according to the static feature points, including: estimating the position and posture of the camera according to the static feature point set M.

[0014] Further, the first scaling factor is the same as the second scaling factor.

[0015] Furthermore, the coordinate values ​​of the ORB feature points are compared with the set U i Before matching the coordinate values ​​in , the method further includes: storing the ORB feature points.

[0016] On the other hand, the present invention provides a visual SLAM system based on a fully convolutional neural network in a dynamic scene, wherein the system is configured to use the visual SLAM method based on a fully convolutional neural network in a dynamic scene described above to estimate the position and posture of the camera.

[0017] Through the technical solution provided by the present invention, the present invention has at least the following technical effects:

[0018] The visual SLAM method based on a full convolutional neural network in the dynamic scene of the present invention is based on the ORB_SLAM2 open source visual SLAM system as a framework, a full convolutional neural network model is constructed, and the full convolutional neural network model is used to identify the dynamic target in the monocular real-time image currently collected by the camera, and the monocular real-time image is subjected to semantic segmentation, and the dynamic target in the image is marked, and finally the dynamic feature points of the monocular real-time image are eliminated by coordinate point mapping, and the static feature points in the monocular real-time image are retained, and the position and posture of the camera are estimated according to the static feature points. The method provided by the present invention can accurately identify dynamic targets and complete semantic segmentation, effectively improve the accuracy and robustness of camera tracking, and improve the positioning and mapping accuracy of visual SLAM in dynamic scenes.

[0019] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a visual SLAM method based on a fully convolutional neural network in a dynamic scene provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the FCN-VGG16 network in the visual SLAM method based on a fully convolutional neural network in a dynamic scene provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of an upsampling operation in a visual SLAM method based on a fully convolutional neural network in a dynamic scene provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of bilinear interpolation in a visual SLAM method based on a fully convolutional neural network in a dynamic scene provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of optimizing the FCN-VGG16 network in a visual SLAM method based on a fully convolutional neural network in a dynamic scene provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0026] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0027] In the present invention, unless otherwise specified, directional words such as "up, down, top, bottom" are usually used with reference to the directions shown in the drawings or are used to describe the relative positional relationships of components in the vertical, perpendicular or gravity direction.

[0028] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0029] Please refer to Figure 1 An embodiment of the present invention provides a visual SLAM method based on a full convolutional neural network in a dynamic scene, the method comprising the following steps: S101: acquiring an image data set; S102: constructing a full convolutional neural network model according to the image data set; S103: using the full convolutional neural network model to perform semantic segmentation on the monocular real-time image currently collected by the camera to obtain a semantic label image; S104: removing the dynamic feature points of the monocular real-time image according to the semantic label image to obtain the static feature points of the monocular real-time image; S105: estimating the position and posture of the camera according to the static feature points.

[0030] Specifically, in the embodiment of the present invention, the ORB_SLAM2 open source visual SLAM system is used as a framework to improve the convolutional neural network, construct a full convolutional neural network, and perform model training on the constructed full convolutional neural network through an image data set, and output the trained full convolutional neural network model. The full convolutional neural network model is used to identify dynamic targets in the monocular real-time image currently collected by the camera, and the monocular real-time image currently collected by the camera is semantically segmented to obtain a semantic label image, and then the dynamic feature points of the monocular real-time image are eliminated according to the semantic label image to obtain the static feature points of the monocular real-time image, and the static feature points are used to estimate the position and posture of the camera.

[0031] The method provided by the present invention can accurately identify dynamic targets and complete semantic segmentation, effectively improve the accuracy and robustness of camera tracking, and improve the positioning and mapping accuracy of visual SLAM in dynamic scenes.

[0032] Furthermore, the use of the fully convolutional neural network model to perform semantic segmentation on the monocular real-time image currently captured by the camera to obtain a semantically labeled image includes: using a convolution kernel with the same input data size as the fully connected layer of the VGG16 network in the fully convolutional neural network model to convert the fully connected layer of the VGG16 network into a convolutional layer to obtain an FCN-VGG16 network; optimizing the FCN-VGG16 network; and using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantically labeled image.

[0033] Specifically, in the embodiment of the present invention, the VGG16 network consists of 13 convolutional layers, 5 pooling layers and 3 fully connected layers. The fully connected layers of the VGG16 network are converted into convolutional layers using convolutional kernels with the same input data size as the fully connected layers of the VGG16 network in the fully convolutional neural network model to obtain the FCN-VGG16 network. Please refer to Figure 2 , Figure 2 Schematic diagram of the FCN-VGG16 network. Figure 2 The output result of the last convolutional layer Conv16 is a heat map, and the number of heat maps represents the number of categories of the heat map. Then the FCN-VGG16 network is optimized, and the optimized FCN-VGG16 network is used to perform binary classification operations on the monocular real-time image of dynamic targets and backgrounds to obtain semantic label images.

[0034] Furthermore, the method also includes: after each convolution calculation in the FCN-VGG16 network, using a linear rectification function as an excitation function to perform sparsity processing on the previous convolution layer; and performing a pooling operation on each pooling layer in the FCN-VGG16 network.

[0035] Specifically, in the implementation mode of the present invention, after each convolution calculation in the FCN-VGG16 network, a linear rectification function is used as an excitation function to perform sparsity processing on the previous convolution layer. The mathematical expression of the linear rectification function is as follows:

[0036] f(x)=max(0,x)

[0037] Where x is the input value and f(x) is the output value. When the input x<0, the output is 0; when the input x≥0, the output is equal to the input. The linear rectification function has simple transformation, low computational complexity, and fast gradient descent convergence speed, which ensures that the output value of the FCN-VGG16 network is positive. At the same time, it increases the sparsity of the FCN-VGG16 network and reduces the overfitting of the network.

[0038] The pooling operation of each pooling layer in the FCN-VGG16 network reduces the size of the input monocular real-time image, and can filter and select the features captured by the convolution layer. The general representation is:

[0039]

[0040] Among them, i,j are the pixel coordinates of the feature, K is the number of channels of the feature map, f,s 0are the convolution kernel size and convolution step size parameters, respectively. In the FCN-VGG16 network, after five pooling operations, the length and width of the heat map are reduced to 1 / 32 of the input monocular real-time image. However, in order to keep the input image and the output semantic segmentation label image the same size and dimension, the heat map needs to be deconvolved 32 times, and the inverse process of pooling is used to achieve the effect of output reconstructing the input, resulting in the FCN-32s-VGG16 network.

[0041] Furthermore, the optimizing the FCN-VGG16 network includes: introducing a skip structure to perform upsampling and fusion operations on each pooling layer in the FCN-VGG16 network to obtain the optimized FCN-VGG16 network.

[0042] Specifically, in the implementation of the present invention, in FCN-VGG16 (FCN-32s-VGG16), the image output by the 32x deconvolution operation is only the features in the convolution kernel of the Conv16 layer. Due to the accuracy problem, the features in the image cannot be restored well. Therefore, a jump structure is introduced to optimize the features, that is, the results of different pooling layers are upsampled and fused. Please refer to Figure 3 In this embodiment, an upsampling operation is performed using an upsampling method based on bilinear interpolation. The input feature map of size N×N is converted into a transition feature map of (2N+1)×(2N+1) using the bilinear interpolation method. Then, a convolution operation is performed on the new feature map using a 2×2 convolution kernel, and finally a new feature map of size 2N×2N is obtained.

[0043] Please refer to Figure 4 , the calculation process of bilinear interpolation is as follows, let the interpolation point p 0 The coordinates are (x 0 ,y 0 ), whose pixel value T 0 Waiting for it. Click p 0 There are 4 pixels around, with coordinates P 1 , P 2 , P 3 , P 4 , the corresponding pixel values ​​are T1, T2, T3 and T4. Therefore, the corresponding relationship is as follows:

[0044] s=x 2 -x 1 ,t=y 2 -y 1

[0045]

[0046] Please refer to Figure 5, based on the FCN-VGG16 network, first perform bilinear interpolation on the Conv16 layer to obtain a double upsampling result, then fuse the result with the Pool4 layer feature to obtain the Fuse-Pool4 layer, and finally perform 16-fold deconvolution on the Fuse-Pool4 layer and output it to obtain the FCN-16s-VGG16 network. Similarly, based on the FCN-16s-VGG16 network, perform bilinear interpolation on the Fuse-Pool4 layer and fuse the features with the Pool3 layer to obtain the Fuse-Pool3 layer, perform 8-fold deconvolution on the Fuse-Pool3 layer feature map and output it to obtain the FCN-8s-VGG16 network, that is, the optimized FCN-VGG16 network.

[0047] Furthermore, the method of using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantic label image includes: determining the number of categories of the heat map of the optimized FCN-VGG16 network; determining the predicted probability that the monocular real-time image belongs to the target category; identifying the dynamic target of the monocular real-time image according to the number of categories of the heat map and the predicted probability; and performing semantic segmentation on the monocular real-time image according to the identified dynamic target to obtain the semantic label image.

[0048] Specifically, in the embodiment of the present invention, the number of categories of the heat map is determined according to the number of optimized FCN-VGG16 network heat maps, the predicted probability that the monocular real-time image belongs to the target category is determined, and the dynamic target of the monocular real-time image is identified according to the number of categories and the predicted probability of the heat map in the following manner:

[0049]

[0050] Among them, M is the number of categories in the heat map, C is the target category of the monocular real-time image, and y ic is a dummy parameter, p ic is the predicted probability that the monocular real-time image belongs to the target category C, L is the loss value, and then the dynamic target of the monocular real-time image is identified according to the size of the loss value. According to the identified dynamic target, the monocular real-time image is semantically segmented to obtain a semantic label image.

[0051] Further, the method of removing the dynamic feature points of the monocular real-time image according to the semantic segmentation result to obtain the static feature points of the monocular real-time image includes: performing image pyramid stratification on the monocular real-time image with a first scaling factor to obtain a plurality of first scaled images; performing image pyramid stratification on the semantic label image with a second scaling factor to obtain a plurality of second scaled images; sequentially removing the dynamic feature points in each first scaled image: performing ORB feature extraction on each first scaled image to obtain the ORB feature points of each first scaled image; outputting image coordinate values ​​of pixel points in the dynamic area of ​​each second scaled image to form a set U i , where i is the number of layers of the second zoomed image, and the dynamic area is the area where the dynamic target is located; the coordinate value of each ORB feature point of the first zoomed image is compared with the corresponding set U i in each first scaled image, remove the coordinate values ​​that are different from the set U i ORB feature points whose coordinate values ​​match those in the first zoomed image; forming a static feature point set M with the ORB feature points retained in each first zoomed image; estimating the position and posture of the camera according to the static feature points, including: estimating the position and posture of the camera according to the static feature point set M.

[0052] Specifically, in the embodiment of the present invention, a semantic label image of two categories of dynamic target and background is obtained through a fully convolutional neural network, a coordinate mapping method based on an image pyramid is used to remove feature points in the dynamic area, and a binary semantic label image is used to limit the dynamic area. The specific operations are as follows:

[0053] The monocular real-time image is layered by an image pyramid with a first scaling factor to obtain a multi-layer first scaling image, and the semantic label image is layered by an image pyramid with a second scaling factor to obtain a multi-layer second scaling image. Furthermore, the first scaling factor is the same as the second scaling factor. Then, the dynamic feature points in each first scaling image are removed in turn: the pixel coordinate values ​​of the dynamic area of ​​the second scaling image are output to form a set U i , extract ORB features from the first scaled image with the same number of layers as the second scaled image to obtain ORB feature points. Traverse all extracted ORB feature points and compare the coordinate values ​​of the ORB feature points with the corresponding set U i Match the coordinate values ​​in: For any ORB feature point P, if the corresponding set U i There is a point P′ in

[0054] P x =P′ x ∪P y =P ′ y

[0055] The ORB feature point P is considered to be a dynamic feature point and is removed. After completing the removal of a single layer of dynamic feature points, continue to remove the next layer. Then, the ORB feature points retained in each first zoomed image form a static feature point set M, and the camera pose is estimated based on the static feature point set M.

[0056] Furthermore, the coordinate values ​​of the ORB feature points are compared with the set U i Before matching the coordinate values ​​in , the method further includes: storing the ORB feature points.

[0057] Specifically, in the implementation manner of the present invention, due to the existence of a quadtree mechanism when extracting ORB feature points, coordinate matching needs to be performed after the ORB feature points are stored and converted into single-layer image coordinate values, otherwise static feature points will be lost due to the elimination of dynamic feature points.

[0058] Another aspect of the present invention provides a visual SLAM system based on a fully convolutional neural network in a dynamic scene, wherein the system is configured to use the visual SLAM method based on a fully convolutional neural network in a dynamic scene described above to estimate the position and posture of a camera. hereinafter represented by the ORB_SLAM2-FCN system.

[0059] The method and system provided by the present invention can accurately identify dynamic targets and complete semantic segmentation, effectively improve the accuracy and robustness of camera tracking, and improve the positioning and mapping accuracy of visual SLAM in dynamic scenes.

[0060] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0061] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0062] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A visual SLAM method based on a fully convolutional neural network in a dynamic scene, characterized in that: The method comprises: Get image dataset; Constructing a fully convolutional neural network model based on the image dataset; The fully convolutional neural network model is used to perform semantic segmentation on the monocular real-time image currently collected by the camera to obtain a semantically labeled image; wherein, the fully convolutional neural network model is used to perform semantic segmentation on the monocular real-time image currently collected by the camera to obtain a semantically labeled image, including: using a convolution kernel with the same input data size as the fully connected layer of the VGG16 network in the fully convolutional neural network model to convert the fully connected layer of the VGG16 network into a convolution layer to obtain an FCN-VGG16 network; optimizing the FCN-VGG16 network; using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantically labeled image; According to the semantic label image, the dynamic feature points of the monocular real-time image are eliminated to obtain the static feature points of the monocular real-time image; wherein, the dynamic feature points of the monocular real-time image are eliminated according to the semantic label image to obtain the static feature points of the monocular real-time image, including: performing image pyramid stratification on the monocular real-time image with a first scaling factor to obtain multiple layers of first scaled images; performing image pyramid stratification on the semantic label image with a second scaling factor to obtain multiple layers of second scaled images; sequentially eliminating the dynamic feature points in each: performing ORB feature extraction on each of the first scaled images to obtain the ORB feature points of each first scaled image; outputting image coordinate values ​​of the pixel points in the dynamic area of ​​each second scaled image to form a set U i , where i is the number of layers of the second zoomed image, and the dynamic area is the area where the dynamic target is located; the coordinate value of each ORB feature point of the first zoomed image is compared with the corresponding set U i in each first scaled image, remove the coordinate values ​​that are different from the set U i ORB feature points whose coordinate values ​​match those in the first zoomed image; forming a static feature point set M with the ORB feature points retained in each first zoomed image; estimating the position and posture of the camera according to the static feature points, including: estimating the position and posture of the camera according to the static feature point set M; The position and posture of the camera is estimated according to the static feature points.

2. The method according to claim 1, characterized in that The method further comprises: After each convolution calculation in the FCN-VGG16 network, a linear rectification function is used as an excitation function to perform sparsity processing on the previous convolution layer; A pooling operation is performed on each pooling layer in the FCN-VGG16 network.

3. The method according to claim 2, characterized in that The optimizing the FCN-VGG16 network includes: A skip structure is introduced to perform upsampling and fusion operations on each pooling layer in the FCN-VGG16 network to obtain the optimized FCN-VGG16 network.

4. The method according to claim 3, characterized in that The method of using the optimized FCN-VGG16 network to perform a binary classification operation of dynamic targets and backgrounds on the monocular real-time image to obtain the semantic label image includes: Determine the number of categories of the heat map of the optimized FCN-VGG16 network; Determining a predicted probability that the monocular real-time image belongs to a target category; Identify dynamic targets in the monocular real-time image according to the number of categories in the heat map and the predicted probability; The monocular real-time image is semantically segmented according to the identified dynamic target to obtain the semantic label image.

5. The method according to claim 4, characterized in that According to the number of categories of the heat map and the predicted probability, the dynamic target of the monocular real-time image is identified in the following manner: Wherein, M is the number of categories of the heat map, C is the target category of the monocular real-time image, and y ic is a dummy parameter, p ic is the predicted probability that the monocular real-time image belongs to the target category C, and L is the loss value; The dynamic target of the monocular real-time image is identified according to the size of the loss value.

6. The method according to claim 5, characterized in that The first scaling factor is the same as the second scaling factor.

7. The method according to claim 6, characterized in that The coordinate values ​​of the ORB feature points are compared with the set U i Before matching the coordinate values ​​in , it also includes: The ORB feature points are stored.

Images