A dynamic environment visual SLAM method based on previous frame memory and DCP network layer

Abstract

The present invention relates to a dynamic environment visual SLAM method based on previous frame memory and DCP network layer, which solves the problem in the prior art that due to the spatial uncertainty of dynamic feature points in a dynamic environment, scale offset is caused by posture estimation, making it difficult to ensure the estimation accuracy of the system in a high dynamic environment. The method includes: acquiring a video image of a dynamic environment, inputting a trained semantic detection model into a current frame image, and obtaining a semantic annotation value of each pixel point in the current frame image; extracting feature points of the current frame image and preprocessing to obtain motion pixel points of the current frame image; obtaining a motion pixel point set of the current frame image based on the motion pixel points of the current frame image and the semantic annotation values ​​of each pixel point of the memory module; fusing and comparing the semantic annotation values ​​of each pixel point of the current frame image with the motion pixel point set of the current frame image, obtaining the dynamic feature points of the current frame image and removing them, and completing the dynamic feature optimization of each frame image.

Description

Technical Field

[0001] The invention belongs to the field of simultaneous positioning and map construction, and in particular relates to a dynamic environment visual SLAM method based on previous frame memory and a DCP network layer. Background Art

[0002] Vision-based simultaneous localization and mapping (VSLAM) is a key technology for mobile robots to perceive environmental information in unknown environments. In the absence of prior information about the environment, VSLAM estimates its own posture during motion through visual sensors and builds an environmental model. When a dynamic object appears in the camera's field of view, the VSLAM system incorporates instantaneous dynamic feature points and static feature points into the system as prior conditions for positioning and mapping the next frame. Due to the spatial uncertainty of dynamic feature points, posture estimation based on these points may lead to scale offset, making it difficult to guarantee the estimation accuracy of the system in a highly dynamic environment.

[0003] The mainstream method in existing SLAM algorithms in dynamic environments is to add dynamic areas to the environment map and perform dynamic target detection and elimination. However, when using the YOLO target detection network to eliminate dynamic moving objects, it only relies on semantic information and does not integrate optical flow information. When the neural network recognition accuracy is poor, dynamic feature points cannot be completely eliminated. Although the semantic SLAM system integrates semantic and optical flow information, the semantic labels in the system are only applied to the current frame, failing to maximize the use of semantic information. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a dynamic environment visual SLAM method based on previous frame memory and DCP network layer, so as to solve the problem that there is spatial uncertainty in dynamic feature points in the existing dynamic environment, and posture estimation based on these points leads to scale offset, making it difficult to ensure the estimation accuracy of the system in a high dynamic environment.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] The present invention provides a dynamic environment visual SLAM method based on previous frame memory and DCP network layer, comprising: acquiring a video image of a dynamic environment, sequentially using the following steps for each frame image in the video image to obtain dynamic feature points of each frame image and remove them, thereby completing dynamic feature optimization of each frame image:

[0007] Input the current frame image into the trained semantic detection model to obtain the semantic annotation value of each pixel in the current frame image; wherein the semantic annotation value is the probability value of the pixel being a dynamic point;

[0008] Extract feature points of the current frame image and perform preprocessing to obtain motion pixel points of the current frame image;

[0009] Based on the motion pixel points of the current frame image and the semantic annotation values ​​of each pixel point of the memory module, a motion pixel point set of the current frame image is obtained; wherein the memory module stores the semantic annotation values ​​of each pixel point in the previous frame image;

[0010] The semantic annotation value of each pixel point of the current frame image is fused and compared with the motion pixel point set of the current frame image to obtain the dynamic feature points of the current frame image.

[0011] Furthermore, the semantic detection model includes a feature extraction module, a feature fusion module and a DCP network module; wherein,

[0012] The current frame image passes through the feature extraction module to extract the basic features of the current frame image to obtain the feature extraction module output image;

[0013] The output image of the feature extraction module is subjected to feature fusion refinement and restoration of the original resolution through the feature fusion module to obtain an output image of the feature fusion module;

[0014] The output image of the feature fusion module passes through the DCP network module to generate a feature map, and then obtains the semantic annotation value of each pixel in the current frame image through the softmax layer.

[0015] Furthermore, the current frame image is extracted by the feature extraction module to extract basic features of the current frame image, including:

[0016] The current frame image is subjected to convolution and pooling operations in the first downsampling unit of the feature extraction module in sequence through a convolution layer, a batch normalization layer, and a pooling layer with a step size of 2 to obtain a first downsampling output image and generate a first pooling index;

[0017] The first down-sampled output image is sequentially subjected to convolution and pooling operations by a convolution layer with a step size of 2, a batch normalization layer, and a pooling layer in a second down-sampling unit of the feature extraction module to obtain a second down-sampled output image and generate a second pooling index;

[0018] The second down-sampling output image is sequentially subjected to convolution and pooling operations through a convolution layer with a step size of 2, two batch normalization layers, and a pooling layer in the third down-sampling unit of the feature extraction module to obtain the feature extraction module output image and generate a third pooling index.

[0019] Furthermore, the output graph of the feature extraction module is passed through the feature fusion module to obtain a feature fusion output graph, including:

[0020] The output image of the feature extraction module is sequentially passed through the upsampling layer and three convolution layers in the first upsampling unit of the feature fusion module to obtain a first upsampling output image; wherein the upsampling layer in the first upsampling unit uses the third pooling index for upsampling;

[0021] The first up-sampled output image is sequentially passed through an up-sampling layer and two convolutional layers in a second up-sampling unit of the feature fusion module to obtain a second up-sampled output image; wherein the up-sampling layer in the second up-sampling unit is up-sampled using a second pooling index;

[0022] The second upsampling output image is sequentially passed through the upsampling layer and two convolutional layers in the third upsampling unit of the feature fusion module to obtain the feature fusion module output image; wherein, the upsampling layer in the third upsampling unit uses the first pooling index for upsampling.

[0023] Furthermore, the output graph of the feature fusion module is passed through the DCP network module to obtain a feature mapping graph, including:

[0024] The feature fusion module output image is spliced ​​with the first down-sampling output image and the second down-sampling output image to obtain a spliced ​​feature image;

[0025] Input the spliced ​​feature map into a three-layer dilated convolution pyramid to obtain a first dilated convolution feature map, a second dilated convolution feature map, and a third dilated convolution feature map, respectively; wherein the first layer of the three-layer dilated convolution pyramid is a basic convolution layer with a convolution kernel of 3×3, the second layer is a low dilated rate convolution layer with a dilation rate of 2 superimposed on the basic convolution layer, and the third layer is a medium dilated rate convolution layer with a dilation rate of 2 superimposed on the low dilated rate convolution layer;

[0026] The first atrous convolution feature map, the second atrous convolution feature map, and the third atrous convolution feature map are respectively passed through a convolution layer with a convolution kernel of 1×1 and batch normalized and then superimposed to obtain a feature map.

[0027] Furthermore, extracting feature points of the current frame image and performing preprocessing to obtain motion pixel points of the current frame image includes:

[0028] Extract feature points of the current frame image;

[0029] Based on the feature points, the optical flow method is used to obtain the feature points corresponding to the previous frame image;

[0030] Based on any 8 pairs of matching feature points, the basic matrix is ​​calculated;

[0031] Based on the basic matrix, the epipolar lines of the feature points of the previous frame image in the current frame image are obtained;

[0032] The distance between each feature point of the current frame image and the corresponding epipolar line is calculated, and when the distance is greater than a preset distance value, the feature point is determined to be a moving pixel point, otherwise, the feature point is determined to be a stationary pixel point.

[0033] Furthermore, obtaining the motion pixel point set of the current frame image includes:

[0034] Based on the semantic annotation value of each pixel in the previous frame image in the memory module, the dynamic point in the previous frame image is obtained;

[0035] Based on the dynamic point in the previous frame image, the optical flow method is used to obtain the pixel point in the current frame image as the memory dynamic point;

[0036] The memorized dynamic points and the motion pixel points of the current frame image constitute a motion pixel point set of the current frame image.

[0037] Furthermore, when the semantic annotation value of a pixel point is greater than a preset value, it is determined to be a dynamic point.

[0038] Furthermore, the semantic annotation value of each pixel point of the current frame image is fused and compared with the motion pixel point set of the current frame image to obtain the dynamic feature points of the current frame image, including:

[0039] The following formula is used to fuse and compare the semantic annotation value of each pixel point of the current frame image with the motion pixel point set of the current frame image to obtain the final semantic state value of each pixel point of the current frame image:

[0040] L c (x,y)=W p S U (x,y)+L S (x,y)

[0041] Among them, L c (x, y) represents the final semantic state value of the pixel point (x, y); W p represents semantic weight; S U (x, y) represents the motion pixel discriminant function of the pixel point (x, y); L S (x,y) represents the semantic annotation value of the pixel point (x,y);

[0042] When the final semantic state value of a pixel point is greater than a preset value, the pixel point is determined to be a dynamic feature point.

[0043] Furthermore, the fusing and comparing the semantic annotation value of each pixel point of the current frame image with the motion pixel point set of the current frame image also includes: storing the semantic annotation value of each pixel point of the current frame image in the memory module.

[0044] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0045] 1. This application adopts a semantic detection model based on the DCP network layer in the target detection network. By fusing the dilated convolutions of large and small receptive fields, the network model can obtain multi-scale feature information while reducing the loss of feature information, thereby improving the efficiency and accuracy of dynamic object recognition, thereby improving the accuracy of visual SLAM dynamic object recognition in dynamic environments.

[0046] 2. This application pre-identifies the dynamic outlier points of the current frame by implanting a previous frame memory module, and fuses them with the dynamic feature points determined by the semantic feature map of the current frame to generate a dynamic outlier point set, thereby reducing the dynamic outlier points missed due to semantic weakening, reducing the pose estimation error of the SLAM system in a dynamic environment, and improving positioning accuracy.

[0047] 3. In the semantic detection model of the present application, the DCP network layer uses the additivity of convolution to add three layers of dilated convolutions symmetrically to the convolution kernel with the largest dilation rate in a symmetrical position, forming a dilated convolution pyramid. Compared with the traditional method of first convolution and then pooling to increase the receptive field, the dilated convolution can obtain a larger receptive field without pooling, so that each convolution output contains a larger range of information, achieving a larger receptive field with fewer parameters.

[0048] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0050] Figure 1 Schematic diagram of a flow chart of a dynamic environment visual SLAM method based on previous frame memory and DCP network layer in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the structure of a semantic detection model in an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the process of the DCP network module of the semantic detection model in an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of the structure of the dilated convolutional pyramid of the DCP network module in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0055] A specific embodiment of the present invention discloses a dynamic environment visual SLAM method based on previous frame memory and DCP network layer, comprising: acquiring a video image of a dynamic environment, sequentially applying the following steps S1 to S4 to each frame image in the video image, obtaining dynamic feature points of each frame image and removing them, completing dynamic feature optimization of each frame image, thereby improving the robustness and stability of the SLAM system in a high dynamic environment.

[0056] Specifically, in the process of vision-based simultaneous positioning and map building, a video image of a dynamic environment is obtained by a visual sensor, and each frame of the video image can be read in sequence using the open source OpenCV.

[0057] Further, such as Figure 1 As shown, steps S1 to S4 include:

[0058] Step S1, input the current frame image into the trained semantic detection model to obtain the semantic annotation value of each pixel in the current frame image; wherein the semantic annotation value is the probability value of the pixel being a dynamic point.

[0059] Further, such as Figure 2 As shown, the semantic detection model includes a feature extraction module, a feature fusion module and a DCP network module; wherein,

[0060] The current frame image passes through the feature extraction module to extract the basic features of the current frame image to obtain the feature extraction module output image.

[0061] Furthermore, the current frame image is subjected to convolution and pooling operations in the first downsampling unit of the feature extraction module, in sequence, through a convolution layer with a step size of 2, a batch normalization layer, and a pooling layer to obtain a first downsampling output image and generate a first pooling index.

[0062] The first down-sampled output image is sequentially subjected to convolution and pooling operations by a convolution layer with a step size of 2, a batch normalization layer, and a pooling layer in the second down-sampling unit of the feature extraction module to obtain a second down-sampled output image and generate a second pooling index.

[0063] The second down-sampling output image is sequentially subjected to convolution and pooling operations through a convolution layer with a step size of 2, two batch normalization layers, and a pooling layer in the third down-sampling unit of the feature extraction module to obtain the feature extraction module output image and generate a third pooling index.

[0064] Specifically, the convolution layer in each downsampling unit in the feature extraction module uses a group of small convolution kernels to learn basic features (such as edges, textures, etc.), laying the foundation for the subsequent learning of complex features. The convolution layer is usually accompanied by an activation function (such as ReLU) to introduce nonlinear characteristics.

[0065] The batch normalization layer + activation function after the convolution layer is used to normalize and excite the extracted features to ensure the stability of the model and avoid overfitting. It should be noted that the number of batch normalization layers is increased in the third downsampling unit of the model to accelerate learning convergence and alleviate the gradient vanishing problem. In deep networks, gradient vanishing is a common problem, especially in the early layers of the network. By applying batch normalization to these layers, the activation values ​​can be rescaled to help the gradient backpropagate more effectively, thereby alleviating this problem.

[0066] The pooling layer after the batch normalization layer is used to reduce the spatial dimension of the feature map while maintaining important information of the features. It can maintain or increase the receptive field while reducing the amount of computation and capture a wider range of contextual information.

[0067] On the other hand, the pooling index generated in each downsampling unit means that for the maximum pooling operation, each output pixel is recorded from which position of the input feature map (that is, which element in which pooling window) the maximum value is selected. That is, the pooling index saves the position information of the mapping from input to output. In each upsampling layer of the feature extraction module, based on these indexes, a high-resolution version of the feature map can be accurately traced back or reconstructed.

[0068] Furthermore, the output image of the feature extraction module is passed through the feature fusion module to perform feature fusion refinement and restore the original resolution to obtain the output image of the feature fusion module.

[0069] Furthermore, the output image of the feature extraction module is sequentially passed through the upsampling layer and three convolution layers in the first upsampling unit of the feature fusion module to obtain a first upsampling output image; wherein the upsampling layer in the first upsampling unit uses the third pooling index for upsampling;

[0070] The first up-sampled output image is sequentially passed through an up-sampling layer and two convolutional layers in a second up-sampling unit of the feature fusion module to obtain a second up-sampled output image; wherein the up-sampling layer in the second up-sampling unit is up-sampled using a second pooling index;

[0071] The second upsampling output image is sequentially passed through the upsampling layer and two convolutional layers in the third upsampling unit of the feature fusion module to obtain the feature fusion module output image; wherein, the upsampling layer in the third upsampling unit uses the first pooling index for upsampling.

[0072] Specifically, the upsampling layer is used to gradually restore the resolution of the feature map to a size close to the input image size, in preparation for generating a high-resolution output.

[0073] The convolution layer in each upsampling unit is used for feature fusion and refinement. It should be noted that in the upsampling process, it does not add new feature information itself, but amplifies the existing feature map through interpolation and other methods. The convolution layer that follows can further process and learn these amplified features to extract finer features, which helps to improve the detail quality and accuracy of the output. Upsampling corresponds to the previous downsampling (max pooling) operation and forms a symmetrical structure. After downsampling, a convolution layer is usually used to extract features. After upsampling, the convolution layer can also be used to restore and supplement these features to achieve feature transfer and fusion.

[0074] In this embodiment, three convolutional layers are used in the first upsampling unit, and two convolutional layers are used in the second upsampling unit and the third upsampling unit, respectively. The reduction in the number of convolutional layers here is used to reduce the burden of calculation. Because the upsampling layer will increase the spatial size of the feature map. As the size of the feature map increases, the computing resources required to process the same receptive field will increase accordingly. In order to balance the computational cost and model complexity, the number of subsequent convolutional layers is reduced, especially in the part close to the output layer, to avoid excessive memory usage and computational burden. At the same time, during the upsampling process, there are often jump connections or feature fusion operations to combine the upsampled features with the features of the corresponding coding layer. These features from the coding layer have been processed by multiple layers of convolution and contain rich local information. Therefore, when operating on the basis of these features, good reconstruction or prediction effects can be achieved without too many additional convolutional layers.

[0075] Furthermore, the feature fusion module output image, the first down-sampling output image and the second down-sampling output image are passed through the DCP network module to generate a feature map, and then the semantic annotation value of each pixel in the current frame image is obtained through the softmax layer.

[0076] Specifically, the DCP network module is used to enhance the receptive field of the model, which helps to capture local and global information in the image and can better adapt to changes in the size and position of objects. Through the DCP network module, feature maps at multiple resolution levels can be generated for subsequent classification tasks.

[0077] Further, the process of the DCP network module is as follows Figure 3 As shown, it includes steps S11 to S13:

[0078] Step S11, the feature fusion module output image is spliced ​​with the first down-sampling output image and the second down-sampling output image to obtain a spliced ​​feature image.

[0079] Specifically, the output image of the feature fusion module is fused with the first down-sampled output image and the second down-sampled output image transmitted via the jump connection, thereby retaining the image feature information lost in the pooling process and increasing the retention of input features. The feature maps of different sizes obtained by the jump connection are expanded by zero padding at the edges to make the sizes of each atlas consistent, and then spliced ​​and fused with the output image of the feature fusion module to obtain a spliced ​​feature map.

[0080] It should be noted that the edge zero padding is to ensure the consistency of the size of feature maps at different levels, because upsampling will cause the image to become larger, while pooling will cause the image to become smaller. In order to unify the size, blank spaces need to be added outside the small-sized feature map to maintain a uniform size.

[0081] Step S12, input the spliced ​​feature map into a three-layer dilated convolution pyramid to obtain a first dilated convolution feature map, a second dilated convolution feature map and a third dilated convolution feature map, respectively; wherein the first layer of the three-layer dilated convolution pyramid is a basic convolution layer with a convolution kernel of 3×3, the second layer is a low dilated rate convolution layer with a dilation rate of 2 superimposed on the basic convolution layer, and the third layer is a medium dilated rate convolution layer with a dilation rate of 2 superimposed on the low dilated rate convolution layer.

[0082] Specifically, Figure 4 The dilated convolution pyramid structure shown in the figure uses the additivity of convolution to add three layers of dilated convolution symmetrically to the convolution kernel with the largest dilation rate to form a dilated convolution pyramid. Dilated convolution injects holes into the convolution kernel of the standard convolution and uses the dilation rate to control the size of the convolution kernel, thereby increasing the receptive field without increasing network parameters and computational scale. The dilation rate refers to the number of parameter intervals in the convolution kernel. Compared with the traditional method of increasing the receptive field by convolution first and then pooling, dilated convolution can obtain a larger receptive field without pooling, so that each convolution output contains a larger range of information.

[0083] Step S13: The first atrous convolution feature map, the second atrous convolution feature map, and the third atrous convolution feature map are respectively passed through a convolution layer with a convolution kernel of 1×1 and batch normalized and then superimposed to obtain a feature map.

[0084] Specifically, after the three feature maps of the dilated convolution pyramid, corresponding weights are assigned to the feature maps under different dilated convolutions through a 1x1 convolution kernel, and batch normalization is used to reduce the risk of network overfitting and accelerate network gradient back propagation.

[0085] Finally, the batch normalization results are superimposed to obtain a feature map containing deep and shallow features.

[0086] It should be noted that the semantic detection model needs to be trained in advance, wherein the training data set includes: sample images and corresponding labels of whether the pixels in the sample images are dynamic or static. The images of the training data set are loaded into the semantic detection model, and the cross entropy loss function is used as the loss function to train the semantic detection model. Through the back propagation and gradient descent optimization algorithms, the model can continuously adjust the parameters to minimize the loss function. After the training, the parameters of the semantic detection model are saved.

[0087] Preferably, the trained semantic detection model can also use network pruning to directly remove the corresponding channels of some convolutional layers and batch normalization layers of the model, eliminate relatively redundant parts in the model, achieve the purpose of lightweight network model, and accelerate the optimization and operation process of the overall model.

[0088] Specifically, network pruning is an effective method for model compression. It reduces the amount of computation and parameters of the model by removing weights or entire channels that contribute less to the output in the network, thereby speeding up the model and saving storage space. The steps for channel pruning for the convolutional layer and the corresponding batch normalization layer are as follows:

[0089] First, evaluate the importance of the channel.

[0090] Specifically, for the convolution layer in the model, the L2 norm of the weight of each convolution kernel (or output channel) is calculated. A larger L2 norm means that the channel contributes more to the model output. For the batch normalization layer, its γ (scale) parameter is calculated. A channel with a larger γ value indicates that the channel has a significant amplification effect on the feature and is often considered to be a more important channel. On the contrary, a channel with a γ value close to 0 may contribute less.

[0091] Secondly, determine the pruning ratio and threshold.

[0092] Specifically, according to the requirements of model compression, a pruning ratio is set, and for example, the ratio may be 50%. Based on the importance of the channels obtained above, a threshold is set, and channels below the threshold are marked as channels to be removed.

[0093] Finally, the pruning operation is performed.

[0094] Specifically, for the marked channels to be removed, in the convolution layer, the selected channels are directly removed, and the corresponding rows and columns are removed; in the batch normalization layer, the corresponding channel information needs to be removed synchronously, including the mean, variance, γ (scaling factor) and β (offset factor) of the corresponding channels to complete the pruning operation.

[0095] Furthermore, the last Softmax layer of the model is a multi-class classifier, which represents the probability of each class as a real-valued vector, where each element is between 0 and 1, and the sum of all elements is equal to 1. The Softmax layer is usually used in the last step of a multi-class classification task to convert the output of the model into a probability distribution. In this embodiment, the output of the Softmax layer is the probability value of the pixel point in the current frame image belonging to a dynamic point.

[0096] Step S2: extracting feature points of the current frame image and performing preprocessing to obtain moving pixel points of the current frame image.

[0097] Further, the method includes the following steps S21 to S25:

[0098] Step S21: extracting feature points of the current frame image.

[0099] Specifically, the feature points of the image are points with unique local properties in the image, that is, "they can represent the image or object in the same or at least very similar invariant form in other similar images containing the same scene or object". The feature point extraction methods include SURF feature detector, SIFT feature detector, FAST feature detector and ORB feature detector.

[0100] Step S22: Based on the feature points, use the optical flow method to obtain the feature points corresponding to the previous frame image.

[0101] Specifically, optical flow is the apparent motion pattern of an image object between two consecutive frames caused by the movement of an object or camera, showing the movement of a point from the current frame to the next frame. In this embodiment, the Lucas-Kanade optical flow is used to iteratively track the feature points using the function cv.calcOpticalFlowPyrLK() in OpenCV.

[0102] Step S23: Based on any 8 pairs of matching feature points, a basic matrix is ​​calculated.

[0103] Specifically, the function cv.findFundamentalMat() in OpenCV is used to calculate the basic matrix in the previous frame image and the current frame image.

[0104] Step S24: based on the basic matrix, obtain the epipolar lines of the feature points of the previous frame image in the current frame image.

[0105] Specifically, the following formula is used to calculate the epipolar lines of each feature point of the previous frame image in the current frame image:

[0106] l 2i =[X i ,Y i ,Z i ] T =FP 1i =F[u 1i ,v 1i ,1] T

[0107] Among them, l 2i Indicates the epipolar line of the i-th feature point of the previous frame image in the current frame; X i , Y i , Z i represents the normal vector of the epipolar line of the i-th feature point of the previous frame image; F represents the basic matrix; u 1i 、v 1i Represents the coordinate value of the i-th feature point in the previous frame image.

[0108] Step S25, calculating the distance between each feature point of the current frame image and the corresponding epipolar line, and determining that the feature point is a moving pixel point when the distance is greater than a preset distance value, otherwise determining that the feature point is a stationary pixel point.

[0109] Specifically, the distance between each feature point of the current frame image and the corresponding epipolar line is calculated using the following formula:

[0110]

[0111] Among them, D j represents the distance between the jth feature point of the current frame image and the corresponding epipolar line; P 2j represents the coordinates of the jth feature point of the current frame image; P 1j Indicates the coordinates of the jth feature point in the previous frame image; X j , Y j Represents the normal vector of the epipolar line corresponding to the j-th feature point in the current frame image.

[0112] Step S3, obtaining a set of motion pixels of the current frame image based on the motion pixels of the current frame image and the semantic annotation values ​​of each pixel in the memory module; wherein the memory module stores the semantic annotation values ​​of each pixel in the previous frame image.

[0113] Furthermore, based on the semantic annotation value of each pixel in the previous frame image in the memory module, the dynamic point in the previous frame image is obtained.

[0114] Specifically, when the semantic annotation value of a pixel point is greater than a preset value, it is determined to be a dynamic point. Exemplarily, the preset value is 0.5.

[0115] Based on the dynamic point in the previous frame image, the optical flow method is used to obtain its pixel point in the current frame image as the memory dynamic point.

[0116] The memorized dynamic points and the motion pixel points of the current frame image constitute a motion pixel point set of the current frame image.

[0117] Step S4: fusing and comparing the semantic annotation value of each pixel point of the current frame image with the motion pixel point set of the current frame image to obtain the dynamic feature points of the current frame image.

[0118] Furthermore, the semantic annotation value of each pixel of the current frame image is fused and compared with the motion pixel set of the current frame image using the following formula to obtain the final semantic state value of each pixel of the current frame image:

[0119] L c (x,y)=W p S U (x,y)+L S (x,y)

[0120] Among them, L c (x, y) represents the final semantic state value of the pixel point (x, y); W p represents the semantic weight, which can be set to 0.5 for example; S U (x, y) represents the motion pixel discriminant function of the pixel point (x, y); L S (x,y) represents the semantic annotation value of the pixel point (x,y).

[0121] Specifically, S. U It represents the motion pixel set of the current frame obtained based on the memory module and the optical flow information. When the pixel of the current frame exists in the dynamic abnormal point set, S U The value of (x, y) is 1. When the pixel point of the current frame does not exist in the dynamic outlier point set, S U The value of (x,y) is 0.

[0122] When the final semantic state value of a pixel is greater than a preset value, the pixel is determined to be a dynamic feature point. Exemplarily, the preset value may be set to 1.

[0123] Furthermore, after the semantic annotation value of each pixel point of the current frame image is stored in the memory module as the previous frame semantics of the next frame image, the next frame image is processed.

[0124] In summary, a dynamic environment visual SLAM method based on previous frame memory and DCP network layer according to an embodiment of the present invention has the following beneficial effects:

[0125] 1. This application adopts a semantic detection model based on the DCP network layer in the target detection network. By fusing the dilated convolutions of large and small receptive fields, the network model can obtain multi-scale feature information while reducing the loss of feature information, thereby improving the efficiency and accuracy of dynamic object recognition, thereby improving the accuracy of visual SLAM dynamic object recognition in dynamic environments.

[0126] 2. This application pre-identifies the dynamic outlier points of the current frame by implanting a previous frame memory module, and fuses them with the dynamic feature points determined by the semantic feature map of the current frame to generate a dynamic outlier point set, thereby reducing the dynamic outlier points missed due to semantic weakening, reducing the pose estimation error of the SLAM system in a dynamic environment, and improving positioning accuracy.

[0127] 3. In the semantic detection model of the present application, the DCP network layer uses the additivity of convolution to add three layers of dilated convolutions symmetrically to the convolution kernel with the largest dilation rate in a symmetrical position, forming a dilated convolution pyramid. Compared with the traditional method of first convolution and then pooling to increase the receptive field, the dilated convolution can obtain a larger receptive field without pooling, so that each convolution output contains a larger range of information, achieving a larger receptive field with fewer parameters.

[0128] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A dynamic environment visual SLAM method based on previous frame memory and DCP network layer, characterized in that: include: Obtain a video image of a dynamic environment, and use the following steps to obtain dynamic feature points of each frame image in the video image in turn and remove them to complete the dynamic feature optimization of each frame image: Input the current frame image into the trained semantic detection model to obtain the semantic annotation value of each pixel in the current frame image; wherein the semantic annotation value is the probability value of the pixel belonging to a dynamic point; the semantic detection model includes a feature extraction module, a feature fusion module and a DCP network module; the current frame image passes through the feature extraction module to extract the basic features of the current frame image to obtain the feature extraction module output image; the feature extraction module output image passes through the feature fusion module to perform feature fusion refinement and restore the original resolution to obtain the feature fusion module output image; the feature fusion module output image passes through the DCP network module to generate a feature map and then passes through the softmax layer to obtain the semantic annotation value of each pixel in the current frame image; Extract feature points of the current frame image and perform preprocessing to obtain motion pixel points of the current frame image; Based on the motion pixel points of the current frame image and the semantic annotation values ​​of each pixel point of the memory module, a motion pixel point set of the current frame image is obtained; wherein the memory module stores the semantic annotation values ​​of each pixel point in the previous frame image; The semantic annotation value of each pixel point of the current frame image is fused and compared with the motion pixel point set of the current frame image to obtain the dynamic feature point of the current frame image, including: using the following formula to fuse and compare the semantic annotation value of each pixel point of the current frame image with the motion pixel point set of the current frame image to obtain the final semantic state value of each pixel point of the current frame image: L c (x,y)=W p S U (x,y)+L S (x,y) Among them, L c (x, y) represents the final semantic state value of the pixel point (x, y); W p represents semantic weight; S U (x, y) represents the motion pixel discriminant function of the pixel point (x, y); L S (x, y) represents the semantic annotation value of the pixel point (x, y); when the final semantic state value of the pixel point is greater than a preset value, the pixel point is judged to be a dynamic feature point.

2. The method according to claim 1, characterized in that: The current frame image is extracted by the feature extraction module to extract basic features of the current frame image, including: The current frame image is subjected to convolution and pooling operations in the first downsampling unit of the feature extraction module in sequence through a convolution layer, a batch normalization layer, and a pooling layer with a step size of 2 to obtain a first downsampling output image and generate a first pooling index; The first down-sampled output image is sequentially subjected to convolution and pooling operations by a convolution layer with a step size of 2, a batch normalization layer, and a pooling layer in a second down-sampling unit of the feature extraction module to obtain a second down-sampled output image and generate a second pooling index; The second down-sampling output image is sequentially subjected to convolution and pooling operations through a convolution layer with a step size of 2, two batch normalization layers, and a pooling layer in the third down-sampling unit of the feature extraction module to obtain the feature extraction module output image and generate a third pooling index.

3. The method according to claim 2, characterized in that: The output graph of the feature extraction module passes through the feature fusion module to obtain a feature fusion output graph, including: The output image of the feature extraction module is sequentially passed through the upsampling layer and three convolution layers in the first upsampling unit of the feature fusion module to obtain a first upsampling output image; wherein the upsampling layer in the first upsampling unit uses the third pooling index for upsampling; The first up-sampled output image is sequentially passed through an up-sampling layer and two convolutional layers in a second up-sampling unit of the feature fusion module to obtain a second up-sampled output image; wherein the up-sampling layer in the second up-sampling unit is up-sampled using a second pooling index; The second upsampling output image is sequentially passed through the upsampling layer and two convolutional layers in the third upsampling unit of the feature fusion module to obtain the feature fusion module output image; wherein, the upsampling layer in the third upsampling unit uses the first pooling index for upsampling.

4. The method according to claim 3, characterized in that: The output graph of the feature fusion module is passed through the DCP network module to obtain a feature map, including: The feature fusion module output image is spliced ​​with the first down-sampling output image and the second down-sampling output image to obtain a spliced ​​feature image; Input the spliced ​​feature map into a three-layer dilated convolution pyramid to obtain a first dilated convolution feature map, a second dilated convolution feature map, and a third dilated convolution feature map, respectively; wherein the first layer of the three-layer dilated convolution pyramid is a basic convolution layer with a convolution kernel of 3×3, the second layer is a low dilated rate convolution layer with a dilation rate of 2 superimposed on the basic convolution layer, and the third layer is a medium dilated rate convolution layer with a dilation rate of 2 superimposed on the low dilated rate convolution layer; The first atrous convolution feature map, the second atrous convolution feature map, and the third atrous convolution feature map are respectively passed through a convolution layer with a convolution kernel of 1×1 and batch normalized and then superimposed to obtain a feature map.

5. The method according to claim 1, characterized in that: Extract the feature points of the current frame image and perform preprocessing to obtain the motion pixel points of the current frame image, including: Extract feature points of the current frame image; Based on the feature points, the optical flow method is used to obtain the feature points corresponding to the previous frame image; Based on any 8 pairs of matching feature points, the basic matrix is ​​calculated; Based on the basic matrix, the epipolar lines of the feature points of the previous frame image in the current frame image are obtained; The distance between each feature point of the current frame image and the corresponding epipolar line is calculated, and when the distance is greater than a preset distance value, the feature point is determined to be a moving pixel point, otherwise, the feature point is determined to be a stationary pixel point.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a moving pixel point set of the current frame image includes: Based on the semantic annotation value of each pixel in the previous frame image in the memory module, the dynamic point in the previous frame image is obtained; Based on the dynamic point in the previous frame image, the optical flow method is used to obtain the pixel point in the current frame image as the memory dynamic point; The memorized dynamic points and the motion pixel points of the current frame image constitute a motion pixel point set of the current frame image.

7. The method according to claim 6, characterized in that: When the semantic annotation value of a pixel point is greater than a preset value, it is determined to be a dynamic point.

8. The method according to claim 1, characterized in that: The fusing and comparing the semantic annotation value of each pixel point of the current frame image with the motion pixel point set of the current frame image also includes: storing the semantic annotation value of each pixel point of the current frame image in the memory module.

Images