A mapping method suitable for dynamic environments

Abstract

The application discloses a mapping method suitable for a dynamic environment, and comprises the following steps: acquiring dynamic frame data; inputting the dynamic frame data into a neural network model for feature extraction to obtain prior knowledge; performing feature point screening on the dynamic frame data according to the prior knowledge to retain static feature points; and performing mapping according to the static feature points; the application filters out dynamic objects by screening static feature points according to prior knowledge, and realizes mapping by static features, thereby avoiding errors caused by dynamic objects in the dynamic environment and improving mapping effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly to a mapping method for dynamic environment. BACKGROUND

[0002] Visual SLAM assumes that the surrounding environment is static during the mapping process, however, there are a large number of dynamic objects in the real scene, such as moving vehicles, bicycles and pedestrians, etc. These dynamic objects lead to false epipolar constraints, which makes the mapping effect of visual SLAM poor in dynamic environment.

[0003] Therefore, how to provide a mapping method suitable for dynamic environment can reduce the mapping error in outdoor dynamic environment, which is a problem that the person skilled in the art needs to solve. SUMMARY

[0004] Therefore, the present application provides a mapping method suitable for dynamic environment, which filters out dynamic objects by screening static feature points based on prior knowledge, and realizes mapping by static features, thereby avoiding the error caused by dynamic objects in dynamic environment and improving the mapping effect.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0006] A mapping method suitable for dynamic environment, comprising the following steps:

[0007] Obtaining dynamic frame data;

[0008] Inputting the dynamic frame data into a neural network model for feature extraction to obtain prior knowledge;

[0009] Screening feature points from the dynamic frame data according to the prior knowledge, and retaining static feature points;

[0010] Mapping according to the static feature points.

[0011] Further, the prior knowledge includes the number of dynamic objects, the dynamic object picture proportion, and the range mark and coordinate information of each dynamic object.

[0012] Further, the feature point screening step includes:

[0013] The current feature points within the range mark are subjected to secondary judgment according to the current frame prior dynamic object picture proportion to obtain static feature points.

[0014] Further, the secondary judgment is specifically:

[0015] Setting a first threshold and a second threshold;

[0016] When the prior dynamic object picture proportion of the current frame is less than a first threshold value and the prior dynamic object quantity of the current frame is less than a second threshold value, the feature points in the range marker are current feature points in the dynamic frame data,

[0017] The current feature points are obtained, and it is determined whether the current feature points are located in the range marker of the prior dynamic object. The current feature points not located in the range marker are considered as static feature points, otherwise, three times of judgment are performed through a geometric method.

[0018] Further, the three times of judgment are performed through a geometric method, specifically:

[0019] An element layer in the periphery of the range marker is selected as an inflation layer;

[0020] When there is no feature point in the inflation layer, it is determined that the feature points in the range marker are static layers;

[0021] When there is a feature point in the inflation layer, the feature points in the range marker of the current frame and the feature points of the inflation layer are matched with the previous frame, an essential matrix is calculated through epipolar geometry, and static feature points are screened according to the essential matrix.

[0022] Further, the static feature points are screened according to the essential matrix, specifically:

[0023] The infinity norm of the essential matrix is calculated, wherein the essential matrix includes a range marker essential matrix E1 and an inflation layer essential matrix E2;

[0024] The ratio r of the infinity norm L1 of the range marker essential matrix E1 to the infinity norm L2 of the inflation layer essential matrix E2 is calculated:

[0025]

[0026] The static feature points are screened according to the ratio r.

[0027] Further, the static feature points are screened according to the ratio r, specifically:

[0028] A threshold value is set. When the ratio r is greater than the threshold value, the prior dynamic object in the range marker is considered as static. When the ratio is less than the threshold value, all feature points of the current frame are matched through binocular feature points, and the best matching points are screened as static feature points.

[0029] Further, the dynamic frame data is input into a neural network model for feature extraction to obtain prior knowledge, specifically:

[0030] An attention mechanism is introduced in the dynamic frame data to obtain an attention weighted feature map;

[0031] Prediction is performed based on the weighted feature map to obtain prior knowledge of dynamic objects.

[0032] Furthermore, an attention mechanism is introduced into the dynamic frame data to obtain attention-weighted feature maps, specifically:

[0033] Global average pooling is performed along both the width and height directions to obtain feature maps corresponding to the width and height directions respectively;

[0034] The feature maps in the width and height directions are combined and then fed into a 1*1 convolutional module, and batch normalization is performed.

[0035] The batch-normalized feature map f1 is fed into the Sigmoid activation function to obtain feature map f2; feature map f2 is then convolved with 1*1 according to its original height and width, and the attention weight g on the width of the feature map is obtained by passing it through the Sigmoid activation function. h Attention weights g at different heights w Then, a feature map y with attention weights in the width and height directions is obtained by weighting. c .

[0036] Furthermore, the neural network is a YOLOv5s network with an attention mechanism. It extracts image information by adding a CA attention mechanism to the Backbone module, then uses the Neck module to extract small targets, then uses the Head module to generate bounding boxes, and finally the YOLO thread obtains coordinate messages by sending a ROS message containing bounding box coordinate information.

[0037] The beneficial effects of this invention are:

[0038] As can be seen from the above technical solution, compared with the prior art, this invention discloses a mapping method suitable for dynamic environments. It filters static feature points by using prior knowledge to eliminate dynamic objects, and the mapping is achieved using static features, thus avoiding errors caused by dynamic objects in dynamic environments and improving the mapping effect. This invention utilizes YOLOv5s with an attention mechanism to obtain prior dynamic objects in the scene. Then, based on the number and proportion of prior dynamic objects in the frame, an adaptive feature point selector is used to select feature points. Finally, the inter-frame positions of feature points within the bounding box and the bounding box dilation layer are matched using the Lucas-Kanade optical flow method, and a geometric method based on the RANSAC algorithm is used to determine whether the region judged by the adaptive feature point selector is a dynamic region, using only static feature points for mapping. This invention is tested on the KITTI dataset and several different dynamic feature point selection strategies are weighed. Experiments show that the improved algorithm has higher accuracy in dynamic environments compared to other existing algorithms. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 The attached figure is a schematic diagram of a mapping method suitable for dynamic environments provided by an embodiment of the present invention;

[0041] Figure 2 The attached figure is a schematic diagram of the neural network structure incorporating the attention mechanism in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention discloses a mapping method suitable for dynamic environments, comprising the following steps:

[0044] S1: Obtain dynamic frame data;

[0045] S2: Input the dynamic frame data into a neural network model for feature extraction to obtain prior knowledge;

[0046] In one embodiment, the number of dynamic objects, the proportion of dynamic objects in the screen, and the range markers and coordinate information of each dynamic object are included.

[0047] Specifically, in ORB-SLAM3, bounding boxes, i.e., the range markers of dynamic objects, are obtained through ROS messages. Each frame's ROS message is a variable-length vector, and the number of elements in the vector represents the number of dynamic objects. The ROS message contains the coordinates of the bounding box. By traversing and superimposing the vectors, the area of ​​the current frame is obtained, and this area is divided by the resolution of the KITTI dataset to obtain the proportion of the dynamic object's area in the current frame. The number of objects is the number of elements in the variable-length vector of each frame's message, and the area proportion is calculated from the bounding box coordinates contained in the message of each frame.

[0048] In this embodiment, the specific steps for feature extraction by the neural network model are as follows:

[0049] like Figure 2 For any intermediate feature tensor X = [x1, x2, ..., xn] in the network C ]∈R C*H*W First, global average pooling is performed along the width and height directions using (H,1) and (1,W) pooling kernels respectively to embed spatial information. The output of the c-th channel with height h can be expressed as:

[0050]

[0051] Where i is the width of the feature map with height h and width i in the c-th channel when performing global average pooling, 0≤i<W;

[0052] The output of the c-th channel with width w can be expressed as:

[0053]

[0054] Where j is the height of the feature map with width w and height j in the c-th channel when global average pooling is performed, 0≤j<H;

[0055] The feature maps in the width and height directions are then concatenated and fed into a 1x1 convolutional module, followed by batch normalization. The batch-normalized feature map f1 is then fed into the Sigmoid activation function to obtain f2 = δ(F1([z h ,z w ])), r is the reduction factor. Then, the feature map f2 is decomposed into two independent tensors according to its original height h and width w. and f h and f w Perform 1x1 convolutions and then apply the sigmoid activation function to obtain the attention weights g of the feature map in width. h =σ(F h (f h Attention weights at different heights g w =σ(F w (f w Then, a feature map with attention weights in the width and height directions is obtained by weighting. Image information is extracted by the Backbone module with CA attention mechanism, then small targets are extracted by the Neck module, then bounding boxes are generated by the Head module, and finally the YOLO thread obtains coordinate messages by sending ROS messages containing bounding box coordinate information.

[0056] S3: Based on the prior knowledge, feature points are filtered in the dynamic frame data, retaining static feature points. After receiving the message from YOLOv5 containing the bounding box coordinates of the prior dynamic objects, the adaptive selector judges based on the number of prior dynamic objects in the current frame and the proportion of prior dynamic objects in the frame. First, it determines whether the current feature point is within the bounding box. Feature points not within the bounding box are considered static points for mapping. If a feature point is within the bounding box, the adaptive feature point selector is used to determine whether this feature point is a static point.

[0057] In the YOLOv5 network, the Neck module uses Spatial Pyramid Pooling (SPPF) to handle multi-scale features. Within the current pyramid level, the feature map is divided into multiple grid cells, and the feature information of each cell is represented using a fixed size. In the next pyramid level, these representations converge into a smaller region. This helps the system process feature information at different scales, thereby improving the system's ability to detect small targets. The Head module generates feature maps at three scales (small, medium, and large) for prediction. First, the original image is gridded. If the center of the target falls into a grid cell, that cell is responsible for detecting the target. Bounding box positions and confidence scores are generated through regression. Finally, Non-Maximum Suppression (NMS) is used to remove redundant bounding boxes and retain the bounding box with the best prediction performance.

[0058] In one embodiment, the specific steps include:

[0059] S31: The current feature point within the range mark is further judged based on the proportion of dynamic objects in the current frame to obtain static feature points.

[0060] The second judgment is specifically as follows:

[0061] S311: Set the first threshold and the second threshold;

[0062] S312: When the proportion of prior dynamic objects in the current frame is less than the first threshold and the number of prior dynamic objects in the current frame is less than the second threshold, the features within the marked range are used to establish the current feature points in the dynamic frame data.

[0063] S313: Obtain the current feature point and determine whether the current feature point is within the range mark of the prior dynamic object. If the current feature point is not within the range mark, it is considered a static feature point. Otherwise, the geometric method is used to make three judgments.

[0064] In this embodiment, three judgments are performed using a geometric method, specifically:

[0065] S3131: Select an element outside the range mark as an expansion layer;

[0066] S3132: When there are no feature points within the expansion layer, the feature points within the marked range are determined to be static layers;

[0067] S3133: When there are feature points in the dilatation layer, the feature points within the range marked in the current frame and the feature points in the dilatation layer are matched with the previous frame. The essential matrix is ​​calculated through epipolar geometry, and static feature points are selected based on the essential matrix.

[0068] Specifically, the static feature points are selected based on the essential matrix as follows:

[0069] Calculate the infinite norm of the essential matrix:

[0070] In the geometric method, a 20-pixel dilation layer is applied around the outer edge of each bounding box. If there are no feature points within the bounding box's dilation layer, the prior dynamic object is considered to be in a low-feature environment, and the feature points within this bounding box are used for mapping. The LK optical flow is used to match the feature points in the current frame's bounding box and its dilation layer with those in the previous frame. First, a set P1 is formed by combining all feature points within each bounding box in the current frame with their corresponding feature points in the previous frame. Then, a set P2 is formed by randomly selecting matching feature point pairs from P1. For a pair of matching feature points in set P2, if feature point X1 is located within the current frame's bounding box and a feature point X2 matches it in the previous frame, let the normalized coordinates of X1 be x1 = [u1, v1, 1]. T The normalized coordinates of X2 are x2 = [u2, v2, 1]. T According to the epipolar constraint, there is an essential matrix. We get (1), and randomly select eight matching points to form eight linear equation systems. Then, from the linear form (2), we obtain the essential matrix E. t Then, feature points in P1 that are not in P2 are selected, and the pixel distances from these feature points to the epipolar lines are calculated. In this paper, we set the threshold to one pixel. If the maximum distance is greater than one pixel, the pair of points is recorded as outside points; if the maximum distance is less than the threshold, the pair of points is recorded as inside points. The number of inside points is then calculated. After all matching points within P1 have been determined, four pairs of matching points are reselected to form P2 to calculate the essential matrix. Finally, when the number of iterations exceeds the threshold t, if the number of inside points exceeds the threshold, the essential matrix with the most inside points is selected; otherwise, the algorithm continues to iterate and add new matching points to find the optimal essential matrix. The essential matrix of the Boundingbox inflation layer is also obtained in this way.

[0071]

[0072]

[0073] After obtaining the essential matrix E1 in the bounding box and the essential matrix E2 in the dilated layer, use ||x|| ∞ =max 1≤i＜8 Find the infinite norm of the essential matrix using |x8|; The algorithm checks if the ratio of the infinity norm L1 of E1 to the infinity norm L2 of E2 is greater than 0.95. If it is greater than a threshold, the prior dynamic object in the bounding box is considered static. If it is less than the threshold, the pixel coordinates of the left-eye feature point in the bounding box are set to (-1, -1). After detecting all feature points in a frame, binocular sparse stereo matching is used. First, a strip search region is established for the left-eye feature points, then coarse matching is performed. The right-eye feature points are matched according to the descriptor, then fine matching is performed using a SAD sliding window, and sub-pixel accuracy is optimized by parabolic fitting. Finally, the WTA algorithm is used to obtain the optimal matching point.

[0074] S4: Construct a map based on the static feature points.

[0075] First, this invention utilizes YOLOv5s with an attention mechanism to obtain prior dynamic objects in the scene. Then, based on the number and proportion of prior dynamic objects in the frame, an adaptive feature point selector is used to select feature points. Finally, a Lucas-Kanade optical flow method is used to match the inter-frame positions of feature points within the bounding box and its dilation layer. A geometric method based on the RANSAC algorithm is used to determine whether the region identified by the adaptive feature point selector is a dynamic region, and only static feature points are used for mapping. This invention is tested on the KITTI dataset, and several different dynamic feature point selection strategies are weighed. Experiments show that the improved algorithm has higher accuracy in dynamic environments compared to other existing algorithms.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0077] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A mapping method suitable for dynamic environments, characterized in that, Includes the following steps: Obtain dynamic frame data; The dynamic frame data is input into a neural network model for feature extraction to obtain prior knowledge; the prior knowledge includes the proportion of dynamic objects in the frame and the range markings and coordinate information of each dynamic object. Based on the prior knowledge, feature points are filtered in the dynamic frame data, and static feature points are retained. The feature point filtering steps include: A secondary judgment is made based on the proportion of prior dynamic objects in the current frame; Three judgments are made using geometric methods, and an element is selected outside the marked range to be set as the expansion layer; When there are no feature points within the expansion layer, the feature points within the marked range are determined to be a static layer; When there are feature points within the dilated layer, the feature points within the range marker of the current frame and the feature points of the dilated layer are matched with those of the previous frame. The essential matrix is ​​calculated using epipolar geometry, and its infinite norm is calculated. The essential matrix includes the range marker essential matrix. and the essential matrix of the expansion layer ; Computation range label essential matrix infinite norm With the essential matrix of the expansion layer infinite norm The ratio r: ； A threshold is set. When the ratio r is greater than the threshold, the prior dynamic object in the range mark is considered to be static. When the ratio is less than the threshold, binocular feature point matching is performed on all feature points of the current frame, and the best matching point is selected as the static feature point. Mapping is constructed based on the static feature points.

2. The mapping method suitable for dynamic environments according to claim 1, characterized in that, The prior knowledge also includes the number of dynamic objects.

3. The mapping method suitable for dynamic environments according to claim 1, characterized in that, The secondary judgment specifically involves setting a first threshold and a second threshold. When the proportion of prior dynamic objects in the current frame is less than a first threshold and the number of prior dynamic objects in the current frame is less than a second threshold, the features within the marked range establish current feature points in the dynamic frame data, obtain the current feature points, and determine whether the current feature points are located within the range marked of the prior dynamic objects. Current feature points that are not located within the range marked are considered as static feature points.

4. The mapping method suitable for dynamic environments according to claim 1, characterized in that, The dynamic frame data is input into a neural network model for feature extraction to obtain prior knowledge. Specifically, an attention mechanism is introduced into the dynamic frame data to obtain an attention-weighted feature map. Prediction is performed based on the weighted feature map to obtain prior knowledge of dynamic objects.

5. A mapping method suitable for dynamic environments according to claim 4, characterized in that, An attention mechanism is introduced into the dynamic frame data to obtain attention-weighted feature maps. Specifically, global average pooling is performed along both the width and height directions to obtain feature maps corresponding to the width and height directions respectively. The feature maps in the width and height directions are combined and then fed into a 1*1 convolutional module, and batch normalization is performed. The feature maps after batch normalization The feature map is obtained by feeding it into the activation function. ; feature map Perform a 1x1 convolution on the original height and width, and then use an activation function to obtain the attention weights of the feature map on the width. Attention weights at different heights Then, a feature map with attention weights in the width and height directions is obtained by weighting. .

6. A mapping method suitable for dynamic environments according to claim 5, characterized in that, The neural network is a YOLOv5s network with an attention mechanism. It extracts image information through the Backbone module with CA attention mechanism, then uses the Neck module to extract small targets, then uses the Head module to generate bounding boxes, and finally the YOLO thread obtains the coordinate message by sending a ROS message containing the bounding box coordinate information.

Images