Three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check

Abstract

The invention discloses a three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check, and the method is characterized in that the method comprises the following steps: firstly improving a YOLOV5-SEG semantic segmentation network, replacing a Conv module of a Backbone part with a GhostConv module, replacing an SPPF module with an SIMSPPF module, and introducing a CBAM module into a Neck part; the method comprises the following steps: acquiring an image, calculating through a convolutional neural network, and segmenting a priori dynamic object pixel by pixel; positioning in the created scene map by using a tracking module; feature points of dynamic objects in the scene are filtered through motion consistency check; generating a trajectory of the camera and a scene map based on the segmented frames; and converting the static map into an octree map which can be used for three-dimensional navigation. The method has the advantages that the picture quality can be improved, accurate reconstruction of a local map is realized, and a three-dimensional map is generated to realize navigation.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a three-dimensional map construction method based on lightweight dynamic semantic segmentation and motion consistency checking. Background Art

[0002] One of the core challenges of 3D navigation is dealing with occlusion and dynamic changes in the environment. For example, in indoor navigation, the movement of furniture or the placement of new items may change the navigation path. In outdoor navigation, such as urban environments, the movement of buildings, vehicles, and pedestrians may affect navigation. 3D navigation technology needs to be able to handle these changes in real time and quickly adapt to new environmental conditions. Similar to image restoration, 3D navigation technology also involves the establishment and optimization of mathematical models. For example, in SLAM technology, the system needs to build an environmental map and update its own position in real time, which requires complex optimization algorithms to process large amounts of sensor data and environmental changes. Summary of the invention

[0003] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is: how to provide a three-dimensional map construction method that can improve image quality, realize accurate reconstruction of local maps and generate three-dimensional maps to achieve navigation.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: A three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check, characterized by comprising the following steps: S1. First, improve the YOLOV5-SEG semantic segmentation network, replace the Conv module in the Backbone part with the GhostConv module, replace the SPPF module with the SIMSPPF module, and introduce the CBAM module in the Neck part; S2, obtain the image, and segment the prior dynamic objects pixel by pixel through convolutional neural network calculation; S3, using the tracking module to locate in the created scene map; S4, filtering feature points of dynamic objects in the scene using motion consistency check; S5, generating camera trajectory and scene map based on the segmented frames; S6. Convert the static map into an octree map that can be used for three-dimensional navigation.

[0005] Furthermore, the image in step 2 is acquired by a monocular or binocular camera or an RGB-D camera; and the improved YOLOV5-SEG semantic segmentation network in step S1 is used to perform semantic segmentation on the image to segment out the prior dynamic objects.

[0006] Furthermore, in step S4, the steps of motion consistency check are as follows: first calculate the optical flow between the current frame and the previous frame to identify the moving feature points in the image; match the feature points of the current frame and the previous frame, and discard the feature points whose pixel difference exceeds the set value; use RANSAC to calculate the basic matrix, use the calculated basic matrix to find the polar lines in the current frame corresponding to the previous frame, and determine whether the distance from each matching point to its corresponding polar line is less than a preset threshold. If the distance is greater than the threshold, the matching point is considered to be moving; count the number of matching points in each object, and if the ratio of the matching point to the total feature points of the object exceeds the set threshold, the object is determined to be a dynamic object.

[0007] Furthermore, in step S4, for the image of the RGB-D camera, based on the depth information of the pixels obtained by the RGB-D camera, the semantic segmentation of the image performed by the neural network and the motion consistency check method are complemented to detect dynamic objects in the image.

[0008] Furthermore, in step S6, the static sparse map constructed by the SLAM algorithm is converted from the original sparse point cloud to the octree map by inserting key points into the octree structure and dividing the point cloud data using the spatial distribution characteristics.

[0009] In summary, the present invention has the advantages of being able to improve image quality, accurately reconstruct local maps, and generate three-dimensional maps for navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of an embodiment of the present invention.

[0011] Figure 2 This is a structural diagram of the lightweight semantic segmentation network designed in the present invention.

[0012] Figure 3 This is the process of the pyramid optical flow method of the present invention.

[0013] Figure 4 Schematic diagram of the octree map of the present invention. DETAILED DESCRIPTION

[0014] The present invention is further described in detail below in conjunction with embodiments.

[0015] When implementing: Figure 1 As shown, a three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check includes: S1. Improve the existing semantic segmentation network to make it lighter, so as to improve the real-time performance and accuracy. In the present invention, the YOLOV5-SEG semantic segmentation network is improved in three parts: 1) the Conv in the Backbone part is replaced by GhostConv to reduce the amount of calculation; 2) the SPPF module is replaced by the SIMSPPF module to improve the calculation efficiency and retain more feature information; 3) the CBAM module is introduced in the Neck part to improve the model's ability to process channel and spatial information.

[0016] S2, obtain the image, and segment the inherent dynamic objects pixel by pixel through convolutional neural network calculation; In the present invention, there are two sources of image acquisition, one is acquired through monocular and binocular cameras, and the other is acquired through RGB-D cameras. After that, the convolutional neural network in S1 is used to perform semantic segmentation of the image, and a priori dynamic objects will be segmented out, such as: people, cars, cats, etc. The network can be trained according to the needs of different situations so that it can self-judge dynamic objects that are likely to appear in the actual environment.

[0017] S3, using a low-cost tracking module to locate in the created scene map; In specific implementation, in step S3: after the previous neural network processing, the prior dynamic objects in the image are removed. At this time, the information in the image becomes less, the features that can be used for tracking and positioning also become fewer, and the corresponding amount of calculation is also reduced. Using this tracking module with short time consumption and low amount of calculation, the camera pose estimation value required for the next step of multi-view geometry judgment is obtained. At the same time, this module is also affected by the subsequent tracking and map reconstruction, and uses the sparse map generated by it for tracking, while continuously updating the map to form a feedback to enhance the accuracy of map reconstruction.

[0018] S4, using a method combining semantic segmentation and motion consistency check to filter out feature points of dynamic objects in the scene; In the specific implementation, in step S4: in order to distinguish the dynamic objects missed by the neural network method in the previous step, that is, the a posteriori dynamic objects, a motion consistency check is performed, and the steps are as follows: 1. Optical flow pyramid calculation: Calculate the optical flow between the current frame and the previous frame to identify moving feature points in the image. Optical flow calculation involves constructing an optical flow pyramid, which is a multi-scale analysis method for detecting motion in an image; 2. Feature point matching and screening: Match the feature points of the current frame with the previous frame, and discard those matching pairs that are too close to the edge of the image or have too large pixel differences. This step helps improve the accuracy of matching and reduce the interference of noise points; 3. Compute the fundamental matrix using RANSAC: Apply the RANSAC algorithm to estimate a fundamental matrix from the matching pairs, which describes the projective transformation between the two views; 4. Calculate epipolar lines and determine matching points: Use the calculated basic matrix to find the epipolar lines in the current frame corresponding to the previous frame, and determine whether the distance from each matching point to its corresponding epipolar line is less than a preset threshold. If the distance is greater than the threshold, the matching point is considered to be moving; 5. Dynamic object recognition: If most of the matching points within an object are judged to be moving, the object is considered a dynamic object.

[0019] Based on the depth information of pixels obtained by the RGB-D camera, the semantic segmentation of the image performed by the neural network and the motion consistency check method are complemented to detect dynamic objects in the image. This makes the segmentation of dynamic objects more accurate. S5, generating camera trajectories and scene maps based on the segmented frames; In this embodiment, in step S5: this part uses the semantic segmentation of the image by the neural network combined with the motion consistency check to complement the image to complete the tracking and mapping tasks. By finding the matching of local map features, the camera of each frame is located by minimizing the reprojection error using pure motion BA to achieve the tracking and positioning effect; by executing local BA to manage the local map and optimize the local map construction task; The present invention introduces a loop based on the generation of camera trajectories and scene maps. Data is exchanged between the generated reconstructed map and the sparse map, and the sparse map is used for tracking. At the same time, the map is continuously updated to form a positive feedback. The reconstructed map is matched with the real scene during the continuous update, eliminating the error between the two and enhancing the accuracy of map reconstruction.

[0020] S6. Convert the static map into an octree map that can be used for three-dimensional navigation.

[0021] The static sparse map constructed by the SLAM algorithm can realize the conversion from the original sparse point cloud to the octree map by inserting key points into the octree structure and dividing the point cloud data using the spatial distribution characteristics. After the octree map structure is optimized, it can effectively represent the environmental space and provide support for three-dimensional navigation. At the same time, the octree map can be quickly updated to reflect environmental changes, providing convenience for operations such as path planning.

[0022] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check, characterized in that: The steps include: S1. First, improve the YOLOV5-SEG semantic segmentation network, replace the Conv module in the Backbone part with the GhostConv module, replace the SPPF module with the SIMSPPF module, and introduce the CBAM module in the Neck part; S2, obtain the image, and segment the prior dynamic objects pixel by pixel through convolutional neural network calculation; S3, using the tracking module to locate in the created scene map; S4, filtering feature points of dynamic objects in the scene using motion consistency check; S5, generating camera trajectory and scene map based on the segmented frames; S6. Convert the static map into an octree map that can be used for three-dimensional navigation.

2. The three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check according to claim 1, characterized in that: The image in step 2 is acquired by using a monocular or binocular camera or an RGB-D camera; and the improved YOLOV5-SEG semantic segmentation network in step S1 is used to perform semantic segmentation on the image to segment out a priori dynamic objects.

3. The three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check according to claim 2, characterized in that: In step S4, the steps of motion consistency check are as follows: first calculate the optical flow between the current frame and the previous frame to identify the moving feature points in the image; match the feature points of the current frame and the previous frame, and discard the feature points whose pixel difference exceeds the set value; use RANSAC to calculate the basic matrix, use the calculated basic matrix to find the polar lines in the current frame corresponding to the previous frame, and determine whether the distance from each matching point to its corresponding polar line is less than a preset threshold. If the distance is greater than the threshold, the matching point is considered to be moving; count the number of matching points in each object, and if the ratio of the matching point to the total feature points of the object exceeds the set threshold, the object is determined to be a dynamic object.

4. The three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check according to claim 3, characterized in that: In step S4, for the image of the RGB-D camera, based on the depth information of the pixels obtained by the RGB-D camera, the semantic segmentation of the image performed by the neural network and the motion consistency check method are complemented to detect dynamic objects in the image.

5. The three-dimensional map construction method based on dynamic semantic segmentation and motion consistency check according to claim 3, characterized in that: In step S6, the static sparse map constructed by the SLAM algorithm is converted from the original sparse point cloud to the octree map by inserting key points into the octree structure and dividing the point cloud data using the spatial distribution characteristics.

Images