A real-time 3D map construction method based on dynamic image acquisition

Abstract

The present invention discloses a real-time construction method for three-dimensional maps based on dynamic image acquisition. The present invention introduces a parameter-free attention mechanism module into the backbone network of the target detection model; the parameter-free attention mechanism module performs a weighted attention operation after the first downsampling and feature enhancement operation of the backbone network. The model is simplified without affecting the overall performance. In addition, the present invention selects five key frame images from the fifteen frames of images before the dynamic image frame based on the offset size; the target area is set in the dynamic image frame and each key frame image, which can ensure that the activity range of the dynamic character can be calculated by the multi-view geometry algorithm, thereby improving the accuracy. In addition, the present invention adjusts the operation area of ​​the multi-view geometry algorithm according to the position and size of the detection frame, reduces the amount of calculation, and allows the computing resources to be biased towards the multi-view geometry algorithm of dynamic culling, thereby improving the robustness of the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual SLAM, and in particular relates to a real-time three-dimensional map construction method based on dynamic image acquisition. Background Art

[0002] Indoor environments often require highly accurate and real-time positioning information to support accurate navigation and movement of devices such as robots and unmanned vehicles. SLAM technology can provide real-time positioning and mapping to meet the needs of these applications. The development of deep learning has brought about many significant changes and innovations, profoundly impacting various fields. Deep learning models have achieved remarkable performance and accuracy in tasks such as image classification, object detection, speech recognition, and natural language processing. Leveraging large-scale datasets and powerful computing resources, deep learning models can achieve superior performance compared to traditional methods. Deep learning has also achieved significant success in image processing and computer vision, and can be applied to environmental perception and understanding tasks in SLAM, such as image semantic segmentation, object detection, and instance segmentation. These technologies can help SLAM systems better identify and understand objects and structures in the environment, improving the accuracy and robustness of environmental perception.

[0003] ORB-SLAM3's map construction is not very descriptive of the real environment, requiring the addition of dense maps and coloring based on the point cloud generated by ORB-SLAM3. Furthermore, dynamic objects appear irregularly in the environment, hindering the construction of the environment. With the development of deep learning, target detection technology uses prior information to detect and locate target objects in images. For other dynamic objects that are not easily detected, multi-view geometry methods are used to detect them, eliminating the interference of dynamic objects in scene map construction and improving the robustness of the system's positioning and mapping. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for real-time construction of a three-dimensional map based on dynamic image acquisition.

[0005] The present invention provides a method for real-time construction of a three-dimensional map based on dynamic image acquisition, comprising the following steps:

[0006] Step 1: The robot equipped with a depth camera moves in the target area, and the depth camera continuously collects images.

[0007] Step 2: Eliminate dynamic objects in the current image.

[0008] 2-1. Input the captured current image into the object detection model to detect whether there are dynamic objects in the current image. If no dynamic objects are detected in the current frame, proceed directly to step 3. If a dynamic object is detected in the current frame, set the current frame as a dynamic image frame and proceed to step 2-2.

[0009] The target detection model includes a backbone network, an intermediate operation part, and an output layer. A non-parametric attention mechanism module is added to the backbone network; the non-parametric attention mechanism module performs a weighted attention operation after the first downsampling and feature enhancement operation of the backbone network. The intermediate operation part includes a pooling module and nine convolution modules connected in sequence. The pooling module in the intermediate operation part is an SPPF pooling layer, the first convolution module is a C2 convolution layer, the second and fourth convolution modules are UpSample convolution layers, the third convolution module is a Concat operation, a C3 convolution layer, and a C2 convolution layer connected in sequence, the fifth, seventh, and ninth convolution modules are a Concat operation and a C3 convolution layer connected in sequence, and the sixth and eighth convolution modules are a C1 convolution layer.

[0010] 2-2. Eliminate dynamic objects in dynamic image frames based on the multi-view geometry module.

[0011] Five keyframe images are selected from multiple frames preceding the dynamic image frame; a target region is set in the dynamic image frame and each keyframe image; and dynamic objects in the target region of the dynamic image frame are extracted and eliminated using a multi-view geometry method.

[0012] Step 3: Track sharp points on the image processed in step 2.

[0013] Step 4: Based on the coordinates of the sharp points in the image obtained by tracking in step 3 and the depth information in the image, map points are added to the 3D map point cloud, and the coordinates of the newly added map points in the 3D map point cloud are solved.

[0014] Step 5: Extract the color information of the newly added map points in the 3D map point cloud. The color information of each map point is projected from the corresponding pixel in the image frame. Then, the map points in the 3D map point cloud are stitched together to form a global dense map.

[0015] Step 6: Determine whether the robot has reached a position it has passed before; if so, execute step 7; if not, continue to repeat steps 2 to 5.

[0016] Step 7: Optimize the current global dense map to obtain the final global dense map.

[0017] Preferably, the detection object of the target detection model described in step 2-1 is the human body.

[0018] Preferably, in step 2-2, a key frame image is selected from fifteen frames of images preceding the dynamic image frame.

[0019] Preferably, the process of selecting the key frame image in step 2-2 is:

[0020] The target detection frame in the dynamic image frame is set to have point A in the upper left corner, point D in the upper right corner, point B in the lower left corner, and point C in the lower right corner. Points A, D, B, and C are recorded as the first, second, third, and fourth key points of the target detection frame in the image, respectively.

[0021] Calculate the offset R for each of the multiple frames before the dynamic image frame i as follows:

[0022]

[0023] Wherein, K1 is the translation coefficient, which is set to 1 in this embodiment; K2 is the vector coefficient, which is set to 25 in this embodiment. ij 、y ij x is the horizontal and vertical coordinates of the jth key point in the i-th frame before the current dynamic image frame. i=1,2,...,15; x 0j 、y 0j is the horizontal and vertical coordinates of the jth key point of the current dynamic image frame. i , BD i AC0 and BD0 are the vectors from point A to point C and from point B to point D in the i-th frame before the current dynamic image frame, respectively. AC0 and BD0 are the vectors from point A to point C and from point B to point D in the current dynamic image frame, respectively.

[0024] Take the offset R i The minimum five frames are used as key frame images.

[0025] As an example, when there are multiple target detection frames in a dynamic image frame, the offset R of each image is i Take the mean of the offsets corresponding to each target detection box.

[0026] Preferably, in step 2-2, the target area is a quadrilateral, and the coordinates of its four corner points are: the horizontal coordinate and the vertical coordinate of the upper left corner point of the target area are the minimum horizontal coordinate and the maximum vertical coordinate of the upper left corner point in the dynamic image frame and each key frame image; the horizontal coordinate and the vertical coordinate of the upper right corner point of the target area are the maximum horizontal coordinate and the maximum vertical coordinate of the upper right corner point in the dynamic image frame and each key frame image; the horizontal coordinate and the vertical coordinate of the lower left corner point of the target area are the minimum horizontal coordinate and the minimum vertical coordinate of the lower left corner point in the dynamic image frame and each key frame image; the horizontal coordinate and the vertical coordinate of the lower right corner point of the target area are the maximum horizontal coordinate and the minimum vertical coordinate of the lower right corner point in the dynamic image frame and each key frame image.

[0027] As a preference, in step 2-2, the process of the multi-view geometry method is as follows: projecting the feature point p of the key frame image into the dynamic image frame, calculating the projection of each feature point p from the key frame image to the dynamic image frame as the feature point p', and calculating the projection depth d for each feature point p' according to the camera motion. proj ; Then calculate the disparity angle α between the back projection of point p and point p'. If the angle is greater than 30°, it is considered a dynamic feature point. Then calculate the reprojection error Δd = d proj -d'; where d' is the depth of the feature point in the current image frame. If the reprojection error Δd is greater than the depth threshold, the feature point p' is determined to be a dynamic feature point.

[0028] Preferably, the three feature maps of different scales in the output layer are input into a bounding box prediction module; the bounding box prediction module generates multiple candidate boxes; and the optimal detection box is selected from these multiple candidate boxes using a non-maximum suppression algorithm. The three feature maps of different scales can utilize a feature pyramid structure. This design aims to improve the object detection model's ability to perceive objects of different scales, thereby enhancing detection accuracy and robustness.

[0029] Preferably, the backbone network comprises five sequentially connected convolutional modules; the first convolutional module in the backbone network is a C0 convolutional layer, the second convolutional module is a sequentially connected C1 convolutional layer, a C3 convolutional layer, and a non-parametric attention mechanism module SimAM, and the third, fourth, and fifth convolutional modules are all sequentially connected C1 convolutional layers and a C3 convolutional layer without the non-parametric attention mechanism module SimAM. The features output by the third convolutional module of the backbone network are additionally passed to the fifth convolutional module in the intermediate operation part; the features output by the fourth convolutional module of the backbone network are additionally passed to the third convolutional module in the intermediate operation part.

[0030] Preferably, the output layer includes three output modules of different sizes. The sizes of the three output modules are 80×80, 40×40, and 20×20, respectively. The first convolution module in the intermediate operation section is additionally transmitted to the ninth convolution module in the intermediate operation section; the third convolution module in the intermediate operation section is additionally transmitted to the seventh convolution module in the intermediate operation section; and the output features of the fifth, seventh, and ninth convolution modules in the intermediate operation section are respectively transmitted to the three output modules of the output layer, forming an 80×80 feature map, a 40×40 feature map, and a 20×20 feature map.

[0031] Preferably, the C0 convolution layer performs a standard convolution operation; the C1 convolution layer is a convolution operation with a convolution kernel of 3*3 and a stride of 2. The C2 convolution layer is a convolution layer operation with a convolution kernel of 1*1 and a stride of 1. The C3 convolution layer is composed of three convolution layers. The SPPF pooling layer is spatial pyramid pooling. Concat is an operation used in deep learning to fuse and connect feature maps of different scales. The UpSample convolution layer is an image upsampling process. PAN is the part of Neck excluding the SPPF pooling layer.

[0032] The present invention has the following beneficial effects:

[0033] 1. This paper introduces a parameterless attention mechanism module into the backbone network of the object detection model. The parameterless attention mechanism module performs weighted attention after the first downsampling and feature enhancement operations of the backbone network. This simplifies the model without affecting the overall performance.

[0034] 2. The present invention simplifies the structure of the target detection model, reduces the size of the model, improves the deployment efficiency of the model in resource-constrained environments, and makes the calculation in the inference phase faster, which is crucial for the deployment of mapping systems with high real-time requirements in the field of mobile robots.

[0035] The present invention selects five keyframe images from the fifteen frames preceding the dynamic image frame based on the offset size. By setting target regions within the dynamic image frame and each keyframe image, the multi-view geometry algorithm can be used to calculate the range of motion of dynamic characters, thereby improving accuracy. Furthermore, the present invention adjusts the computational region of the multi-view geometry algorithm based on the position and size of the detection frame, reducing the computational load and shifting computing resources toward the multi-view geometry algorithm for dynamic culling, thereby improving the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of the present invention.

[0037] Figure 2 Schematic diagram of the structure of the target detection model in the present invention.

[0038] Figure 3 This is the first principle diagram of the multi-view geometry algorithm in the present invention.

[0039] Figure 4 This is the second principle diagram of the multi-view geometry algorithm in the present invention.

[0040] Figure 5 This is the first experimental result diagram of thread tracking in the present invention.

[0041] Figure 6 This is the second experimental result diagram of thread tracking in the present invention.

[0042] Figure 7 This is the third experimental result diagram of thread tracking in the present invention.

[0043] Figure 8 Schematic diagram of the global dense map constructed for the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] A real-time 3D map construction method based on dynamic image acquisition uses a mapping system with multiple independent threads: dynamic object culling, tracking, local mapping, dense map construction, and loop detection. When the loop detection thread detects that the robot has returned to its starting point, it optimizes the current 3D map to produce the final 3D map. The mapping system is mounted on a mobile robot equipped with a depth camera.

[0046] In this embodiment, the mapping system first uses the target detection model and the human body in the positioning image. For other dynamic objects that move with the human body and are difficult to detect using target detection, a multi-view geometry method is used to detect them, eliminating the interference of dynamic objects in the scene map construction. Then, a dense map is constructed using 3D point cloud stitching, and voxel filtering is used to effectively remove noise. Global BA optimizes map details.

[0047] like Figure 1 As shown, the real-time construction method of the three-dimensional map based on dynamic image acquisition includes the following steps:

[0048] Step 1: Build a target detection model based on the lightweight model yolov5-ghost.

[0049] The development of target detection algorithms has made continuous breakthroughs in reasoning ability and accuracy, but the ability of models to be deployed on low-computing power devices is still unsatisfactory. In the field of VSLAM (Visual Simultaneous Localization and Mapping), real-time requirements are relatively high, and a large number of image frames need to be quickly calculated in complex environments. This requires that the overall algorithm module meet basic operating conditions. Therefore, the use of lightweight neural networks is necessary for the system. YOLOv5 is a target detection algorithm. Due to its speed and accuracy, YOLO is one of the most famous object detection algorithms. This embodiment improves on the basis of YOLOv5 to improve accuracy and reduce computational complexity.

[0050] 1-1. Improve the backbone module

[0051] Visual attention is an innate ability of the human brain. When we see an image, we first quickly scan it and then focus on the target area that requires our attention. In deep learning, the attention mechanism is an important technical means used to enhance the model's attention to input data, improving its performance and generalization ability. The attention mechanism can help models better learn and understand the dependencies between data in areas such as sequence data and image data, thereby improving model performance. Traditional attention mechanisms typically include channel attention, spatial attention, and mixed attention, but they all require additional parameters to calculate weights. To make the backbone module more accurate and the model simpler without affecting overall performance, an attention mechanism module SimAM is added to the first C3 layer of the backbone module without introducing additional parameters. The SimAm module implements an efficient attention mechanism without parameters.

[0052] SPPF (Spatial Pyramid Pooling with Fusion) is an improved spatial pyramid pooling method used to speed up deep learning models. Compared to traditional SPP (Spatial Pyramid Pooling), SPPF introduces multiple small pooling kernels and cascades them together to replace a single large pooling kernel. This method improves speed while retaining the original functionality. SPPF can more efficiently capture feature information at different scales and achieves better performance when processing large images. This method significantly improves the efficiency of deep learning models when processing large-scale data while maintaining model accuracy.

[0053] 1-2 Improve the neck module

[0054] GhostNet is a lightweight neural network architecture designed to improve model performance and speed. This is crucial for using deep learning models with limited computing resources. The lightweight Ghost module is deployed as a convolutional module within a PAN (Path Aggregation Network). Ghost achieves similar functionality to the original convolutional network using fewer parameters. This helps reduce model size and improves deployment efficiency in resource-constrained environments. Furthermore, the reduced number of parameters allows for faster computation during the inference phase, which is crucial for deploying VSLAM algorithms, which require high real-time performance, in the mobile robotics field.

[0055] 1-3 Select non-maximum suppression algorithm

[0056] NMS (non-maximum suppression) is an important part of the object detection process and is used to search for local maxima. During the target detection process, a large number of candidate boxes will be generated at the same location. NMS can be used to find the best target bounding box and eliminate the remaining bounding boxes. However, in the case of dense boxes, using NMS will miss the target, so Soft-NMS (flexible non-maximum suppression algorithm) is used to find the best target bounding box to improve the accuracy, robustness and adaptability of target detection to various targets. The improved lightweight model yolov5-ghost system block diagram is shown in Figure 2 .

[0057] like Figure 2 As shown in the figure, based on the above improvements, the obtained target detection model includes a backbone network (Backbone), an intermediate operation part (Neck) and an output layer (Head). Figure 2 In the figure, the gray filled box represents the backbone network (Backbone), the blue filled box represents the intermediate operation part (Neck), and the yellow filled box represents the output layer (Head).

[0058] The backbone network consists of five sequentially connected convolutional modules. The first convolutional module is the C0 convolutional layer. The second convolutional module is the sequentially connected C1 convolutional layer, C3 convolutional layer, and the non-parametric attention mechanism module SimAM. The third, fourth, and fifth convolutional modules are all sequentially connected C1 convolutional layer and C3 convolutional layer without the non-parametric attention mechanism module SimAM. In the second convolutional module, the non-parametric attention mechanism module SimAM performs a weighted attention operation on the feature map output by the C3 convolutional layer, and the resulting feature map is input to the third convolutional module.

[0059] The intermediate operation part includes a pooling module and nine convolution modules connected in sequence. The pooling module in the intermediate operation part is the SPPF pooling layer, the first convolution module is the C2 convolution layer, the second and fourth convolution modules are UpSample convolution layers, the third convolution module is the Concat convolution layer, C3 convolution layer, and C2 convolution layer connected in sequence, the fifth, seventh, and ninth convolution modules are the Concat convolution layer and C3 convolution layer connected in sequence, and the sixth and eighth convolution modules are the C1 convolution layer.

[0060] The output layer includes three output modules of different sizes, 80×80, 40×40, and 20×20 respectively.

[0061] The features output by the third convolutional module of the backbone network are additionally passed to the fifth convolutional module in the intermediate operation part;

[0062] The features output by the fourth convolutional module of the backbone network are additionally passed to the third convolutional module in the intermediate operation part;

[0063] The first convolution module in the intermediate operation part is additionally passed to the ninth convolution module in the intermediate operation part;

[0064] The third convolution module in the intermediate operation part is additionally passed to the seventh convolution module in the intermediate operation part;

[0065] The output features of the fifth, seventh, and ninth convolutional modules in the intermediate operations are passed to the three output modules of the output layer, forming 80×80 feature maps, 40×40 feature maps, and 20×20 feature maps, respectively. In the post-processing stage of the YOLOv5 system, the non-maximum suppression algorithm is used to select the optimal detection box from the many candidate boxes of the target object and mark the detection box on the image.

[0066] The C0 convolutional layer performs standard convolution operations; the C1 convolutional layer uses a 3*3 kernel with a stride of 2 and is commonly used as an image downsampling layer. The C2 convolutional layer uses a 1*1 kernel with a stride of 1. The C3 convolutional layer is composed of three convolutional layers and increases the network depth and receptive field, improving feature extraction capabilities. The SPPF pooling layer performs spatial pyramid pooling. Concat is an operation used in deep learning to fuse and connect feature maps of different scales. The UpSample convolutional layer performs image upsampling. PAN is the portion of Neck excluding the SPPF pooling layer.

[0067] During the object detection model's operation, the image first enters the C0 convolutional layer in the backbone network. This layer replaces the Focus module in the conventional YOLOv5 model and performs preliminary processing and feature extraction on the input image, aiming to simplify the system. In the standard convolution, the image undergoes one downsampling process, followed by four downsampling and feature enhancement operations. During these four downsampling and feature enhancement operations, the first downsampling and feature enhancement process is selected to apply the parameter-free attention mechanism (SimAM). This helps the model better understand and process complex input information, thereby improving accuracy and efficiency. Therefore, the backbone network undergoes a total of five downsampling processes before the image is input to the Neck module for further feature enhancement. The Neck module employs a "two-tower" process called PAN. The first "tower" further enhances high-level semantic information, while the second "tower" transmits and enhances low-level details, which facilitates localization. Deploying the lightweight Ghost model throughout the entire system, including the backbone network, would inevitably significantly reduce detection accuracy. Therefore, deploying the lightweight Ghost model only in the PAN is the optimal choice, balancing performance and speed. In the first tower, the image is passed to Spatial Pyramid Pooling (SPPF) to fuse features from different levels. The image then undergoes a C2 convolution layer and is upsampled. The upsampled result is concatenated with the image from the fourth downsampling operation in the backbone network. The result then undergoes a C3 convolution, a C2 convolution, and an upsampling operation. This result is then concatenated and convolved with the result from the third downsampling operation in the backbone network, resulting in an 80x80 feature map. In the second tower, the 80x80 feature map undergoes a C1 convolution, which is then concatenated and convolved with the result from the third operation in the first tower, resulting in a 40x40 feature map. The 40x40 feature map undergoes a C1 convolution and is concatenated and convolved with the result from the first operation in the first tower, resulting in a 20x20 feature map. The resulting three feature maps are then used for prediction.

[0068] The three feature maps of different scales from the output layer are fed into the bounding box prediction module. This module first generates multiple anchor boxes in the image and then predicts the categories and offsets of these anchor boxes based on the trained model parameters, ultimately generating predicted candidate bounding boxes. The non-maximum suppression algorithm selects the optimal detection box from these candidate boxes. The three feature maps of different scales enable the use of a feature pyramid structure. This design aims to improve the object detection model's ability to perceive objects of different scales, thereby enhancing detection accuracy and robustness.

[0069] Step 2: The dynamic object culling thread culls the dynamic objects in the current image frame.

[0070] Since dynamic objects appear irregularly in the environment scene, it affects the construction of the environment. And the appearance of dynamic objects in the ORB-SLAM3 system may cause the stability of certain feature points in the image to decrease. This is because the movement of dynamic objects may cause their appearance in consecutive frames to change, making the extracted feature points unstable or difficult to match. And extracting feature points on dynamic objects will continuously increase the cumulative error in the system when matching postures, affecting the accuracy of posture evaluation. In order to solve the impact of dynamic objects and the demand for dense scene mapping in dynamic scenes. This embodiment uses ORB-SLAM3 as the benchmark version, makes improvements in its tracking thread, adds a target detection module and a multi-view geometry module based on YOLOv5-ghost, and eliminates the impact of dynamic objects on the system.

[0071] The robot equipped with a depth camera moves in the mapped area, and the depth camera continuously collects images according to the preset sampling rate. Each time an image is collected, the following processing is performed:

[0072] 2-1 Use the target detection model to detect the human body in the current frame image.

[0073] In typical indoor scenes, people are the most prevalent dynamic objects. When a person is detected in the object detection thread, a detection box is used to outline the person. However, since the person only occupies a portion of the detection box, the feature points within the detection box cannot be completely removed. Therefore, a single object detection module cannot completely eliminate the impact of dynamic objects. Taking actual conditions into consideration, a thread for dynamic object detection within the detection box has been added to avoid excessive removal of static feature points within the detection box, which could lead to positioning loss and loss of mapping accuracy.

[0074] To preserve static feature points within the detection frame, a thread is designed as follows: First, the input image frame is detected using the YOLOv5-ghost module. If a person appears, the frame is identified as a dynamic object and sent to the next thread, which then performs a detailed classification of the frame. Regions outside the dynamic frame are then merged with the feature extraction thread, retaining the feature points outside the frame. Objects within the frame are then subjected to multi-view geometry analysis. Dynamic objects within the frame are removed to preserve static points.

[0075] If no human body is detected in the current frame image, the current frame image is set as a static image frame and directly input into the tracking thread of step 3; if a human body is detected in the current frame image, the current frame image is set as a dynamic image frame, and the dynamic image frame is input into step 2-2 to remove dynamic objects.

[0076] 2-2. Eliminate dynamic objects in dynamic image frames based on the multi-view geometry module.

[0077] 2-2-1. Select key frame image

[0078] In existing multi-view geometry judgment, the image frames with the largest overlap between the previous 5 frames and the current frame are generally selected, and the overlap needs to consider the distance and rotation between the new frame and each key frame.

[0079] In this embodiment, five key frame images are screened from the first fifteen frames of the dynamic image frame. The screening process is as follows:

[0080] Set the lower left corner of the current dynamic image frame as the origin. The target detection frame in the dynamic image frame has point A at the upper left corner, point D at the upper right corner, point B at the lower left corner, and point C at the lower right corner. Points A, D, B, and C are recorded as the first, second, third, and fourth key points of the target detection frame in the image, respectively.

[0081] Calculate the offset R for the first fifteen frames of the dynamic image frame i as follows:

[0082]

[0083] Wherein, K1 is the translation coefficient, which is set to 1 in this embodiment; K2 is the vector coefficient, which is set to 25 in this embodiment. ij 、y ij x is the horizontal and vertical coordinates of the jth key point in the i-th frame before the current dynamic image frame. i=1,2,...,15; x 0j 、y 0j is the horizontal and vertical coordinates of the jth key point of the current dynamic image frame. i , BD i AC0 and BD0 are the vectors from point A to point C and from point B to point D in the i-th frame before the current dynamic image frame, respectively. AC0 and BD0 are the vectors from point A to point C and from point B to point D in the current dynamic image frame, respectively.

[0084] When there are multiple target detection frames in a dynamic image frame, the offset R of each image i Take the mean of the offsets corresponding to each target detection box.

[0085] In the first fifteen frames of the dynamic image frame, take the offset R i The minimum five frames are used as key frame images.

[0086] Compared to other keyframe selection methods, this embodiment differs in its approach: it uses geometric calculations to determine the five image frames with the smallest range of variation and selects them as keyframes. This ensures that the range of motion of dynamic characters can be calculated by the multi-view geometry algorithm, and their dynamic feature points are eliminated by the algorithm.

[0087] The area used is also different. The common area of ​​image frame activity in the keyframe is divided and multi-view geometry operations are performed only on the common area. This reduces the amount of computation and allocates computing resources to the multi-view geometry algorithm for dynamic culling, improving the robustness of the system.

[0088] The key frame overlaps with the current frame to the greatest extent, making a choice for subsequent multi-view geometry calculations. Figure 3 It can be seen that the original algorithm has a small number of feature points in the dynamic feature point elimination. As shown in Table 1, after running the TUM dataset for 30 seconds, the improved system effectively reduces the appearance of dynamic feature points and achieves better results.

[0089] Table 1 Comparative experimental results of improved key frame extraction

[0090]

[0091] 2-2-2. Extract the target area from the dynamic image frame. The specific process is as follows:

[0092] The amount of calculation for multi-view geometry calculation of the entire area of ​​the five key frame images is greater than that of the local area operation. In order to reduce the amount of calculation and the operation pressure of the system, only one common area in these five key frames is selected for multi-view geometry calculation.

[0093] The target area is an irregular quadrilateral with four corner points, and the values ​​are as follows: the horizontal coordinate and vertical coordinate of the first corner point (upper left corner point) of the target area are the minimum horizontal coordinate value and the maximum vertical coordinate value of the key point A in the dynamic image frame and the five key frame images; the horizontal coordinate and vertical coordinate of the second corner point (upper right corner point) of the target area are the maximum horizontal coordinate value and the maximum vertical coordinate value of the key point D in the dynamic image frame and the five key frame images; the horizontal coordinate and vertical coordinate of the third corner point (lower left corner point) of the target area are the minimum horizontal coordinate value and the minimum vertical coordinate value of the key point B in the dynamic image frame and the five key frame images; the horizontal coordinate and vertical coordinate of the fourth corner point (lower right corner point) of the target area are the maximum horizontal coordinate value and the minimum vertical coordinate value of the key point C in the dynamic image frame and the five key frame images.

[0094] Eliminate dynamic objects in the target area of ​​the dynamic image frame.

[0095] Afterwards, the five key frame images and dynamic image frames are processed using a multi-view geometry method, and the dynamic objects in each target detection frame in the dynamic image frame are removed. The image after removing the dynamic objects is input into the tracking thread of step three.

[0096] like Figure 4 and 5As shown in the figure, the principle of the multi-view geometry method is: project the feature point p in the key frame with the highest similarity to the current frame, calculate the projection of each feature point p from the key frame to the current frame as the feature point p', and calculate the projection depth d for each feature point p' according to the camera motion. proj The 3D point corresponding to each feature point is point P in the figure. Then calculate the disparity angle α between the back projection of point p and point p'. If the angle is greater than 30°, it is considered a dynamic feature point. Taking into account the reprojection error, the reprojection error Δd=d is calculated for the depth. proj -d'; where d' is the depth of the feature point in the current image frame. Multi-view geometry is used to extract feature points from key frames close to the current frame. When the difference between the projected depth and the actual depth is greater than a depth threshold, the point is considered a dynamic feature point. In this embodiment, the depth threshold is set to 0.4.

[0097] Step 3: Track the thread.

[0098] Perform sharp point tracking on the image processed in step 2.

[0099] The tracking thread receives the image frames and IMU data processed in step 2, calculates the camera's motion between adjacent image frames and the current pose, and minimizes the error by matching the reprojection of map features. In visual inertial mode, inertial residuals are included in the optimization to estimate the body velocity and IMU bias. When tracking is lost, the tracking thread attempts to relocalize the current frame in the Atlas multi-map. If the relocalization is successful, tracking will be resumed and the active map will be switched if necessary. Otherwise, after a period of time, the active map is stored as an inactive map, and a new active map is initialized from scratch. Then track the local map, relocalize, and create keyframes for the local mapping thread.

[0100] Step 4: Local mapping thread.

[0101] Based on the coordinates of the sharp points in the image obtained by tracking in step 3 and the depth information in the image, map points are added to the 3D map point cloud, and the world coordinates of the newly added map points in the 3D map point cloud are solved.

[0102] The local mapping thread is started when the ORB-SLAM3 system is initialized. It serves the loop detection thread, receives feature points and depth information from keyframes processed by the tracking thread, builds and updates the local map, and performs local map optimization to eliminate accumulated trajectory errors.

[0103] The local mapping thread receives keyframes passed by the tracking thread and internally decides whether to accept them based on whether to process them and whether to perform BA optimization. If they are rejected, the tracking thread cancels the creation of the keyframe. When a new keyframe is inserted into the local mapping thread, the keyframes in the local map and the connections between them are updated. BA optimization is also performed on the poses and map points of the keyframes in the current local map. Finally, redundant keyframes and their corresponding map points are deleted.

[0104] Step 5: Dense map construction thread

[0105] The system extracts the color information of newly added map points in the 3D map point cloud; the color information of each map point is projected from the corresponding pixel in the image frame. The map points in the 3D map point cloud are then stitched together to form a global dense map. Simultaneously, the system continuously uses the global bundle adjustment (BA) algorithm to optimize the map points and colorize the point cloud based on the colors of the feature points.

[0106] 5-1 Convert the two-dimensional coordinates of the RGB image and depth image into a three-dimensional point cloud

[0107] Constructing a dense map requires converting the two-dimensional coordinates in the camera into coordinates in the three-dimensional world. The purpose is to obtain richer scene information so as to more accurately describe the geometric structure and topological relationship of the environment and thus construct a three-dimensional point cloud. The world coordinate system is now defined as P w =[x w ,y w ,z w ], assume the image frame pixel coordinates (u, v) and the spatial depth value d, and obtain the transformation in the world coordinate system according to the camera pinhole model imaging principle.

[0108]

[0109] in s is the ratio of the actual distance to the spatial distance d, and in this embodiment, s=1000; Represents the camera's intrinsic parameters, which can be obtained through camera calibration. R is the camera rotation matrix; t is the camera translation matrix, which is estimated by the tracking thread and obtained through loop closure optimization.

[0110] 5-2 3D point cloud stitching and filtering

[0111] This embodiment uses PCL to stitch point clouds and generate a dense map. The coordinates of each point cloud are first solved and then colored, and the point cloud of each image frame is projected into the three-dimensional world coordinate system for fusion. In the process of point cloud stitching, the calculation becomes complicated due to the increasing number of point clouds. In addition, because noise appears in the image frame during the camera shooting process, the camera positioning error occurs, which affects the accuracy of point cloud stitching. If the noise causes positioning deviation, the stitched point cloud may be offset or deformed. Noise points will also be confused with real feature points, making the matching process more complicated and time-consuming. Therefore, it is necessary to remove noise points. This embodiment uses a voxel filter to remove noise.

[0112] The same object can appear frequently from different perspectives between adjacent image frames, displaying areas of overlapping perspectives. Generating a point cloud in this area produces a large number of similar point clouds. This redundant and complex information consumes a large amount of computing resources. The main principle of voxel filtering is to divide the point cloud data into a uniform cubic grid (voxels) and retain only one representative point in each voxel, thereby effectively filtering out noise points, completing spatial downsampling, and solving redundant point cloud data, providing a better foundation for subsequent point cloud processing and analysis.

[0113] Step 6: Loopback Detection Thread

[0114] The bag-of-words model is used to determine whether the robot has reached a location it has previously passed. If so, step seven is executed. If not, steps two to five are repeated.

[0115] The loop detection thread primarily detects and handles loops. Accumulated errors are a common problem in SLAM systems, and VSALM primarily addresses this through visual loop closures. The scene is described using a bag-of-words model, and when two bags of words are very similar, they are considered the same scene. This creates a constraint that, when applied backwards, eliminates accumulated errors. Loops return to previously passed locations or approximate previously passed trajectories. In these cases, the system must be able to detect these loops and perform appropriate processing to correct for any drift or errors in the map.

[0116] Step 7: Dense loop closure mapping

[0117] The current global dense map is optimized according to the global BA algorithm to obtain the final global dense map.

[0118] In this step, global BA can further optimize the map geometry by optimizing the camera pose and the positions of map points. For dense maps, global BA may optimize the position of each voxel or pixel in the map to improve the consistency and accuracy of the map.

[0119] The keyframe optimization operation is first performed in the local mapping thread, and then the bag-of-words model is used in the loop thread to detect whether there is a loop, and then the graph optimization model is used to perform global BA to optimize the structure of the entire map, including optimizing the position of all map points and the pose of all keyframes. By optimizing the map density, global BA can ensure that the points in the map are more evenly distributed, thereby improving the density of the map. By reducing the reprojection error, global BA can ensure that the position and pose of the points and keyframes in the map are more accurate. Optimizing map density and accuracy can improve reconstruction efficiency. Making the points in the dense map more evenly distributed and the position more accurate can make the subsequent reconstruction and map update process more efficient. The influence of dynamic feature points on walking people is removed in the image frame, and the results are as follows. Figure 5 、 6 and Figure 7 The final result of dense mapping is shown in Figure 8 .

Claims

1. A method for real-time construction of a three-dimensional map based on dynamic image acquisition, characterized by: The following steps are involved: Step 1: The robot equipped with a depth camera moves in the target area, and the depth camera continuously collects images; Step 2: Eliminate dynamic objects in the current image; 2-1. The current image captured is input into the target detection model to detect whether there is a dynamic object in the current image; if no dynamic object is detected in the current frame image, step three is executed directly; if a dynamic object is detected in the current frame image, the current frame image is set as a dynamic image frame and step 2-2 is executed; The target detection model includes a backbone network, an intermediate operation part and an output layer; a non-parametric attention mechanism module is added to the backbone network; the non-parametric attention mechanism module performs a weighted attention operation after the first downsampling and feature enhancement operation of the backbone network; the intermediate operation part includes a pooling module and nine convolution modules connected in sequence; the pooling module in the intermediate operation part is an SPPF pooling layer, the first convolution module is a C2 convolution layer, the second and fourth convolution modules are UpSample convolution layers, the third convolution module is a Concat operation, a C3 convolution layer and a C2 convolution layer connected in sequence, the fifth, seventh and ninth convolution modules are a Concat operation and a C3 convolution layer connected in sequence, and the sixth and eighth convolution modules are a C1 convolution layer; 2-2. Eliminate dynamic objects in dynamic image frames based on the multi-view geometry module; Selecting five key frame images from multiple frames preceding the dynamic image frame; setting a target area in the dynamic image frame and each key frame image; extracting and eliminating dynamic objects in the target area of ​​the dynamic image frame using a multi-view geometry method; Step 3: Track sharp points on the image processed in step 2; Step 4: Based on the coordinates of the sharp points in the image obtained by tracking in step 3 and the depth information in the image, map points are added to the 3D map point cloud, and the coordinates of the newly added map points in the 3D map point cloud are solved; Step 5: Extract the color information of the newly added map points in the 3D map point cloud; the color information of each map point is obtained by projecting the corresponding pixel point in the image frame; then, the map points in the 3D map point cloud are spliced ​​to form a global dense map; Step 6: Determine whether the robot has reached a location it has previously passed; if so, proceed to step 7; if not, continue to repeat steps 2 to 5; Step 7: Optimize the current global dense map to obtain the final global dense map.

2. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: The detection object of the target detection model described in step 2-1 is the human body.

3. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: In step 2-2, a key frame image is selected from fifteen frames of images preceding the dynamic image frame.

4. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1 or 3, characterized in that: The process of selecting key frame images in step 2-2 is as follows: The target detection frame in the dynamic image frame is set to have point A in the upper left corner, point D in the upper right corner, point B in the lower left corner, and point C in the lower right corner; point A, point D, point B, and point C are recorded as the first, second, third, and fourth key points of the target detection frame in the image respectively; Calculate the offset for each of the multiple frames preceding the dynamic image frame as follows: ； in, is the translation coefficient, which takes a value of 1; is the vector coefficient, and its value is 25; 、 are the horizontal and vertical coordinates of the jth key point in the i-th frame before the current dynamic image frame; i=1,2,...,15; 、 are the horizontal and vertical coordinates of the jth key point of the current dynamic image frame; 、 are respectively the vector from point A to point C and the vector from point B to point D in the i-th frame image before the current dynamic image frame; 、 are the vector from point A to point C and the vector from point B to point D in the current dynamic image frame respectively; Get offset The minimum five frames are used as key frame images.

5. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 4, characterized in that: When there are multiple target detection frames in a dynamic image frame, the offset of each image Take the mean of the offsets corresponding to each target detection box.

6. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: In step 2-2, the target area is a quadrilateral, and the coordinates of its four corner points are: the horizontal coordinate and vertical coordinate of the upper left corner point of the target area are the minimum horizontal coordinate and the maximum vertical coordinate of the upper left corner point in the dynamic image frame and each key frame image; The horizontal coordinate and vertical coordinate of the upper right corner point of the target area are the maximum horizontal coordinate and vertical coordinate of the upper right corner points in the dynamic image frame and each key frame image; The horizontal coordinate and vertical coordinate of the lower left corner point of the target area are the minimum horizontal coordinate and vertical coordinate of the lower left corner point in the dynamic image frame and each key frame image; The horizontal coordinate and vertical coordinate of the lower right corner point of the target area are the maximum horizontal coordinate and the minimum vertical coordinate of the lower right corner point in the dynamic image frame and each key frame image.

7. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: In step 2-2, the process of the multi-view geometry method is as follows: project the feature point p of the key frame image to the dynamic image frame, calculate the projection of each feature point p from the key frame image to the dynamic image frame as the feature point p', and calculate the projection depth of each feature point p' according to the camera motion. ; Then calculate the disparity angle α between the back projection of point p and point p'. If the angle is greater than 30°, it is considered a dynamic feature point; then calculate the reprojection error about the depth ;in, is the depth of the feature point in the current image frame; if the reprojection error If the depth is greater than the depth threshold, the feature point p' is determined to be a dynamic feature point.

8. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: The three feature maps of different scales of the output layer are input into a bounding box prediction module; the bounding box prediction module generates multiple candidate boxes; and the optimal detection box is selected from the multiple candidate boxes through a non-maximum suppression algorithm.

9. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 1, characterized in that: The backbone network includes five convolution modules connected in sequence; the first convolution module in the backbone network is the C0 convolution layer, the second convolution module is the C1 convolution layer, C3 convolution layer and the non-parameter attention mechanism module SimAM connected in sequence, and the third, fourth and fifth convolution modules are all C1 convolution layers and C3 convolution layers without the non-parameter attention mechanism module SimAM connected in sequence; the features output by the third convolution module of the backbone network are additionally passed to the fifth convolution module in the intermediate operation part; the features output by the fourth convolution module of the backbone network are additionally passed to the third convolution module in the intermediate operation part.

10. The method for real-time construction of a three-dimensional map based on dynamic image acquisition according to claim 9, characterized in that: The output layer includes three output modules of different sizes; the sizes of the three output modules are 80×80, 40×40, and 20×20 respectively; the first convolution module in the intermediate operation part is additionally transmitted to the ninth convolution module in the intermediate operation part; the third convolution module in the intermediate operation part is additionally transmitted to the seventh convolution module in the intermediate operation part; the output features of the fifth, seventh, and ninth convolution modules in the intermediate operation part are respectively transmitted to the three output modules of the output layer, forming an 80×80 feature map, a 40×40 feature map, and a 20×20 feature map; The C0 convolution layer performs a standard convolution operation; the C1 convolution layer is a convolution operation with a convolution kernel of 3*3 and a stride of 2; the C2 convolution layer is a convolution layer operation with a convolution kernel of 1*1 and a stride of 1; the C3 convolution layer is composed of three convolution layers; the SPPF pooling layer is spatial pyramid pooling; Concat is an operation used in deep learning to fuse feature maps of different scales; the UpSample convolution layer is the image upsampling process; and PAN is the part of Neck excluding the SPPF pooling layer.

Images