Three-dimensional map reconstruction method, device and equipment without dynamic object interference

Abstract

The application discloses a kind of three-dimensional map reconstruction methods, devices and equipment without dynamic object interference, realize high-precision three-dimensional map reconstruction based on dynamic barrier shelter background repair.This application combines deep learning technology, multi-view information fusion and prior frame data, dynamic barrier detection, segmentation and elimination scheme, while designing high-precision background repair strategy.Detection and segmentation dynamic object through deep learning model, accurately eliminate the influence of dynamic object by combining motion consistency, to avoid mismatching and pose drift under dynamic scene.In addition, by using the fusion of multi-view information and historical frame data, the static background obscured by dynamic objects is repaired, the high-precision completion of the background and the accurate restoration of the three-dimensional map are realized, thereby effectively constructing a three-dimensional map without dynamic object interference, enhancing the accuracy and robustness of the map, especially in complex dynamic environment, can maintain high stability.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and equipment for reconstructing a three-dimensional map without interference from dynamic objects. Background Art

[0002] After nearly three decades of development, Simultaneous Localization and Mapping (SLAM) technology has become a key technology in robotics, automation, and computer vision research. SLAM is ubiquitous in various fields, including micro aerial vehicles (MAVs), unmanned ground vehicles (UGVs), autonomous driving, virtual reality (VR), and augmented reality (AR), all of which rely on SLAM to provide reliable positioning and mapping results.

[0003] Currently known simultaneous localization and mapping technologies all assume that the scene they are working in is static and rigid. When these systems operate in dynamic scenes, the erroneous data associations caused by the static scene assumption can severely impair the accuracy and stability of the system. Generally speaking, the presence of dynamic objects in a scene divides all features into two categories: static features and dynamic features. Detecting and rejecting dynamic features is a key research question. Previous research can be divided into three categories: geometric information methods, semantic information methods, and methods that combine geometric and semantic information.

[0004] Geometric information methods use multi-view geometric constraints to detect dynamic objects. The idea is that a static feature point in the current image must lie on the epipolar line corresponding to the same feature point in the previous image. If the distance between a feature point and the corresponding epipolar line exceeds a set empirical threshold, the feature point is considered dynamic. Kundu et al.'s algorithm has advantages such as high speed and strong scene generalization. However, it lacks a high-level understanding of the scene, making it difficult to determine the empirical threshold, resulting in low accuracy. Semantic information methods use deep learning techniques to acquire semantic information about dynamic regions. They employ the YOLO object detection method to obtain prior semantic information about dynamic objects in the scene. Dynamic feature points are then removed based on this semantic information to improve tracking accuracy. However, YOLO's method of extracting semantic information through bounding boxes can lead to some static feature points being mistakenly identified as outliers and removed. In recent years, researchers have begun exploring methods that integrate geometric and semantic information. For example, for RGB-D cameras, the semantic segmentation results of Mask R-CNN are currently combined with multi-view geometric constraints to detect dynamic objects and remove outliers. Although this method has made some progress in SLAM systems in dynamic scenes, it still has the following shortcomings in terms of dynamic obstacle interference: (1) In the process of matching a large number of dynamic features in the environment. Since dynamic features are prone to rapid changes, a large number of mismatches lead to an increase in data noise, which seriously affects the accuracy of map construction. Reducing the number of mismatches and improving the accuracy of feature matching, accurately identifying and matching dynamic feature points is one of the main difficulties. (2) Dynamic obstacles are constantly changing in the scene, and the motion trajectory of dynamic objects is unstable. It is necessary to ensure that the algorithm can accurately identify and track obstacles while having sufficient robustness to cope with the dynamic changes of obstacles. It is particularly difficult to design a dynamic target detection and tracking algorithm with strong adaptability and high real-time performance. (3) Background repair for parts occluded by dynamic obstacles and reconstruction of high-precision three-dimensional maps. Background repair technology not only fills the texture of the occluded part. There are technical difficulties in synchronously reconstructing texture and depth information to maintain the authenticity of the occluded area. Background repair requires not only filling the texture details of the occluded part, but also recovering the depth information of the area while removing the object, thereby restoring the authenticity of the information of the occluded part. There are certain research difficulties.

[0005] In summary, existing 3D map reconstruction technology cannot effectively construct high-precision, dynamic object-free 3D maps when faced with dynamic object interference, and cannot meet the urgent demand for high-quality 3D maps in various fields. Summary of the Invention

[0006] The embodiments of the present invention provide a method, apparatus and device for reconstructing a three-dimensional map without dynamic object interference, which can effectively construct a high-precision three-dimensional map without dynamic object interference when facing dynamic object interference.

[0007] An embodiment of the present invention provides a method for reconstructing a three-dimensional map without interference from dynamic objects, comprising:

[0008] The environment data is collected by a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information;

[0009] Performing preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using the Mask R-CNN network to generate a dynamic region mask to separate the dynamic region from the static background; the dynamic region mask is used to mark the dynamic object region in the target image;

[0010] performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy;

[0011] Combining the processed dynamic region mask, using a target detection algorithm to identify dynamic objects in the target image again and generate a semantic segmentation mask;

[0012] Eliminate dynamic areas based on the semantic segmentation mask, and combine multi-view geometric verification technology to filter feature points of the target image that do not conform to the dynamic background, and retain the static parts;

[0013] Using multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation, background repair is performed on the target image's occluded area by the dynamic object to restore static background information;

[0014] The corresponding key frames of the repaired target image are selected to optimize the scene representation and camera pose, and the static information of the key frames is used to construct an octree map. The optimized scene representation and camera pose are integrated into the octree map to complete the three-dimensional map construction without interference from dynamic objects.

[0015] As an improvement to the above solution, after collecting environmental data by the depth camera to obtain the target image, the method further includes:

[0016] The target image is preprocessed to obtain a preprocessed target image; the preprocessing includes image grayscale, scale adjustment, denoising and smoothing operations.

[0017] As an improvement to the above solution, the Mask R-CNN network is used to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image to generate a dynamic region mask to separate the dynamic region from the static background, including:

[0018] Input the target image into the Mask R-CNN network and generate candidate regions through the region proposal network; the candidate regions are regions in the image that may contain dynamic objects;

[0019] For each candidate region, the Mask R-CNN network performs bounding box regression, category classification, and pixel-level segmentation, generating a bounding box, category information, and a dynamic region mask for the dynamic object. The bounding box includes the location coordinates of the dynamic object, the category information includes the category label of the dynamic object, and the dynamic region mask is used to mark the dynamic object region in the image.

[0020] The dynamic area is separated from the static background according to the generated dynamic area mask; the dynamic object area marked by the dynamic area mask will be excluded from the static background.

[0021] As an improvement to the above solution, performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy includes:

[0022] Obtaining a dynamic region mask generated by performing preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network;

[0023] Performing morphological operations on the generated dynamic region mask, including erosion and dilation, to remove noise at the mask boundary and small misjudgment areas; the morphological operations are used to improve the segmentation accuracy of the dynamic region mask;

[0024] The bounding box of the dynamic object is updated using Kalman filtering or optical flow method to ensure accurate tracking of the position of the dynamic obstacle in consecutive frames. The position update is based on the bounding box information of the dynamic object to ensure the position consistency of the dynamic object in consecutive frames.

[0025] The processed dynamic area mask is used for subsequent dynamic object removal and background restoration; the dynamic object area marked by the processed dynamic area mask will be excluded from the 3D map construction process and used to guide the background restoration of the area blocked by the dynamic object.

[0026] As an improvement to the above solution, the dynamic area mask after the combination processing is used to identify the dynamic objects in the target image again using the target detection algorithm and generate a semantic segmentation mask, including:

[0027] Using the processed dynamic region mask as input, the target image is processed using an object detection algorithm to generate a bounding box and category information of the dynamic object; the bounding box includes the position coordinates of the dynamic object in the target image, and the category information includes the category label of the dynamic object;

[0028] The processed dynamic region mask and the generated bounding box and category information of the dynamic object are used as input. The Mask R-CNN network performs pixel-by-pixel semantic segmentation on the target image to generate a semantic segmentation mask for the dynamic object. The semantic segmentation mask is used to mark the dynamic object area in the image.

[0029] As an improvement to the above solution, the method of eliminating dynamic areas based on the semantic segmentation mask and combining multi-view geometric verification technology to screen feature points of the target image that do not conform to the dynamic background and retain the static parts includes:

[0030] Obtaining a generated semantic segmentation mask; the semantic segmentation mask is used to mark the dynamic object area in the target image;

[0031] Eliminate dynamic regions in the target image according to the semantic segmentation mask; the dynamic regions are dynamic object regions marked by the semantic segmentation mask;

[0032] Combined with multi-view geometric verification technology, feature points in the target image that do not conform to the dynamic background are screened out; the multi-view geometric verification technology calculates the projection error between the key frame and the current frame to screen out points that do not conform to the dynamic background characteristics;

[0033] The filtered static feature points are retained; the static feature points are used for subsequent three-dimensional map construction.

[0034] As an improvement to the above solution, the method utilizes multi-view views and historical frames of the target image to perform background restoration on the target image's occluded area by the dynamic object through multi-view information fusion and depth data interpolation to restore static background information, including:

[0035] Acquire a historical frame of the target image; the historical frame is an image frame captured before the target image, and contains static background information of the area blocked by the dynamic object;

[0036] Integrate the information of different perspectives in the multi-perspective view, perform fusion operation on the static part information of the target image under different perspectives, and obtain the static part information after multi-perspective fusion;

[0037] For an area in the target image that is blocked by a dynamic object, using a depth data interpolation method, using known depth information of an unblocked area as a reference, interpolating the depth data of the blocked area to obtain depth information of the area;

[0038] Based on the static part information after multi-view fusion and the depth information obtained by depth data interpolation, the background repair operation is performed on the area occluded by the dynamic object, and the repaired information is used as the final static background information.

[0039] As an improvement to the above solution, the corresponding keyframes of the restored target image are selected to optimize the scene representation and camera pose, and the static information of the keyframes is used to build an octree map. The optimized scene representation and camera pose are integrated into the octree map to complete the construction of a 3D map without interference from dynamic objects, including:

[0040] Obtaining a restored target image, the restored target image including static background information of a region occluded by the dynamic object, the restored target image being obtained by performing background restoration on the region occluded by the dynamic object in the target image by utilizing multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation;

[0041] Selecting corresponding key frames of the restored target image, wherein the key frames are representative image frames in the restored target image and are screened from the restored target image by calculating inter-frame overlap and scene complexity;

[0042] Based on the static feature points and depth information in the key frames, the geometric structure of the scene is optimized using multi-view geometric constraints to obtain the optimized scene representation;

[0043] Through multi-view geometric constraints and the projection relationship of feature points in different key frames, the camera's rotation and translation parameters are optimized to obtain the optimized camera pose;

[0044] Convert the static information in different keyframes into nodes or units of the octree, present the three-dimensional space in a hierarchical manner, and construct an octree map;

[0045] The optimized scene representation and the information corresponding to the camera posture are integrated with the corresponding nodes or units in the octree map according to the structural rules of the octree to complete the construction of a three-dimensional map without interference from dynamic objects.

[0046] Another embodiment of the present invention provides a three-dimensional map reconstruction device without dynamic object interference, including:

[0047] An image acquisition module is used to acquire environmental data through a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information;

[0048] An image separation module is configured to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network to generate a dynamic region mask to separate the dynamic region from the static background; the dynamic region mask is used to mark the dynamic object region in the target image;

[0049] a processing module, configured to perform segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy;

[0050] A recognition module, configured to combine the processed dynamic region mask and use a target detection algorithm to re-identify dynamic objects in the target image and generate a semantic segmentation mask;

[0051] A screening module is used to eliminate dynamic areas based on the semantic segmentation mask, and to screen feature points of the target image that do not conform to the dynamic background by combining multi-view geometric verification technology, and retain static parts;

[0052] A restoration module is used to perform background restoration on the target image's occluded area by the dynamic object by using multi-view views and historical frames of the target image through multi-view information fusion and depth data interpolation to restore static background information;

[0053] The reconstruction module is used to select the corresponding key frames of the restored target image to optimize the scene representation and camera pose, and use the static information of the key frames to build an octree map, integrate the optimized scene representation and camera pose into the octree map, and complete the construction of a 3D map without interference from dynamic objects.

[0054] Another embodiment of the present invention provides a three-dimensional map reconstruction device without interference from dynamic objects, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the three-dimensional map reconstruction method without interference from dynamic objects described in the above-mentioned embodiment of the invention.

[0055] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0056] First, a depth camera captures environmental data to obtain a target image, including continuous image frames or RGB-D images with depth information. Based on this data, the Mask R-CNN network performs preliminary detection and pixel-by-pixel semantic segmentation of dynamic objects, generating a dynamic region mask. This allows for a preliminary separation of dynamic regions from the static background. This is due to the powerful feature extraction and classification capabilities of the Mask R-CNN network, which effectively distinguishes different objects and provides important dynamic region information for subsequent processing. The dynamic region mask is then processed for segmentation accuracy, employing techniques such as morphological operations to refine the boundaries of dynamic regions, improve segmentation accuracy, and ensure the accuracy of subsequent processing. Combined with the processed dynamic region mask, an object detection algorithm is then used to re-identify dynamic objects and generate a semantic segmentation mask, further refining the labeling of dynamic objects. This combined information from the processed mask allows for focusing on possible dynamic regions, improving recognition accuracy. Dynamic regions are then eliminated based on the semantic segmentation mask, and feature points that do not correspond to the dynamic background are screened using multi-view geometric verification techniques. Multi-view geometric verification leverages the geometric relationships between multiple viewpoints to ensure accurate screening, thereby retaining the static regions. Then, using multi-view views and historical frames, multi-view information fusion and depth data interpolation are used to inpaint areas occluded by dynamic objects. This leverages the complementarity of multi-view information and the advantages of depth data interpolation to restore more complete and accurate static background information. Finally, keyframes are selected from the inpainted target image, the scene representation and camera pose are optimized, and an octree map is constructed using the static information from the keyframes. This optimized information is then integrated into the octree. The octree structure facilitates hierarchical storage and representation of three-dimensional space, completing the construction of a 3D map free of dynamic object interference. The optimization of keyframes and the use of the octree during the construction process effectively integrate static information and improve the quality of map construction. As can be seen from the above analysis, the embodiments of the present invention, through technical means such as the Mask R-CNN network, segmentation accuracy processing, multi-view geometric verification technology, multi-view information fusion, depth data interpolation, and octree map construction, effectively address the problems of mismatching, pose drift, and background information loss caused by dynamic object interference in dynamic scenes, significantly improving the accuracy of 3D map reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a method for reconstructing a three-dimensional map without interference from dynamic objects provided by one embodiment of the present invention;

[0058] Figure 2 This is a technical roadmap provided by an embodiment of the present invention;

[0059] Figure 3 This is a technical idea diagram provided by an embodiment of the present invention;

[0060] Figure 4 This is a technical research framework diagram provided by an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the principle of multi-view geometric detection of dynamic points provided by an embodiment of the present invention;

[0062] Figure 6 2 is a diagram of a Mask R-CNN framework for instance segmentation provided by an embodiment of the present invention;

[0063] Figure 7 This is a dynamic obstacle segmentation effect diagram provided by an embodiment of the present invention;

[0064] Figure 8 This is a flow chart of a dynamic object culling algorithm provided by one embodiment of the present invention;

[0065] Figure 9 This is a dynamic feature elimination effect diagram provided by an embodiment of the present invention;

[0066] Figure 10 This is a diagram showing the principle of octree mapping provided by an embodiment of the present invention;

[0067] Figure 11 This is an octree map construction effect diagram provided by an embodiment of the present invention;

[0068] Figure 12 is a schematic diagram of a background restoration process provided by an embodiment of the present invention;

[0069] Figure 13 This is a three-dimensional mapping effect diagram provided by an embodiment of the present invention;

[0070] Figure 14 1 is a schematic structural diagram of a three-dimensional map reconstruction device without dynamic object interference provided by one embodiment of the present invention;

[0071] Figure 15 The figure is a schematic structural diagram of a three-dimensional map reconstruction device without dynamic object interference provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0073] See also Figure 1, is a flow chart of a method for reconstructing a three-dimensional map without interference from dynamic objects provided by one embodiment of the present invention. The method for reconstructing a three-dimensional map without interference from dynamic objects comprises:

[0074] The environment data is collected by a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information;

[0075] Performing preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using the Mask R-CNN network to generate a dynamic region mask to separate the dynamic region from the static background; the dynamic region mask is used to mark the dynamic object region in the target image;

[0076] performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy;

[0077] Combining the processed dynamic region mask, using a target detection algorithm to identify dynamic objects in the target image again and generate a semantic segmentation mask;

[0078] Eliminate dynamic areas based on the semantic segmentation mask, and combine multi-view geometric verification technology to filter feature points of the target image that do not conform to the dynamic background, and retain the static parts;

[0079] Using multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation, background repair is performed on the target image's occluded area by the dynamic object to restore static background information;

[0080] The corresponding key frames of the repaired target image are selected to optimize the scene representation and camera pose, and the static information of the key frames is used to construct an octree map. The optimized scene representation and camera pose are integrated into the octree map to complete the three-dimensional map construction without interference from dynamic objects.

[0081] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0082] First, a depth camera captures environmental data to obtain a target image, including continuous image frames or RGB-D images with depth information. Based on this data, the Mask R-CNN network performs preliminary detection and pixel-by-pixel semantic segmentation of dynamic objects, generating a dynamic region mask. This allows for a preliminary separation of dynamic regions from the static background. This is due to the powerful feature extraction and classification capabilities of the Mask R-CNN network, which effectively distinguishes different objects and provides important dynamic region information for subsequent processing. The dynamic region mask is then processed for segmentation accuracy, employing techniques such as morphological operations to refine the boundaries of dynamic regions, improve segmentation accuracy, and ensure the accuracy of subsequent processing. Combined with the processed dynamic region mask, an object detection algorithm is then used to re-identify dynamic objects and generate a semantic segmentation mask, further refining the labeling of dynamic objects. This combined information from the processed mask allows for focusing on possible dynamic regions, improving recognition accuracy. Dynamic regions are then eliminated based on the semantic segmentation mask, and feature points that do not correspond to the dynamic background are screened using multi-view geometric verification techniques. Multi-view geometric verification leverages the geometric relationships between multiple viewpoints to ensure accurate screening, thereby retaining the static regions. Then, using multi-view views and historical frames, multi-view information fusion and depth data interpolation are used to inpaint areas occluded by dynamic objects. This leverages the complementarity of multi-view information and the advantages of depth data interpolation to restore more complete and accurate static background information. Finally, keyframes are selected from the inpainted target image, the scene representation and camera pose are optimized, and an octree map is constructed using the static information from the keyframes. This optimized information is then integrated into the octree. The octree structure facilitates hierarchical storage and representation of three-dimensional space, completing the construction of a 3D map free of dynamic object interference. The optimization of keyframes and the use of the octree during the construction process effectively integrate static information and improve the quality of map construction. As can be seen from the above analysis, the embodiments of the present invention, through technical means such as the Mask R-CNN network, segmentation accuracy processing, multi-view geometric verification technology, multi-view information fusion, depth data interpolation, and octree map construction, effectively address the problems of mismatching, pose drift, and background information loss caused by dynamic object interference in dynamic scenes, significantly improving the accuracy of 3D map reconstruction.

[0083] For example, sensors such as RGB cameras, depth cameras, or lidar are used to collect environmental data, generating a series of continuous image frames or RGB-D images with depth information. These data will provide the basis for dynamic obstacle detection and segmentation.

[0084] As an improvement to the above embodiment, after collecting environmental data by the depth camera to obtain the target image, the method further includes:

[0085] The target image is preprocessed to obtain a preprocessed target image; the preprocessing includes image grayscale, scale adjustment, denoising and smoothing operations.

[0086] In this embodiment, to reduce the impact of image noise and illumination variations on subsequent detection, each image frame is preprocessed. These operations typically include image grayscale conversion, rescaling, denoising, and smoothing to reduce the interference of noise and illumination variations on detection. This allows the preprocessed image to more clearly reflect the shape and details of the target object. The preprocessed image can be expressed as: .

[0087] As an improvement to the above embodiment, the Mask R-CNN network is used to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image to generate a dynamic region mask to separate the dynamic region from the static background, including:

[0088] Input the target image into the Mask R-CNN network and generate candidate regions through the region proposal network; the candidate regions are regions in the image that may contain dynamic objects;

[0089] For each candidate region, the Mask R-CNN network performs bounding box regression, category classification, and pixel-level segmentation, generating a bounding box, category information, and a dynamic region mask for the dynamic object. The bounding box includes the location coordinates of the dynamic object, the category information includes the category label of the dynamic object, and the dynamic region mask is used to mark the dynamic object region in the image.

[0090] The dynamic area is separated from the static background according to the generated dynamic area mask; the dynamic object area marked by the dynamic area mask will be excluded from the static background.

[0091] In this embodiment, the powerful image analysis capabilities of the Mask R-CNN network are leveraged to perform multi-dimensional processing on the target image, accurately separating dynamic areas from the static background. This lays the foundation for the subsequent construction of a high-precision 3D map free of dynamic object interference. The technical effect is demonstrated by the ability, through a series of ordered operations, to accurately identify the areas containing dynamic objects within a complex target image, generate corresponding bounding boxes, category information, and dynamic area masks, and achieve a clear demarcation between the dynamic area and the static background, effectively preventing dynamic object information from being mixed with the static background. This ensures that subsequent map construction can be carried out based on accurate static background information, improving the accuracy and quality of the entire 3D map reconstruction.

[0092] Specifically, the first step is to input the target image into the Mask R-CNN network. As an advanced and mature deep learning architecture, its Region Proposal Network (RPN) takes the lead. Based on information such as pixel features in the target image, the RPN uses the feature extraction capabilities of a Convolutional Neural Network (CNN) to slide a window across the image. Through a series of operations such as convolution and pooling, it generates numerous candidate regions that may contain dynamic objects. These candidate regions are essentially local areas in the image that have been initially screened and are considered to have a high probability of containing dynamic objects. Their purpose is to narrow the scope of subsequent processing, focusing on parts of the image that are more likely to contain dynamic objects, avoiding indiscriminate and computationally intensive processing of the entire image, thereby improving processing efficiency and accuracy. In the second step, for each candidate region, the Mask R-CNN network's branching structure further performs key operations such as bounding box regression, category classification, and pixel-level segmentation. In the bounding box regression operation, the specially designed regression module in the Mask R-CNN network is utilized. This module, based on the rich feature information extracted from the candidate regions, learns the deviation between the true boundary position of the dynamic object in the image and the boundary of the candidate region. It then applies an appropriate loss function (such as the SmoothL1 loss function) to continuously optimize and adjust the bounding box coordinate parameters, ultimately generating a precise bounding box for the dynamic object. This bounding box contains the exact coordinates of the dynamic object in the target image. For example, with the upper left corner of the image as the origin, the positive x-axis is horizontally to the right, and the positive y-axis is vertically downward. The bounding box coordinate information (x1, y1, x2, y2) clearly defines the specific location range of the dynamic object on the two-dimensional image plane, providing a quantitative basis for subsequent accurate spatial position determination of the dynamic object. For category classification, the classification branch of the Mask R-CNN network matches and determines features within each candidate region based on the feature patterns of various objects previously learned from a large-scale annotated dataset. It uses structures such as multi-layer fully connected layers to map the feature vectors of the candidate area to the corresponding category space, and ultimately outputs the category information of the dynamic object, that is, to clarify the specific category label to which it belongs, such as pedestrians, vehicles, animals, and other specific categories. This classification result is of great significance for subsequent targeted processing and analysis based on the characteristics of different types of dynamic objects, as well as distinguishing the impact of different dynamic objects in complex scenes. The pixel-level segmentation operation is achieved through the unique mask generation branch in the Mask R-CNN network. This branch will perform a detailed semantic analysis on each pixel in the candidate area, assign a corresponding semantic value based on the degree of association between the pixel and the dynamic object, and finally generate a dynamic area mask after a complex calculation and mapping process.The mask marks the dynamic object area in the image with pixel-level accuracy. For example, in the mask image, the pixels belonging to the dynamic object are marked with a specific value (such as 1), and the pixels belonging to the static background are marked with another value (such as 0). In this way, the boundary between the dynamic area and the static background can be clearly demarcated with extremely high resolution, providing an accurate basis for the subsequent complete separation of the two. The last step is to separate the dynamic area from the static background based on the generated dynamic area mask. Specifically, at the level of the entire target image, according to the pixel values ​​marked by the dynamic area mask, those parts marked as dynamic object areas (that is, the corresponding pixel values ​​are specific values ​​representing dynamic objects) are clearly separated from the overall image, so that they are completely isolated from the static background area. These dynamic object areas marked by the dynamic area mask will be strictly excluded from the static background, thereby ensuring the purity and integrity of the static background information. This provides accurate and dynamic-free static background basic data for subsequent operations such as further dynamic object recognition based on the octree data structure, feature point screening using multi-view geometry verification technology, and background repair, ensuring that the entire 3D map reconstruction process can proceed in the correct and precise direction.

[0093] As an improvement to the above embodiment, performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy includes:

[0094] Obtaining a dynamic region mask generated by performing preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network;

[0095] Performing morphological operations on the generated dynamic region mask, including erosion and dilation, to remove noise at the mask boundary and small misjudgment areas; the morphological operations are used to improve the segmentation accuracy of the dynamic region mask;

[0096] The bounding box of the dynamic object is updated using Kalman filtering or optical flow method to ensure accurate tracking of the position of the dynamic obstacle in consecutive frames. The position update is based on the bounding box information of the dynamic object to ensure the position consistency of the dynamic object in consecutive frames.

[0097] The processed dynamic area mask is used for subsequent dynamic object removal and background restoration; the dynamic object area marked by the processed dynamic area mask will be excluded from the 3D map construction process and used to guide the background restoration of the area blocked by the dynamic object.

[0098] In this embodiment, after obtaining the initial dynamic area mask, morphological operations are used to remove noise at the mask boundary and small areas of misjudgment. At the same time, the position of the dynamic object boundary box is updated with the help of Kalman filtering or optical flow method to ensure the accuracy of position tracking in continuous frames. Ultimately, the processed dynamic area mask can more accurately mark the dynamic object area, effectively reduce the errors caused by insufficient accuracy of the dynamic area mask in subsequent processing links, improve the accuracy of dynamic object removal and background restoration, and thus enhance the quality and accuracy of the entire three-dimensional map construction, so that it can better serve various application scenarios that rely on high-precision maps.

[0099] Specifically, the Mask R-CNN network performs preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image to generate a dynamic region mask. This mask is a preliminary marker of the dynamic object regions obtained by analyzing the image using a deep learning network in the previous step. However, it may have certain accuracy issues, such as unclear boundaries and small-scale misclassifications, necessitating further processing. Next, the generated dynamic region mask is subjected to morphological operations, primarily erosion and dilation. The erosion operation involves convolving the dynamic region mask with a structuring element (typically a matrix of a specific shape and size, such as a rectangle or circle). During this operation, the structuring element is slid across the mask image. If any portion of the area covered by the structuring element is background pixels, the corresponding center pixel is reclassified as a background pixel. This gradually "erodes" the boundary pixels of the dynamic region, removing boundary noise and isolated, small areas misclassified as dynamic objects, resulting in clearer and more regular boundaries. The dilation operation, in contrast, also slides a structuring element across the mask image. However, if the area covered by the structuring element contains pixels belonging to a dynamic object, the corresponding central pixel in that area is identified as a dynamic object pixel. This method fills holes within the dynamic region or compensates for dynamic object pixels that may have been over-removed by the erosion operation, improving the integrity of the dynamic region, further optimizing the quality of the dynamic region mask, and enhancing segmentation accuracy. The dynamic object's bounding box is then updated using a Kalman filter or optical flow method. The Kalman filter is an algorithm based on a linear system state equation that optimally estimates the system state using input and output observation data. In this scenario, it predicts the position of the dynamic object's bounding box in the current frame based on state information such as the position and velocity of the dynamic object's bounding box in the previous frame (this information can be obtained from previous processing steps) combined with observation data in the current frame (such as features detected in the image that may be related to the dynamic object). The bounding box is then corrected based on the actual observed new position information. This iterative update ensures accurate tracking of the dynamic object's position across consecutive frames, ensuring its positional consistency and avoiding positional deviations caused by factors such as object motion and image noise. The optical flow method primarily calculates the speed and direction of pixel movement based on grayscale changes in an image, thereby inferring the motion of dynamic objects. By analyzing the correspondence between pixels in adjacent frames, the exact position of the dynamic object's bounding box in different frames is determined, and the bounding box position is updated. This allows the dynamic object to be accurately located in successive image frames, providing accurate dynamic object position information for subsequent processing. Finally, the dynamic area mask after this processing is used for subsequent dynamic object removal and background restoration.The processed dynamic area mask has improved accuracy, and the dynamic object areas it marks can be accurately identified. In the subsequent 3D map construction process, these accurately marked dynamic object areas will be strictly excluded to prevent them from interfering with static background information and map construction. At the same time, the dynamic area mask can serve as a guide when performing background repair in areas occluded by dynamic objects. For example, based on the range of the marked dynamic area, the surrounding unobstructed static background information and other relevant data are rationally utilized, and the background of the occluded area is restored through the corresponding repair algorithm. This makes the background repair work more accurate and effective, laying a solid foundation for the ultimate construction of a high-precision 3D map without interference from dynamic objects.

[0100] As an improvement to the above embodiment, the combined processed dynamic area mask uses a target detection algorithm to identify dynamic objects in the target image again and generate a semantic segmentation mask, including:

[0101] Using the processed dynamic region mask as input, the target image is processed using an object detection algorithm to generate a bounding box and category information of the dynamic object; the bounding box includes the position coordinates of the dynamic object in the target image, and the category information includes the category label of the dynamic object;

[0102] The processed dynamic region mask and the generated bounding box and category information of the dynamic object are used as input. The Mask R-CNN network performs pixel-by-pixel semantic segmentation on the target image to generate a semantic segmentation mask for the dynamic object. The semantic segmentation mask is used to mark the dynamic object area in the image.

[0103] In this embodiment, by combining the processed dynamic region mask with an object detection algorithm, accurate identification and semantic segmentation of dynamic objects in the target image are achieved, thereby generating a high-quality semantic segmentation mask. Specifically, the target image is first processed using an object detection algorithm (such as YOLO, Faster R-CNN, or SSD) to generate bounding boxes and category information for the dynamic objects, clarifying their location and type. Then, using the processed dynamic region mask and the generated bounding boxes and category information as input, a pixel-by-pixel semantic segmentation operation is performed through the Mask R-CNN network to generate a semantic segmentation mask for the dynamic objects. This mask accurately marks the dynamic object areas in the image, ensuring that the edges of the dynamic objects are clear and separated from the background. Through this technical solution, the system can effectively reduce the interference of dynamic objects on map construction, improve the accuracy and stability of the 3D map, and provide a reliable foundation for subsequent static background inpainting and map construction.

[0104] Specifically, the object detection algorithm first processes the target image using the processed dynamic region mask as input, generating bounding boxes and category information for dynamic objects. During this process, the object detection algorithm extracts and analyzes features from the target image based on the information provided by the dynamic region mask regarding areas where dynamic objects may be present. For feature extraction, various feature descriptors, such as gradient-based features, texture features, or color histograms, may be used. By extracting these features from regions that may contain dynamic objects and combining them with machine learning or deep learning classification methods, the algorithm determines the dynamic object category of the region and determines the location of its bounding box in the target image. Specifically, the object detection algorithm uses the dynamic region mask as prior information to narrow the detection range, searching for features only in areas marked as dynamic. To generate the bounding box, the algorithm applies a regression algorithm (such as linear regression or bounding box regression from deep learning) based on the extracted feature information to find the bounding box that best fits the dynamic object and accurately determines its location coordinates, thereby accurately delineating the dynamic object's range. Generating category information relies on a classification algorithm, potentially using a support vector machine (SVM) or a deep learning classifier (such as a fully connected layer plus a softmax function). The extracted features are mapped to different category labels to determine the category of dynamic objects, such as pedestrians, vehicles, and animals. Next, the Mask R-CNN network uses the processed dynamic region mask and the generated bounding boxes and category information of the dynamic objects as input to perform pixel-by-pixel semantic segmentation on the target image, generating a semantic segmentation mask for the dynamic objects. In this step, the Mask R-CNN network leverages its powerful convolutional neural network architecture to first extract and fuse features from the input. It then fuses features at different levels through multiple convolutional and pooling layers, enabling the network to capture richer semantic information in the image. During pixel-by-pixel semantic segmentation, the Mask R-CNN network leverages the previously obtained bounding box and category information to refine the segmentation task down to the pixel level. For each pixel, the network predicts the probability of it belonging to a dynamic object and assigns a corresponding category label to each pixel belonging to a dynamic object, ultimately generating a semantic segmentation mask. A fully convolutional network (FCN) structure might be used here to map the network output from the feature map back to the pixel size of the original image. Upsampling and deconvolution operations ensure that the semantic segmentation mask has the same resolution as the original image. For example, in the branch structure of the Mask R-CNN network, one branch is responsible for generating bounding boxes and category information, while the other branch is responsible for generating pixel-by-pixel mask information.In this way, the final semantic segmentation mask can accurately mark the dynamic object area in the image and accurately mark each pixel as belonging to a dynamic object or a static background, providing a more refined basis for subsequent dynamic area elimination and precise extraction of the static background, and helping to achieve more accurate three-dimensional map construction without interference from dynamic objects.

[0105] As an improvement to the above embodiment, the method of eliminating dynamic areas based on the semantic segmentation mask and combining multi-view geometric verification technology to screen feature points of the target image that do not conform to the dynamic background and retain the static parts includes:

[0106] Obtaining a generated semantic segmentation mask; the semantic segmentation mask is used to mark the dynamic object area in the target image;

[0107] Eliminate dynamic regions in the target image according to the semantic segmentation mask; the dynamic regions are dynamic object regions marked by the semantic segmentation mask;

[0108] Combined with multi-view geometric verification technology, feature points in the target image that do not conform to the dynamic background are screened out; the multi-view geometric verification technology calculates the projection error between the key frame and the current frame to screen out points that do not conform to the dynamic background characteristics;

[0109] The filtered static feature points are retained; the static feature points are used for subsequent three-dimensional map construction.

[0110] In this embodiment, semantic segmentation masks and multi-view geometric verification techniques are used to accurately remove dynamic areas from the target image and filter out feature points that match the static background characteristics, ensuring that the subsequent 3D map construction process is based solely on pure static information. Specifically, dynamic areas are marked with precise semantic segmentation masks, effectively removing them from the target image. Multi-view geometric verification techniques are also used to further filter out feature points that may be misjudged as static but do not actually match the dynamic background. This ensures that the retained static areas are highly accurate, providing a high-quality static information source for constructing a 3D map free of dynamic object interference, significantly improving the quality and accuracy of the final 3D map and meeting the demand for high-precision maps.

[0111] Specifically, this step begins by obtaining a generated semantic segmentation mask. This semantic segmentation mask, obtained through the preceding steps, represents dynamic object regions in the target image using a specific pixel labeling scheme. For example, different pixel values ​​may be used to distinguish between dynamic and static regions, potentially marking pixels in dynamic regions as 1 and pixels in static regions as 0. Next, dynamic regions in the target image are removed based on the semantic segmentation mask. This is done by iterating over each pixel in the target image. When a pixel is detected as dynamic in the semantic segmentation mask (e.g., with a pixel value of 1), it is removed from the target image or set to a background pixel value, thereby removing the dynamic regions. This may involve a pixel-level screening and processing algorithm to ensure that dynamic regions are completely excluded and prevent them from interfering with subsequent processing. Next, multi-view geometry verification techniques are used to filter feature points in the target image that do not correspond to the dynamic background. In this step, the principles of multi-view geometry are applied to calculate the projection matrix between the keyframe and the current frame. Specifically, a projection formula based on camera intrinsic and extrinsic parameters is used to project feature points from the keyframe onto the current frame, and then the projection error is calculated. The projection formula here can be ,in is the camera intrinsic parameter matrix, is the external parameter matrix, is the coordinate of the feature point in three-dimensional space, It is the two-dimensional coordinate projected to the current frame. After calculating the projection error, a reasonable error threshold is set. For feature points whose projection error exceeds the threshold, they are judged as feature points that do not conform to the dynamic background. When calculating the projection error, the Euclidean distance formula is used. To calculate the distance between the projected point and the actual point, use it as an error metric to filter out points that do not conform to the dynamic background features. Finally, retain the filtered static feature points. These static feature points have been strictly screened to exclude dynamic areas and feature points that do not conform to the dynamic background, ensuring that they can truly reflect the static background information and provide a reliable foundation for subsequent three-dimensional map construction. When storing and managing these static feature points, data structures such as arrays or linked lists may be used to store them in order so that subsequent algorithms can easily access and use this information, thereby providing accurate static information support for building high-quality three-dimensional maps. Through this series of steps, the static part of the target image is accurately extracted, avoiding the adverse effects of dynamic objects and their possible misjudged feature points on subsequent map construction, which helps to ultimately achieve high-precision three-dimensional map construction without interference from dynamic objects.

[0112] As an improvement to the above embodiment, the method of using multi-view views and historical frames of the target image to perform background restoration on the target image's occluded area by the dynamic object through multi-view information fusion and depth data interpolation to restore static background information includes:

[0113] Acquire a historical frame of the target image; the historical frame is an image frame captured before the target image, and contains static background information of the area blocked by the dynamic object;

[0114] Integrate the information of different perspectives in the multi-perspective view, perform fusion operation on the static part information of the target image under different perspectives, and obtain the static part information after multi-perspective fusion;

[0115] For an area in the target image that is blocked by a dynamic object, using a depth data interpolation method, using known depth information of an unblocked area as a reference, interpolating the depth data of the blocked area to obtain depth information of the area;

[0116] Based on the static part information after multi-view fusion and the depth information obtained by depth data interpolation, the background repair operation is performed on the area occluded by the dynamic object, and the repaired information is used as the final static background information.

[0117] This embodiment utilizes multi-perspective views and historical frames of the target image, fusion of multi-perspective information, and interpolation of depth data to address the problem of background restoration in areas of the target image occluded by dynamic objects. This approach aims to restore complete and accurate static background information, providing a high-quality static background foundation for the subsequent construction of a 3D map free of dynamic object interference. This embodiment fully leverages the information advantages of multiple perspectives and historical frames. Through fusion and interpolation, it fills in missing static background information obscured by dynamic objects, significantly improving the completeness and accuracy of this static background information, thereby enhancing the overall 3D map reconstruction quality and ensuring superior results in practical application scenarios.

[0118] Specifically, the first fundamental step is to acquire historical frames of the target image. These frames are image frames captured before the target image. They contain important information, particularly static background information that may not have been occluded by the dynamic object in the previous frames. This information is crucial for background restoration in areas currently occluded by the dynamic object. In practice, these historical frames may be stored in a data storage system, such as a database or file system, and organized by timestamp or other identifiers to facilitate rapid subsequent search and retrieval. Next, the information from different perspectives in the multi-view view is integrated, and the static portion of the target image from different perspectives is fused to obtain the multi-view fused static portion. Image registration techniques are employed in this process to ensure accurate alignment of images from different perspectives. For example, feature-based image registration methods extract feature points (such as SIFT, SURF, or ORB feature points) and use feature matching algorithms (such as RANSAC) to find correspondences between images from different perspectives, thereby achieving precise image alignment. For the fusion of static information, the weighted average method can be used to assign different weights according to the reliability or importance of different perspectives, and perform weighted summation of the static information under each perspective. For each pixel position, the fused pixel value can be expressed as ,in It is The pixel value of the viewing angle, is the corresponding weight, and . This can combine the advantages of different perspectives and improve the accuracy and robustness of the static part information. Then, for the area in the target image that is occluded by the dynamic object, the depth information is processed using the depth data interpolation method. Based on the known depth information of the unoccluded area, a variety of algorithms can be used to interpolate the depth information of the occluded area, such as bilinear interpolation, bicubic interpolation or spline function-based interpolation methods. Taking bilinear interpolation as an example, for a certain pixel point in the occluded area , its depth value The four surrounding pixels with known depth values ​​can be The calculation formula is based on existing methods. This approach utilizes the depth information of the surrounding unobstructed areas to reasonably estimate the depth of the occluded area, providing depth-dimensional information support for subsequent background restoration. Finally, based on the static information from multi-view fusion and the depth information obtained through depth data interpolation, background restoration is performed on the area occluded by the dynamic object. A background restoration algorithm based on texture synthesis may be employed, using the fused static information as texture information. Combined with the depth information, the pixel values ​​of the occluded area are restored by copying, filling, or deforming the texture. For example, the relative positional relationship between the occluded area and the surrounding unobstructed areas is determined based on depth information. Based on this relationship, appropriate texture blocks are selected from the static information from multi-view fusion and copied or deformed to fill the occluded area, ensuring a natural transition between the restored information and the surrounding static background. Ultimately, the restored information serves as the final static background information. This provides a complete and accurate static background for subsequent 3D map construction, avoiding the impact of occluded areas by dynamic objects and ensuring high-quality 3D map construction.

[0119] As an improvement to the above embodiment, the corresponding key frames of the restored target image are selected to optimize the scene representation and camera pose, and the static information of the key frames is used to construct an octree map. The optimized scene representation and camera pose are integrated into the octree map to complete the construction of a three-dimensional map without interference from dynamic objects, including:

[0120] Obtaining a restored target image, the restored target image including static background information of a region occluded by the dynamic object, the restored target image being obtained by performing background restoration on the region occluded by the dynamic object in the target image by utilizing multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation;

[0121] Selecting corresponding key frames of the restored target image, wherein the key frames are representative image frames in the restored target image and are screened from the restored target image by calculating inter-frame overlap and scene complexity;

[0122] Based on the static feature points and depth information in the key frames, the geometric structure of the scene is optimized using multi-view geometric constraints to obtain the optimized scene representation;

[0123] Through multi-view geometric constraints and the projection relationship of feature points in different key frames, the camera's rotation and translation parameters are optimized to obtain the optimized camera pose;

[0124] Convert the static information in different keyframes into nodes or units of the octree, present the three-dimensional space in a hierarchical manner, and construct an octree map;

[0125] The optimized scene representation and the information corresponding to the camera posture are integrated with the corresponding nodes or units in the octree map according to the structural rules of the octree to complete the construction of a three-dimensional map without interference from dynamic objects.

[0126] In this embodiment, keyframes are selected from the restored target image, and the information in the keyframes is then used to optimize the scene representation and camera pose using multi-view geometric constraints. This optimized information is then integrated into the octree map, thereby completing the construction of a 3D map free of interference from dynamic objects. Based on the completed background restoration, this embodiment achieves a precise representation of the 3D scene by carefully selecting keyframes and optimizing the scene and camera pose. Ultimately, the static information is presented in the form of an octree map. This not only enables efficient storage and visualization of 3D information, but also eliminates interference from dynamic objects, providing high-quality maps for applications in fields such as autonomous driving and virtual reality, and improving the accuracy and reliability of scene reconstruction.

[0127] Specifically, first, the inpainted target image is acquired. This step builds on the previous background inpainting work by utilizing multi-viewpoint views and historical frames of the target image. Multi-view information fusion and depth data interpolation techniques are used to inpaint the background of areas occluded by dynamic objects. In specific implementations, multi-view information fusion may employ feature point matching, such as using the SIFT or SURF feature point algorithms, to find matching feature points between images from different viewpoints. Information from these matching points is then fused together. Depth data interpolation utilizes known depth information. For occluded areas, depth data is filled in based on the depth information of surrounding unobstructed areas using algorithms such as bilinear interpolation or bicubic interpolation. Ultimately, the inpainted target image contains complete static background information. Next, keyframes corresponding to the inpainted target image are selected. Representative keyframes are selected by calculating inter-frame overlap and scene complexity. Inter-frame overlap can be calculated using feature point matching results, measuring the number of matching points or the proportion of matching points to the total number of feature points. Scene complexity can be assessed based on image texture complexity, depth variation, and edge information. For example, gradient operators (such as the Sobel operator) can be used to calculate gradient magnitude and direction, which can be used as indicators of scene complexity. Taking these factors into consideration, keyframes are selected from the restored target image to avoid redundant information and ensure that the keyframes effectively reflect the key scene information. Then, based on the static feature points and depth information in the keyframes, multi-view geometric constraints are used to optimize the scene's geometric structure, resulting in an optimized scene representation. In this process, multi-view geometric constraints reconstruct 3D points in the scene based on the position and depth information of feature points in different keyframes using triangulation principles, thereby optimizing the scene's geometric structure. For example, corresponding feature points from multiple sets of keyframes are used to calculate the coordinates of points in 3D space using triangulation formulas based on their projection relationships at different viewpoints. These 3D points are then used to adjust the scene's geometric elements, such as planes and curves, to achieve an optimized scene representation. For camera pose optimization, the camera's rotation and translation parameters are optimized through multi-view geometric constraints and the projection relationship of feature points in different keyframes to obtain the optimized camera pose. In the specific implementation, projection equations are established based on the projection position relationship of feature points in different keyframes, such as using the essential matrix or homography matrix, and then the rotation and translation parameters of the camera are obtained by solving these equations. Optimization algorithms such as nonlinear least squares can be used to minimize the error between the actual projection point and the projection point calculated based on the current camera pose, and the camera pose is continuously optimized iteratively. The static information in different keyframes is converted into nodes or units of an octree, and the three-dimensional space is presented in a hierarchical manner to construct an octree map.The specific operation involves recursively partitioning the three-dimensional space into eight subspaces. For each keyframe, the static information is assigned to the corresponding octree node or cell based on its 3D spatial position. For a 3D point (x, y, z), its coordinates are used to determine which octree cell it belongs to and store it in the corresponding cell. By continuously partitioning the space, a hierarchical structure is formed. This structure can efficiently store and manage 3D information and flexibly display maps of varying precision based on the required level of detail. Finally, the optimized scene representation and camera pose information are integrated with the corresponding nodes or cells in the octree map according to the structural rules of the octree. For the optimized scene representation, geometric structure information (such as points, lines, and surfaces) is added to the corresponding octree nodes or cells, while camera pose information can be stored in the octree metadata or associated with the octree nodes through additional data structures. In this way, the scene representation and camera pose information are integrated into the octree map, and finally the three-dimensional map construction without dynamic object interference is completed, ensuring that the entire three-dimensional map contains both accurate scene information and accurate camera perspective information, providing complete, accurate and dynamic interference-free map information for subsequent applications.

[0128] To facilitate understanding of the above embodiment, the following detailed description is given here:

[0129] This embodiment proposes a 3D map reconstruction method that combines theoretical research with synthetic dataset verification for occluded background restoration technology in dynamic scenes. Synthetic datasets are used to simulate a variety of typical dynamic scenes, and the performance of the proposed method is systematically verified and compared. The accuracy of dynamic object tracking and background restoration is evaluated, and the method is continuously adjusted and improved to ensure high-precision 3D map reconstruction in complex dynamic scenes. The technical route of this embodiment is shown in Figure 2 .

[0130] To better eliminate the influence of dynamic obstacles and fully reconstruct the dynamic background in dynamic scenes, this embodiment leverages the advantages of deep learning in object detection. It combines a convolutional neural network (CNN) with prior information and depth information from the static scene to eliminate dynamic objects and thereby restore the static background obscured by dynamic obstacles. To detect dynamic objects, the experiment employed an object detection algorithm and a convolutional neural network to perform semantic segmentation on the input image. The masks obtained from the semantic segmentation network effectively label dynamic objects in the image and serve as stable and reliable prior constraints to guide the dynamic SLAM system.

[0131] Taking the original RGB image as input, a specialized dynamic processing process is used to remove dynamic objects and output a binary mask of potential dynamic or movable objects in the image. In each mapping iteration, keyframes are selected to optimize the scene representation and camera pose and remove dynamic objects. For the removed dynamic targets, the occluded background is repaired using static information obtained from the previous viewpoint to synthesize a realistic image without dynamic targets. The repaired image contains more scene information, making the map presentation more accurate and enhancing the stability of camera tracking. See the technical concept architecture for details. Figure 3 .

[0132] In view of the complexity and diversity of dynamic scenes, and to effectively solve the core problem of building 3D maps in dynamic scenes, this embodiment focuses on the detection and segmentation of dynamic targets and the reconstruction of static backgrounds. The following problems are often encountered in the process of mapping dynamic environments. Ultimately, a system is built that can efficiently and accurately generate 3D maps in dynamic environments, providing a solid technical foundation for intelligent systems to perceive the environment. The main research of each part is as follows:

[0133] 1) Detection and segmentation of dynamic obstacles: In dynamic environments, traditional SLAM systems often fail to distinguish between dynamic and static feature points, leading to mismatches and decreased map positioning accuracy. The motion interference of dynamic obstacles (such as pedestrians and vehicles) is the main cause of these problems. This embodiment introduces deep learning models (such as SegNet) and motion consistency detection technology to accurately identify dynamic targets, segment them, and remove dynamic feature points. Combined with semantic segmentation technology, the system can distinguish between static backgrounds and dynamic obstacles in real time, eliminating feature points of dynamic objects during map construction, effectively reducing the pose drift problem caused by interference from dynamic obstacles, and enhancing the system's robustness and adaptability in complex dynamic environments.

[0134] 2) Static background reconstruction: The occlusion of dynamic obstacles will cause the loss of information in the background area, which will affect the integrity and coherence of the three-dimensional map. In order to restore the occluded static background information, this embodiment designs a background repair strategy that combines multi-perspective information and prior frame data. When processing scenes occluded by dynamic objects, the system makes full use of the information fusion in multi-perspective images and the texture and depth data in the historical frames to complete and restore the background area with high precision. By performing state tracking and depth reasoning on the occluded area of ​​the dynamic target, the accurate restoration of the background texture and depth features of the occluded area is ensured, thereby significantly improving the detail expression and overall accuracy of the three-dimensional map, providing strong technical support for the construction of high-quality maps in complex dynamic scenes, thereby restoring a static three-dimensional map that can significantly reduce the interference of dynamic obstacles.

[0135] The main framework of this embodiment combines the object detection and instance segmentation capabilities of Mask R-CNN to achieve high-precision 3D map construction in dynamic environments. Through the deep neural network structure of Mask R-CNN, the system is able to first detect and segment dynamic objects in the scene. Its region proposal network generates candidate regions, while subsequent branches are responsible for performing bounding box regression, category classification, and pixel-level segmentation. This allows accurate identification of dynamic objects in complex scenes and generates detailed segmentation masks to ensure that the edges of dynamic objects are clear and clearly separated from the background.

[0136] After obtaining the segmentation masks for dynamic objects, the system further filters and eliminates dynamic features in the scene using multi-view geometric features, filtering out objects that do not conform to the static background characteristics, significantly improving the stability of the 3D reconstruction process. During this process, Mask R-CNN's mask and feature attributes significantly improve the accuracy of scene parsing, ensuring that interfering features do not affect map construction.

[0137] After removing dynamic objects, the system needs to repair the occluded background areas to ensure the integrity of the 3D map. With the help of depth camera data and multi-view information, the system uses semantic segmentation and static information from historical frames to complete the background. Through deep learning feature synthesis technology, it can ensure the true restoration of the background after removing dynamic objects, providing a stable and accurate foundation for subsequent 3D reconstruction. The main research framework is shown in Figure 4 .

[0138] Step 1: Dynamic content recognition and segmentation based on Mask R-CNN and multi-view geometry

[0139] In the case of RGB-D, multi-view geometry is used to improve dynamic content segmentation from two aspects. To this end, it is necessary to know the camera pose, and a low-cost tracking module has been implemented to localize the camera within the created scene map. The general dynamic and static obstacle detection and segmentation method mainly consists of the following steps: data acquisition and preprocessing, dynamic obstacle detection, and dynamic object recognition.

[0140] Use sensors such as RGB cameras, depth cameras, or lidar to collect environmental data and generate a series of continuous image frames. Or RGB-D images with depth information. This data will provide the basis for dynamic obstacle detection and segmentation. To reduce the impact of image noise and lighting changes on subsequent detection, each frame of the image must be preprocessed. These operations typically include image grayscale, rescaling, denoising, and smoothing to reduce the interference of noise and lighting changes on detection. This allows the preprocessed image to more clearly reflect the shape and details of the target object. The preprocessed image can be expressed as:

[0141] (1)

[0142] At the same time, when performing obstacle detection, a deep learning-based target detection network such as YOLO, FasterR-CNN or SSD is used to perform preliminary recognition of dynamic objects. Different models have their own advantages: YOLO is faster and suitable for scenes with high frame rate requirements, while FasterR-CNN has higher accuracy and is suitable for complex environments. , the bounding box of the dynamic object can be extracted from the image and category information .in, Represents the bounding box of the detected dynamic object, including the position coordinates ,and Indicates the category information of dynamic objects, such as vehicles or pedestrians. That is:

[0143] (2)

[0144] To detect dynamic objects, Mask R-CNN is used to obtain pixel-by-pixel semantic segmentation of the image. This is the state-of-the-art for object instance segmentation. Mask R-CNN can simultaneously obtain pixel-by-pixel semantic segmentation and instance labels. However, instance labels may be used in future work to track different moving objects. Mask R-CNN takes a raw RGB image as input. The idea is to segment classes that are likely to be dynamic or movable (people, bicycles, cars, motorcycles, airplanes, buses, trains, trucks, boats, birds, cats, dogs, horses, sheep, cows, elephants, bears, zebras, and giraffes). It is believed that for most environments, the likely dynamic objects are included in this list. If additional classes are needed, the network trained on MSCOCO can be fine-tuned using new training data.

[0145] To detect dynamic objects, Mask R-CNN is used to obtain pixel-by-pixel semantic segmentation of the image. This is the state-of-the-art for object instance segmentation. Mask R-CNN can simultaneously obtain pixel-by-pixel semantic segmentation and instance labels. However, instance labels may be used in future work to track different moving objects. Mask R-CNN takes a raw RGB image as input. The idea is to segment classes that are likely to be dynamic or movable (people, bicycles, cars, motorcycles, airplanes, buses, trains, trucks, boats, birds, cats, dogs, horses, sheep, cows, elephants, bears, zebras, and giraffes). It is believed that for most environments, the likely dynamic objects are included in this list. If additional classes are needed, the network trained on MSCOCO can be fine-tuned using new training data.

[0146] After obtaining dynamic obstacles through target detection, semantic dynamic segmentation is further used to segment dynamic objects at the pixel level. Current segmentation methods include DeepLab, U-Net or Mask R-CNN network models. Among them, Mask R-CNN can generate accurate object masks after target detection. Generate a mask from the dynamic regions extracted from the image ,Right now:

[0147] (3)

[0148] This mask The dynamic object areas in the image are marked to ensure that these areas are not mistakenly included in the map background. After generating the mask, it is used to separate the dynamic and static areas in the image. The image is passed to the background repair module for subsequent background repair, while the static area is directly used for map construction. At the same time, by accumulating static area information from previous frames (i.e., building a static prior information library), static and dynamic obstacles can be better distinguished during the detection process.

[0149] The generated mask is processed to improve segmentation accuracy. For example, the mask can be refined through morphological operations (such as erosion and dilation) to remove boundary noise and small areas of misjudgment, thereby improving the segmentation of dynamic obstacles. Based on the target's motion and the update of the mask, the Kalman filter or optical flow method can be used to update the position of the dynamic object's bounding box to ensure accurate tracking of the dynamic obstacle's position in consecutive frames. The state update and measurement update formulas of the Kalman filter are:

[0150] (4)

[0151] (5)

[0152] in is the state vector (including position and velocity), A is the state transfer matrix, B is the control input matrix, H is the measurement matrix, and Denote process noise and measurement noise, respectively. By using Mask R-CNN, most dynamic objects can be segmented without tracking and mapping. However, some objects cannot be detected by this method because they are not dynamic a priori but movable. Figure 5 Schematic diagram of the principle of multi-view geometry detection of dynamic points:

[0153] When detecting motion between keyframes (KF), it is necessary to select several KFs with the highest overlap with the current frame (CF) from the KF database. The upper limit of the KF database is generally set to 20. The larger the database, the more difficult it is to initialize the system and the speed of frame search will be affected. The number of overlapping KFs selected will also affect the operating speed of the system and the accuracy of dynamic object detection. Calculate each key point Projection from the previous keyframe to the current frame to obtain the keyframe and their projected depth For each key point, its corresponding 3D point is ,calculate and The angle between the back projections of the point is the disparity angle a. If this angle is larger than 30°, the point may be occluded and will be ignored from then on. In the TUM dataset, static objects with a disparity larger than 30° are considered dynamic due to their viewpoint differences. Taking into account the reprojection error, the current frame is obtained The depths of the remaining keypoints in (directly from the depth measurement) and compare them to Compare. If the difference Exceeding the threshold , then the key point A dynamic object.

[0154] Step 2: Introduction to Mask R-CNN segmentation algorithm:

[0155] Mask R-CNN is a deep learning model that integrates object detection, instance segmentation, and bounding box localization, making it suitable for object recognition and segmentation tasks in complex and dynamic environments. It extends Faster R-CNN by adding a branch for each region of interest (RoI) to predict a segmentation mask, building on the existing classification and bounding box regression branches. The mask branch is a small FCN applied to each RoI, predicting the segmentation mask in a pixel-topixel manner. Given the Faster R-CNN framework, Mask R-CNN is easy to implement and train, facilitating a wide range of flexible architecture designs. Furthermore, the mask branch imposes minimal computational overhead, enabling fast systems and rapid experimentation. Its architecture is based on a two-stage design, first localizing objects and then performing pixel-level segmentation. In the first stage, Mask R-CNN generates candidate regions using a region proposal network (RPN), marking potential object locations. In the second stage, the model further extracts features from these regions and, through a branched structure, performs bounding box regression, category classification, and pixel-level segmentation, respectively. Compared with other detection models, Mask R-CNN has more refined pixel-level segmentation capabilities and is suitable for accurately distinguishing between background and dynamic objects. Figure 6 shown.

[0156] It plays an important role in the process of dynamic object removal and background restoration. First, the segmentation branch of Mask R-CNN can accurately identify the contours of dynamic objects, thereby generating pixel-level masks to separate these dynamic objects from the background. Secondly, this fine segmentation mask provides a basis for subsequent background restoration, so that after removing dynamic objects, the system can accurately restore the obscured background information. Specifically, the dynamic background reconstruction in the study uses the segmentation results of Mask R-CNN for feature extraction and screening, and combines semantic segmentation with multi-view geometric information to ensure the integrity and authenticity of the background after dynamic objects are removed. The dynamic object segmentation effect diagram is as follows Figure 7 As shown, the red area is the segmentation effect using the Mask R-CNN network.

[0157] Step 3: Dynamic obstacle removal based on semantic segmentation method

[0158] The above method can better segment moving objects. This embodiment mainly uses the semantic segmentation network to detect and segment dynamic obstacles. The semantic segmentation network can better eliminate prior dynamic objects, but the effect of distinguishing non-prior dynamic objects in the scene is limited. A multi-view geometry algorithm is used for further processing. The proposed dynamic object elimination algorithm process is as follows: Figure 8As shown in the figure, dynamic points are detected based on the motion relationship between keyframes and keyframes. After being detected, the points are divided into those with semantic information and those without by determining whether they have semantic labels. Semantic contour search is performed on the semantic map for feature points with semantic information, and region growing is performed on the depth map for dynamic points without semantic information. This fully utilizes semantic information to reduce the number of region growing seed points and improve the system's operating efficiency. The dynamic object mask without semantic information is fused with the dynamic object mask with semantic information to obtain a complete dynamic object mask.

[0159] The RGB-D camera is used to obtain RGB images and depth images. The RGB images are processed by the semantic segmentation network to obtain pixel-level semantic information. The semantic information is used to remove the feature points of the prior dynamic objects in the image. The multi-view geometry algorithm is used to further detect the feature points corresponding to non-prior dynamic objects. The detection results of the multi-view geometry and the semantic segmentation network are cross-validated to obtain the complete dynamic area. After filtering out the feature points in the dynamic area, the tracking thread is entered to obtain a more accurate pose. The dynamic feature removal effect is as follows: Figure 9 As shown, the red box marks the detected dynamic features.

[0160] Step 4: Build a 3D map based on the static background repair of the dynamic environment of the octree map

[0161] Octree is a data structure with eight child nodes, and its name comes from its unique way of dividing space. The reason for dividing space into eight sub-areas can be figuratively understood as cutting a cube once on three orthogonal planes, thereby dividing the entire cube into eight small cubes of equal volume. It is widely used in fields such as 3D mapping, environmental modeling, and spatial retrieval. Octree-based 3D map construction and static background restoration is an innovative method that combines space segmentation and dynamic scene processing, which can achieve efficient map construction in complex dynamic environments. Figure 10 The main principle diagram for building an octree graph.

[0162] The octree recursively divides the three-dimensional space into small cubic nodes to form a hierarchical structure. This method can not only dynamically adjust the resolution of the map to adapt to the complexity of the scene, but also reduce storage requirements and support efficient processing of sparse scenes. At the same time, it facilitates rapid retrieval and positioning of specific spatial areas to meet the needs of real-time map construction. In dynamic scenes, dynamic objects (such as pedestrians, vehicles, etc.) interfere with the accuracy and integrity of map construction. Through target detection algorithms (such as Mask R-CNN or YOLO), dynamic objects can be identified and semantic segmentation masks can be generated to exclude dynamic areas from map construction. Combined with multi-view geometric verification technology, feature points can be further screened to ensure that only static parts are retained, laying the foundation for static background repair. The specific mapping effect of the octree is as follows: Figure 11 shown.

[0163] Static background inpainting effectively reconstructs areas occluded by dynamic objects by fusing historical keyframes with multi-view information. Leveraging multi-frame data stored in an octree, occluded areas can be restored to a static background consistent with the scene logic. Missing depth data is filled in using interpolation or depth compensation methods. Combined with the classification results of the semantic segmentation network, realistic inpainted images are generated based on semantic features. This inpainting method enhances map accuracy and camera tracking stability, ensuring map applicability in dynamic environments.

[0164] During specific implementation, the RGB and depth channels of the previous key frame are projected onto the dynamic area of ​​the current frame based on the known position information of the previous and next frames. In this way, the static background information can be mapped to the current frame, thereby filling the blank areas left after the dynamic objects are removed. In this process, some areas remain blank due to lack of correspondence, while other areas may not be drawn because the required part of the scene has not yet appeared in the key frame, or has appeared but lacks effective depth information. During the background repair process, the system will use the depth camera to capture dynamic frame data from a global perspective for each dynamic object that has been removed. In order to further optimize the coherence of the repair, multi-perspective cross-frame information transmission is used to obtain the motion trend of the scene. By combining Mask R-CNN, multi-perspective information fusion and historical information in the prior frame, accurate restoration of the dynamically occluded area is achieved, and a high-precision three-dimensional map without dynamic objects is generated, ensuring stability and accuracy in dynamic environments. This process removes moving content from the generated map, presenting a natural, realistic image effect without dynamic interference. The static background repair process is as follows Figure 12 shown.

[0165] This embodiment not only enhances the visual quality of the synthesized image, but also has important significance for application scenarios such as virtual reality and augmented reality, as well as tasks such as relocalization and camera tracking after map creation. In specific implementation, the RGB and depth channels of the previous key frame are projected to the dynamic area of ​​the current frame based on the known position information of the previous and next frames. In this way, the static background information can be mapped to the current frame, thereby filling the blank area left after the dynamic objects are removed. However, in this process, some areas remain blank due to lack of correspondence, while other areas may not be drawn because the required part of the scene has not yet appeared in the key frame, or has appeared but lacks valid depth information.

[0166] In order to deal with these problems, a method combining deep learning and motion consistency is used to better restore the content of blank areas. This method not only improves the integrity and realism of the synthesized image, but also provides more stable static background information for the subsequent dynamic SLAM system. It can clearly show the successful segmentation and deletion of dynamic content, and how most of the segmented areas are correctly integrated into the static background information. Through such a background repair process, high-quality images without dynamic interference can be generated, providing more reliable visual data support for various applications. The mask of the dynamic object is replaced by the It is used to remove dynamic interference and only retain data in static areas to build a map. In the process of map construction, the selection of key frames is crucial. By selecting appropriate key frames, the scene representation and camera posture can be optimized, and the accuracy and consistency of the map can be improved. At the same time, combined with depth information and image data, point cloud fusion technology is used to merge static information in different frames into a complete three-dimensional map. The mapping effect of this embodiment is shown as follows Figure 13 shown.

[0167] The solution of this embodiment has practical feasibility, not only promoting the development of intelligent robotics technology but also demonstrating its value and potential in practical applications. By optimizing the dynamic target detection algorithm, the plan combines the powerful recognition capabilities of deep learning with the stability of traditional image processing technology to achieve accurate recognition and tracking of dynamic targets. The innovation of this algorithm lies in its ability to effectively separate dynamic elements from images through multiple steps, such as motion detection, depth analysis, classification and labeling, and tracking verification, thereby reducing errors in the three-dimensional map construction process.

[0168] See also Figure 14 , is a schematic diagram of the structure of a 3D map reconstruction device without dynamic object interference provided by one embodiment of the present invention. The 3D map reconstruction device without dynamic object interference includes:

[0169] The image acquisition module 10 is used to acquire environmental data through a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information;

[0170] An image separation module 11 is configured to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network to generate a dynamic region mask to separate the dynamic region from the static background; the dynamic region mask is used to mark the dynamic object region in the target image;

[0171] A processing module 12 is configured to perform segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy;

[0172] The recognition module 13 is used to combine the processed dynamic area mask and use the target detection algorithm to re-identify the dynamic objects in the target image and generate a semantic segmentation mask;

[0173] A screening module 14 is configured to remove dynamic areas based on the semantic segmentation mask and filter feature points of the target image that do not conform to the dynamic background by combining a multi-view geometric verification technique, and retain static parts;

[0174] A restoration module 15 is configured to perform background restoration on the target image's occluded area by the dynamic object by utilizing the multi-view views and the target image's historical frames through multi-view information fusion and depth data interpolation, thereby restoring the static background information.

[0175] The reconstruction module 16 is used to select the corresponding key frames of the restored target image to optimize the scene representation and camera pose, and use the static information of the key frames to build an octree map, integrate the optimized scene representation and camera pose into the octree map, and complete the construction of a three-dimensional map without interference from dynamic objects.

[0176] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0177] First, a depth camera captures environmental data to obtain a target image, including continuous image frames or RGB-D images with depth information. Based on this data, the Mask R-CNN network performs preliminary detection and pixel-by-pixel semantic segmentation of dynamic objects, generating a dynamic region mask. This allows for a preliminary separation of dynamic regions from the static background. This is due to the powerful feature extraction and classification capabilities of the Mask R-CNN network, which effectively distinguishes different objects and provides important dynamic region information for subsequent processing. The dynamic region mask is then processed for segmentation accuracy, employing techniques such as morphological operations to refine the boundaries of dynamic regions, improve segmentation accuracy, and ensure the accuracy of subsequent processing. Combined with the processed dynamic region mask, an object detection algorithm is then used to re-identify dynamic objects and generate a semantic segmentation mask, further refining the labeling of dynamic objects. This combined information from the processed mask allows for focusing on possible dynamic regions, improving recognition accuracy. Dynamic regions are then eliminated based on the semantic segmentation mask, and feature points that do not correspond to the dynamic background are screened using multi-view geometric verification techniques. Multi-view geometric verification leverages the geometric relationships between multiple viewpoints to ensure accurate screening, thereby retaining the static regions. Then, using multi-view views and historical frames, multi-view information fusion and depth data interpolation are used to inpaint areas occluded by dynamic objects. This leverages the complementarity of multi-view information and the advantages of depth data interpolation to restore more complete and accurate static background information. Finally, keyframes are selected from the inpainted target image, the scene representation and camera pose are optimized, and an octree map is constructed using the static information from the keyframes. This optimized information is then integrated into the octree. The octree structure facilitates hierarchical storage and representation of three-dimensional space, completing the construction of a 3D map free of dynamic object interference. The optimization of keyframes and the use of the octree during the construction process effectively integrate static information and improve the quality of map construction. As can be seen from the above analysis, the embodiments of the present invention, through technical means such as the Mask R-CNN network, segmentation accuracy processing, multi-view geometric verification technology, multi-view information fusion, depth data interpolation, and octree map construction, effectively address the problems of mismatching, pose drift, and background information loss caused by dynamic object interference in dynamic scenes, significantly improving the accuracy of 3D map reconstruction.

[0178] It is understandable that the embodiment of the apparatus for reconstructing a three-dimensional map without interference from dynamic objects may refer to the relevant contents of the embodiment of the method for reconstructing a three-dimensional map without interference from dynamic objects, and will not be described in detail here.

[0179] See also Figure 15, is a schematic diagram of a device for reconstructing a three-dimensional map without dynamic object interference, provided in one embodiment of the present invention. The device for reconstructing a three-dimensional map without dynamic object interference in this embodiment includes: a processor 100, a memory 101, and a computer program stored in the memory 101 and executable on the processor 100, such as a three-dimensional map reconstruction program without dynamic object interference. When the processor 100 executes the computer program, it implements the steps of each of the aforementioned embodiments of the method for reconstructing a three-dimensional map without dynamic object interference. Alternatively, when the processor 100 executes the computer program, it implements the functions of each module / unit in each of the aforementioned device embodiments.

[0180] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the dynamic object interference-free 3D map reconstruction device.

[0181] The 3D map reconstruction device without dynamic object interference can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The 3D map reconstruction device without dynamic object interference can include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a 3D map reconstruction device without dynamic object interference and does not constitute a limitation on the 3D map reconstruction device without dynamic object interference. The device can include more or fewer components than shown, or a combination of certain components, or different components. For example, the 3D map reconstruction device without dynamic object interference can also include input and output devices, network access devices, buses, and the like.

[0182] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor serves as the control center of the dynamic object interference-free 3D map reconstruction device, connecting various components of the dynamic object interference-free 3D map reconstruction device using various interfaces and circuits.

[0183] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the dynamic object interference-free 3D map reconstruction device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0184] If the integrated modules / units of the dynamic object interference-free 3D map reconstruction device are implemented as software functional units and sold or used as standalone products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard drive, a magnetic disk, an optical disk, computer memory, read-only memory (ROM), random access memory (RAM), an electrical carrier signal, a telecommunications signal, and software distribution media. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0185] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0186] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A three-dimensional map reconstruction method without dynamic object interference, characterized in that: include: The environment data is collected by a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information; Perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using the Mask R-CNN network to generate a dynamic region mask to separate the dynamic region from the static background; The dynamic area mask is used to mark the dynamic object area in the target image; performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy; Combining the processed dynamic region mask, using a target detection algorithm to identify dynamic objects in the target image again and generate a semantic segmentation mask; Eliminate dynamic areas based on the semantic segmentation mask, and combine multi-view geometric verification technology to filter feature points of the target image that do not conform to the dynamic background, and retain the static parts; Using multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation, background repair is performed on the target image's occluded area by the dynamic object to restore static background information; Selecting the corresponding keyframes of the restored target image to optimize the scene representation and camera pose, and using the static information of the keyframes to construct an octree map, integrating the optimized scene representation and camera pose into the octree map to complete the construction of a three-dimensional map without interference from dynamic objects; The method of using the multi-view views and the historical frames of the target image to perform background repair on the area of ​​the target image blocked by the dynamic object through multi-view information fusion and depth data interpolation to restore the static background information includes: Acquire a historical frame of the target image; the historical frame is an image frame captured before the target image, and contains static background information of the area blocked by the dynamic object; Integrate the information of different perspectives in the multi-perspective view, perform fusion operation on the static part information of the target image under different perspectives, and obtain the static part information after multi-perspective fusion; For an area in the target image that is blocked by a dynamic object, using a depth data interpolation method, using known depth information of an unblocked area as a reference, interpolating the depth data of the blocked area to obtain depth information of the area; Based on the static part information after multi-view fusion and the depth information obtained by depth data interpolation, the background repair operation is performed on the area blocked by the dynamic object, and the repaired information is used as the final static background information; The method comprises: selecting the corresponding key frames of the restored target image to optimize the scene representation and camera pose, constructing an octree map using the static information of the key frames, integrating the optimized scene representation and camera pose into the octree map, and completing the construction of a three-dimensional map without interference from dynamic objects, including: Obtaining a restored target image, the restored target image including static background information of a region occluded by the dynamic object, the restored target image being obtained by performing background restoration on the region occluded by the dynamic object in the target image by utilizing multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation; Selecting corresponding key frames of the restored target image, wherein the key frames are representative image frames in the restored target image and are screened from the restored target image by calculating inter-frame overlap and scene complexity; Based on the static feature points and depth information in the key frames, the geometric structure of the scene is optimized using multi-view geometric constraints to obtain the optimized scene representation; Through multi-view geometric constraints and the projection relationship of feature points in different key frames, the camera's rotation and translation parameters are optimized to obtain the optimized camera pose; Convert the static information in different keyframes into nodes or units of the octree, present the three-dimensional space in a hierarchical manner, and construct an octree map; The optimized scene representation and the information corresponding to the camera posture are integrated with the corresponding nodes or units in the octree map according to the structural rules of the octree to complete the construction of a three-dimensional map without interference from dynamic objects.

2. The method for reconstructing a three-dimensional map without dynamic object interference according to claim 1, wherein: After collecting environmental data through the depth camera to obtain the target image, the method further includes: The target image is preprocessed to obtain a preprocessed target image; the preprocessing includes image grayscale, scale adjustment, denoising and smoothing operations.

3. The method for reconstructing a three-dimensional map without dynamic object interference according to claim 1, wherein: The Mask R-CNN network is used to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image to generate a dynamic region mask to separate the dynamic region from the static background, including: Input the target image into the Mask R-CNN network and generate candidate regions through the region proposal network; the candidate regions are regions in the image that may contain dynamic objects; For each candidate region, the Mask R-CNN network performs bounding box regression, category classification, and pixel-level segmentation, generating a bounding box, category information, and a dynamic region mask for the dynamic object. The bounding box includes the location coordinates of the dynamic object, the category information includes the category label of the dynamic object, and the dynamic region mask is used to mark the dynamic object region in the image. The dynamic area is separated from the static background according to the generated dynamic area mask; the dynamic object area marked by the dynamic area mask will be excluded from the static background.

4. The method for reconstructing a three-dimensional map without dynamic object interference according to claim 1, wherein: The performing segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy includes: Obtaining a dynamic region mask generated by performing preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network; Performing morphological operations on the generated dynamic region mask, including erosion and dilation, to remove noise at the mask boundary and small misjudgment areas; the morphological operations are used to improve the segmentation accuracy of the dynamic region mask; The bounding box of the dynamic object is updated using Kalman filtering or optical flow method to ensure accurate tracking of the position of the dynamic obstacle in consecutive frames. The position update is based on the bounding box information of the dynamic object to ensure the position consistency of the dynamic object in consecutive frames. The processed dynamic area mask is used for subsequent dynamic object removal and background restoration; the dynamic object area marked by the processed dynamic area mask will be excluded from the 3D map construction process and used to guide the background restoration of the area blocked by the dynamic object.

5. The method for reconstructing a three-dimensional map without dynamic object interference according to claim 1, wherein: The combined processed dynamic area mask uses a target detection algorithm to identify dynamic objects in the target image again and generate a semantic segmentation mask, including: Using the processed dynamic region mask as input, the target image is processed using an object detection algorithm to generate a bounding box and category information of the dynamic object; the bounding box includes the position coordinates of the dynamic object in the target image, and the category information includes the category label of the dynamic object; The processed dynamic region mask and the generated bounding box and category information of the dynamic object are used as input. The Mask R-CNN network performs pixel-by-pixel semantic segmentation on the target image to generate a semantic segmentation mask of the dynamic object. The semantic segmentation mask is used to mark the dynamic object area in the image.

6. The method for reconstructing a three-dimensional map without dynamic object interference according to claim 1, wherein: The process of eliminating the dynamic area based on the semantic segmentation mask and screening the feature points of the target image that do not conform to the dynamic background by combining the multi-view geometric verification technology to retain the static part includes: Obtaining a generated semantic segmentation mask; the semantic segmentation mask is used to mark the dynamic object area in the target image; Eliminate dynamic regions in the target image according to the semantic segmentation mask; the dynamic regions are dynamic object regions marked by the semantic segmentation mask; Combined with multi-view geometric verification technology, feature points in the target image that do not conform to the dynamic background are screened out; the multi-view geometric verification technology calculates the projection error between the key frame and the current frame to screen out points that do not conform to the dynamic background characteristics; The filtered static feature points are retained; the static feature points are used for subsequent three-dimensional map construction.

7. A three-dimensional map reconstruction device without dynamic object interference, characterized in that: include: An image acquisition module is used to acquire environmental data through a depth camera to obtain a target image; the target image includes continuous image frames or an RGB-D image with depth information; An image separation module is used to perform preliminary dynamic object detection and pixel-by-pixel semantic segmentation on the target image using a Mask R-CNN network, generate a dynamic region mask, and separate the dynamic region from the static background; The dynamic area mask is used to mark the dynamic object area in the target image; a processing module, configured to perform segmentation accuracy processing on the generated dynamic area mask to improve segmentation accuracy; A recognition module, configured to combine the processed dynamic region mask and use a target detection algorithm to re-identify dynamic objects in the target image and generate a semantic segmentation mask; A screening module is used to eliminate dynamic areas based on the semantic segmentation mask, and to screen feature points of the target image that do not conform to the dynamic background by combining multi-view geometric verification technology, and retain static parts; A restoration module is used to perform background restoration on the target image's occluded area by the dynamic object by using multi-view views and historical frames of the target image through multi-view information fusion and depth data interpolation to restore static background information; A reconstruction module is used to select the corresponding key frames of the restored target image to optimize the scene representation and camera pose, and use the static information of the key frames to build an octree map, integrate the optimized scene representation and camera pose into the octree map, and complete the construction of a 3D map without interference from dynamic objects; The repair module is specifically used for: Acquire a historical frame of the target image; the historical frame is an image frame captured before the target image, and contains static background information of the area blocked by the dynamic object; Integrate the information of different perspectives in the multi-perspective view, perform fusion operation on the static part information of the target image under different perspectives, and obtain the static part information after multi-perspective fusion; For an area in the target image that is blocked by a dynamic object, using a depth data interpolation method, using known depth information of an unblocked area as a reference, interpolating the depth data of the blocked area to obtain depth information of the area; Based on the static part information after multi-view fusion and the depth information obtained by depth data interpolation, the background repair operation is performed on the area blocked by the dynamic object, and the repaired information is used as the final static background information; The reconstruction module is specifically used for: Obtaining a restored target image, the restored target image including static background information of a region occluded by the dynamic object, the restored target image being obtained by performing background restoration on the region occluded by the dynamic object in the target image by utilizing multi-view views and historical frames of the target image, through multi-view information fusion and depth data interpolation; Selecting corresponding key frames of the restored target image, wherein the key frames are representative image frames in the restored target image and are screened from the restored target image by calculating inter-frame overlap and scene complexity; Based on the static feature points and depth information in the key frames, the geometric structure of the scene is optimized using multi-view geometric constraints to obtain the optimized scene representation; Through multi-view geometric constraints and the projection relationship of feature points in different key frames, the camera's rotation and translation parameters are optimized to obtain the optimized camera pose; Convert the static information in different keyframes into nodes or units of the octree, present the three-dimensional space in a hierarchical manner, and construct an octree map; The optimized scene representation and the information corresponding to the camera posture are integrated with the corresponding nodes or units in the octree map according to the structural rules of the octree to complete the construction of a three-dimensional map without interference from dynamic objects.

8. A three-dimensional map reconstruction device without dynamic object interference, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for reconstructing a three-dimensional map without interference from dynamic objects as described in any one of claims 1 to 6 is implemented.

Images