Indoor dynamic environment three-dimensional reconstruction method and system based on deep learning

Abstract

The invention discloses an indoor dynamic environment three-dimensional reconstruction method and system based on deep learning, and the method comprises the steps: employing SuperPoint and LightGlue to carry out the extraction and matching of feature points, carrying out the precise recognition of a dynamic target in combination with YOLOv11 and SAM technologies, and carrying out the implementation of an image pyramid feature selection strategy, and can uniformly reserve key feature points layer by layer, thereby achieving the precise recognition of a dynamic target. The feature points of the dynamic object are effectively eliminated, so that the efficiency is further improved, and finally, a high-precision, high-robustness and high-real-time visual synchronous localization and mapping (VSLAM) system for high-precision localization and mapping is realized in an indoor dynamic environment. The method is suitable for the fields of robot navigation, unmanned vehicles, indoor positioning and navigation and the like in indoor environments with weak GPS signals and more dynamic objects, such as large underground parking lots, shopping malls, office buildings and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of high-precision positioning and map construction of indoor dynamic environments, and specifically to a method and system for three-dimensional reconstruction of indoor dynamic environments based on deep learning. Background Art

[0002] With the increasing popularity of smart buildings, underground spaces, and large indoor venues (such as underground parking lots, large shopping malls, and office buildings), the demand for high-precision indoor positioning and three-dimensional mapping is growing. Because Global Positioning System (GPS) signals cannot penetrate buildings, traditional satellite-based navigation technologies are completely ineffective in indoor environments. While wireless positioning methods such as Wi-Fi, geomagnetic interference, and Bluetooth have partially supplemented positioning needs, they generally suffer from the following issues: high hardware deployment costs, susceptibility to signal obstruction or interference, complex maintenance, and limited positioning accuracy (typically in the meter range), making it difficult to achieve centimeter-level, real-time, and robust positioning.

[0003] Visual Simultaneous Localization and Mapping (VSLAM) technology uses a single or multiple cameras to perceive the environment, simultaneously localizing itself and mapping the environment while in motion. With its low deployment cost, high information density, and strong environmental adaptability, it has become a mainstream solution for indoor autonomous vehicles, robots, and smart devices. However, indoor environments often feature numerous dynamic objects, significant lighting variations, and repetitive textures, placing higher demands and challenges on the robustness and accuracy of VSLAM systems.

[0004] ORB-SLAM3 is one of the most representative open source VSLAM systems. It is based on ORB feature points (OrientedFAST and Rotated BRIEF, oriented fast feature points and rotation brief descriptors), can support multiple data sources such as monocular, binocular, RGB-D cameras and inertial measurement units (IMU), and has strong real-time performance and good positioning accuracy. This method uses modules such as feature extraction, feature matching, three-dimensional reconstruction of map points, and loop detection, and is suitable for static background environments. Its core advantage lies in the high system efficiency achieved through a multi-threaded architecture (tracking thread, mapping thread, loop detection thread). However, its robustness is poor in weak texture, low light or dynamic environments, which often leads to tracking loss or increased positioning error.

[0005] There are two main categories of SLAM techniques for dynamic environments. The first category consists of geometric methods, including frame differencing, background subtraction, and optical flow, which detect dynamic targets based on pixel-level changes between consecutive frames. These methods have low computational complexity, but are less robust to lighting variations and weak textures, resulting in high false detection and missed detection rates, making them particularly effective in complex indoor scenes. The second category consists of deep learning-based methods, such as DS-SLAM (which incorporates the SegNet semantic segmentation network), DynaSLAM (which combines Mask R-CNN for pixel-level segmentation), and YOLOv4-SLAM (which employs a single-stage object detection network). These methods can more accurately identify and remove feature points in dynamic target areas, improving localization accuracy. However, these methods generally suffer from high computational complexity and slow inference speed, making them difficult to meet the real-time requirements of embedded devices or mobile platforms (typically requiring a frame rate of ≥30 FPS). Furthermore, despite their high detection speed, the YOLO family of methods can only output a minimum bounding rectangle. Compared to semantic segmentation, their detection boxes often contain a large amount of redundant regions, making them more likely to cover irrelevant background in scenes with dense dynamic targets.

[0006] In summary, the existing technology mainly has the following defects:

[0007] 1. Insufficient robustness of feature point extraction: Traditional ORB feature points are insensitive to weak textures, low light, dynamic occlusion, lens blur, etc. The number of feature points extracted is small or the matching failure rate is high, resulting in positioning interruption or reduced accuracy.

[0008] 2. Incomplete elimination of dynamic targets: Existing methods mostly eliminate dynamic targets based on rectangular detection boxes (Bounding Box), which cannot accurately segment the boundaries of dynamic areas. This leads to the incorrect elimination of static background feature points or incomplete elimination of dynamic area feature points, which in turn pollutes map construction.

[0009] 3. Heavy computational burden and poor real-time performance: Semantic segmentation networks based on deep learning (such as SegNet and Mask R-CNN) have high computational complexity and long inference time, making it difficult to achieve real-time operation on resource-constrained embedded platforms.

[0010] 4. Limited performance of the target detection model: Although the existing YOLOv4 is fast, it is inferior to the latest YOLOv11 in detection accuracy and model lightweightness. Summary of the Invention

[0011] This application provides a deep learning-based three-dimensional reconstruction method and system for indoor dynamic environments to solve the above problems existing in the prior art.

[0012] According to the first aspect, an embodiment provides a method for three-dimensional reconstruction of an indoor dynamic environment based on deep learning, the method comprising:

[0013] Collect indoor environment video stream data and perform frame sampling;

[0014] The sampled frame image is downsampled multiple times and an image pyramid is constructed;

[0015] SuperPoint visual feature point extraction and feature point screening are performed on each layer of the image in the image pyramid to obtain the feature point set of each layer;

[0016] YOLOv dynamic target detection is performed on each layer of the image pyramid, and the obtained dynamic target detection box is segmented and masked to obtain a global dynamic mask;

[0017] The obtained global dynamic mask is used to discriminate the feature point sets of each layer of the image pyramid to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer;

[0018] Perform LightGlue feature matching and camera tracking on the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix;

[0019] The optimal camera pose transformation matrix of all key frames is obtained, and the local 3D point cloud generated by combining the depth image information corresponding to each key frame is used to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene.

[0020] Furthermore, SuperPoint visual feature point extraction and feature point screening are performed on each layer of the image in the image pyramid to obtain the feature point set of each layer, specifically including:

[0021] Input each layer of the image pyramid into the SuperPoint network for feature point detection and extract the initial feature point set and its descriptor;

[0022] The extracted initial feature points are screened on each layer of the image, including: dividing each layer of the image into a uniform grid, and sorting the feature points output by the SuperPoint network in each grid unit according to the confidence level, retaining only the top several feature points with relatively high confidence levels; at the same time, setting an upper limit on the total number of feature points in each layer of the image, and further screening out a set of feature points with relatively high confidence levels through global sorting.

[0023] Furthermore, YOLOv dynamic target detection is performed on each layer of the image pyramid, specifically including:

[0024] YOLOv11 is used as the dynamic target detection network. Based on a one-stage framework, the CSPDarknet-v7 backbone network is used for feature extraction. Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PAN) are used to enhance multi-scale feature fusion capabilities, significantly improving detection accuracy while maintaining a high frame rate.

[0025] Suppose the input image is I(x,y), where (x,y) is the horizontal and vertical coordinates of each pixel in image I. The YOLOv11 network generates a set of candidate detection boxes B at the output layer:

[0026]

[0027] Among them, b i is the i-th candidate detection frame, N is the total number of candidate detection frames, (x i ,y i ) is the center coordinate of the candidate detection frame, (w i ,h i ) is the width and height of the candidate detection box, p i is the category confidence; after non-maximum suppression, the detection box with confidence higher than the preset threshold is retained as the preliminary positioning result of the dynamic target.

[0028] Furthermore, the obtained dynamic target detection frame is segmented and masked, and a global dynamic mask is obtained, which specifically includes:

[0029] Each detection box b i It is input into the SAM model as a prompt to further accurately segment the dynamic area;

[0030] The SAM model first performs high-resolution encoding on the input image I(x,y) to generate an image feature map F img , and by prompting the encoder to detect the box b i Mapped to vector v i , then F img and v i The fused input mask decoder generates a binary semantic mask corresponding to the dynamic target:

[0031]

[0032] Among them, M i (x, y) represents the segmentation mask of the target corresponding to the i-th detection box. Pixels with a value of 1 belong to the dynamic target area, and pixels with a value of 0 belong to the background or non-target area.

[0033] Take the union of all target masks to get the global dynamic mask:

[0034]

[0035] Among them, M dyn (x,y) represents the global dynamic mask, covering all areas in the input image that are identified as dynamic objects; Represents the logical OR operation, that is, for all single target masks M i The union operation is performed on the values ​​of (x, y) at each pixel position (x, y). If the value of any target mask at the corresponding pixel position is 1, the value of the global mask at the corresponding pixel position is also 1, indicating that the pixel at the corresponding position belongs to the dynamic target area; conversely, if the values ​​of all masks at the corresponding pixel position are 0, the global mask value is 0, indicating that the pixel at the corresponding position belongs to the static background environment area; i takes values ​​from 1 to N, where N is the total number of dynamic targets detected.

[0036] Furthermore, the obtained global dynamic mask is used to discriminate the feature point sets of each layer of the image pyramid to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer, specifically including:

[0037] For each feature point position (x j ,y j ), check whether it falls within the dynamic target mask area:

[0038]

[0039] IsDynamic represents whether it is a dynamic target, (x j ,y j ) is the position of each feature point. If the mask value M of the feature point position dyn (x j ,y j )=1, then the judgment result of IsDynamic is 1, indicating that the corresponding feature point is located in the dynamic target area, and it will be removed; if the mask value M of the feature point position dyn (x j ,y j )=1, the judgment result of IsDynamic is 0, indicating that the corresponding feature point is located in the static background environment area, and it is retained; thereby ensuring that the feature points finally retained are all static background environment feature points, avoiding matching errors caused by dynamic targets.

[0040] Furthermore, the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames are matched with LightGlue features and camera tracking to obtain the optimal camera pose transformation matrix, which specifically includes:

[0041] The static background environment feature points and their descriptors extracted from each layer of the image pyramid are used as input data for the LightGlue model;

[0042] Let the current frame image be I t , the previous frame image is I t-1 , LightGlue interactively models the feature descriptors of the two frames and outputs a set of matching pairs Where (i k ,j k ) represents the index of the kth pair of matching feature points, s k is the matching confidence score, K is the total number of matching pairs;

[0043] Further filter the matching pairs output by the LightGlue model and set the confidence threshold τ m , only keep the k >τ m The matching pair is taken as the final matching result;

[0044] The obtained matching pairs are then used to restore the camera pose in the tracking thread of the SLAM system. Assuming that the feature point positions corresponding to the matching pairs have been corrected to the original image scale through the image pyramid level, the camera motion estimation problem is constructed based on the matching pairs, and the camera pose transformation matrix is ​​optimized by minimizing the reprojection error. Right now:

[0045]

[0046] Among them, t is the current frame, t-1 is the previous frame, k is the index of the matching feature point pair, and i and j represent the index of the matched feature points in the current frame and the previous frame in the matching pair respectively. ′ is the number of matching pairs after screening; Indicates the feature points of the previous frame The corresponding three-dimensional points are obtained by triangulation; is the two-dimensional observation position of the feature point matched in the current frame; π(·) is the camera projection function, which is used to project the three-dimensional point onto the image plane; is the camera pose transformation matrix to be optimized, which represents the camera motion from the previous frame to the current frame, including the rotation matrix R and the translation vector t; is the square of the Euclidean distance, which represents the reprojection error; the error function E is solved by nonlinear optimization method to obtain the optimal camera pose transformation matrix matrix It represents the relative transformation between the camera pose of the current frame and the previous frame, reflecting the position and orientation of the camera in the local space.

[0047] Furthermore, the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames are matched with LightGlue features and camera tracking to obtain the optimal camera pose transformation matrix, which specifically includes:

[0048] In the SLAM tracking thread, the camera pose transformation results are further fused with the historical pose information, and the camera trajectory is smoothed through filtering methods or graph optimization frameworks to reduce the cumulative error; at the same time, if the number of matching pairs obtained is lower than the preset threshold or the reprojection error exceeds the preset threshold, the relocalization module is triggered to restore tracking continuity.

[0049] Furthermore, the optimal camera pose transformation matrix of all key frames is obtained, and the local 3D point cloud generated by combining the depth image information corresponding to each key frame is used to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene. Specifically, the following steps are performed:

[0050] Get the camera poses of all keyframes from the SLAM system in Represents the transformation matrix of the t-th frame keyframe relative to the world coordinate system, including the rotation matrix and translation vector, which reflects the position and orientation of the camera in the global space. Obtained by transforming from the camera coordinate system to the world coordinate system;

[0051] Then extract the depth image D corresponding to each key frame t , using the intrinsic parameter matrix K of the RGB-D camera and the depth information, the two-dimensional image pixels are converted into local three-dimensional point cloud data. For each pixel point (u, v) in the depth image, the corresponding three-dimensional point X t (u,v) is calculated as follows:

[0052]

[0053] Among them D t (u,v) is the depth value of the pixel (u,v), [x,y,z] is the position coordinate of the camera in three-dimensional space, f is the focal length of the camera, (c x ,c y ) is the camera principal point coordinate; after conversion, the depth image of each key frame is converted into a three-dimensional point cloud in the local coordinate system

[0054] Then according to the pose of the key frame Each local point cloud Convert to the world coordinate system and perform preliminary splicing to obtain the global point cloud P global , the specific conversion process is:

[0055]

[0056] in It is the point cloud data after conversion to the world coordinate system;

[0057] By accumulating and stitching the point cloud data of all key frames, a global three-dimensional representation of the indoor scene is preliminarily constructed.

[0058] Furthermore, the optimal camera pose transformation matrix of all key frames is obtained, and the local 3D point cloud generated by combining the depth image information corresponding to each key frame is used to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene. Specifically, the following steps are also included:

[0059] The iterative closest point ICP algorithm is used to finely align the point clouds of adjacent key frames, optimize the relative poses between point clouds, and further reduce the cumulative error;

[0060] Apply voxel grid filtering to downsample the global point cloud. By dividing the point cloud into fixed-size voxel grids and retaining the representative points within each grid, the point cloud density is effectively reduced while preserving the main geometric structure of the scene.

[0061] Using the global dynamic mask M generated based on the SAM model dyn (x, y), combined with depth information and camera projection model, the pixels in the dynamic area are mapped to the 3D point cloud, and the point cloud data belonging to the dynamic target is identified and eliminated. Specifically, for each 3D point Determine its pixel position (u, v) on the original image by back-projection. If M dyn If (u, v) = 1, the corresponding point is determined to belong to the dynamic target area and is removed, ensuring that the final 3D map only contains static background environment information.

[0062] According to a second aspect, an embodiment provides a deep learning-based 3D reconstruction system for an indoor dynamic environment, the system comprising:

[0063] Image acquisition module, used to collect indoor environment video stream data and perform frame sampling;

[0064] An image pyramid construction module is used to downsample the sampled frame image multiple times and construct an image pyramid;

[0065] The feature extraction module is used to extract and filter SuperPoint visual feature points for each layer of the image in the image pyramid to obtain the feature point set of each layer;

[0066] The dynamic target detection module is used to perform YOLOv dynamic target detection on each layer of the image pyramid, perform SAM segmentation and mask generation on the obtained dynamic target detection box, and obtain a global dynamic mask;

[0067] The dynamic target feature point elimination module is used to use the obtained global dynamic mask to identify the feature point sets of each layer of the image pyramid, so as to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer;

[0068] The feature matching and camera tracking module is used to perform LightGlue feature matching and camera tracking on the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix;

[0069] The 3D map construction module is used to obtain the optimal camera pose transformation matrix for all key frames, and combine the local 3D point cloud generated by the depth image information corresponding to each key frame to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene.

[0070] According to a third aspect, an embodiment provides an electronic device, the device comprising: a processor and a memory;

[0071] The memory is used to store one or more program instructions;

[0072] The processor is used to run one or more program instructions to execute the steps of a method for three-dimensional reconstruction of an indoor dynamic environment based on deep learning as described in any of the above items.

[0073] According to the fourth aspect, an embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a deep learning-based three-dimensional reconstruction method for an indoor dynamic environment as described in any one of the above items are implemented.

[0074] This application provides a method and system for 3D reconstruction of indoor dynamic environments based on deep learning. It uses SuperPoint and LightGlue to extract and match feature points, and combines YOLOv11 and SAM technology to accurately identify dynamic targets. In addition, by implementing an image pyramid feature selection strategy, it can evenly retain key feature points layer by layer and effectively remove feature points of dynamic objects, thereby further improving efficiency. Ultimately, a visual simultaneous localization and mapping (VSLAM) system with high precision, high robustness, and high real-time performance is achieved in indoor dynamic environments. It is suitable for robot navigation, unmanned vehicles, indoor positioning and navigation in indoor environments with weak GPS signals and many dynamic objects, such as large underground parking lots, shopping malls, and office buildings. It has the following beneficial effects:

[0075] (1) Improved robustness of feature points: The SuperPoint+LightGlue method enhances the positioning stability of the system in high-dynamic, low-texture environments, avoiding tracking loss in scenes with changing lighting and repeated textures caused by traditional methods such as the ORB method.

[0076] (2) Accurate elimination of dynamic targets: YOLOv11+SAM joint detection and semantic segmentation, the dynamic area recognition accuracy is higher than the traditional detection box method, and the static point contamination rate is significantly reduced.

[0077] (3) The system has strong real-time performance: pyramid sampling + lightweight network, and only one visual sensor is used for image input. Compared with binocular sensors and depth sensors, it has less data volume and low computational complexity. It is suitable for running in resource-constrained environments such as embedded devices and mobile robots, meeting real-time requirements.

[0078] (4) High-precision scene map construction: A three-dimensional point cloud map can be constructed without interference from dynamic objects, with high integrity and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A flowchart of a method for 3D reconstruction of an indoor dynamic environment based on deep learning provided by one embodiment of the present invention;

[0080] Figure 2 A flowchart of a method for 3D reconstruction of an indoor dynamic environment based on deep learning provided by one embodiment of the present invention;

[0081] Figure 3 A schematic diagram of an image pyramid in a method for 3D reconstruction of an indoor dynamic environment based on deep learning provided by one embodiment of the present invention;

[0082] Figure 4 A flowchart of SuperPoint environmental feature point extraction in a deep learning-based 3D reconstruction method for indoor dynamic environments provided by one embodiment of the present invention;

[0083] Figure 5 A flowchart for constructing a 3D point cloud map of an indoor scene in a deep learning-based 3D reconstruction method for an indoor dynamic environment provided by one embodiment of the present invention;

[0084] Figure 6 A comparison diagram of key frame point cloud stitching before and after in a deep learning-based 3D reconstruction method for indoor dynamic environments provided by one embodiment of the present invention;

[0085] Figure 7 A structural diagram of a deep learning-based 3D reconstruction system for indoor dynamic environments provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0086] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0087] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0088] Explanation of relevant terms:

[0089] 1. VSLAM (Visual Simultaneous Localization and Mapping): Visual Simultaneous Localization and Mapping uses cameras to obtain environmental information, locate the vehicle in real time, and construct a map of the environment.

[0090] 2. ORB-SLAM3 (oriented FAST and rotated BRIEF simultaneous localization and mapping 3): A third-generation feature point-based SLAM algorithm that supports monocular, binocular, and RGB-D cameras and can integrate inertial measurement unit (IMU) data for high-precision positioning and mapping.

[0091] 3. YOLOv11 (You Only Look Once version 11): A state-of-the-art deep learning model for real-time object detection, with higher accuracy and lighter weight, suitable for fast recognition of dynamic targets.

[0092] 4. SAM (SegmentAnything Model): A pre-trained general image segmentation model that performs fine pixel-level segmentation of target areas in an image by receiving hint information (such as points, bounding boxes, etc.).

[0093] 5. SuperPoint: An unsupervised method for image keypoint detection and description extraction based on deep neural networks

[0094] 6.LightGlue: A feature matching algorithm based on the Transformer architecture that achieves high robustness and accurate matching through global context modeling.

[0095] 5.IMU (Inertial Measurement Unit): Inertial measurement unit, including accelerometer and gyroscope, used to measure the acceleration and angular velocity information of the device.

[0096] 6. Image pyramid: A multi-scale image representation method that obtains image sequences of different resolutions by downsampling the image multiple times.

[0097] 7. Point cloud map: A spatial geometric representation composed of a large number of three-dimensional points, used to describe the three-dimensional structure of the environment.

[0098] The first embodiment of the present invention provides a method for 3D reconstruction of indoor dynamic environments based on deep learning, which uses SuperPoint and LightGlue to extract and match feature points, and combines YOLOv11 and SAM technology to accurately identify dynamic targets. In addition, by implementing the image pyramid feature selection strategy, it is possible to evenly retain key feature points layer by layer and effectively remove feature points of dynamic objects, thereby further improving efficiency. Ultimately, a high-precision, high-robustness and high-real-time VSLAM system is achieved. Figure 1 and Figure 2 Provide detailed explanation.

[0099] like Figure 1 As shown, in step S100, indoor environment video stream data is collected and frame sampling is performed.

[0100] like Figure 1 As shown, in step S200, the sampled frame image is downsampled multiple times and an image pyramid is constructed.

[0101] Specifically, an image pyramid is a method of obtaining a multi-scale representation by continuously downsampling the original image, which can extract more robust features at different scales. In this embodiment, an 8-layer image pyramid is constructed for the input color image. Figure 3The process of constructing an image from downsampling the original image to the eighth layer is demonstrated. The resolution of each layer decreases, with a scaling factor of 1.2. To balance detection accuracy and computational efficiency, a uniform sampling strategy is used to ensure a uniform distribution of feature points across the image, using a fixed number of samples per layer (e.g., 1000). SuperPoint visual feature point extraction is then performed on each layer, improving the robustness of feature detection to scale variations.

[0102] Assuming that the original image size is W0×H0 and the scaling factor is α, the image size of the i-th layer can be expressed as follows:

[0103]

[0104] like Figure 1 As shown, in step S300, SuperPoint visual feature point extraction and feature point screening are performed on each layer of the image in the image pyramid to obtain a feature point set of each layer.

[0105] In this embodiment, feature point extraction and screening utilizes the deep learning-based SuperPoint algorithm, combined with image pyramid structures and dynamic mask information, to achieve robust extraction and optimized distribution of feature points in static environments. SuperPoint is an efficient feature point detection and description method that simultaneously outputs feature point locations and corresponding descriptors through a pre-trained neural network. It exhibits strong scale invariance and adaptability to low-light and weak-texture environments, demonstrating greater robustness in complex scenes compared to the traditional ORB algorithm.

[0106] The above steps specifically include:

[0107] S310: Input each layer of the image pyramid into the SuperPoint network for feature point detection, and extract the initial feature point set and its descriptor.

[0108] Specifically, an eight-layer image pyramid is constructed for the input image, with a scaling factor of 1.2. This generates a sequence of images from high to low resolution, capturing multi-scale feature information. The specific construction method is as described above. For each layer of the pyramid, the SuperPoint algorithm is applied to detect feature points, extracting an initial set of feature points and their descriptors. These feature points cover regions of varying scales in the image, providing a rich set of candidate points for subsequent matching and localization.

[0109] S320, screening the extracted initial feature points on each layer image, including: dividing each layer image into uniform grids, and sorting the feature points output by the SuperPoint network in each grid unit according to the confidence, and only retaining the top several feature points with relatively high confidence; at the same time, setting an upper limit of the total number of feature points of each layer image, and further screening out a set of feature points with relatively high confidence through global sorting.

[0110] Specifically, to ensure the uniform distribution of feature points and control the amount of calculation, the feature points are screened on each layer image. The image is divided into uniform grids (for example, the image is divided into 30x30 grid units), and the feature points output by SuperPoint are sorted according to the confidence in each grid unit, and only the top several feature points with the highest confidence are retained, thereby ensuring the uniformity of the feature points in space. At the same time, an upper limit of the total number of feature points of each layer image is set (for example, 1000 or adjusted according to experiments), and a set of feature points with the highest confidence is further screened out through global sorting, taking into account accuracy and efficiency.

[0111] As shown in Figure 1 , in step S400, YOLOv dynamic target detection is performed on each layer image in the image pyramid, SAM segmentation and mask generation are performed on the obtained dynamic target detection frame, and a global dynamic mask is obtained.

[0112] In this embodiment, the dynamic target detection and semantic mask generation combine the technical advantages of YOLOv11 (You Only Look Once version 11) and SAM (Segment Anything Model), and realize a complete process from efficient target detection to accurate pixel-level segmentation. The specific process is shown in Figure 4 .

[0113] The above steps specifically include:

[0114] S410, YOLOv dynamic target detection is performed on each layer image in the image pyramid, specifically including:

[0115] S411, YOLOv11 is used as the dynamic target detection network, which is based on a one-stage (one-stage) framework, uses a CSPDarknet-v7 backbone network for feature extraction, and enhances the multi-scale feature fusion capability through spatial pyramid pooling SPP and path aggregation network PAN, while maintaining high frame rate and significantly improving detection accuracy.

[0116] S412, assuming that the input image is I(x,y), the YOLOv11 network generates a set of candidate detection frame sets B at the output layer:

[0117]

[0118] Among them, b i is the i-th candidate detection frame, N is the total number of candidate detection frames, (x i ,y i ) is the center coordinate of the candidate detection frame, (w i ,h i ) is the width and height of the candidate detection box, p i is the category confidence; after non-maximum suppression (NMS), the confidence level higher than the threshold τ is retained d The detection box is taken as the preliminary positioning result of the dynamic target. d It is a preset confidence lower limit (for example, set to 0.5 or tuned according to scene experiments) used to filter out detection results with low confidence, ensuring that only high-confidence dynamic target detection boxes are retained, reducing the impact of false detections on subsequent segmentation.

[0119] In addition, the present invention can also adopt other lightweight target detection models: such as MobileNet-SSD, NanoDet, etc., and select the most suitable model according to the needs of the specific application scenario and the hardware performance. It can also combine depth information for dynamic object detection: using the depth information of the RGB-D camera, dynamic objects are identified by detecting depth changes. This method can avoid relying on deep learning models and reduce the computational burden, but it may be affected when the performance of the depth sensor is limited. Optical flow method can also be used to assist in dynamic object detection: combining optical flow method and target detection to detect and eliminate moving objects. The optical flow method has relatively low requirements for computing resources, but is sensitive to changes in illumination and needs to be weighed according to the application scenario.

[0120] S420, performing SAM segmentation and mask generation on the obtained dynamic target detection frame to obtain a global dynamic mask, specifically including:

[0121] S421, each detection box b i It is input into the SAM model as a prompt for further accurate segmentation of dynamic regions.

[0122] SAM is a prompt-based image segmentation model that has excellent generalization capabilities thanks to a large amount of pre-training data. It can accept various forms of prompts (such as points, rectangles, and text) to guide segmentation and generate accurate pixel-level segmentation masks, thus supporting fine-grained object segmentation in complex scenes.

[0123] S422, the SAM model first performs high-resolution encoding on the input image I(x,y) to generate an image feature map F img , and by prompting the encoder to detect the box b iMapped to vector v i , then F img and v i The fused input mask decoder generates a binary semantic mask corresponding to the dynamic target:

[0124]

[0125] Among them, M i (x,y) represents the segmentation mask of the target corresponding to the i-th detection box. Pixels with a value of 1 belong to the dynamic target area, and pixels with a value of 0 belong to the background or non-target area.

[0126] S423, take the union of all target masks to obtain the global dynamic mask:

[0127]

[0128] Among them, M dyn (x,y) represents the global dynamic mask, covering all areas in the input image that are identified as dynamic objects; Represents the logical OR operation, that is, for all single target masks M i The union operation is performed on the values ​​of (x, y) at each pixel position (x, y). If the value of any target mask at the corresponding pixel position is 1, the value of the global mask at the corresponding pixel position is also 1, indicating that the pixel at the corresponding position belongs to the dynamic target area; conversely, if the values ​​of all masks at the corresponding pixel position are 0, the global mask value is 0, indicating that the pixel at the corresponding position belongs to the static background environment area; i takes values ​​from 1 to N, where N is the total number of dynamic targets detected.

[0129] This global dynamic mask can accurately fit the boundaries of dynamic targets. Compared with the traditional rectangular box elimination method, it significantly reduces the false elimination of redundant background information, thereby effectively excluding feature points in the dynamic area in the subsequent feature point extraction and matching stages, improving the accuracy of static environment information and the overall robustness of the SLAM system.

[0130] In addition, the dynamic semantic segmentation of the present invention can adopt an RGB-D segmentation method based on depth change, but the recognition efficiency may be reduced.

[0131] like Figure 1 As shown, in step S500, the obtained global dynamic mask is used to discriminate the feature point sets of each layer of the image pyramid to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer.

[0132] In this embodiment, in order to avoid the interference of dynamic targets on feature point matching and positioning, the global dynamic mask M generated by the aforementioned SAM model is used. dyn (x,y) is used to identify the filtered feature points.

[0133] The above steps specifically include:

[0134] For each feature point position (x j ,y j ), check whether it falls within the dynamic target mask area:

[0135]

[0136] IsDynamic represents whether it is a dynamic target, (x j ,y j ) is the position of each feature point. If the mask value M of the feature point position dyn (x j ,y j )=1, then the judgment result of IsDynamic is 1, indicating that the corresponding feature point is located in the dynamic target area, and it will be removed; if the mask value M of the feature point position dyn (x j ,y j )=1, the judgment result of IsDynamic is 0, indicating that the corresponding feature point is located in the static background environment area, and it is retained; thereby ensuring that the feature points finally retained are all static background environment feature points, avoiding matching errors caused by dynamic targets.

[0137] Through the aforementioned SuperPoint feature extraction and pyramid feature point screening method, this embodiment achieves uniform distribution and quantity control of feature points on multi-scale images, while effectively eliminating interfering feature points in dynamic areas, providing high-quality static environment feature data for subsequent feature matching and pose estimation, and further improving the positioning accuracy and robustness of the SLAM system in complex dynamic environments.

[0138] like Figure 1 As shown, in step S600, LightGlue feature matching and camera tracking are performed on the feature points of the static background environment area of ​​each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix.

[0139] In this embodiment, the feature matching and tracking module uses the LightGlue algorithm as the core method to achieve efficient matching of feature points between images and restore the camera pose in combination with the tracking thread of the SLAM system. LightGlue is a feature matching framework based on deep learning. It uses the Transformer architecture to perform global context modeling of feature descriptors. Compared with traditional matching methods (such as nearest neighbor matching), it can achieve more robust matching results in complex scenarios such as illumination changes, perspective changes, and dynamic interference, providing a reliable foundation for continuous tracking of SLAM systems.

[0140] The above steps specifically include:

[0141] S610, using static background environment feature points and their descriptors extracted from each layer of the image pyramid as input data for the LightGlue model;

[0142] S620, set the current frame image to I t , the previous frame image is I t-1 , LightGlue interactively models the feature descriptors of the two frames and outputs a set of matching pairs Where (i k ,j k ) represents the index of the kth pair of matching feature points, s k is the matching confidence score, and K is the total number of matching pairs.

[0143] S630: To further improve the matching quality, filter the matching pairs output by LightGlue and set the confidence threshold τ m (For example, set it to 0.7 or adjust it according to experiments), and only keep the k >τ m The matching pairs are taken as the final matching results.

[0144] S640, then use the obtained matching pairs to restore the camera pose in the tracking thread of the SLAM system. Assuming that the feature point positions corresponding to the matching pairs have been corrected to the original image scale through the image pyramid level, the camera motion estimation problem is constructed based on the matching pairs, and the camera pose transformation matrix is ​​optimized by minimizing the reprojection error. Right now:

[0145]

[0146] Among them, K ′ is the number of matching pairs after screening; Indicates the feature points of the previous frame The corresponding three-dimensional points are obtained by triangulation; is the two-dimensional observation position of the feature point matched in the current frame; π(·) is the camera projection function, which is used to project the three-dimensional point onto the image plane; is the camera pose transformation matrix to be optimized, which represents the camera motion from the previous frame to the current frame, including the rotation matrix R and the translation vector t; is the square of the Euclidean distance, which represents the reprojection error; the error function E is solved by nonlinear optimization methods (such as the Levenberg-Marquardt algorithm) to obtain the optimal camera pose transformation matrix matrix represents the relative transformation of the current frame and the previous frame camera pose, reflecting the position and orientation of the camera in the local space, and the next step also needs to be transformed from the camera coordinate system to the world coordinate system.

[0147] In the SLAM tracking thread, the pose transformation result is further fused with the historical pose information, and the camera trajectory is smoothed through filtering methods (such as extended Kalman filtering) or graph optimization framework to reduce cumulative errors. At the same time, if the number of matching pairs is insufficient or the pose estimation quality is low (for example, the re-projection error exceeds the preset threshold), the re-localization module is triggered to restore the tracking continuity.

[0148] Through the feature matching and tracking module based on LightGlue, the embodiment realizes efficient and robust feature matching in dynamic environment and weak texture scene, and combines the pose estimation method of minimizing re-projection error, which significantly improves the stability of SLAM system tracking and the accuracy of camera pose recovery.

[0149] As shown in Figure 1 , in step S700, the optimal camera pose transformation matrix of all key frames is obtained, and the local three-dimensional point cloud generated by combining the depth image information corresponding to each key frame is used for point cloud splicing and three-dimensional map construction, to generate an indoor scene three-dimensional point cloud map.

[0150] In the embodiment, the point cloud splicing and three-dimensional map construction module uses all key frame poses calculated by the aforementioned SLAM system to generate complete scene three-dimensional map data in combination with the depth information collected by the RGB-D camera, and the specific construction flowchart is as shown in Figure 5 .

[0151] The above steps specifically include:

[0152] S710, obtaining the camera pose of all key frames from the SLAM system , wherein represents the transformation matrix of the t-th key frame relative to the world coordinate system, including a rotation matrix and a translation vector, reflecting the position and orientation of the camera in the global space, and the camera pose transformation matrix is obtained by transforming from the camera coordinate system to the world coordinate system;

[0153] S720, then extracting the depth image D t corresponding to each key frame, using the intrinsic matrix K (including focal length and principal point offset parameters) of the RGB-D camera and the depth information to convert the two-dimensional image pixel points into local three-dimensional point cloud data, for each pixel point (u, v) in the depth image, the corresponding three-dimensional point X t (u, v) is calculated by the following method:

[0154]

[0155] Among them D t (u,v) is the depth value of the pixel (u,v), f is the focal length of the camera, (c x ,c y ) is the camera principal point coordinate; after conversion, the depth image of each key frame is converted into a three-dimensional point cloud in the local coordinate system

[0156] S730, then according to the pose of the key frame Each local point cloud Convert to the world coordinate system and perform preliminary splicing to obtain the global point cloud P global , the specific conversion process is:

[0157]

[0158] in It is the point cloud data after conversion to the world coordinate system;

[0159] S740, by accumulating and stitching the point cloud data of all key frames, a global 3D representation of the indoor scene is preliminarily constructed. Figure 6 The point cloud part corresponding to the key frame is displayed after the coordinate system is converted to obtain the spliced ​​scene point cloud map. Figure 6 (a) is the key frame point cloud (part), Figure 6 (b) is the scene point cloud map (partial) after stitching.

[0160] To improve stitching accuracy, the Iterative Closest Point (ICP) algorithm is used to finely align the point clouds of adjacent keyframes, optimize the relative pose between point clouds, and further reduce cumulative errors. The ICP algorithm iteratively calculates the closest point pairs and minimizes the distance between them, optimizing the transformation relationship between point clouds and ensuring that the stitched point cloud data is more spatially consistent. In addition, to reduce the impact of noise and redundant data, voxel grid filtering is applied to downsample the global point cloud. By dividing the point cloud into a fixed-size voxel grid and retaining the representative points within each grid, the point cloud density is effectively reduced while preserving the main geometric structure of the scene.

[0161] Finally, the global dynamic mask M generated by the aforementioned SAM model is used dyn (x, y), combined with depth information and camera projection model, the pixels in the dynamic area are mapped to the 3D point cloud, and the point cloud data belonging to the dynamic target is identified and removed. Specifically, for each 3D point Determine its pixel position (u, v) on the original image by back-projection. If M dyn(u,v) = 1, it is determined that the point belongs to the dynamic area and is removed. Through this step, it is ensured that the final three-dimensional map only contains static environment information.

[0162] Through the above point cloud splicing and three-dimensional map construction process, the embodiment generates high-quality scene three-dimensional map data, effectively eliminates the interference of dynamic targets, and improves the accuracy and consistency of the map through ICP optimization and filtering denoising, thereby providing a reliable spatial reference for subsequent navigation, path planning and positioning tasks of the SLAM system.

[0163] The three-dimensional reconstruction method for indoor dynamic environment based on deep learning provided by the embodiment has the following characteristics:

[0164] 1. Multi-layer image pyramid feature extraction and matching based on SuperPoint-LightGlue: using the deep learning feature point extraction network SuperPoint and the attention mechanism matching network LightGlue, the feature robustness and matching accuracy in weak texture and low light environments are significantly improved.

[0165] 2. Pyramid uniform sampling and dynamic mask removal strategy: by uniformly sampling a fixed number of feature points in the image pyramid and removing dynamic area feature points based on a dynamic mask, it is ensured that the feature points are reasonably distributed and free from dynamic point interference.

[0166] 3. YOLOv11+SAM joint dynamic target semantic mask generation and application: combining real-time target detection of YOLOv11 and arbitrary target pixel-level segmentation of SAM, high-precision and real-time dynamic area segmentation and feature removal are realized.

[0167] Corresponding to the above disclosed three-dimensional reconstruction method for indoor dynamic environment based on deep learning, the embodiment of the present application also discloses a three-dimensional reconstruction system for indoor dynamic environment based on deep learning, as shown in Figure 7 , which specifically comprises:

[0168] An image acquisition module for acquiring indoor environment video stream data and performing frame sampling;

[0169] An image pyramid construction module for performing multiple down-sampling on the sampled frame images and constructing an image pyramid;

[0170] A feature extraction module for performing SuperPoint visual feature point extraction and feature point screening on each layer of image in the image pyramid to obtain a feature point set of each layer;

[0171] a dynamic target detection module, configured to perform YOLOv dynamic target detection on each layer of image in the image pyramid, perform SAM segmentation and mask generation on the obtained dynamic target detection frame, and obtain a global dynamic mask;

[0172] a dynamic target feature point elimination module, configured to use the obtained global dynamic mask to distinguish feature point sets of each layer of the image pyramid, so as to eliminate feature points in a dynamic target region of each layer and retain feature points in a static background environment region of each layer;

[0173] a feature matching and camera tracking module, configured to perform LightGlue feature matching and camera tracking on the feature points in the static background environment region of each layer of the image pyramid of the two continuous frames of images, to obtain an optimal camera pose transformation matrix;

[0174] a three-dimensional map construction module, configured to obtain the optimal camera pose transformation matrix of all key frames, and combine a local three-dimensional point cloud generated by using depth image information corresponding to each key frame, to perform point cloud splicing and three-dimensional map construction, and generate an indoor scene three-dimensional point cloud map.

[0175] It should be noted that the detailed description of the indoor dynamic environment three-dimensional reconstruction system based on deep learning provided in the embodiments of the present application can refer to the related description of the indoor dynamic environment three-dimensional reconstruction method based on deep learning provided in the embodiments of the present application, which will not be repeated here.

[0176] In addition, the embodiments of the present application also provide an electronic device, which comprises a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of the indoor dynamic environment three-dimensional reconstruction method based on deep learning according to any one of the above.

[0177] It should be noted that the detailed description of the electronic device provided in the embodiments of the present application can refer to the related description of the indoor dynamic environment three-dimensional reconstruction method based on deep learning provided in the embodiments of the present application, which will not be repeated here.

[0178] In addition, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program; when the computer program is executed by a processor, the steps of the indoor dynamic environment three-dimensional reconstruction method based on deep learning according to any one of the above are implemented.

[0179] It should be noted that the detailed description of the computer readable storage medium provided in the embodiments of the present application can refer to the related description of the indoor dynamic environment three-dimensional reconstruction method based on deep learning provided in the embodiments of the present application, which will not be repeated here.

[0180] Those skilled in the art can understand that all or part of the functions of various methods in the above embodiments can be realized by hardware or by a computer program. When all or part of the functions in the above embodiments are realized by a computer program, the program can be stored in a computer readable storage medium, which can include a read-only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, and the like. The above functions are realized by executing the program by a computer. For example, the program is stored in a memory of a device, and the above functions are realized by executing the program in the memory by a processor. In addition, when all or part of the functions in the above embodiments are realized by a computer program, the program can also be stored in a storage medium such as a server, another computer, a disk, an optical disk, a flash disk, or a mobile hard disk, and is saved in a memory of a local device by downloading or copying, or the system of the local device is updated, and the above functions are realized by executing the program in the memory by a processor.

[0181] The above application of specific examples to the present application is described, which is only used to help understand the present application and does not limit the present application. For those skilled in the art, according to the idea of the present application, a number of simple deductions, deformations or substitutions can be made.

Claims

1. A method for 3D reconstruction of indoor dynamic environments based on deep learning, characterized in that: The method comprises: Collect indoor environment video stream data and perform frame sampling; The sampled frame image is downsampled multiple times and an image pyramid is constructed; SuperPoint visual feature point extraction and feature point screening are performed on each layer of the image in the image pyramid to obtain the feature point set of each layer; YOLOv dynamic target detection is performed on each layer of the image pyramid, and the obtained dynamic target detection box is segmented and masked to obtain a global dynamic mask; The obtained global dynamic mask is used to discriminate the feature point sets of each layer of the image pyramid to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer; Perform LightGlue feature matching and camera tracking on the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix; The optimal camera pose transformation matrix of all key frames is obtained, and the local 3D point cloud generated by combining the depth image information corresponding to each key frame is used to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene.

2. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 1, wherein: SuperPoint visual feature point extraction and feature point screening are performed on each layer of the image in the image pyramid to obtain the feature point set of each layer, including: Input each layer of the image pyramid into the SuperPoint network for feature point detection and extract the initial feature point set and its descriptor; The extracted initial feature points are screened on each layer of the image, including: dividing each layer of the image into a uniform grid, and sorting the feature points output by the SuperPoint network in each grid unit according to the confidence level, retaining only the top several feature points with relatively high confidence levels; at the same time, setting an upper limit on the total number of feature points in each layer of the image, and further screening out a set of feature points with relatively high confidence levels through global sorting.

3. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 1, wherein: YOLOv dynamic target detection is performed on each layer of the image pyramid, including: YOLOv11 is used as the dynamic target detection network. Based on a one-stage framework, the CSPDarknet-v7 backbone network is used for feature extraction. Spatial Pyramid Pooling (SPP) and Path Aggregation Network (PAN) are used to enhance multi-scale feature fusion capabilities, significantly improving detection accuracy while maintaining a high frame rate. Suppose the input image is I(x,y), where (x,y) is the horizontal and vertical coordinates of each pixel in image I. The YOLOv11 network generates a set of candidate detection boxes B at the output layer: Among them, b i is the i-th candidate detection frame, N is the total number of candidate detection frames, (x i ,y i ) is the center coordinate of the candidate detection box, (w i ,h i ) is the width and height of the candidate detection box, p i is the category confidence; after non-maximum suppression, the detection box with confidence higher than the preset threshold is retained as the preliminary positioning result of the dynamic target.

4. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 3, wherein: Perform SAM segmentation and mask generation on the obtained dynamic target detection frame, and obtain a global dynamic mask, specifically including: Each detection box b i It is input into the SAM model as a prompt to further accurately segment the dynamic area; The SAM model first performs high-resolution encoding on the input image I(x,y) to generate an image feature map F img , and by prompting the encoder to detect the box b i Mapped to vector v i , then F img and v i The fused input mask decoder generates a binary semantic mask corresponding to the dynamic target: Among them, M i (x,y) represents the segmentation mask of the target corresponding to the i-th detection box. Pixels with a value of 1 belong to the dynamic target area, and pixels with a value of 0 belong to the background or non-target area. Take the union of all target masks to get the global dynamic mask: Among them, M dyn (x,y) represents the global dynamic mask, covering all areas in the input image that are identified as dynamic objects; Represents the logical OR operation, that is, for all single target masks M i The union operation is performed on the values ​​of (x, y) at each pixel position (x, y). If the value of any target mask at the corresponding pixel position is 1, the value of the global mask at the corresponding pixel position is also 1, indicating that the pixel at the corresponding position belongs to the dynamic target area; conversely, if the values ​​of all masks at the corresponding pixel position are 0, the global mask value is 0, indicating that the pixel at the corresponding position belongs to the static background environment area; i takes values ​​from 1 to N, where N is the total number of dynamic targets detected.

5. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 4, wherein: The obtained global dynamic mask is used to discriminate the feature point sets of each layer of the image pyramid to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer, including: For each feature point position (x j ,y j ), check whether it falls within the dynamic target mask area: IsDynamic represents whether it is a dynamic target, (x j ,y j ) is the position of each feature point. If the mask value M of the feature point position dyn (x j ,y j )=1, then the judgment result of IsDynamic is 1, indicating that the corresponding feature point is located in the dynamic target area, and it will be removed; if the mask value M of the feature point position dyn (x j ,y j )=1, the judgment result of IsDynamic is 0, indicating that the corresponding feature point is located in the static background environment area, and it is retained; thereby ensuring that the feature points finally retained are all static background environment feature points, avoiding matching errors caused by dynamic targets.

6. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 1, wherein: Perform LightGlue feature matching and camera tracking on the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix, specifically including: The static background environment feature points and their descriptors extracted from each layer of the image pyramid are used as input data for the LightGlue model; Let the current frame image be I t , the previous frame image is I t-1 , LightGlue interactively models the feature descriptors of the two frames and outputs a set of matching pairs Where (i k ,j k ) represents the index of the kth pair of matching feature points, s k is the matching confidence score, K is the total number of matching pairs; Further filter the matching pairs output by the LightGlue model and set the confidence threshold τ m , only keep the k >τ m The matching pair is taken as the final matching result; The obtained matching pairs are then used to restore the camera pose in the tracking thread of the SLAM system. Assuming that the feature point positions corresponding to the matching pairs have been corrected to the original image scale through the image pyramid level, the camera motion estimation problem is constructed based on the matching pairs, and the camera pose transformation matrix is ​​optimized by minimizing the reprojection error. Right now: Among them, t is the current frame, t-1 is the previous frame, k is the index of the matching feature point pair, and i and j represent the index of the matched feature points in the current frame and the previous frame in the matching pair respectively. ′ is the number of matching pairs after screening; Indicates the feature points of the previous frame The corresponding three-dimensional points are obtained by triangulation; is the two-dimensional observation position of the feature point matched in the current frame; π(·) is the camera projection function, which is used to project the three-dimensional point onto the image plane; is the camera pose transformation matrix to be optimized, which represents the camera motion from the previous frame to the current frame, including the rotation matrix R and the translation vector t; is the square of the Euclidean distance, which represents the reprojection error; the error function E is solved by nonlinear optimization method to obtain the optimal camera pose transformation matrix matrix It represents the relative transformation between the camera pose of the current frame and the previous frame, reflecting the position and orientation of the camera in the local space.

7. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 6, wherein: Perform LightGlue feature matching and camera tracking on the static background environment area feature points of each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix, which specifically includes: In the SLAM tracking thread, the camera pose transformation results are further fused with the historical pose information, and the camera trajectory is smoothed through filtering methods or graph optimization frameworks to reduce the cumulative error; at the same time, if the number of matching pairs obtained is lower than the preset threshold or the reprojection error exceeds the preset threshold, the relocalization module is triggered to restore tracking continuity.

8. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 6, wherein: Obtain the optimal camera pose transformation matrix for all keyframes, and combine the local 3D point cloud generated by the depth image information corresponding to each keyframe to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene. Specifically, this includes: Get the camera poses of all keyframes from the SLAM system in Represents the transformation matrix of the t-th frame keyframe relative to the world coordinate system, including the rotation matrix and translation vector, which reflects the position and orientation of the camera in the global space. Obtained by transforming from the camera coordinate system to the world coordinate system; Then extract the depth image D corresponding to each key frame t , using the intrinsic parameter matrix K of the RGB-D camera and the depth information, the two-dimensional image pixels are converted into local three-dimensional point cloud data. For each pixel point (u, v) in the depth image, the corresponding three-dimensional point X t (u,v) is calculated as follows: Among them D t (u,v) is the depth value of the pixel (u,v), [x,y,z] is the position coordinate of the camera in three-dimensional space, f is the focal length of the camera, (c x ,c y ) is the camera principal point coordinate; after conversion, the depth image of each key frame is converted into a three-dimensional point cloud in the local coordinate system Then according to the pose of the key frame Each local point cloud Convert to the world coordinate system and perform preliminary splicing to obtain the global point cloud P global , the specific conversion process is: in It is the point cloud data after conversion to the world coordinate system; By accumulating and stitching the point cloud data of all key frames, a global three-dimensional representation of the indoor scene is preliminarily constructed.

9. The method for 3D reconstruction of an indoor dynamic environment based on deep learning according to claim 8, wherein: Obtain the optimal camera pose transformation matrix for all keyframes, and combine the local 3D point cloud generated by the depth image information corresponding to each keyframe to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene. Specifically, it also includes: The iterative closest point ICP algorithm is used to finely align the point clouds of adjacent key frames, optimize the relative poses between point clouds, and further reduce the cumulative error; Apply voxel grid filtering to downsample the global point cloud. By dividing the point cloud into fixed-size voxel grids and retaining the representative points within each grid, the point cloud density is effectively reduced while preserving the main geometric structure of the scene. Using the global dynamic mask M generated based on the SAM model dyn (x, y), combined with depth information and camera projection model, the pixels in the dynamic area are mapped to the 3D point cloud, and the point cloud data belonging to the dynamic target is identified and eliminated. Specifically, for each 3D point Determine its pixel position (u, v) on the original image by back-projection. If M dyn If (u, v) = 1, the corresponding point is determined to belong to the dynamic target area and is removed, ensuring that the final 3D map only contains static background environment information.

10. A deep learning-based 3D reconstruction system for indoor dynamic environments, characterized in that: The system comprises: Image acquisition module, used to collect indoor environment video stream data and perform frame sampling; An image pyramid construction module is used to downsample the sampled frame image multiple times and construct an image pyramid; The feature extraction module is used to extract and filter SuperPoint visual feature points for each layer of the image in the image pyramid to obtain the feature point set of each layer; The dynamic target detection module is used to perform YOLOv dynamic target detection on each layer of the image pyramid, perform SAM segmentation and mask generation on the obtained dynamic target detection box, and obtain a global dynamic mask; The dynamic target feature point elimination module is used to use the obtained global dynamic mask to identify the feature point sets of each layer of the image pyramid, so as to eliminate the feature points in the dynamic target area of ​​each layer and retain the feature points in the static background environment area of ​​each layer; The feature matching and camera tracking module is used to perform LightGlue feature matching and camera tracking on the feature points of the static background environment area in each layer of the image pyramid of two consecutive frames to obtain the optimal camera pose transformation matrix; The 3D map construction module is used to obtain the optimal camera pose transformation matrix for all key frames, and combine the local 3D point cloud generated by the depth image information corresponding to each key frame to perform point cloud stitching and 3D map construction to generate a 3D point cloud map of the indoor scene.

Images