Open world construction site reconstruction method based on video and point cloud dual modal data

Abstract

The application discloses an open world construction scene reconstruction method based on video and point cloud dual-mode data, and belongs to the technical field of machine vision and artificial intelligence, and comprises the following steps: assembling a laser radar and an industrial camera, anchoring a single visual angle, and synchronously collecting three-dimensional point cloud and two-dimensional color RGB image dual-mode data in a construction scene; using an automatic pixel-level calibration algorithm to calibrate an external parameter matrix for the collected point cloud and image, and visualizing the color point cloud in real time after recording and registration; based on an input open world text prompt word, segmenting a specific object in a construction scene from a real-time captured color point cloud instance; and accurately mapping the segmented object region to the three-dimensional point cloud through the external parameter matrix to realize accurate analysis of the three-dimensional space. The application improves the accuracy and integrity of three-dimensional construction scene reconstruction, improves the intelligent level of construction scene reconstruction, and can adapt to the dynamically changing environment and new objects in the open world.

Description

Technical Field

[0001] This invention relates to the field of machine vision and artificial intelligence, and in particular to a method for reconstructing open-world construction scenes based on dual-modal data of video and point cloud. Background Technology

[0002] In the field of computer vision, rapid 3D construction scene reconstruction technology is one of the key research topics. Currently, common 3D reconstruction methods mainly include image rendering and compositing, surveying and modeling, and radar point cloud scanning. However, with the rapid development of social production technology, the demand for applications that can quickly scan and reconstruct 3D construction scenes in various engineering projects is increasing. In practical engineering applications, due to the long reconstruction time, large computational load, and time-consuming and labor-intensive nature of image rendering and compositing and surveying and modeling, radar point cloud scanning is usually more commonly used to reconstruct 3D construction scenes.

[0003] With the rapid development of computer vision and artificial intelligence technologies, 3D construction scene reconstruction technology has been widely applied in many fields such as building construction, industrial inspection, and virtual reality. Traditional 3D construction scene reconstruction methods mainly rely on single-modal data acquisition, such as LiDAR or camera images. While these methods can achieve 3D construction scene reconstruction to a certain extent, they have many limitations. For example, when using only LiDAR data for reconstruction, although relatively accurate depth information can be obtained, data loss or noise interference is likely to occur when dealing with complex construction scenes; while relying solely on camera image data for reconstruction can provide rich texture information, it lacks depth information, resulting in an inaccurate and incomplete 3D construction scene reconstruction.

[0004] In recent years, with the continuous development of deep learning technology, instance segmentation technology has made significant progress in the field of image processing. Instance segmentation technology can accurately segment each object in an image and assign it a unique identifier, providing new ideas and methods for solving the problem of 3D construction scene reconstruction in open-world construction scenarios. However, most existing instance segmentation technologies are concentrated in the field of 2D images, and there are still many technical challenges in processing 3D point cloud data. How to effectively apply instance segmentation technology to 3D point cloud data and achieve rapid reconstruction of open-world construction scenes has become one of the important problems that urgently need to be solved in the field of computer vision and artificial intelligence.

[0005] Meanwhile, the development of open-world real-time object recognition technology has provided a new solution to this problem. Open-world recognition refers to the system's ability to identify novel objects not seen during the training phase in a constantly changing environment. This technology is particularly important in field applications because the environment and objects on-site are often in dynamic change, and traditional closed-world recognition methods are difficult to adapt to such dynamic changes. Therefore, researching and developing a rapid reconstruction system for open-world 3D construction scenes based on dual-modal data of video and point clouds is of great significance for improving on-site safety and monitoring efficiency, and has broad application prospects. Summary of the Invention

[0006] The purpose of this invention is to provide an open-world construction scene reconstruction method based on dual-modal data of video and point cloud. This method addresses the problems of traditional construction scene manual inspection using only simple methods such as two-dimensional video streams, which require inspectors to maintain high concentration for extended periods and are prone to fatigue or negligence. Furthermore, traditional methods cannot effectively handle the complex and ever-changing environment and objects on-site, making them unsuitable for real-time object recognition in open-world construction scenarios. This invention aims to effectively improve the efficiency of object instance segmentation by integrating the advantages of dual-modal data from LiDAR and industrial cameras, combined with open-world real-time object recognition technology.

[0007] To achieve the above objectives, this invention provides an open-world construction scene reconstruction method based on dual-modal data of video and point cloud, comprising the following steps: S1. By assembling a lidar and an industrial camera and anchoring a single viewpoint, dual-modal data of 3D point cloud and 2D color RGB image are simultaneously acquired in the construction scene. S2. Feature extraction is performed on the collected construction scene point cloud and images. The external parameter matrix is ​​accurately calibrated using an automatic pixel-level calibration algorithm, and the recorded and registered color point cloud is visualized in real time. S3. Segment specific objects in a construction scene based on real-time captured color point cloud instances using input open-world text prompts; S4. The segmented object region is accurately mapped to a 3D point cloud using an extrinsic parameter matrix to achieve accurate analysis in 3D space.

[0008] Preferably, in S1, the dual-modal data synchronous acquisition includes the following steps: S11. Ensure that the point cloud data collected by the lidar corresponds in time with the RGB image data captured by the industrial camera by recording and matching the timestamps of the two. S12. Control the lidar and industrial camera to start and stop data acquisition simultaneously to ensure the synchronization of data acquisition; S13. Set the LiDAR and industrial camera to collect data at the same time interval to ensure that the point cloud and RGB image of each frame are consistent in time. S14. Achieve a one-to-one correspondence between point cloud data and RGB image data within the same frame, providing accurate data pairs for subsequent dual-modal data registration; S15. Using the data collected synchronously above, prepare a benchmark dataset for dual-modal data registration to support high-precision point cloud and image fusion.

[0009] Preferably, in S2, feature extraction includes the following steps: S201. Extract the point cloud data and RGB image data of the construction scene recorded in S1; S202. Downsample the collected raw point cloud data to reduce the amount of data and improve the efficiency of subsequent processing, while retaining key geometric features. S203. Apply statistical methods to identify and remove outliers in the point cloud. The noise reduction process helps to improve the quality of point cloud data. S204. Determine the calibration range of the point cloud data, i.e. the region of interest, and focus the analysis while eliminating interference from irrelevant data.

[0010] Preferably, in S2, the specific process includes the following steps: S21. Use a calibration board to take pictures to obtain the intrinsic parameter matrix of the industrial camera for calibration of the construction scene; S22. Apply an automatic pixel-level calibration algorithm to extract edge feature information and calibrate the extrinsic parameter matrix of the collected construction scene dataset; S23. Apply the calibrated extrinsic parameter matrix and combine it with the timestamp matching in S1 on site to merge the recorded LiDAR point cloud and the RGB image recorded by the industrial camera in the same frame, so as to realize the real-time four-dimensional color point cloud visualization of the recorded data on site. The four-dimensional specifically refers to three-dimensional coordinates plus one-dimensional color information.

[0011] Preferably, step S3 specifically includes the following steps: S31. Utilize prompt words in open-source text sets to construct a semantic understanding model based on an advanced semantic understanding architecture; to achieve efficient semantic parsing and understanding of text data related to construction scenarios; the semantic understanding architecture is used to process open-world text data, including but not limited to the Transformer architecture, the BERT autoencoder semantic understanding model, the semantic understanding model based on graph neural networks, and the semantic understanding model based on knowledge graphs; S32. Input open-world text data related to the construction scenario into the semantic understanding model; S33. Based on the collected low-latency real-time four-dimensional color point cloud data, the input open-world text prompts are applied in real time to perform object instance segmentation on each frame of the image.

[0012] Preferably, in S4, specifically: using the calibrated extrinsic parameter matrix and four-dimensional color point cloud data, the image results after segmentation of the construction scene object instance are mapped to the color point cloud of the corresponding frame in real time.

[0013] This invention also provides an open-world construction scene reconstruction system based on dual-modal data of video and point cloud, comprising: Assembly and acquisition module: Simultaneously acquire dual-modal data of 3D point cloud and 2D color RGB image within the construction scene by assembling a lidar and an industrial camera and anchoring a single viewpoint; Recording and Visualization Module: For the collected point clouds and images of the construction scene, the extrinsic parameter matrix is ​​accurately calibrated using an automatic pixel-level calibration algorithm, and the recorded and registered color point clouds are visualized in real time. Open World Recognition Module: Segment specific objects in a construction scene based on real-time captured color point cloud instances using input open world text prompts; 3D Scene Mapping Module: Accurately maps segmented object regions to 3D point clouds using extrinsic parameter matrices to achieve precise analysis of 3D space.

[0014] Preferably, the lidar acquires point cloud data in real time at high frequency, providing sufficient data density for rapid acquisition; the lidar and industrial camera acquire data through a time synchronization mechanism to ensure the consistency of dual-modal data in time, providing a guarantee for high-precision fusion; the lidar is connected to edge computing devices through a standard interface and integrated into the system of this invention. The industrial camera provides high-resolution color images, capturing detailed information on-site; it is designed with a wide dynamic range to maintain image quality under different lighting conditions; it supports high frame rate acquisition, providing a smooth video stream for real-time construction scene segmentation and reconstruction; and it connects to edge computing devices through a standard interface and is integrated into the system of this invention.

[0015] This invention provides an edge computing device, comprising: The processor is designed for edge computing to handle complex computing tasks, including machine learning and computer vision. The memory stores a computer-executable program, which, when executed by the processor, causes the processor to execute an open-world construction scene reconstruction method based on dual-modal data of video and point cloud. The communication interface supports multiple communication protocols to achieve efficient data transmission between devices; Power management includes a power management system to adapt to energy constraints in edge computing environments; The expansion interface is used to support the connection and expansion of external devices, including LiDAR, industrial cameras, mice and keyboards.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an open-world construction scene reconstruction method based on dual-modal data of video and point cloud.

[0017] Therefore, the open-world construction scene reconstruction method based on dual-modal data of video and point cloud, as described above, has the following beneficial effects: 1) Improve reconstruction accuracy and completeness: By fusing dual-modal data of LiDAR point cloud and industrial camera images, the advantages of both types of data are fully utilized to effectively compensate for the information loss problem of single-modal data in complex construction scenarios, thereby improving the accuracy and completeness of 3D construction scene reconstruction and more realistically and accurately restoring the 3D structure and details of the construction scene.

[0018] 2) Improve automation and intelligence by introducing open-world real-time object recognition technology. Based on the input open-world text prompts, the real-time captured color point cloud is segmented into instances and accurately mapped to the 3D point cloud to achieve accurate analysis of 3D space. This enables full-process automation from data acquisition to object recognition, segmentation and 3D reconstruction, improving the efficiency and automation of construction scene reconstruction, while adapting to the dynamically changing environment and new objects in the open world.

[0019] 3) Real-time performance and efficiency: Optimize data acquisition, processing, and analysis processes to achieve low-latency real-time reconstruction and visualization. High-frequency data acquisition, combined with automatic pixel-level calibration algorithms and the powerful processing capabilities of edge computing devices, ensures real-time processing of large amounts of dual-modal data and rapid generation of four-dimensional color point clouds for visualization, meeting the requirements of real-time monitoring and rapid response at construction sites.

[0020] 4) Excellent scalability: The system's software architecture is also easy to expand and upgrade, allowing for the convenient addition of new functional modules or optimization and improvement of existing modules to adapt to ever-changing application needs and technological developments.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is a flowchart of an embodiment of the present invention; Figure 2 This is an overall logical framework diagram of an embodiment of the present invention; Figure 3 This is a system composition block diagram according to an embodiment of the present invention; Figure 4This is a schematic diagram of the hardware structure of an edge computing device according to an embodiment of the present invention; Figure 5 This is a visualization result of the original point cloud collected by Livox lidar at the construction site according to an embodiment of the present invention; Figure 6 This is a visualization result of the original color RGB image captured by an MVS camera at the construction site according to an embodiment of the present invention; Figure 7 This is the registration intrinsic and extrinsic parameter matrix obtained by applying the edge feature extraction algorithm in this embodiment of the invention to register a single-frame laboratory-acquired image and point cloud; Figure 8 This is a single-frame color point cloud of the construction site obtained after applying the registration matrix in an embodiment of the present invention (corresponding to...). Figure 4 and Figure 5 Visualization; Figure 9 This is a two-dimensional image instance segmentation mask extracted after the Groungding-DINO open-world set prompts the input text "person" (construction worker) and "excavator" (excavator) in the embodiment of the present invention, including an accuracy index; a is a two-dimensional image, and b is the obtained segmentation mask; Figure 10 This embodiment of the invention visualizes the three-dimensional segmentation mask result obtained by reprojecting a two-dimensional image mask back into the original point cloud. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] Example 1 This invention provides a method for reconstructing open-world construction scenes based on dual-modal data of video and point cloud, the process of which is as follows: Figure 1 As shown, Figure 2 This is the overall logical framework diagram of this embodiment. Figure 1 The process is integrated at the system level, and with Figure 1 They complement each other, enabling simultaneous understanding of methodological processes and system logic. Specifically, they include the following steps: S1. Simultaneous acquisition of dual-modal data—3D point cloud and 2D color RGB image—of the construction scene by assembling a LiDAR and an industrial camera and anchoring a single-viewpoint. The simultaneous acquisition of dual-modal data includes the following steps: S11. Ensure that the point cloud data collected by the lidar corresponds in time with the RGB image data captured by the industrial camera by recording and matching the timestamps of the two. S12. Control the lidar and industrial camera to start and stop data acquisition simultaneously to ensure the synchronization of data acquisition; S13. Set the LiDAR and industrial camera to collect data at the same time interval to ensure that the point cloud and RGB image of each frame are consistent in time. S14. Achieve a one-to-one correspondence between point cloud data and RGB image data within the same frame, providing accurate data pairs for subsequent dual-modal data registration; S15. Using the data collected synchronously above, prepare a benchmark dataset for dual-modal data registration to support high-precision point cloud and image fusion.

[0026] In this embodiment, the Livox Avia LiDAR and the HikRobot MVS industrial camera are fixed at the same viewing angle using a rigid bracket, ensuring that their fields of view are aligned and the overlap area is greater than 80%. Microsecond-level time synchronization is achieved via the PTP (Precision Time Protocol), with the acquisition frequency set to 10Hz. Each frame of data includes: 3D point cloud: Real-time scanning, covering the construction scene area within the field of view of the lidar; RGB image: 2448×2880 resolution, corresponding to the real-time construction scene in 3D point cloud.

[0027] The recorded 3D point cloud and RGB image data were published via ROS2 topics. The topic names for recordings from the Livox Avia LiDAR and HikRobot MVS industrial cameras are as follows: / livox / lidar (sensor_msgs / PointCloud2) / camera / color / image_raw (sensor_msgs / Image) Frame-level synchronization is achieved through a timestamp alignment mechanism. Before system startup, a parallel shell script (sync_record.sh) sends a "start" command to both the LiDAR node and the industrial camera node. Internally, the script calls the `roslaunch` command to simultaneously start the driver and synchronously terminates recording upon receiving the SIGINT signal, ensuring a time deviation of less than 1 ms between the first and last frames. The raw data generated during recording is centrally stored in ROS.bag format, with file names including the start timestamp for easy offline retrieval. During playback, the `rosbag play` tool publishes the topics ` / livox / lidar` and ` / camera / color / image_raw` in the order of the original timestamps. The ROS Time Synchronizer filter subscribes to these two topics, performing frame alignment with a 33 ms time window, and outputting synchronized PointCloud2 and Image message pairs. Finally, the script `extract_sync_frames.sh` writes the matched frames sequentially to the corresponding folders (` / dataset / YYYY-MM-DD / pcd` and ` / rgb`), generating timestamped index files, forming a construction scene dataset that can be directly used for extrinsic parameter calibration and network training. In this embodiment, Figure 5 Visualization results of the original point cloud collected by Livox lidar at the construction site; Figure 6 Visualization results of raw color RGB images captured by an MVS camera at the construction site.

[0028] S2. Feature extraction is performed on the collected construction scene point cloud and images. An automatic pixel-level calibration algorithm is used to accurately calibrate the extrinsic parameter matrix, enabling real-time visualization of the recorded and registered color point cloud. Specifically, this includes the following steps: S21. Use a calibration board to take pictures to obtain the intrinsic parameter matrix of the industrial camera for calibration of the construction scene; S22. Apply an automatic pixel-level calibration algorithm to extract edge feature information and calibrate the extrinsic parameter matrix of the collected construction scene dataset; S23. Apply the calibrated extrinsic parameter matrix and combine it with the timestamp matching in S1 on site to merge the recorded LiDAR point cloud and the RGB image recorded by the industrial camera in the same frame, so as to realize the real-time four-dimensional color point cloud visualization of the recorded data on site. The four-dimensional specifically refers to three-dimensional coordinates plus one-dimensional color information.

[0029] Feature extraction specifically includes the following steps: S201. Extract the point cloud data and RGB image data of the construction scene recorded in S1; S202. Downsample the collected raw point cloud data to reduce the amount of data and improve the efficiency of subsequent processing, while retaining key geometric features. S203. Apply statistical methods to identify and remove outliers in the point cloud. The noise reduction process helps to improve the quality of point cloud data. S204. Determine the calibration range of the point cloud data, i.e. the region of interest, and focus the analysis while eliminating interference from irrelevant data.

[0030] The specific operation in this embodiment is as follows: First, PCL is applied to process the original point cloud as follows: Point cloud downsampling was performed using a VoxelGrid filter with a voxel size of 0.05m; StatisticalOutlierRemoval was used to remove outliers from the point cloud, with the number of neighbors set to 50 and the standard deviation factor set to 1.0. The point cloud processed in the above two steps is then cropped, with the forward view direction as the center and the cropping range being [missing information]. x ∈[0,30m], y ∈[-15m,15m], z ∈[-2m,5m], while retaining the data of the construction scene of interest, it removes the interference of edge construction scenes and remote construction scenes, reducing the amount of data while accelerating the efficiency of point cloud registration and processing.

[0031] Next, Zhang Zhengyou's chessboard method was used to calibrate the camera intrinsic parameters, obtaining the intrinsic parameter matrix. K : ; in,( f x , f y () is the focal length (unit: pixels), c x , c y () are the coordinates of the main point.

[0032] An automatic pixel-level calibration algorithm based on edge feature matching is adopted, which eliminates the need for extrinsic parameter matrix calibration using a calibration board. The specific process is as follows: The LiDAR point cloud is voxel-segmented, and RANSAC is used to fit the local plane to extract depth-continuous edges. The Canny algorithm is used to extract 2D edges from the RGB image, and a kD tree is constructed. The LiDAR edge points are projected onto the image plane, and a residual function is constructed. ; Where π represents the pinhole projection model, and f is the distortion correction function. and Let L be the normal vector and position point of the image edge, and let L be the coordinate system of the lidar, C be the coordinate system of the camera, and C be the two-dimensional coordinate system of the image plane.

[0033] This represents the external parameters between the lidar and the camera to be calibrated.

[0034] The extrinsic parameter matrix typically contains rotation and translation information, used to describe the spatial position and orientation of one coordinate system relative to another. This represents a 3×3 rotation matrix from the lidar coordinate system L to the camera coordinate system C. This represents a 3×1 translation vector from the origin of the lidar coordinate system L to the origin of the camera coordinate system C. denoted as a special Euclidean group, which is a group consisting of all rotation matrices and translation vectors used to describe rigid body transformations in three-dimensional space.

[0035] During the calibration process, the goal of this embodiment is to accurately determine the parameters so that point cloud data and image data can be accurately converted between different coordinate systems. By minimizing reprojection error or other optimization objectives, the optimal rotation matrix and translation vector are solved, thereby achieving precise alignment between the LiDAR and the camera.

[0036] In this embodiment, the Levenberg-Marquardt algorithm is used to solve for the optimal extrinsic parameters, thereby minimizing the residual function.

[0037] Finally, the recorded and registered color point cloud is visualized in real time, and the point cloud and the image are fused with the calibrated extrinsic matrix to form a four-dimensional color point cloud. x , y , z (RGB), and achieve real-time visualization through RViz2 software. Figure 7 This is the registration intrinsic and extrinsic parameter matrix obtained after registering a single-frame laboratory-acquired image and point cloud using an edge feature extraction algorithm. Figure 8 To visualize the single-frame color point cloud of the construction site obtained after applying the registration matrix, the corresponding... Figure 4 and Figure 5 .

[0038] S3. Segment specific objects in the construction scene based on real-time captured color point cloud instances using input open-world text prompts. This includes the following steps: S31. Utilize prompt words from open-source text collections to construct a semantic understanding model based on an advanced semantic understanding architecture; to achieve efficient semantic parsing and understanding of text data related to construction scenarios; the semantic understanding architecture is used to process open-world text data, including the Transformer architecture, the BERT autoencoder semantic understanding model, the graph neural network-based semantic understanding model, and the knowledge graph-based semantic understanding model, etc. S32. Input open-world text data related to the construction scenario into the semantic understanding model; S33. Based on the collected low-latency real-time four-dimensional color point cloud data, the input open-world text prompts are applied in real time to perform object instance segmentation on each frame of the image.

[0039] In this embodiment, the algorithm for segmenting specific objects in a construction scene based on real-time captured color point cloud instances using input open-world text prompts employs the open-set object detection algorithm GroundingDINO1.5 Edge model.

[0040] The GroundingDINO1.5 model is based on a Transformer architecture and is used to process image and language data. Its deep early fusion strategy combines language and image features through a cross-attention mechanism during the feature extraction stage. Large-scale pre-training and self-training are performed on the Grounding-20M dataset, and self-training is conducted using pseudo-labeled data. The loss function optimizes bounding box regression and enhances the classification ability between predicted objects and language labels during training. GroundingDINO1.5 Edge is optimized for edge devices, using EfficientViT-L1 as the image backbone for fast multi-scale feature extraction.

[0041] Specifically, the GroundingDINO1.5 Edge model takes as input RGB images of the construction scene captured by an industrial camera and natural language prompts, and outputs as bounding boxes of different categories of objects to be segmented, category confidence scores, and pixel-level masks. Figure 9 The two-dimensional image instance segmentation mask, including accuracy metrics, is extracted after applying the Groungding-DINO open-world set prompt text "person" (construction worker) and "excavator" (excavator) to this embodiment.

[0042] Based on the input open-world text prompts, the real-time captured images of specific objects in the construction scene are segmented from real-time captured color point cloud instances. The real-time captured images are the RGB image stream of the real-time construction scene recorded by an industrial camera in S1.

[0043] S4. The segmented object region is accurately mapped to the 3D point cloud using an extrinsic parameter matrix to achieve accurate 3D spatial analysis. Specifically, using the calibrated extrinsic parameter matrix and 4D color point cloud data, the image results of the segmented construction scene object instance are mapped to the corresponding frame's color point cloud in real time.

[0044] In this embodiment, the image mask obtained in S3 is projected onto the point cloud space through an extrinsic parameter matrix, specifically as follows: For each pixel (u,v) within the mask, backproject it onto the spatial ray P. c : ; Further transformation to the lidar coordinate system P l :P l =( ) -1 ·P c Match the nearest point in the point cloud and assign a semantic label.

[0045] The output is a semantic color point cloud, which supports real-time rendering and interaction. The semantic labels of the point cloud include various objects that are required to be segmented by the open-world text prompts input in S3. Figure 10 This embodiment visualizes the three-dimensional segmentation mask result obtained by reprojecting the two-dimensional image mask back into the original point cloud.

[0046] Figure 3 The diagram shows the components of an open-world construction scene reconstruction system based on dual-modal data of video and point cloud, including an assembly and acquisition module 310, a recording and visualization module 320, an open-world recognition module 330, and a 3D scene mapping module 340.

[0047] The assembly and acquisition module 310 performs operation S1, simultaneously acquiring dual-modal data of 3D point cloud and 2D color RGB image within the construction scene by assembling a LiDAR and an industrial camera and anchoring them in a single-viewpoint configuration. The LiDAR acquires point cloud data at high frequency in real time, providing sufficient data density for rapid acquisition. The LiDAR and industrial camera acquire data through a time synchronization mechanism, ensuring temporal consistency of the dual-modal data and guaranteeing high-precision fusion. The LiDAR connects to the edge computing device via a standard interface and is integrated into the system of this invention. The industrial camera provides high-resolution color images, capturing detailed information from the scene; it is designed with a wide dynamic range to maintain image quality under different lighting conditions; it supports high frame rate acquisition, providing a smooth video stream for real-time construction scene segmentation and reconstruction; and it connects to the edge computing device via a standard interface and is integrated into the system of this invention.

[0048] The recording and visualization module 320 performs the S2 operation, which uses an automatic pixel-level calibration algorithm to accurately calibrate the extrinsic parameter matrix for the collected construction scene point cloud and images, and realizes real-time visualization of the recorded and registered color point cloud.

[0049] The open world recognition module 330 performs the S3 operation, segmenting specific objects in the construction scene based on the real-time captured color point cloud instances using the input open world text prompts.

[0050] The 3D scene mapping module 340 executes the S4 operation to accurately map the segmented object region to the 3D point cloud through the extrinsic parameter matrix to achieve accurate analysis of the 3D space.

[0051] This embodiment also illustrates an edge computing device, such as... Figure 4 As shown, device 400 includes a processor 410, a readable storage medium 420, and a communication interface 430. Device 400 can execute the aforementioned method for rapid reconstruction of open-world 3D construction scenes based on dual-modal data of video and point clouds.

[0052] The processor 410 includes an NVIDIA Jetson Orin Nano system-on-a-module (SoM) that integrates an ARM Cortex-A78AE CPU and an Ampere GPU, providing up to 40 TOPS of AI computing power and supporting TensorRT, CUDA, and OpenCV acceleration libraries. It also features 8 GB of onboard LPDDR5 memory for caching point clouds, images, and intermediate features. The processor 410 can function as a single processing unit or be expanded into a multi-node cluster via NVLink or Ethernet to perform various tasks in parallel, such as LiDAR point cloud preprocessing, industrial camera image inference, and security risk assessment.

[0053] The readable storage medium 420 can be any medium that contains, stores, transmits, propagates, or transfers instructions. It includes a 256 GB NVMe SSD (PCIe 3.0 x4) and a 64 GB eMMC 5.1 flash memory, used for storing historical video recordings, GroundingDINO 1.5 Edge weight files, and Docker images, respectively; an M.2 2242 slot is also provided for quick on-site replacement of larger capacity SSDs. The storage medium 420 can also be a vibration-resistant industrial-grade SD card or a SATA DOM to adapt to high-dust, high-vibration environments.

[0054] The communication interface 430 supports Gigabit Ethernet, 802.11ac Wi-Fi, and LTE / 5G daughter cards for remotely pushing alarm JSON to the monitoring platform via the MQTT(S) protocol. It also provides 4×USB 3.2 Gen1 ports, 1×HDMI 2.0 port, and 2×RS-485 ports, allowing connection to LiDAR, industrial cameras, audible and visual alarm lights, and PLCs, enabling heterogeneous access for multiple sensors. The interface driver layer incorporates a TSN (Time-Sensitive Networking) protocol stack, ensuring that the time error between point cloud and image frames on the transmission link is <1 ms, providing a guarantee for subsequent pixel-level fusion.

[0055] The edge computing unit used in this embodiment is an NVIDIA Jetson Orin Nano 8GB, running Ubuntu 22.04 with a kernel version of Linux 6.2.0-jetson. It supports ROS2, OpenCV, TensorRT, and PyTorch, and is started as a systemd service, supporting breakpoint resume and logging.

[0056] The computer program 421 can be configured to have one or more program modules, such as 421A acquisition module, 421B calibration module, 421C open-world recognition module, and 421D mapping module. When these program modules are executed by the processor 410, the device 400 completes the entire process of lidar-industrial camera time synchronization, automatic extrinsic parameter calibration, open-world text instance segmentation and reasoning, and semantic point cloud mapping. The module division and number are not fixed; those skilled in the art can add, delete, or merge functional modules according to on-site needs, as long as the functionality can still be achieved. Figure 1 and Figure 2 The methods and processes shown are all within the protection scope of this invention.

[0057] Therefore, this invention adopts the above-mentioned open-world construction scene reconstruction method based on dual-modal data of video and point cloud. By fusing the precise depth information of LiDAR point cloud with the rich texture information of industrial camera images, it effectively compensates for the information loss problem of single-modal data in complex construction scenes, significantly improving the accuracy and completeness of 3D construction scene reconstruction, and can more realistically and accurately restore the 3D structure and details of the scene. It introduces open-world real-time object recognition technology, which can segment objects in real-time color point cloud based on input text prompts, and accurately map the segmentation results to 3D point cloud with the help of calibrated extrinsic matrix to achieve spatial analysis. It realizes full-process automation from data acquisition, object recognition to 3D reconstruction, greatly improving the intelligence level of construction scene reconstruction, and can adapt to the dynamically changing environment and new objects in the open world. At the same time, by optimizing the data acquisition and processing process and combining the high-efficiency computing power of edge computing devices, it achieves low-latency real-time reconstruction and 4D color point cloud visualization, meeting the needs of construction sites for real-time monitoring and rapid response. In addition, the system software architecture has good scalability, which facilitates the addition of new functional modules or optimization of existing modules to adapt to the ever-changing application needs and technological developments.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An open-world construction scene reconstruction method based on dual-modal data of video and point cloud, characterized in that, Includes the following steps: S1. Simultaneous acquisition of dual-modal data: 3D point cloud and 2D color RGB image, using a LiDAR and industrial camera, anchored in a single viewpoint. The dual-modal data acquisition achieves microsecond-level time synchronization via the PTP protocol, with an acquisition frequency of 10Hz. The time deviation between the first and last frames recorded is less than 1ms, and frame alignment is performed using a 33ms time window. The specific steps include: S11. Ensure that the point cloud data collected by the lidar corresponds in time with the RGB image data captured by the industrial camera by recording and matching the timestamps of the two. S12. Control the lidar and industrial camera to start and stop data acquisition simultaneously to ensure the synchronization of data acquisition; S13. Set the LiDAR and industrial camera to collect data at the same time interval to ensure that the point cloud and RGB image of each frame are consistent in time. S14. Achieve a one-to-one correspondence between point cloud data and RGB image data within the same frame, providing accurate data pairs for subsequent dual-modal data registration; S15. Using the data collected synchronously above, prepare a benchmark dataset for dual-modal data registration to support high-precision point cloud and image fusion. S2. Feature extraction is performed on the collected construction scene point clouds and images. An automatic pixel-level calibration algorithm is used to accurately calibrate the extrinsic parameter matrix, enabling real-time visualization of the recorded and registered color point clouds. This includes the following steps: S21. Use a calibration board to take pictures to obtain the intrinsic parameter matrix of the industrial camera for calibration of the construction scene; S22. Apply an automatic pixel-level calibration algorithm to extract edge feature information and calibrate the extrinsic parameter matrix of the collected construction scene dataset; S23. Apply the calibrated external parameter matrix and combine it with the timestamp matching in S1 on site to merge the recorded LiDAR point cloud and the RGB image recorded by the industrial camera in the same frame to realize real-time four-dimensional color point cloud visualization of the recorded data on site. The four-dimensional specifically refers to three-dimensional coordinates plus one-dimensional color information. S3. Segmenting specific objects in a construction scene based on real-time captured color point cloud instances using input open-world text prompts; the instance segmentation employs an open-set object detection and segmentation model optimized for edge devices, using EfficientViT-L1 as the image backbone for fast multi-scale feature extraction; specifically including the following steps: S31. Utilize prompt words in open-source text sets to construct a semantic understanding model based on an advanced semantic understanding architecture; to achieve efficient semantic parsing and understanding of text data related to construction scenarios; the semantic understanding architecture is used to process open-world text data, including but not limited to the Transformer architecture, the BERT autoencoder semantic understanding model, the semantic understanding model based on graph neural networks, and the semantic understanding model based on knowledge graphs; S32. Input open-world text data related to the construction scenario into the semantic understanding model; S33. Based on the collected low-latency real-time four-dimensional color point cloud data, the input open-world text prompts are applied in real time to perform object instance segmentation on each frame of the image. S4. The segmented object region is accurately mapped to the 3D point cloud through the extrinsic parameter matrix to achieve accurate 3D spatial analysis. Specifically, using the calibrated extrinsic parameter matrix and 4D color point cloud data, the image results of the segmented construction scene object instance are mapped to the color point cloud of the corresponding frame in real time. Among them, each pixel in the segmentation mask is back-projected to the spatial ray, transformed to the lidar coordinate system, matched with the nearest point in the color point cloud of the corresponding frame, and assigned a semantic label.

2. The method for reconstructing open-world construction scenes based on dual-modal data of video and point cloud as described in claim 1, characterized in that, In S2, feature extraction includes the following steps: S201. Extract the point cloud data and RGB image data of the construction scene recorded in S1; S202. Downsample the collected raw point cloud data to reduce the amount of data and improve the efficiency of subsequent processing, while retaining key geometric features. S203. Apply statistical methods to identify and remove outliers in the point cloud. The noise reduction process helps to improve the quality of point cloud data. S204. Determine the calibration range of the point cloud data, i.e. the region of interest, and focus the analysis while eliminating interference from irrelevant data.

3. An open-world construction scene reconstruction system based on dual-modal data of video and point cloud for implementing the method of any one of claims 1-2, characterized in that, include: Assembly and acquisition module: Simultaneously acquire dual-modal data of 3D point cloud and 2D color RGB image within the construction scene by assembling a lidar and an industrial camera and anchoring a single viewpoint; Recording and Visualization Module: For the collected point clouds and images of the construction scene, the extrinsic parameter matrix is ​​accurately calibrated using an automatic pixel-level calibration algorithm, and the recorded and registered color point clouds are visualized in real time. Open World Recognition Module: Segment specific objects in a construction scene based on real-time captured color point cloud instances using input open world text prompts; 3D Scene Mapping Module: Accurately maps segmented object regions to 3D point clouds using extrinsic parameter matrices to achieve precise analysis of 3D space.

4. The open-world construction scene reconstruction system based on dual-modal data of video and point cloud as described in claim 3, characterized in that, include: The lidar acquires point cloud data in real time at high frequency, providing sufficient data density for rapid acquisition; the lidar and industrial camera acquire data through a time synchronization mechanism to ensure the consistency of dual-modal data in time, providing a guarantee for high-precision fusion; the lidar is connected to edge computing devices through a standard interface and integrated into the system of this invention. The industrial camera provides high-resolution color images, capturing detailed information on-site; it is designed with a wide dynamic range to maintain image quality under different lighting conditions; it supports high frame rate acquisition, providing a smooth video stream for real-time construction scene segmentation and reconstruction; and it connects to edge computing devices through a standard interface and is integrated into the system of this invention.

5. An edge computing device for implementing the method according to any one of claims 1-2, characterized in that, include: The processor is designed for edge computing to handle complex computing tasks, including machine learning and computer vision. The memory stores a computer-executable program, which, when executed by the processor, causes the processor to execute an open-world construction scene reconstruction method based on dual-modal data of video and point cloud. The communication interface supports multiple communication protocols to achieve efficient data transmission between devices; Power management includes a power management system to adapt to energy constraints in edge computing environments; The expansion interface is used to support the connection and expansion of external devices, including LiDAR, industrial cameras, mice and keyboards.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the open-world construction scene reconstruction method based on video and point cloud dual-modal data as described in any one of claims 1-2.

Images