A Method and System for Constructing a Depth Completion Dataset for Rail Transit Tunnel Environments

Abstract

This invention discloses a method and system for constructing a depth completion dataset for rail transit tunnel environments. The method includes: S1: Synchronizing the acquired data through a synchronous sampling module to obtain multi-source data with unified sampling times; S2: Performing 3D modeling of the tunnel environment based on the synchronized multi-source data, and generating a dense point cloud map through a point cloud registration strategy; S3: Dynamically constructing the field of view of each frame of camera image based on the current pose of the camera, and obtaining dense point clouds within the field of view based on the local dense point cloud corresponding to the current viewpoint of the camera; S4: Projecting the dense point clouds within the field of view onto the image plane to generate a long-distance dense depth map, and projecting the current frame's LiDAR point cloud onto the same image plane to generate a sparse depth map; S5: Matching image data, sparse depth maps, and dense depth maps according to timestamps to generate a dataset. This invention aims to obtain a long-distance depth completion dataset for rail transit tunnel environments.

Description

Technical Field

[0001] This invention relates to the field of computer vision and data processing technology, specifically to a method and system for constructing a depth completion dataset for rail transit tunnel environments. Background Technology

[0002] Depth map datasets are the data foundation for training neural network models for depth completion or depth estimation. Currently, there are many publicly available and typical depth map datasets, such as KITTI, NYUv2, and ScanNet. These datasets provide a wealth of paired image and depth information, which not only strongly supports the development of various depth estimation and completion algorithms but also provides reliable evidence for algorithm performance evaluation, greatly promoting research and applications in related fields.

[0003] High-quality depth completion datasets typically rely on meticulously designed multi-source data acquisition devices and dense depth map construction methods. However, current depth map dataset acquisition devices and construction methods are mostly designed for close-range open environments such as highways or indoor spaces, making it difficult to adapt to the complex environmental conditions of long-distance tunnel scenarios. Long-distance tunnel scenarios have a series of unique characteristics. For example, in practical applications of rail transit, due to factors such as limited sensor installation locations and data transmission delays, the synchronous acquisition of image and point cloud data faces significant challenges, making it difficult to ensure consistency in sampling time between the two. In addition, in long-distance target recognition in rail transit, data-driven depth completion or estimation models rely on depth map datasets from even more distant locations. Furthermore, in long-distance curved road scenarios, when point cloud data is converted into dense depth maps, the phenomenon of background point clouds penetrating foreground walls occurs, causing abrupt changes in the distance between adjacent point clouds, resulting in depth map anomalies.

[0004] The aforementioned characteristics of tunnel scenarios make it difficult to obtain high-quality long-distance depth completion datasets, thus limiting the training and application of depth completion and estimation algorithms in rail transit tunnel environments. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for constructing a depth completion dataset for rail transit tunnel environments, in view of the above-mentioned problems in the prior art. The present invention aims to obtain a long-distance depth completion dataset for rail transit tunnel environments.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for constructing a depth completion dataset for rail transit tunnel environments includes the following steps:

[0008] Step S01: The inertial data collected by the inertial measurement unit, the image data acquired by the camera, and the point cloud data collected by the lidar are processed using the synchronous sampling module to obtain multi-source data with unified sampling time.

[0009] Step S02: Based on the synchronized point cloud data and inertial data, perform 3D modeling of the tunnel interior environment, and generate a globally consistent dense point cloud map through point cloud registration strategy;

[0010] Step S03: Dynamically construct the field of view of each frame of camera image according to the current pose of the camera, and obtain the dense point cloud within the field of view based on the local dense point cloud map corresponding to the current view of the camera. The field of view is represented in the form of a view frustum. By judging the observability of the point cloud within the view frustum, the point cloud is divided into point cloud within the field of view and point cloud outside the field of view.

[0011] Step S04: Project the dense point cloud within the visible field of view onto the image plane to generate a distant dense depth map, and project the current frame LiDAR point cloud onto the same image plane to generate a sparse depth map;

[0012] Step S05: Match image data, sparse depth map and dense depth map according to timestamp to generate a ternary dataset containing image, sparse depth map and dense ground truth depth map.

[0013] Optionally, after the microcontroller of the synchronous sampling module receives the time synchronization from the main controller, it parses the point cloud frame timestamp through the radar data parsing module. The trigger signal modulation module calculates the modulation signal based on the point cloud frame timestamp. After receiving the modulation signal, the microcontroller sends a trigger signal to control the camera data acquisition. At the same time as sending the trigger signal, it records the current timestamp and appends the timestamp to the received point cloud data, image data, and inertial data to generate time-stamped point cloud data, image data, and inertial data. The time-stamped point cloud data, image data, and inertial data are then input into the sampling timestamp verification module for consistency verification to obtain time-aligned point cloud data, image data, and inertial data.

[0014] Optionally, the trigger signal modulation module calculates the modulation signal based on the point cloud frame timestamp, including: after the lidar receives the data acquisition signal, it performs a scanning operation to generate raw point cloud data; the lidar data parsing module obtains the acquisition timestamp of the raw point cloud frame and the timestamp of the raw point cloud frame received after a certain transmission delay by parsing the raw point cloud data; the trigger signal modulation module calculates the acquisition timestamp of the next period point cloud frame based on the fixed lidar sampling period, the acquisition timestamp of the raw point cloud frame, and the timestamp of the received raw point cloud frame; the trigger signal modulation module modulates the signal based on the acquisition timestamp of the next period point cloud frame; and the microcontroller sends a trigger signal to control camera sampling at the acquisition timestamp of the next period point cloud frame based on the modulation signal, thereby dynamically realizing the synchronous acquisition of point cloud data and camera data.

[0015] Optionally, in step S02, the process of constructing a dense point cloud map includes:

[0016] Step S201: Associate an initial pose with each frame of point cloud data and calculate the horizontal distance between adjacent frames. Based on the horizontal distance threshold, dynamically select a set of forward-backward frames that are adjacent to the current key frame from the continuous point cloud frames. The key frame is the point cloud frame that is closest to the timestamp of each image.

[0017] Step S202: Merge keyframes and neighboring frames of keyframes to form local point cloud blocks covering the continuous structure of the tunnel and generate a local map;

[0018] Step S203: Based on the initial pose, align multiple frames of point clouds in the local map using a point cloud matching algorithm to obtain an optimized local map. Then, fuse the optimized local maps frame by frame to obtain a globally consistent tunnel dense point cloud map.

[0019] Optionally, in step S03, the method for performing line-of-sight analysis on the visible field of the point cloud includes:

[0020] Step S301: Calculate the view frustum parameters corresponding to the camera based on the camera extrinsic parameters to form the spatial viewable field model of the current point cloud frame;

[0021] Step S302: Transform the local dense point cloud corresponding to the current view of the camera to the camera coordinate system, and divide the view frustum space grid according to the vertical angular resolution and the horizontal angular resolution;

[0022] Step S303: Starting from the camera's optical center, traverse the local dense point cloud data within the camera's view frustum along the rays of each grid direction, marking the foreground visible point cloud and the background invisible point cloud.

[0023] Optionally, in step S303, the foreground visible point cloud is the point cloud point closest to the camera optical center in each grid space, and the remaining point cloud points are background invisible point clouds.

[0024] Optionally, in step S04, the method for generating a distant dense depth map involves projecting the point cloud within the visible field of view onto the image plane using camera extrinsic and intrinsic parameter matrices, assigning a depth value to each pixel using the nearest neighbor fill method, and prohibiting interpolation and filtering, thereby obtaining a continuous dense depth map that is pixel-level aligned with the original camera image.

[0025] Optionally, it also includes culling of point clouds outside the visible field of view to eliminate the anomaly of distant background point cloud penetration and improve the quality of dense depth maps.

[0026] This invention also provides a system for constructing a depth completion dataset for rail transit tunnel environments, including...

[0027] The synchronous sampling module is used to process the inertial data collected by the inertial measurement unit, the image data acquired by the camera, and the point cloud data collected by the lidar to obtain multi-source data with uniform sampling time.

[0028] The point cloud mapping module is used to perform three-dimensional modeling of the tunnel interior environment based on synchronized point cloud data and inertial data, and generate a globally consistent dense point cloud map through point cloud registration strategy.

[0029] The visible field of view analysis module is used to dynamically construct the visible field of view range of each frame of camera image according to the current pose of the camera, and obtain the dense point cloud within the visible field of view based on the local dense point cloud map corresponding to the current view of the camera. The visible field of view is represented in the form of a view frustum. By judging the observability of the point cloud within the view frustum, the point cloud is divided into point cloud within the visible field of view and point cloud outside the visible field of view.

[0030] The dense depth map conversion module is used to project dense point clouds within the visible field of view onto the image plane to generate a long-distance dense depth map;

[0031] The sparse depth map conversion module is used to project the current frame of LiDAR point cloud onto the same image plane to generate a sparse depth map;

[0032] The depth completion dataset generation module is used to match image data, sparse depth maps, and dense depth maps based on timestamps to generate a ternary dataset containing images, sparse depth maps, and dense ground truth depth maps.

[0033] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute, via a processor, the method for constructing a depth completion dataset for a rail transit tunnel environment.

[0034] Compared with the prior art, the present invention can mainly achieve the following beneficial effects:

[0035] 1. This invention integrates inertial data collected by the inertial measurement unit, image data acquired by the camera, and point cloud data acquired by the lidar through a synchronous sampling module, ensuring the consistency of multi-source data at the sampling time and achieving millisecond-level high-precision spatiotemporal synchronization of point clouds and images. Using the synchronized inertial and point cloud data, a 3D model of the tunnel interior is created. A point cloud registration strategy is employed to generate a globally consistent dense point cloud map. The visible field of view of the camera image is dynamically constructed based on the camera's current attitude, obtaining a local dense point cloud map corresponding to the camera's current viewpoint. The point cloud within and outside the visible field of view is divided into points using a view frustum. The dense point cloud within the visible field of view is projected onto the image plane to generate a long-distance dense depth map. Simultaneously, the sparse point cloud acquired by the lidar in real time is projected onto the same image plane to generate a sparse depth map. Through timestamp matching, the image data, sparse depth map, and dense depth map are integrated into a ternary dataset.

[0036] 2. This invention improves the sensing distance and accuracy of dense depth maps of rail transit tunnel environments, provides high-quality data support for the training and evaluation of long-distance depth completion models, and promotes the development of related technologies in the field of long-distance tunnel environment depth completion or estimation.

[0037] 3. This invention uses the origin of the lidar coordinate system as the observation point for visual field analysis, separates foreground point clouds and background point clouds, and effectively removes invisible background point clouds in tunnel and curve scenes, thus avoiding anomalies in dense depth maps. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method for constructing a depth completion dataset in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of the data time synchronization acquisition module in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of the visible field of view analysis in an embodiment of the present invention.

[0041] Figure 4 This is a diagram illustrating the effect of removing point clouds outside the visible field of view in an embodiment of the present invention.

[0042] Figure 5 This is a schematic diagram illustrating an example of a depth dataset in an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0044] like Figure 1As shown, this embodiment provides a method for constructing a depth completion dataset for rail transit tunnel environments, including the following steps:

[0045] Step S01: The inertial data collected by the inertial measurement unit, the image data acquired by the camera, and the point cloud data collected by the lidar are processed using the synchronous sampling module to obtain multi-source data with unified sampling time.

[0046] Step S02: Based on the synchronized point cloud data and inertial data, perform 3D modeling of the tunnel interior environment, and generate a globally consistent dense point cloud map through point cloud registration strategy;

[0047] Step S03: Dynamically construct the field of view of each frame of camera image according to the current pose of the camera, and obtain the dense point cloud within the field of view based on the local dense point cloud map corresponding to the current view of the camera. The field of view is represented in the form of a view frustum. By judging the observability of the point cloud within the view frustum, the point cloud is divided into point cloud within the field of view and point cloud outside the field of view.

[0048] Step S04: Project the dense point cloud within the visible field of view onto the image plane to generate a distant dense depth map, and simultaneously project the current frame LiDAR point cloud onto the same image plane to generate a sparse depth map;

[0049] Step S05: Match image data, sparse depth map and dense depth map according to timestamp to generate a ternary dataset containing image, sparse depth map and dense ground truth depth map.

[0050] Depth completion technology, in particular, is a key technique that uses neural network models to predict and fill in the depth of missing regions based on existing sparse depth information, combined with image data or prior scene knowledge, thereby generating a complete and dense depth map. This technology is widely used in fields such as autonomous driving, environmental perception, 3D reconstruction, and augmented reality, and holds an important position in computer vision.

[0051] In this embodiment, after the microcontroller of the synchronous sampling module receives the time synchronization from the main controller, it parses the point cloud frame timestamp through the radar data parsing module. The trigger signal modulation module calculates the modulation signal based on the point cloud frame timestamp. After receiving the modulation signal, the microcontroller sends a trigger signal to control the camera data acquisition. At the same time as sending the trigger signal, it records the current timestamp and appends the timestamp to the received point cloud data, image data, and inertial data to generate time-stamped point cloud data, image data, and inertial data. The time-stamped point cloud data, image data, and inertial data are then input into the sampling timestamp verification module for consistency verification to obtain time-aligned point cloud data, image data, and inertial data.

[0052] This embodiment achieves time synchronization of data from three types of sensors—Inertial Measurement Unit (IMU, a sensor device integrating an accelerometer and gyroscope, capable of measuring the acceleration and angular velocity of an object in real time)—through a synchronous acquisition module. By fusing data with other sensors, the IMU can effectively improve the robustness and accuracy of the system, especially in situations where image information is unreliable, such as low light or missing features; IMU data can serve as an important supplement, enhancing system stability. This ensures the consistency of sampling times for multi-source data. Figure 2 As shown, the synchronous acquisition module includes:

[0053] The main controller is responsible for functions such as time synchronization, driving multi-sensor data acquisition, and storing acquired data on disk. It uses the NTP time synchronization protocol to synchronize the MCU (Microcontroller Unit, also known as a single-chip microcomputer or microcontroller, is a miniature computing device integrating a processor core, memory, and peripheral interfaces, widely used in embedded system design) time system. It's important to understand that the time synchronization protocol is not limited to NTP; it can also be PTP or other time synchronization protocols.

[0054] LiDAR (LiDAR) is used to acquire sparse point cloud data and trigger the transmission of modulated signals. The LiDAR sampling frequency is 10Hz, and each frame of point cloud data is assigned a precise timestamp. The clock source for the timestamp is the MCU (Microcontroller Unit). Understandably, the choice of LiDAR sampling frequency can be made based on the specific application requirements.

[0055] The camera is used to acquire image data. Upon receiving a modulation signal from the LiDAR driver, the camera triggers exposure sampling to ensure that the exposure time of the image and point cloud frames is consistent, and the camera's sampling frequency is 10Hz, the same as the LiDAR's sampling frequency. The clock source for the timestamp is the MCU.

[0056] The inertial measurement unit (IMU) is used to acquire inertial attitude data of the device. The IMU samples at 1000Hz, providing high-frequency, high-precision attitude information, and adds a high-precision timestamp to each data frame. The timestamp clock source is the MCU.

[0057] The sampling timestamp verification module is used to verify the consistency of the sampling timestamps of the collected point cloud data, image data, and inertial data. Data that is time-aligned and passes the integrity verification is written to disk and stored on the main controller's hard drive.

[0058] In this embodiment, the trigger signal modulation module calculates the modulation signal based on the point cloud frame timestamp, including: after the lidar receives the data acquisition signal, it performs a scanning operation to generate raw point cloud data; the lidar data parsing module obtains the acquisition timestamp T1 of the raw point cloud frame by parsing the raw point cloud data, and the timestamp T2 of the raw point cloud frame received by the lidar data parsing module after a certain transmission delay; the trigger signal modulation module calculates the acquisition timestamp T3 of the next period point cloud frame based on the fixed lidar sampling period T0, using the acquisition timestamp T1 of the raw point cloud frame and the timestamp T2 of the received raw point cloud frame, with the calculation formula being: T3=T0+T1-T2; the trigger signal modulation module modulates the signal based on the acquisition timestamp T3 of the next period point cloud frame; and the microcontroller unit sends a trigger signal to control camera sampling at the acquisition timestamp T3 of the next period point cloud frame based on the modulation signal, thereby dynamically realizing the synchronous acquisition of point cloud data and camera data.

[0059] In step S02 of this embodiment, the process of constructing a dense point cloud map includes:

[0060] Step S201: Associate an initial pose with each frame of point cloud data and calculate the horizontal distance between adjacent frames. Based on the horizontal distance threshold, dynamically select a set of forward-backward frames that are adjacent to the current key frame from the continuous point cloud frames. The key frame is the point cloud frame that is closest to the timestamp of each image.

[0061] Step S202: Merge keyframes and neighboring frames of keyframes to form local point cloud blocks covering the continuous structure of the tunnel and generate a local map;

[0062] Step S203: Based on the initial pose, align multiple frames of point clouds in the local map using a point cloud matching algorithm to obtain an optimized local map. Then, fuse the optimized local maps frame by frame to obtain a globally consistent tunnel dense point cloud map.

[0063] This embodiment uses time-synchronized IMU and LiDAR data to create a 3D map of the tunnel's interior environment through a point cloud mapping module. This module utilizes inertial measurement data to assist in point cloud registration strategies, achieving long-distance, highly stable 3D spatial construction of the tunnel based on low-frequency point clouds. The final output is a globally consistent dense point cloud, providing a structural basis for line-of-sight analysis and depth map conversion.

[0064] In specific application embodiments, constructing a globally consistent dense point cloud map includes:

[0065] Step 1: Assume that each frame of point cloud corresponds to a pose. The pose is determined by the translation vector. and rotation matrix By estimating the pose of each frame of the point cloud, the position and orientation of the lidar in space can be determined.

[0066] Step 2: Laser odometry displacement from Frame calculation, defining the horizontal distance between two frames. for:

[0067] (1)

[0068] Step 3: Set the current keyframe as ;

[0069] Step 4: From the current keyframe Begin by searching forward for a set of frames. The frame set satisfies the following conditions:

[0070] (2)

[0071] From the current keyframe Begin by searching for a set of frames backward. The frame set satisfies the following conditions:

[0072] (3)

[0073] Step 5: Merge keyframes and their neighboring frames to form local point cloud blocks covering the continuous structure of the tunnel, generating a local map. The final set of mapping frames is as follows:

[0074] (4)

[0075] In step S03 of this embodiment, the method for performing line-of-sight analysis on the visible field of the point cloud includes:

[0076] Step S301: Calculate the view frustum parameters corresponding to the camera based on the camera extrinsic parameters to form a spatial viewable model of the current camera image;

[0077] Step S302: Transform the local dense point cloud corresponding to the current view of the camera to the camera coordinate system, and divide the view frustum space grid according to the vertical angular resolution and the horizontal angular resolution;

[0078] Step S303: Starting from the optical center of the camera, traverse the point cloud data within the camera's view frustum along the rays of each grid direction, and mark the foreground visible point cloud and the background invisible point cloud.

[0079] In specific application embodiments, the field of view of each frame of camera image is dynamically constructed based on the camera's current pose, and dense point clouds within the field of view are obtained based on the local dense point cloud map corresponding to the camera's current viewpoint. The field of view is represented in the form of a view frustum. By determining the observability of the point cloud within the view frustum, the point cloud is divided into point clouds within the field of view and point clouds outside the field of view, such as... Figure 3As shown. The line-of-sight analysis specifically includes the following steps:

[0080] Step 1: Based on the camera extrinsic parameters and the pose information of the current frame, calculate the corresponding frustum parameters of the camera (including the field of view, near plane, far plane, etc.) to form a spatial visible field model of the current frame;

[0081] Step 2: Convert the local dense point cloud map to the current frame's camera coordinate system, and adjust it according to the vertical angular resolution. With horizontal angular resolution The view frustum is divided into multiple smaller grid spaces. ;

[0082] Step 3: Starting from the camera's optical center, traverse the point cloud within the view frustum along rays from each grid direction. For each grid space... If multiple points exist, the closest point is selected as the foreground visible point cloud, and the remaining points are selected as the background invisible point cloud. Finally, the point clouds inside and outside the visible field of view are marked based on the interaction between each point cloud and the ray.

[0083] like Figure 4 (a) Figure 4 As shown in (b), this embodiment also includes a process for removing point clouds outside the visible field of view to eliminate the abnormal penetration of distant background point clouds and improve the quality of dense depth maps.

[0084] In step S04 of this embodiment, the method for generating a distant dense depth map is to project the point cloud within the visible field of view onto the image plane through the camera's extrinsic and intrinsic parameter matrices, assign a depth value to each pixel using the nearest neighbor fill method, and prohibit interpolation and filtering processing, thereby obtaining a continuous dense depth map that is aligned with the original camera image at the pixel level.

[0085] Specifically, based on camera extrinsic and intrinsic parameter information, a local 3D point cloud map matching the image frame is projected onto the image plane to form a long-distance dense depth map. The specific steps are as follows:

[0086] Step 1: Let the 3D point be in the world coordinate system as Transformed to the camera coordinate system using an extrinsic parameter matrix, 3D points The process of projecting onto a two-dimensional image plane can be expressed as follows:

[0087] (5)

[0088] in, , , These are the row and column numbers after projection onto a two-dimensional plane. It is a scale factor. , These are the focal lengths in the horizontal and vertical directions, respectively. , Principal point coordinates , These are the external parameter rotation matrix and translation vector, respectively;

[0089] Step 2: For each pixel coordinate, fill the depth value using the nearest neighbor method, without using interpolation or filtering algorithms;

[0090] Step 3: The final output is a dense depth map that is strictly aligned with the original camera image at the pixel level and has continuous depth values.

[0091] In this embodiment, the same method described above for projecting a local 3D point cloud map matching the image frame onto the image plane to form a distant dense depth map is used to project sparse point cloud frame data matching the image frame onto the image plane to form a sparse depth map.

[0092] like Figure 5 (a) Figure 5 (b) Figure 5 As shown in (c), this embodiment matches RGB images, sparse depth maps, and dense depth maps into pairs based on the timestamp nearest neighbor principle. Through the collaborative work of the above modules, a ternary dataset containing images, sparse depth maps, and dense ground truth depth maps can be automatically generated. This achieves the construction of high-quality tunnel scene depth completion samples and has strong scalability and engineering application value.

[0093] Compared to existing highway environment depth map dataset construction techniques, this embodiment enables the construction of long-distance depth map datasets in rail transit tunnel environments to support long-distance depth completion or depth estimation model training for urban rail transit. Furthermore, compared to direct dense depth map conversion, the introduction of view-through analysis can address the anomaly problem of long-distance background point cloud penetration, generating high-quality dense depth maps.

[0094] This embodiment combines high-frequency, high-precision IMU measurement data and uses precise constraints based on degradation features to achieve long-distance point cloud mapping within the tunnel under low-speed operation of the system. By generating a long-distance dense depth map through projection transformation, it realizes the construction of a continuous dense point cloud map for subway tunnel scenes of more than 10km.

[0095] This embodiment also provides a system for constructing a depth completion dataset for rail transit tunnel environments, including...

[0096] The synchronous sampling module is used to process the inertial data collected by the inertial measurement unit, the image data acquired by the camera, and the point cloud data collected by the lidar to obtain multi-source data with uniform sampling time.

[0097] The point cloud mapping module is used to perform three-dimensional modeling of the tunnel interior environment based on synchronized point cloud data and inertial data, and generate a globally consistent dense point cloud map through point cloud registration strategy.

[0098] The visible field of view analysis module is used to dynamically construct the visible field of view range of each frame of camera image according to the current pose of the camera, and obtain the dense point cloud within the visible field of view based on the local dense point cloud map corresponding to the current view of the camera. The visible field of view is represented in the form of a view frustum. By judging the observability of the point cloud within the view frustum, the point cloud is divided into point cloud within the visible field of view and point cloud outside the visible field of view.

[0099] The dense depth map conversion module is used to project dense point clouds within the visible field of view onto the image plane to generate a long-distance dense depth map;

[0100] The sparse depth map conversion module is used to project the current frame of LiDAR point cloud onto the same image plane to generate a sparse depth map;

[0101] The depth completion dataset generation module is used to match image data, sparse depth maps, and dense depth maps based on timestamps to generate a ternary dataset containing images, sparse depth maps, and dense ground truth depth maps.

[0102] This embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to be executed by a processor as a method for constructing a depth completion dataset for a rail transit tunnel environment.

[0103] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a depth completion dataset for a rail transit tunnel environment, characterized in that, The method comprises the following steps: Step S01: processing the inertial data collected by the inertial measurement unit, the image data acquired by the camera and the point cloud data collected by the laser radar by using a synchronous sampling module to obtain multi-source data unified at the sampling time; Step S02: according to the synchronized point cloud data and the inertial data, performing three-dimensional modeling on the internal environment of the tunnel, and generating a globally consistent dense point cloud map through a point cloud registration strategy; Step S03: dynamically constructing a visible domain range of each frame of camera image according to the current attitude of the camera, and obtaining dense point clouds in the visible domain based on the local dense point cloud map corresponding to the current view angle of the camera, wherein the visible domain is in the form of a view frustum, and the point clouds are divided into point clouds in the visible domain and point clouds outside the visible domain by judging the observability of the point clouds in the view frustum; Step S04: projecting the dense point clouds in the visible domain to an image plane to generate a long-distance dense depth map, and projecting the current frame of laser radar point clouds to the same image plane to generate a sparse depth map; Step S05: matching the image data, the sparse depth map and the dense depth map according to the time stamp to generate a ternary data set containing the image, the sparse depth map and the dense depth map.

2. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 1, characterized in that, After the micro control unit of the synchronous sampling module receives the time service of the main controller, the point cloud frame time stamp is analyzed through the radar data analysis module, the trigger signal modulation module calculates the modulation signal according to the point cloud frame time stamp, after the micro control unit receives the modulation signal, the trigger signal is sent to control the camera data acquisition, and at the same time the current time stamp is recorded, the time stamp is attached to the received point cloud data, image data and inertial data, and the point cloud data, image data and inertial data with time stamp are generated, the point cloud data, image data and inertial data with time stamp are input into the sampling time stamp verification module for consistency verification, and the time aligned point cloud data, image data and inertial data are obtained.

3. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 2, characterized in that, The trigger signal modulation module calculates the modulation signal according to the point cloud frame time stamp, which comprises: after the laser radar receives the data acquisition signal, the original point cloud data is generated through scanning operation, the radar data analysis module analyzes the original point cloud data to obtain the acquisition time stamp of the original point cloud frame, and the time stamp of the original point cloud frame is received after a certain transmission delay, the trigger signal modulation module calculates the acquisition time stamp of the next cycle point cloud frame according to the fixed laser radar sampling period, the acquisition time stamp of the original point cloud frame and the time stamp of the original point cloud frame received, the trigger signal modulation module modulates the signal according to the acquisition time stamp of the next cycle point cloud frame, and the micro control unit sends the trigger signal for controlling the camera sampling at the acquisition time stamp of the next cycle point cloud frame according to the modulation signal, so as to dynamically realize the synchronous acquisition of the point cloud data and the camera data.

4. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 1, characterized in that, In step S02, the process of constructing the dense point cloud map comprises: Step S201: associating an initial pose with each frame of point cloud data, calculating the horizontal distance between adjacent frames, dynamically selecting a forward-backward frame set adjacent to the current key frame from the continuous point cloud frames based on the horizontal distance threshold, and the key frame is the point cloud frame closest to the time stamp of each image; Step S202: merge the key frame and the adjacent frames of the key frame to form a local point cloud block covering the continuous structure of the tunnel and generate a local map; Step S203: align the multiple frames of point cloud in the local map by a point cloud matching algorithm according to the initial pose to obtain an optimized local map, and fuse the optimized local map frame by frame to obtain a globally consistent dense point cloud map of the tunnel.

5. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 1, characterized in that, In the step S03, the method for visibility analysis of the point cloud viewable domain comprises: Step S301: calculate the parameters of the view cone corresponding to the camera according to the camera external parameters to form a spatial viewable domain model of the current camera image; Step S302: convert the local dense point cloud corresponding to the current view angle of the camera to the camera coordinate system, and divide the view cone space grid according to the vertical angle resolution and the horizontal angle resolution; Step S303: from the camera optical center, traverse the local dense point cloud data in the camera view cone along the rays in each grid direction, and mark the foreground viewable point cloud and the background non-viewable point cloud.

6. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 5, characterized in that, In the step S303, the foreground viewable point cloud is the point cloud point closest to the camera optical center in each grid space, and the remaining point cloud points are the background non-viewable point cloud.

7. The method for constructing a deep completion dataset for rail transit tunnel environment according to claim 1, characterized in that, In the step S04, the generation method of the long-distance dense depth map is that the point cloud in the viewable domain is projected to the image plane through the camera external parameters and internal parameters matrix, each pixel is assigned a depth value through the nearest neighbor filling method, and interpolation and filtering processing is prohibited, so that a continuous dense depth map is obtained which is pixel-level aligned with the original camera image.

8. The method for constructing a deep completion dataset for rail transit tunnel environment according to any one of claims 1-7, characterized in that, It also includes a rejection processing of the point cloud outside the viewable domain to eliminate the penetration anomaly of the long-distance background point cloud and improve the quality of the dense depth map.

9. A system for constructing a depth completion dataset for rail transit tunnel environments, characterized in that, It comprises: a synchronous sampling module for processing the inertial data collected by the inertial measurement unit, the image data obtained by the camera and the point cloud data collected by the laser radar to obtain multiple source data unified at the sampling time; a point cloud mapping module for three-dimensional modeling of the tunnel internal environment according to the synchronized point cloud data and inertial data, and generating a globally consistent dense point cloud map through a point cloud registration strategy; a viewable domain visibility analysis module for dynamically constructing the viewable domain range of each frame of camera image according to the current pose of the camera, and obtaining the dense point cloud in the viewable domain based on the local dense point cloud map corresponding to the current view angle of the camera, wherein the viewable domain is represented in the form of a view cone, and the point cloud is divided into the point cloud in the viewable domain and the point cloud outside the viewable domain by judging the observability of the point cloud in the view cone; a dense depth map conversion module for projecting the dense point cloud in the viewable domain to the image plane to generate a long-distance dense depth map; a sparse depth map conversion module for projecting the current frame of laser radar point cloud to the same image plane to generate a sparse depth map; a depth completion dataset generation module for matching the image data, the sparse depth map and the dense depth map according to the time stamp to generate a three-element dataset containing the image, the sparse depth map and the dense depth map.

10. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the method for constructing a depth completion dataset for a rail transit tunnel environment according to any one of claims 1-8 by a processor.

Images