Method, medium and device for displaying three-dimensional point cloud frames and two-dimensional image frames in fusion
By constructing an accurate spatial pose transformation matrix and perspective projection algorithm, combined with depth-level coloring, the problems of misalignment and wasted computing resources in the fusion of LiDAR and camera data were solved, and efficient and intuitive fusion display of 3D point clouds and 2D images was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHENGDU SIWI POWER ELECTRONICS TECH
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, the extrinsic parameter calibration accuracy of lidar and cameras is insufficient, resulting in misalignment between point clouds and images. Furthermore, the large volume of lidar data and the lack of filtering lead to a waste of computing resources, making it difficult for humans to accurately judge the distance to obstacles.
By loading precise intrinsic and extrinsic parameters, a spatial pose transformation matrix for the LiDAR and camera is constructed. A perspective projection algorithm is used to map the 3D point cloud onto a 2D pixel plane. A depth-level coloring mechanism is introduced to filter effective point cloud data, and a message subscription mechanism is used to synchronize the data.
It achieves high-precision alignment and fusion of 3D point clouds and 2D images, enhances the depth perception capability of environmental awareness and system efficiency, reduces invalid rendering operations, and improves the robustness and real-time performance of the system.
Smart Images
Figure CN122134990A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, multi-sensor data fusion, and autonomous driving environmental perception technology. More specifically, it relates to a method, medium, and apparatus for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames. Background Technology
[0002] With the rapid development of autonomous driving technology and intelligent robots, multi-sensor fusion perception has become a key technology for achieving high-precision environmental understanding. Among the many sensor configurations, the combination of LiDAR and camera is the most common. LiDAR can provide high-precision three-dimensional spatial point cloud data, accurately measure distances, and is not easily affected by changes in lighting. Cameras, on the other hand, can provide high-resolution two-dimensional texture images containing rich color and semantic information. By fusing the data from these two sensors, complementary advantages can be achieved, enhancing the system's ability to perceive the environment.
[0003] However, these two types of sensors differ significantly in data format and spatial dimension: LiDAR outputs sparse, discrete three-dimensional coordinates, while cameras output a dense two-dimensional pixel matrix. In practical applications and development / debugging, the following are the main technical challenges: First, verifying multi-sensor joint calibration is difficult. The accuracy of the extrinsic parameters of the LiDAR and camera directly determines the performance of the fusion system; some methods do not fully consider the accuracy of the LiDAR and camera extrinsic parameter calibration when performing coordinate system transformation, resulting in misalignment between point cloud and image pixels; Moreover, since lidar data is massive and contains a large number of points outside the field of view, direct processing without effective filtering would waste computing resources and interfere with the display effect of the fused image.
[0004] Finally, because cameras lack depth perception capabilities, it is difficult for vehicle operators to accurately judge the distance to obstacles ahead based solely on the two-dimensional images from the camera. Although lidar contains distance information, the real-time point cloud image from the lidar is not synchronized with the camera image, making it difficult for human observers to match the point cloud with objects in the real scene.
[0005] Therefore, there is an urgent need for an accurate and intuitive method for fusing and displaying 3D point cloud frames and 2D image frames based on perspective projection. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide a method, medium, and apparatus for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames; The solution adopted by this invention to solve the technical problem is: on the one hand: A method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames specifically includes the following steps: Step S1: Initialize the fusion processing node, obtain and load the camera's intrinsic and extrinsic parameters, and establish a communication link between the fusion processing node and the camera and LiDAR. Step S2: Construct the spatial pose transformation matrix of the LiDAR coordinate system relative to the camera coordinate system; Step S3: Acquire two-dimensional image frame data and time-correlated three-dimensional point cloud frame data; Step S4: Convert the received 3D point cloud frame data into an traversable point cloud structure, traverse the discrete points in the point cloud structure; use the spatial pose transformation matrix to transform the discrete points from the lidar coordinate system to the camera coordinate system to obtain 3D discrete points. Step S5: Use the perspective projection algorithm to project the three-dimensional discrete points in the camera coordinate system onto the pixel plane to obtain undistorted normalized coordinates, and then map them to pixel coordinates to form the projected pixel points. Step S6: Filter the projected pixels and extract the pixels that fall within the effective range of the image; Step S7: Based on the depth information of the 3D point cloud frame in the camera coordinate system, generate the corresponding colored pixels according to the preset depth grading coloring rules. Step S8: Overlay colored pixels onto the two-dimensional image frame data to generate and publish the fused image frame.
[0007] In some possible implementations, in step S4, before transforming the discrete points from the lidar coordinate system to the camera coordinate system using the spatial pose transformation matrix, a validity check is performed on each discrete point to filter out unqualified discrete points.
[0008] In some possible implementations, step S2, constructing the spatial pose transformation matrix of the lidar coordinate system relative to the camera coordinate system, specifically refers to: The 3×3 rotation matrix from the lidar coordinate system to the camera coordinate system is obtained through calibration. Translation vector Construct a 4×4 homogeneous transformation matrix The formula is as follows: .
[0009] In some possible implementations, step S4 specifically includes the following steps: Step S41: Set the coordinates of discrete points in the lidar coordinate system as follows The coordinates of the corresponding point in the camera coordinate system are ; like and If the coordinate value in the graph is not numerical or is infinity, then the graph will be... and corresponding Filter out; Step S42: ... Convert to The formula is: ; Expanding the calculation as follows: .
[0010] In some possible implementations, step S5 specifically includes the following steps: Step S51: Calculate normalized coordinates ,in, , .
[0011] Step S52: Map normalized coordinates to pixel coordinates : ; in, , Focus the camera. , The coordinates of the main point.
[0012] In some possible implementations, in step S6, the criterion for selecting the projected pixels is: Let the image resolution be... , H is the image width, and H is the image height.
[0013] ; Points that do not meet any of the above conditions are considered invalid and are discarded.
[0014] In some possible implementations, step S7, the method for generating colored pixels according to a preset depth-level coloring rule, specifically includes the following steps: S71: Preset minimum display depth and maximum display depth ; S72: Obtain the depth value in the camera coordinate system corresponding to the current pixel. ; S73: Depth value Normalization to The interval is used to obtain the normalized depth value. The calculation formula is as follows: ; like Then let ; like Then let ; S74: Uses the HSV color space mapping table to... Mapped to RGB color values This is used as the drawing color for that pixel.
[0015] In some possible implementations, in step S3, the time-correlated three-dimensional point cloud frame data refers to finding the point cloud frame with the smallest absolute value of the timestamp difference with the image frame in the cached lidar point cloud frame queue when a two-dimensional image frame is received.
[0016] on the other hand: This invention provides a computer terminal storage medium storing computer terminal executable instructions, characterized in that the computer terminal executable instructions are used to execute the above-described method for fusion display of three-dimensional point cloud frames and two-dimensional image frames.
[0017] On the other hand: This invention provides a computing device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the three-dimensional point cloud frame and two-dimensional image frame fusion display method as described above.
[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention solves the spatial alignment problem by loading precisely calibrated intrinsic and extrinsic parameters and utilizing the principle of perspective projection to accurately map three-dimensional spatial points onto a two-dimensional pixel plane. This invention introduces a depth-level coloring mechanism, which displays different colors according to the distance of the point cloud from the camera, significantly enhancing the depth perception capability of the fused image. This invention filters out invalid discrete points before projection and filters out pixels outside the image range after projection, reducing invalid rendering operations and ensuring system efficiency and real-time performance. This invention acquires data through a message subscription mechanism and time correlation, effectively solving the data misalignment problem caused by time asynchrony among multiple sensors and improving the robustness of the system. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the workflow of the method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames in this invention. Detailed Implementation
[0020] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, "a" or "one," etc., do not indicate a quantity limitation, but rather indicate the existence of at least one. In the implementation of this application, "and / or" describes the association relationship of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more. For example, multiple positioning posts refer to two or more positioning posts. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0021] The present invention will now be described in detail.
[0022] Example 1: like Figure 1 As shown, this embodiment provides a method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames; This embodiment is based on a robot operating system environment and implemented using C++. The hardware platform includes a wire-controlled four-wheel motion structure chassis, a mechanical rotating LiDAR, and an RGB industrial camera, all rigidly connected. A method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames includes the following steps: Step S1: Initialize the fusion processing node, acquire and load the camera's intrinsic and extrinsic parameters, and establish a communication link between the fusion processing node and the camera and lidar. Step S2: Establish the spatial pose transformation matrix of the lidar coordinate system relative to the camera coordinate system; Step S3: Based on the message subscription mechanism, acquire two-dimensional image frame data and time-related three-dimensional point cloud frame data; Step S4: Convert the received 3D point cloud frame data into a traversable point cloud structure, traverse the discrete points in the structure, perform a validity check on each discrete point, filter out unqualified discrete points, and then use the spatial pose transformation matrix described in S2 to transform the discrete points from the lidar coordinate system to the camera coordinate system to obtain 3D discrete points. Step S5: Use the perspective projection algorithm to project the 3D discrete points in the camera coordinate system onto the pixel plane to obtain undistorted normalized coordinates, and then map them to pixel coordinates; Step S6: Filter the projected pixels and extract the pixels that fall within the effective range of the image; Step S7: Based on the depth information of the 3D point cloud frame in the camera coordinate system, generate pixels of corresponding colors according to the preset depth grading coloring rules. Step S8: Overlay the colored pixels onto the two-dimensional image frame data to generate and publish the fused image frame.
[0023] This embodiment establishes a precise spatial pose transformation matrix for LiDAR and camera, and combines a strict time correlation mechanism and perspective projection algorithm to perform validity detection, field of view selection, and depth-level coloring of 3D point clouds. This achieves high-precision alignment and visualization overlay of multi-sensor data in the spatiotemporal dimension, effectively solving the problems of misalignment and lag in data fusion in dynamic scenes, and significantly enhancing the intuitiveness, accuracy, and real-time performance of environmental perception.
[0024] In this embodiment, the process of establishing the spatial pose transformation matrix of the lidar coordinate system relative to the camera coordinate system in step S2 is as follows: The 3×3 rotation matrix from the lidar coordinate system to the camera coordinate system is obtained through calibration. Translation vector Construct a 4×4 homogeneous transformation matrix The formula is as follows: ; This embodiment achieves a geometric benchmark for multi-sensor spatial fusion by pre-calculating and constructing a homogeneous transformation matrix, ensuring that the lidar point cloud can be mathematically and accurately transformed into the camera's viewpoint.
[0025] In this embodiment, step S4 specifically includes the following steps: Step S41: Let the coordinates of the discrete point in the lidar coordinate system be... The coordinates of the corresponding point in the camera coordinate system are ; like and If the coordinates in the graph are non-numeric (NaN) or infinity (Inf), then... and corresponding Filter out; Step S42: ... Convert to The formula is: ; Expanding the calculation as follows: .
[0026] This embodiment utilizes the homogeneous transformation matrix in step S2. The point cloud is transformed from the radar coordinate system to the camera coordinate system, and abnormal data such as non-numerical values and infinity are often contained in the raw LiDAR data, avoiding crashes caused by program calculation errors and greatly improving the stability of the system.
[0027] In this embodiment, step S5 specifically includes the following steps: Step S51: Calculate normalized coordinates ,in, , .
[0028] Step S52: Map normalized coordinates to pixel coordinates : ; in, , Focus the camera. , The coordinates of the main point.
[0029] This embodiment uses the principle of perspective projection and combines it with camera intrinsic parameters to simulate the process of light entering the camera, realizing the dimensionality reduction mapping from the three-dimensional world to the two-dimensional plane, ensuring that the position of the point cloud on the image coincides with the texture features of the image itself.
[0030] In this embodiment, the lidar has a 360° field of view, while the camera's field of view is only 60°. Step S6 filters out a large number of points that fall behind the camera and outside the field of view. The criteria for filtering projected pixels are as follows: Let the image resolution be... , H is the image width, and H is the image height.
[0031] ; Points that do not meet any of the above conditions are considered invalid and are discarded.
[0032] This embodiment eliminates a large number of points that fall behind the camera and outside the field of view by filtering pixels, avoiding subsequent coloring and drawing operations on invalid points, which significantly improves rendering speed and frame rate.
[0033] In this embodiment, step S7 specifically includes the following steps: Step S71: Preset minimum display depth and maximum display depth ; Step S72: Obtain the depth value of the camera coordinate system corresponding to the current pixel. ; Step S73: Set the depth value Normalization to The interval is used to obtain the normalized depth value. The calculation formula is as follows: ; like Then let ; like Then let ; Step S74: Use the HSV color space mapping table to... Mapped to RGB color values This is used as the drawing color for that pixel.
[0034] This embodiment uses color-coded depth information to enable the human eye to identify foreground obstacles and background environment, thereby improving the efficiency of human-computer interaction and the accuracy of environmental understanding.
[0035] When in use, this invention initializes the fusion node and loads the camera's intrinsic and extrinsic parameters to construct a precise spatial pose transformation matrix from the lidar coordinate system to the camera coordinate system, laying a rigorous mathematical foundation for subsequent data unification. In the data acquisition stage, this invention innovatively adopts a time association strategy based on a message subscription mechanism. By comparing timestamps, it strictly selects the three-dimensional point cloud frame that is closest to the current two-dimensional image frame in time, thereby effectively eliminating ghosting and positional lag phenomena in dynamic scenes caused by inconsistent sensor acquisition frequencies, and ensuring high synchronization of data in the time dimension.
[0036] This invention introduces a point cloud validity detection mechanism in its processing flow. Before coordinate transformation, it automatically identifies and removes abnormal noise points containing non-numerical or infinite values, preventing program crashes or calculation errors. Subsequently, it uses a perspective projection algorithm to simulate the imaging principle of a camera, reducing the dimension of the discrete point cloud in three-dimensional space to a two-dimensional pixel plane. A field-of-view filtering mechanism then removes redundant points falling outside the effective image range, significantly reducing the computational overhead of subsequent rendering stages.
[0037] To enhance the intuitiveness of human-computer interaction and environmental perception, this invention further proposes an adaptive hierarchical coloring scheme based on depth information. By normalizing the depth values of the point cloud in the camera coordinate system and mapping them to the HSV color space, the fused pixels can exhibit smooth color transitions based on distance. Finally, these pixels with depth and color information are precisely superimposed on the original 2D image to generate a real-time fused image that possesses both high-resolution texture features and accurate 3D depth information.
[0038] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.
Claims
1. A method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames, characterized in that, Specifically, the following steps are included: Step S1: Initialize the fusion processing node, obtain and load the camera's intrinsic and extrinsic parameters, and establish a communication link between the fusion processing node and the camera and LiDAR. Step S2: Construct the spatial pose transformation matrix of the LiDAR coordinate system relative to the camera coordinate system; Step S3: Acquire two-dimensional image frame data and time-correlated three-dimensional point cloud frame data; Step S4: Convert the received 3D point cloud frame data into an traversable point cloud structure, traverse the discrete points in the point cloud structure; use the spatial pose transformation matrix to transform the discrete points from the lidar coordinate system to the camera coordinate system to obtain 3D discrete points. Step S5: Use the perspective projection algorithm to project the three-dimensional discrete points in the camera coordinate system onto the pixel plane to obtain undistorted normalized coordinates, and then map them to pixel coordinates to form the projected pixel points. Step S6: Filter the projected pixels and extract the pixels that fall within the effective range of the image; Step S7: Based on the depth information of the 3D point cloud frame in the camera coordinate system, generate the corresponding colored pixels according to the preset depth grading coloring rules. Step S8: Overlay colored pixels onto the two-dimensional image frame data to generate and publish the fused image frame.
2. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 1, characterized in that, In step S4, before transforming the discrete points from the lidar coordinate system to the camera coordinate system using the spatial pose transformation matrix, a validity check is performed on each discrete point to filter out unqualified discrete points.
3. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 1, characterized in that, The step S2, which involves constructing the spatial pose transformation matrix of the lidar coordinate system relative to the camera coordinate system, specifically refers to: The 3×3 rotation matrix from the lidar coordinate system to the camera coordinate system is obtained through calibration. Translation vector Construct a 4×4 homogeneous transformation matrix The formula is as follows: 。 4. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 3, characterized in that, Step S4 specifically includes the following steps: Step S41: Set the coordinates of discrete points in the lidar coordinate system as follows The coordinates of the corresponding point in the camera coordinate system are ; like and If the coordinate value in the graph is not numerical or is infinity, then the graph will be... and corresponding Filter out; Step S42: ... Convert to The formula is: ； Expanding the calculation as follows: 。 5. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 4, characterized in that, Step S5 specifically includes the following steps: Step S51: Calculate normalized coordinates ,in, , ; Step S52: Map normalized coordinates to pixel coordinates : ； in, , Focus the camera. , The coordinates of the main point.
6. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 5, characterized in that, In step S6, the criteria for selecting the projected pixels are as follows: Let the image resolution be... , H is the image width, and H is the image height; ； Points that do not meet any of the above conditions are considered invalid and are discarded.
7. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 6, characterized in that, In step S7, the method for generating colored pixels according to a preset depth-level coloring rule specifically includes the following steps: S71: Preset minimum display depth and maximum display depth ; S72: Obtain the depth value in the camera coordinate system corresponding to the current pixel. ; S73: Depth value Normalization to The interval is used to obtain the normalized depth value. The calculation formula is as follows: ； like Then let ; like Then let ; S74: Uses the HSV color space mapping table to... Mapped to RGB color values This is used as the drawing color for that pixel.
8. The method for fusing and displaying three-dimensional point cloud frames and two-dimensional image frames according to claim 1, characterized in that, In step S3, the time-correlated three-dimensional point cloud frame data refers to finding the point cloud frame with the smallest absolute value of the timestamp difference with the image frame in the cached lidar point cloud frame queue when a two-dimensional image frame is received.
9. A computer terminal storage medium, characterized in that, The device stores computer terminal executable instructions, characterized in that the computer terminal executable instructions are used to execute the method for fusion display of three-dimensional point cloud frames and two-dimensional image frames as described in any one of claims 1-8.
10. A computing device, characterized in that, include: At least one processor; The at least one processor is also connected in communication with a memory, wherein the memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for fusion display of three-dimensional point cloud frames and two-dimensional image frames as described in any one of claims 1-8.