Color point cloud data correction method, device and computer equipment

Abstract

The present application is applicable to the field of intelligent transportation technology and provides a color point cloud data correction method, device and computer equipment. The method includes: obtaining initial color point cloud data; wherein, the initial color point cloud data is obtained based on the fusion of point cloud data and image data; inputting the image data and the initial color point cloud data into a preset color correction network, and outputting target color point cloud data, thereby ensuring the integrity and accuracy of the color assignment of the point cloud data.

Description

Technical Field

[0001] The present application belongs to the field of intelligent transportation technology, and in particular relates to a method, device and computer equipment for correcting color point cloud data. Background Art

[0002] Color point clouds consist of point data in three-dimensional space and corresponding color information (typically from RGB images). They hold significant potential in environmental perception, road sign recognition, building modeling, and urban planning. For example, in environmental perception, color point clouds can help vehicles more accurately identify objects such as traffic signs, pedestrians, and vehicles.

[0003] However, due to the varying startup times and sampling frequencies of the sensors, significant time deviations exist, leading to inaccurate mapping between point cloud data and image data. This is especially true in dynamic scenes, where the time intervals between different sensors are large, making it difficult for motion compensation to accurately predict positions, leading to deviations in color assignment. Summary of the Invention

[0004] The present invention provides a method, apparatus, and computer device for correcting color point cloud data. These methods address the existing technical problem of inaccurate mapping between point cloud data and image data due to significant time deviations caused by varying sensor startup times and sampling frequencies. This is particularly true in dynamic scenes, where the time intervals between different sensors are large, making it difficult for motion compensation to accurately predict positions, leading to deviations in color assignment.

[0005] In a first aspect, an embodiment of the present application provides a method for correcting color point cloud data, comprising:

[0006] Acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data;

[0007] The image data and the initial color point cloud data are input into a preset color correction network, and target color point cloud data is output.

[0008] In a possible implementation of the first aspect, obtaining initial color point cloud data includes:

[0009] Acquiring the point cloud data and the image data;

[0010] The point cloud data and the image data are fused to generate the initial color point cloud data; wherein the fusion process includes data fusion and / or feature fusion.

[0011] In a possible implementation of the first aspect, the training process of the color correction network includes:

[0012] Acquire test image data and test color point cloud data corresponding to the test image data;

[0013] Inputting the test image data and the test color point cloud data into an initial color correction network, and outputting a test result;

[0014] The initial color correction network is optimized based on the test results and preset label data to obtain the color correction network.

[0015] In a possible implementation of the first aspect, the method further includes:

[0016] Obtain target image data;

[0017] performing a simulation transformation operation on the target image data to generate the test image data;

[0018] The analog transformation operation includes at least one of the following: a linear mapping operation, a gamma transformation operation, a light source simulation operation, a frequency domain adjustment operation, and an adversarial network generation operation.

[0019] In a possible implementation of the first aspect, the initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information;

[0020] The step of inputting the image data and the initial color point cloud data into a preset color correction network and outputting target color point cloud data comprises:

[0021] The point cloud reflection intensity, the point cloud spatial position information and the image data are input into the color correction network, and the target color point cloud data is output.

[0022] In a possible implementation of the first aspect, the method further includes:

[0023] Performing semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result;

[0024] Map data is generated based on the target color point cloud data and the annotation results.

[0025] In a possible implementation of the first aspect, the method further includes:

[0026] Get the motion information of the object;

[0027] Calculating the relative time difference between the point cloud data and the image data of each camera;

[0028] Calculating an offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference;

[0029] The point cloud data is adjusted based on the offset to generate adjusted point cloud data.

[0030] In a second aspect, an embodiment of the present application provides a color point cloud data correction device, comprising:

[0031] An acquisition module, configured to acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data;

[0032] The correction module is used to input the image data and the initial color point cloud data into a preset color correction network and output target color point cloud data.

[0033] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the color point cloud data correction method described in any one of the first aspects above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the color point cloud data correction method described in any one of the first aspects above is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer device, enables the computer device to execute the color point cloud data correction method described in any one of the above-mentioned first aspects.

[0036] Compared to the prior art, the present embodiment has the following advantages: by inputting image data and initial color point cloud data into a preset color correction network, target color point cloud data is output. The color correction network combines the initial color point cloud data (such as point cloud reflection intensity and spatial position information) to correct the color information of the initially assigned point cloud in a high-dimensional space, generating the final color of each point in the point cloud (i.e., the target color point cloud data), thereby improving the accuracy and reliability of the color point cloud data.

[0037] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flowchart of a color point cloud data correction method provided by an embodiment of the present application;

[0040] Figure 2 Schematic diagram of the structure of the color point cloud data correction device provided in an embodiment of the present application;

[0041] Figure 3 It is a structural diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0043] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0044] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0045] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0046] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0048] With the rapid development of autonomous driving technology and the increasing maturity of intelligent transportation systems, high-precision maps have become a critical and indispensable component of autonomous driving and traffic management. Traditional high-precision maps are primarily constructed using the geometric ranging capabilities of lidar. However, these maps, consisting solely of point cloud data, lack color information, making them difficult to meet the semantic requirements of complex applications such as environmental perception, object recognition, and dynamic map updates. Assigning accurate and stable color information to point cloud data would significantly enhance the applicability and intelligence of high-precision maps in a variety of scenarios. Currently, point cloud color assignment technology faces the following major challenges: time synchronization. Due to the varying startup times and sampling frequencies of sensors, significant time skew exists, leading to inaccurate mapping between point cloud data and image data. In dynamic scenes, the long time intervals between different sensors make it difficult for motion compensation to accurately predict positions, resulting in inaccurate color assignments.

[0049] In order to solve the above problems, Figure 1 A schematic flowchart of a color point cloud data correction method provided by the present application is shown.

[0050] S101, obtaining initial color point cloud data; wherein, the initial color point cloud data is obtained based on the fusion of point cloud data and image data.

[0051] Among them, point cloud data is a set of points distributed in three-dimensional space, and each point contains its coordinates in space.

[0052] In the embodiments of the present application, point cloud data can be collected by laser radar, depth camera, etc. Among them, laser radar is one of the main sources of point cloud data. It obtains the three-dimensional coordinates of an object by emitting laser pulses and measuring the reflection time or phase difference. Laser radar can quickly generate high-precision point cloud data and is widely used in autonomous driving, terrain mapping, and three-dimensional modeling. In autonomous driving, point cloud data is used for environmental perception, obstacle detection, and high-precision map construction. The point cloud generated by the laser radar can provide three-dimensional information of the vehicle's surroundings in real time, helping the vehicle make decisions.

[0053] Image data is two-dimensional data captured by cameras, containing rich color information (typically represented by RGB values), which provides visual detail and semantic information for high-precision maps. Image data can be acquired using a variety of camera devices, including standard RGB cameras and panoramic cameras. This image data is used in high-precision map construction to enhance environmental perception, assist in semantic understanding of point cloud data, and improve map visualization.

[0054] Color point cloud data is a data format that combines point cloud data in three-dimensional space with color information. This data not only contains the position of each point in space, but also the color information associated with each point.

[0055] The initial color point cloud data mentioned above refers to the preliminary result obtained during the fusion process of point cloud data and image data. It contains spatial position and color information. Initial color point cloud data may have some issues, such as inaccurate color assignment, information loss, or data inconsistency. To improve data quality, further processing of the initial color point cloud data is usually required. The detailed processing process is described below.

[0056] There are various ways to obtain the initial color point cloud data. You can directly obtain the initial color point cloud data that has been preliminarily fused; or you can obtain the point cloud data and image data separately and then perform fusion processing.

[0057] S102: Input the image data and the initial color point cloud data into a preset color correction network, and output target color point cloud data.

[0058] The color correction network is a pre-built network used to improve the accuracy of point cloud color information. This network combines the spatial location information and point cloud reflection intensity of the point cloud data to correct the color information of the point cloud data, thereby generating accurate color point cloud data that is closer to the real scene, namely the target color point cloud data.

[0059] In this embodiment of the present application, target color point cloud data is output by inputting image data and initial color point cloud data into a preset color correction network. This color correction network not only considers the color information of the image data but also incorporates the initial color point cloud data (such as point cloud reflectance intensity and spatial position information) to correct the color information of the initially assigned point cloud in a high-dimensional space, generating the final color of each point in the cloud (i.e., the target color point cloud data), thereby improving the accuracy and reliability of the color point cloud data. With its precise three-dimensional spatial coordinates and rich and accurate color information, the target color point cloud data provides an intuitive and detailed foundational spatial framework and visual feature expression for map construction.

[0060] In an optional embodiment, the color correction network in the embodiments of this application can employ a multimodal deep neural network architecture to fuse the 3D geometric information of point cloud data with the color information of image data. The specific network structure may include an input layer and feature extraction module, a feature fusion module, a color prediction and correction module, and an output module. The following description of the color correction network is merely an example; those skilled in the art may adjust and optimize the network structure based on actual needs to accommodate different application scenarios and data types.

[0061] The following are the introductions to the input layer and feature extraction module:

[0062] Input layer: The network receives multiple types of input data, such as three types of input data, including: image data, point cloud reflection intensity, and point cloud spatial position information.

[0063] Image feature extraction: Extract multi-scale image features (including shallow texture features and deep semantic features, etc.).

[0064] Point cloud feature extraction: It can extract local geometric features (for example, based on the spatial position information of the point cloud, construct a local neighborhood through farthest point sampling and sphere query, and extract geometric properties such as curvature and normal vector), reflection intensity features (joining the reflection intensity value with the geometric features to form a point cloud feature vector), etc.

[0065] The feature fusion module is introduced as follows:

[0066] The attention mechanism can be used to achieve efficient fusion of image and point cloud features:

[0067] Spatial alignment: Project the point cloud onto the image plane and establish point-pixel correspondence.

[0068] Image-to-point cloud attention: Calculate the weight distribution of image features to point cloud features, and enhance the color information related to the point cloud geometry.

[0069] Point cloud to image attention: Use point cloud reflection intensity to filter reliable areas in the image and suppress noise interference.

[0070] Feature concatenation and dimensionality reduction: The fused features are reduced in dimension through a fully connected layer, and the fused feature vector is output.

[0071] The color prediction and correction module is introduced as follows:

[0072] Color prediction head: It consists of N layers (e.g., 3 layers) of fully connected layers, which inputs the fused feature vector and outputs the original RGB color value of each point cloud point (N×3).

[0073] Geometric constraint unit: introduces spatial continuity loss to constrain the smooth transition of colors of adjacent point clouds and avoid color jumps.

[0074] Reflection intensity guidance unit: Dynamically adjusts the color prediction weight according to the reflection intensity value, such as reducing the brightness gain in high-reflection areas and enhancing the color saturation in low-reflection areas.

[0075] The output module is described as follows:

[0076] Convert the predicted raw RGB color values ​​to a standardized point cloud format and ensure numerical validity.

[0077] In an optional embodiment, S101 acquires initial color point cloud data, including:

[0078] Acquire the point cloud data and the image data; perform fusion processing on the point cloud data and the image data to generate the initial color point cloud data; wherein the fusion processing includes data fusion and / or feature fusion.

[0079] For the relevant description of the point cloud data and the image data, please refer to the relevant description in the previous embodiment, which will not be repeated here.

[0080] In the embodiment of the present application, a data-level fusion method and a feature-level fusion method may be used to generate the initial color point cloud data.

[0081] The data-level fusion method includes: aligning the coordinate systems of the point cloud data and the image data through sensor calibration; and mapping the pixel values ​​in the image data to each point in the point cloud to generate the initial color point cloud data. Specifically, sensor calibration aligns the point cloud data and image data to the same coordinate system. Then, the color information (i.e., RGB values) in the image data is mapped to each point in the point cloud. For example, a corresponding RGB value is directly assigned based on the projected position of the point cloud point in the image. Each point in the point cloud not only contains its spatial position but also color information, forming a color point cloud.

[0082] The feature-level fusion method includes: extracting features from the point cloud data to obtain point cloud features; extracting features from the image data to obtain image features; and fusing the point cloud features with the image features to form a colored point cloud. Optionally, a convolutional neural network is used to extract features from the point cloud and image, and an attention mechanism is used to calculate weights between the features to achieve feature fusion and obtain a colored point cloud.

[0083] In an optional embodiment, fixed-mount sensors such as lidar and cameras must be accurately calibrated before data collection. The purpose of calibration is to establish an accurate spatial correspondence between the point cloud data and the image data, thereby ensuring that the point cloud can be correctly projected onto the image plane.

[0084] Among them, the camera's internal parameter calibration can be completed through Zhang Zhengyou calibration method or deep learning-based calibration method to accurately obtain the camera's internal parameters, including focal length, optical center position, and distortion coefficient.

[0085] Among them, the internal parameter calibration of the lidar can be completed based on the target calibration method or the deep learning-based calibration method, so as to accurately obtain the internal parameters of the lidar, including scanning angle deviation, distance deviation, point cloud reflection intensity calibration, etc.

[0086] Extrinsic calibration between the camera and LiDAR can be performed using either manually selected control point calibration or a checkerboard calibration method. These methods determine the relative position and posture of the two sensors, establishing a spatial correspondence between the point cloud data and the image data. The results of this extrinsic calibration enable the LiDAR point cloud data to be accurately projected onto the camera's image plane, providing the foundation for subsequent multi-sensor data fusion.

[0087] In an optional embodiment, a sensor-equipped vehicle travels within a scene at a relatively low speed (e.g., under 40 km / h) to collect data from sensors such as lidar and cameras. During data collection, time synchronization is required between the data from different sensors to generate matching relationships between the different sensor data. Specifically, using the lidar timestamp as a reference, data frames from other sensors within the preceding and following half of the sampling period can be searched. The data frame with the closest time is selected for synchronization. This minimizes the time interval between the data frames from other sensors and the current lidar data frame, avoiding data alignment difficulties caused by excessive time intervals.

[0088] In an optional embodiment, after the above data is collected, projection motion compensation is further required. Specifically, the object's motion information is obtained; the relative time difference between the point cloud data and the image data of each camera is calculated; the offset of the point cloud data relative to the image data of each camera is calculated based on the motion information and the relative time difference; and the point cloud data is adjusted based on the offset to generate adjusted point cloud data.

[0089] Among them, motion information (including vehicle speed, acceleration, and angle, etc.) is obtained from the Inertial Measurement Unit (IMU) and the Global Positioning System (GPS).

[0090] In an embodiment of the present application, the relative time difference between the current point cloud data and the image data from different cameras is calculated. Based on this time difference and the acquired motion information, the vehicle's motion changes during this period are inferred, and the offset of the point cloud data relative to the image data from each camera is determined. In addition, to more accurately predict the vehicle's speed and angle of motion, interpolation is used to fill the time gaps, improve the accuracy of motion compensation, and obtain more accurate motion information. For example, if the vehicle's motion state at two time points is known, interpolation can be used to estimate the vehicle's state at any time between these two time points. Subsequently, based on the calculated offset and the accurate motion information obtained through interpolation, the position of the point cloud data is adjusted to compensate for the offset caused by vehicle motion. This ensures that the point cloud data and image data are more accurately aligned in space, providing a reliable foundation for subsequent multi-sensor data fusion. Through the above steps, data alignment issues caused by sensor data asynchrony and vehicle motion can be effectively resolved, improving the accuracy and reliability of environmental perception in autonomous driving systems.

[0091] The motion-compensated point cloud data is then projected onto the image plane using the camera's intrinsic and extrinsic parameters. The specific steps are as follows: The point cloud data is converted from the LiDAR coordinate system to the camera coordinate system using extrinsic parameters (rotation matrix and translation vector). The point cloud data is projected from the camera coordinate system to the pixel coordinate system of the image plane using the camera's intrinsic parameters. A mapping relationship is established between each point in the point cloud data and different image pixels. The RGB values ​​of the corresponding pixels are extracted from the image data. The extracted RGB values ​​are assigned to the corresponding point cloud points, thereby adding initial color information to the point cloud and obtaining the initial colored point cloud data.

[0092] It's important to note that in a multi-view camera system, each camera covers a different viewing angle. To achieve comprehensive environmental perception, point cloud data must be fused with image data from multiple cameras. The specific steps are as follows: The point cloud data is projected onto the image plane of each camera, and a mapping relationship is established between the point cloud points and the image pixels. The RGB values ​​of the corresponding pixels are extracted from the image data of each camera. The color information is then fused according to specific rules (such as average, weighted average, or selecting the clearest view angle) to assign more accurate color information to the point cloud points.

[0093] In an optional embodiment, the training process of the color correction network includes:

[0094] Step a1: Acquire test image data and test color point cloud data corresponding to the test image data.

[0095] Test image data refers to image data that has undergone specific processing or has not been processed, and is used to evaluate and verify the performance of the color correction network. This data is usually derived from actual application scenarios or generated through simulation to ensure that it can reflect the various conditions and changes in the real environment.

[0096] In the embodiment of the present application, the test image data is generated by acquiring target image data and performing a simulation transformation operation on the target image data;

[0097] The analog transformation operation includes at least one of the following: a linear mapping operation, a gamma transformation operation, a light source simulation operation, a frequency domain adjustment operation, and an adversarial network generation operation. Other related operations are also included within the scope of protection of this application, but are not limited to the above analog transformation operations.

[0098] The target image data refers to image data collected under normal lighting conditions. These data are used as preset labeled data. They provide a benchmark for training and optimizing the color correction network, are used to evaluate the test results of the network output, and guide the adjustment and optimization of the network.

[0099] In the embodiments of the present application, a variety of image simulation transformation operations are performed to simulate different lighting conditions, environmental changes, or sensor characteristics, thereby providing more challenging and diverse data for training the color correction network. Specifically, the simulation transformation operations include at least one of the following:

[0100] The linear mapping operation changes the brightness and contrast of an image by adjusting the pixel value range. This operation can simulate the effects of images under different exposure conditions and enhance the network's ability to adapt to brightness changes.

[0101] The gamma transform operation changes the brightness distribution of an image. By adjusting the image's gamma value, it enhances dark details or highlights. This operation can simulate the effects of images under different light intensities, improving the network's robustness to lighting changes.

[0102] Light source simulation simulates image effects under different lighting environments by changing the direction, intensity, and color of the lighting in the image. For example, it can simulate scenes under direct sunlight, diffuse reflections on a cloudy day, or artificial light sources. This operation not only changes the brightness and contrast of the image but also introduces shadows and reflections, enhancing the network's adaptability to complex lighting conditions.

[0103] Frequency domain adjustments modify the image's frequency distribution by processing its spectrum. For example, high-pass filtering enhances edge details, while low-pass filtering smooths the image. This operation can simulate the imaging characteristics of different sensors or the effects of environmental noise, improving the network's robustness to varying imaging conditions.

[0104] The Generative Adversarial Network (GAN) operation uses a generative adversarial network to generate realistic image data. Through adversarial training of the generator and discriminator, synthetic images are produced that are indistinguishable from real images. This operation can simulate images under extreme lighting conditions, rare scenes, or sensor failures, enhancing the network's generalization ability for complex scenes.

[0105] In the embodiments of the present application, the test image data generated by the simulated transformation operations described above can provide diverse input for training the color correction network. This data covers not only images under normal lighting conditions, but also images in complex scenes such as strong light, low light, and shadows. By training on this diverse data, the color correction network can learn a wider range of color correction strategies, thereby better adapting to different lighting conditions and sensor characteristics.

[0106] Step a2: input the test image data and the test color point cloud data into the initial color correction network and output the test result.

[0107] The test result refers to the corrected color point cloud data output after the initial color correction network processes the test image data and the test color point cloud data.

[0108] In this embodiment of the present application, after obtaining test image data, the data fusion described in the above embodiment is performed on the test image data to generate test color point cloud data. The test image data (including image data from multiple viewing angles under different lighting conditions, such as overexposure and low light) and the test color point cloud data (including point cloud spatial position information, point cloud reflection intensity, and initial color information) are input into the initial color correction network. The network performs feature extraction and fusion processing, and then outputs the test results.

[0109] Step a3: Optimizing the initial color correction network based on the test results and preset label data to obtain the color correction network.

[0110] The preset labeled data refers to image data captured under normal lighting conditions. This data serves as the ground truth. It reflects the true color information of the scene and is the basis for evaluating and optimizing the color correction network. Optionally, the preset labeled data is the target image data mentioned above.

[0111] In an embodiment of the present application, the accuracy of the test results is evaluated by calculating the error between the test results and the labeled data (such as the mean square error, structural similarity index, etc.), and the initial color correction network is optimized. After multiple iterative optimizations, a color correction network that can accurately correct color deviations is finally obtained.

[0112] In the existing technology, point cloud color assignment technology still has the following problems:

[0113] Lack of spatial information: The color correction algorithm only predicts the color under normal lighting at the image data level, but ignores spatial information, which leads to poor prediction accuracy in some areas.

[0114] Insufficient input information: Existing color correction methods rely solely on bright and dark light images for prediction, lacking other reliable information to assist, resulting in inaccurate color prediction results.

[0115] In order to solve the above problems, an embodiment of the present application also provides a color point cloud data correction method, wherein the initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; the point cloud reflection intensity, the point cloud spatial position information and the image data are input into the color correction network, and the target color point cloud data is output.

[0116] Point cloud reflection intensity is the reflectivity of each point in the point cloud data to the laser pulse. It is used in color correction networks for the following purposes: when point cloud data lacks RGB color information, it can provide a black and white photo-like visualization of the point cloud. Based on point cloud reflection intensity, point cloud data can be classified or noise filtered to remove points within a specific intensity range. Point cloud reflection intensity can also serve as an auxiliary feature, helping the network better understand the material and structure of objects in the scene.

[0117] In color correction networks, point cloud spatial position information is used to provide precise 3D locations for each point in space, helping the network understand the shape and layout of objects. This information is used to project the color information of image data onto the point cloud, generating a colored point cloud. Combining point cloud reflection intensity with spatial position information enables more accurate semantic segmentation and object detection.

[0118] In an embodiment of the present application, a multidimensional color correction network is utilized. This multidimensional color correction network considers three dimensions: point cloud reflection intensity, point cloud spatial position information, and color information. It corrects the color information of initial color point cloud data in a high-dimensional space, generating accurate target color point cloud data. The inclusion of point cloud spatial position information provides three-dimensional geometric constraints for color assignment, resulting in more natural color transitions. By incorporating geometric information into the point cloud data, the network can more stably output corrected color information when handling lighting variations. Even under extreme lighting conditions (such as overexposure or low light), the network can leverage geometric information to assist in color correction, enhancing system robustness. For example, color changes at the edges or in shadows of an object can be smoothed using spatial position information, avoiding color jumps or discontinuities. Furthermore, the point cloud spatial position information helps the network predict more consistent colors across different poses and angles. For example, when a vehicle views the same object from different directions, spatial position information ensures consistent color across different viewing angles, preventing color deviations caused by perspective changes. Furthermore, the spatial position information of the point cloud helps the network understand the shape and layout of objects in the scene, enabling more accurate color assignment. For example, the network can adjust the brightness and contrast of colors based on the surface curvature and orientation of objects to better reflect the scene. This spatial position information can also be used to identify shadows and reflections, and adjust colors to simulate realistic lighting effects. For example, the brightness of shadows can be reduced, while the contrast of reflections can be increased.

[0119] Point cloud reflection intensity is introduced. This provides material characteristics of each point, helping the network predict color more accurately. For example, points with higher point cloud reflection intensity typically correspond to smooth or reflective surfaces, and their colors may be closer to the true color. Point cloud reflection intensity may correspond to rough or highly absorbent surfaces, and their colors may require adjustment. Point cloud reflection intensity can reduce the impact of lighting conditions on color prediction. For example, in strong or low light conditions, point cloud reflection intensity can help the network adjust the brightness and contrast of colors to more accurately reflect the scene. Point cloud reflection intensity helps the network perceive the material characteristics of objects, leading to more accurate color predictions. For example, point cloud reflection intensity can distinguish between different materials such as metal, plastic, and vegetation, and the network can adjust the color hue and saturation based on this information. Furthermore, point cloud reflection intensity, combined with spatial position information, can enhance the network's semantic understanding of the scene. For example, the network can identify objects such as traffic signs, pedestrians, and vehicles based on point cloud reflection intensity and spatial position information, and assign them more accurate colors.

[0120] By fusing the 3D geometry of point cloud data with the color information of image data, the color correction network achieves a more comprehensive understanding of the physical characteristics of the scene. Using pre-set labeled data (real colors under normal lighting conditions) as optimization targets, the network gradually corrects color deviations through error calculation and parameter adjustment.

[0121] For ease of understanding, this is described in conjunction with specific scenarios. The spatial position information (coordinates and normal vectors) of the point cloud data can be input into the network to enable the network to perceive local geometric structures (such as planes and curved surfaces) and avoid color prediction errors caused by perspective occlusion in two-dimensional images. For example, when adjacent point clouds belong to the same object surface in space, their colors should transition smoothly (such as the curved surface of a car body). The network achieves color consistency through spatial position constraints. In addition, areas with high reflection intensity (such as metal and glass) are prone to "white overflow" in overexposed scenes. The network combines the reflection intensity of the point cloud to suppress color distortion in overly bright areas. Areas with low reflection intensity (such as asphalt pavement) lose details in dim light. By associating the reflection intensity of the point cloud with the spatial position, the true color (such as the color of the lane line) is restored.

[0122] In summary, by incorporating point cloud spatial position information and point cloud reflection intensity, the color correction network can significantly improve the effectiveness of color assignment and correction. The point cloud spatial position information ensures natural color transitions and perspective consistency, while the point cloud reflection intensity reduces the difficulty of color prediction and improves its accuracy.

[0123] In an optional embodiment, the method further includes: performing semantic annotation and / or structured annotation on the target color point cloud data to generate annotation results; and generating map data based on the target color point cloud data and the annotation results.

[0124] Semantic annotation involves assigning semantic category labels to each point or point cloud segment in the target color point cloud data. These labels can represent the categories of different objects in the point cloud data, such as roads, buildings, traffic signs, pedestrians, and vehicles. The primary goal of semantic annotation is to enable machines to understand the semantic content of point cloud data, thereby making more accurate decisions in applications such as autonomous driving and intelligent transportation.

[0125] Structured annotation refers to the more granular annotation of point cloud data, not only marking the object category but also the structure and attributes of the object. For example, building annotation (marking the building's height, number of floors, door and window locations, etc.) and road annotation (marking the road's width, number of lanes, traffic sign locations, etc.)

[0126] The annotation results are usually in the form of labels, which can be the category label of each point in the point cloud data, or the bounding box, polygon outline or other geometric attributes of the object.

[0127] In this embodiment of the application, a high-precision map is generated based on the target color point cloud data and annotation results. The high-precision map not only contains the 3D geometric information of the point cloud, but also contains rich semantic information and structured data, which can provide more accurate environmental perception and decision support for autonomous vehicles.

[0128] In an embodiment of the present application, in order to construct a high-precision map, it is necessary to fuse colored point cloud data from different times, different sensors (such as lidar, camera) or different perspectives. Specifically, it includes: aligning the data of different sensors into a unified coordinate system to ensure the consistency of the data in time and space. Use sensor calibration technology (such as external parameter calibration of lidar and camera) to establish the spatial relationship between sensors. Perform multi-sensor data fusion on the aligned data to generate colored point cloud data. Merge the colored point cloud data into a unified coordinate system, combine the annotation results, and construct a high-precision map of the entire scene.

[0129] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0130] Corresponding to the color point cloud data correction method described in the above embodiment, Figure 2 A structural block diagram of the color point cloud data correction device provided in an embodiment of the present application is shown. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0131] Reference Figure 2 , the color point cloud data correction device includes:

[0132] An acquisition module, configured to acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data;

[0133] The correction module is used to input the image data and the initial color point cloud data into a preset color correction network and output target color point cloud data.

[0134] In one possible implementation, a module is obtained for:

[0135] Acquiring the point cloud data and the image data;

[0136] The point cloud data and the image data are fused to generate the initial color point cloud data; wherein the fusion process includes data fusion and / or feature fusion.

[0137] In a possible implementation, the color point cloud data correction device further includes a training module for:

[0138] Acquire test image data and test color point cloud data corresponding to the test image data;

[0139] Inputting the test image data and the test color point cloud data into an initial color correction network, and outputting a test result;

[0140] The initial color correction network is optimized based on the test results and preset label data to obtain the color correction network.

[0141] In a possible implementation, the color point cloud data correction device further includes a generation module, which is used to:

[0142] Obtain target image data;

[0143] performing a simulation transformation operation on the target image data to generate the test image data;

[0144] The analog transformation operation includes at least one of the following: a linear mapping operation, a gamma transformation operation, a light source simulation operation, a frequency domain adjustment operation, and an adversarial network generation operation.

[0145] In one possible implementation, the initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; the correction module is configured to:

[0146] The point cloud reflection intensity, the point cloud spatial position information and the image data are input into the color correction network, and the target color point cloud data is output.

[0147] In a possible implementation, the color point cloud data correction device further includes a map generation module, which is used to:

[0148] Performing semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result;

[0149] Map data is generated based on the target color point cloud data and the annotation results.

[0150] In a possible implementation, the color point cloud data correction device further includes an adjustment module for:

[0151] Get the motion information of the object;

[0152] Calculating the relative time difference between the point cloud data and the image data of each camera;

[0153] Calculating an offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference;

[0154] The point cloud data is adjusted based on the offset to generate adjusted point cloud data.

[0155] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0157] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0158] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0159] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0161] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0162] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0166] Figure 3 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 3 As shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 3 Only one is shown), a memory 21 and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned embodiments of the color point cloud data correction method when executing the computer program 22.

[0167] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that Figure 3 The computer device is merely an example and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, etc.

[0168] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0169] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 21 may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped with the computer device. Furthermore, the memory 21 may include both an internal storage unit of the computer device and an external storage device. The memory 21 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0170] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0171] The personal information processed by the Applicant will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The Applicant will treat the user's personal information and its processing with a high degree of diligence.

[0172] The Applicant attaches great importance to the security of user personal information and has taken reasonable and feasible security measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

Claims

1. A color point cloud data correction method, characterized in that: include: Acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; Inputting the image data and the initial color point cloud data into a preset color correction network, and outputting target color point cloud data; The color correction network is used to correct the color information of the point cloud data by combining the color information of the image data with the point cloud reflection intensity and spatial position information of the initial color point cloud data; The color correction network includes a color prediction and correction module, which is used to correct the initial color point cloud by introducing spatial continuity loss in the constructed network to constrain the smooth transition of colors of adjacent point clouds; and dynamically adjust the color prediction weights based on the reflection intensity value, reducing the brightness gain in high-reflection areas and enhancing the color saturation in low-reflection areas.

2. The color point cloud data correction method according to claim 1, characterized in that: The obtaining of initial color point cloud data includes: Acquiring the point cloud data and the image data; The point cloud data and the image data are fused to generate the initial color point cloud data; wherein the fusion process includes data fusion and / or feature fusion.

3. The color point cloud data correction method according to claim 1, characterized in that: The training process of the color correction network includes: Acquire test image data and test color point cloud data corresponding to the test image data; Inputting the test image data and the test color point cloud data into an initial color correction network, and outputting a test result; The initial color correction network is optimized based on the test results and preset label data to obtain the color correction network.

4. The color point cloud data correction method according to claim 3, characterized in that: The method further comprises: Obtain target image data; performing a simulation transformation operation on the target image data to generate the test image data; The analog transformation operation includes at least one of the following: a linear mapping operation, a gamma transformation operation, a light source simulation operation, a frequency domain adjustment operation, and an adversarial network generation operation.

5. The color point cloud data correction method according to any one of claims 1 to 4, characterized in that: The initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; The step of inputting the image data and the initial color point cloud data into a preset color correction network and outputting target color point cloud data comprises: The point cloud reflection intensity, the point cloud spatial position information and the image data are input into the color correction network, and the target color point cloud data is output.

6. The color point cloud data correction method according to claim 5, characterized in that: The method further comprises: Performing semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result; Map data is generated based on the target color point cloud data and the annotation results.

7. The color point cloud data correction method according to claim 5, characterized in that: The method further comprises: Get the motion information of the object; Calculating the relative time difference between the point cloud data and the image data of each camera; Calculating an offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference; The point cloud data is adjusted based on the offset to generate adjusted point cloud data.

8. A color point cloud data correction device, characterized in that: include: An acquisition module, configured to acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; A correction module is used to input the image data and the initial color point cloud data into a preset color correction network and output target color point cloud data; the color correction network is used to combine the color information of the image data with the point cloud reflection intensity and spatial position information of the initial color point cloud data to correct the color information of the point cloud data; the color correction network includes a color prediction and correction module, which is used to correct the initial color point cloud by the following method: introducing spatial continuity loss in the constructed network to constrain the smooth transition of colors of adjacent point clouds; and dynamically adjusting the color prediction weight according to the reflection intensity value, reducing the brightness gain in high-reflection areas, and enhancing the color saturation in low-reflection areas.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer program product, characterized in that When the computer program product is run on a computer device, the computer device is caused to perform the method according to any one of claims 1 to 7.

Images