Color point cloud data correction method and device and computer equipment

Abstract

The invention is suitable for the technical field of intelligent transportation, and provides a color point cloud data correction method and device and computer equipment, and the method comprises the steps: obtaining initial color point cloud data; wherein the initial color point cloud data is obtained based on fusion of point cloud data and image data; and inputting the image data and the initial color point cloud data into a preset color correction network, and outputting target color point cloud data, thereby ensuring completeness and accuracy of color assignment of the point cloud data.

Description

Technical Field

[0001] This application belongs to the field of intelligent transportation technology, and particularly relates to a method, device, and computer device for correcting color point cloud data. Background Art

[0002] A color point cloud is composed of point cloud data in three-dimensional space and corresponding color information (usually from an RGB image). Color point clouds have important application prospects in fields such as environmental perception, road sign recognition, building modeling, and urban planning. For example, in environmental perception, color point clouds can help vehicles more accurately identify targets such as traffic signs, pedestrians, and vehicles.

[0003] However, due to the different startup times and sampling frequencies of sensors, there are obvious time deviations, which makes the mapping between point cloud data and image data inaccurate. Especially in dynamic scenarios, the time intervals between different sensors are large, and it is difficult to accurately predict positions for motion compensation, resulting in problems with color assignment deviations. Summary of the Invention

[0004] Embodiments of this application provide a method, device, and computer device for correcting color point cloud data, which can solve the technical problem in the prior art that due to the different startup times and sampling frequencies of sensors, there are obvious time deviations, which makes the mapping between point cloud data and image data inaccurate. Especially in dynamic scenarios, the time intervals between different sensors are large, and it is difficult to accurately predict positions for motion compensation, resulting in deviations in color assignment.

[0005] In a first aspect, embodiments of this application provide a method for correcting color point cloud data, including: Obtain initial color point cloud data; wherein, the initial color point cloud data is obtained by fusing point cloud data and image data; Input the image data and the initial color point cloud data into a preset color correction network, and output target color point cloud data.

[0006] In a possible implementation manner of the first aspect, the obtaining of the initial color point cloud data includes: Obtain the point cloud data and the image data; Perform a fusion process on the point cloud data and the image data to generate the initial color point cloud data; wherein, the fusion process includes data fusion and / or feature fusion.

[0007] In a possible implementation manner of the first aspect, the training process of the color correction network includes: Obtain test image data and corresponding test color point cloud data; Input the test image data and the test color point cloud data into the initial color correction network to output a test result; Optimize the initial color correction network based on the test result and preset label data to obtain the color correction network.

[0008] In a possible implementation manner of the first aspect, the method further includes: Obtain target image data; Perform a simulation transformation operation on the target image data to generate the test image data; Wherein, the simulation transformation operation includes at least one of the following: linear mapping operation, gamma transformation operation, light source simulation operation, frequency domain adjustment operation, adversarial network generation operation.

[0009] In a possible implementation manner of the first aspect, the initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; The inputting the image data and the initial color point cloud data into a preset color correction network to output target color point cloud data includes: Input the point cloud reflection intensity, the point cloud spatial position information and the image data into the color correction network to output the target color point cloud data.

[0010] In a possible implementation manner of the first aspect, the method further includes: Perform semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result; Generate map data based on the target color point cloud data and the annotation result.

[0011] In a possible implementation manner of the first aspect, the method further includes: Obtain the motion information of an object; Calculate the relative time difference between the point cloud data and the image data of each camera; Calculate the offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference; Adjust the point cloud data based on the offset to generate adjusted point cloud data.

[0012] In a second aspect, an embodiment of the present application provides a color point cloud data correction device, including: An acquisition module, configured to acquire initial color point cloud data; wherein, the initial color point cloud data is obtained by fusing point cloud data and image data; A correction module, configured to input the image data and the initial color point cloud data into a preset color correction network to output target color point cloud data.

[0013] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for correcting color point cloud data described in any one of the above first aspects is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method for correcting color point cloud data described in any one of the above first aspects is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the computer program product runs on a computer device, the computer device is enabled to execute the method for correcting color point cloud data described in any one of the above first aspects.

[0016] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: By inputting image data and initial color point cloud data into a preset color correction network, target color point cloud data is output. The color correction network combines the initial color point cloud data (such as point cloud reflection intensity and spatial position information) to correct the color information of the initially assigned point cloud in a high-dimensional space, generating the final color of each point cloud point (i.e., the target color point cloud data), thereby improving the accuracy and reliability of the color point cloud data.

[0017] It can be understood that the beneficial effects of the above second aspect to fifth aspect can refer to the relevant descriptions in the above first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a schematic flowchart of a method for correcting color point cloud data provided by an embodiment of the present application; Figure 2 is a schematic structural diagram of a device for correcting color point cloud data provided by an embodiment of the present application; Figure 3 is a schematic structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0021] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0022] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the specification and claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" depending on the context.

[0024] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0025] The reference to "one embodiment" or "some embodiments" or the like described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] In the current context where autonomous driving technology is developing rapidly and intelligent transportation systems are becoming increasingly mature, high-precision maps have become an extremely crucial and indispensable part of the fields of autonomous driving and traffic management. Traditional high-precision maps are mainly constructed by leveraging the geometric ranging function of lidar. However, with only point cloud data and lacking color information, it is difficult to meet the demand for semantic information in complex application scenarios such as environmental perception, target recognition, and dynamic map updates. If real and stable color information can be assigned to the point cloud data, the applicability and intelligence level of high-precision maps in various scenarios will be significantly improved. Currently, the point cloud color assignment technology mainly has the following problems: the time synchronization problem. Due to the different startup times and sampling frequencies of sensors, there are obvious time deviations, which makes the mapping between point cloud data and image data inaccurate. Especially in dynamic scenarios, the time intervals of different sensors are relatively large, and it is difficult to accurately predict the position through motion compensation, resulting in color assignment deviations.

[0027] To solve the above problems, Figure 1 Fig. shows a schematic flowchart of a method for correcting colored point cloud data provided by this application.

[0028] S101, obtain initial colored point cloud data; wherein, the initial colored point cloud data is obtained by fusing point cloud data and image data.

[0029] Among them, point cloud data is a set composed of a group of points distributed in three-dimensional space, and each point contains its coordinates in space.

[0030] In the embodiments of this application, point cloud data can be collected by lidar, depth cameras, etc. Among them, lidar is one of the main sources of point cloud data. It obtains the three-dimensional coordinates of objects by emitting laser pulses and measuring the reflection time or phase difference. Lidar can quickly generate high-precision point cloud data and is widely used in autonomous driving, terrain mapping, and 3D modeling. In autonomous driving, point cloud data is used for environmental perception, obstacle detection, and high-precision map construction. The point cloud generated by lidar can provide real-time three-dimensional information about the vehicle's surrounding environment and help the vehicle make decisions.

[0031] Among them, image data is two-dimensional data captured by a camera, containing rich color information (usually represented by RGB values), and can provide visual details and semantic information for high-precision maps. Image data can be obtained through various camera devices, including ordinary RGB cameras, panoramic cameras, etc. These image data are used in high-precision map construction to enhance environmental perception, assist in semantic understanding of point cloud data, and improve the visualization effect of the map.

[0032] Among them, the colored point cloud data is a data form that combines the point cloud data in three-dimensional space with color information. This data not only contains the position of each point in space but also the color information associated with each point.

[0033] Among them, the above-mentioned initial colored point cloud data refers to the preliminary result obtained during the fusion process of point cloud data and image data, which contains spatial position information and color information. There may be some problems with the initial colored point cloud data, such as inaccurate color assignment, information loss, or data inconsistency, etc. To improve the data quality, it is usually necessary to further process the initial colored point cloud data. For the specific processing process, please refer to the following text.

[0034] Among them, there are various ways to obtain the initial colored point cloud data. It is possible to directly obtain the initially fused initial colored point cloud data; it is also possible to obtain the point cloud data and image data separately and then perform the fusion process.

[0035] S102, input the image data and the initial colored point cloud data into a preset color correction network, and output the target colored point cloud data.

[0036] Among them, the color correction network is a pre-constructed network for improving the accuracy of the point cloud color information. This network combines the spatial position information of the point cloud data and the point cloud reflection intensity to correct the color information of the point cloud data, so as to generate accurate colored point cloud data closer to the real scene, that is, the target colored point cloud data.

[0037] In the embodiment of the present application, by inputting the image data and the initial colored point cloud data into a preset color correction network, the target colored point cloud data is output. The color correction network not only considers the color information of the image data but also combines the initial colored point cloud data (such as point cloud reflection intensity, spatial position information) to correct the color information of the initially assigned point cloud in the high-dimensional space, generating the final color of each point cloud point (that is, the target colored point cloud data), thereby improving the accuracy and reliability of the colored point cloud data. With its accurate three-dimensional space coordinates and rich and accurate color information, the target colored point cloud data provides an intuitive and detailed basic spatial framework and visual feature expression for map construction.

[0038] In an optional embodiment, the color correction network in the embodiment of the present application can adopt a multi-modal deep neural network architecture to fuse the three-dimensional geometric information of the point cloud data and the color information of the image data. The specific network structure may include: an input layer, a feature extraction module, a feature fusion module, a color prediction and correction module, and an output module. The following description of the color correction network is only an example, and those skilled in the art can adjust and optimize the network structure according to actual needs to adapt to different application scenarios and data types.

[0039] Among them, the relevant introduction of the input layer and the feature extraction module is as follows: Input layer: The network receives multiple types of input data, such as three types of input data, including: image data, point cloud reflection intensity, and point cloud spatial position information.

[0040] Image feature extraction: Extract multi-scale image features (including shallow texture features and deep semantic features, etc.).

[0041] Point cloud feature extraction: Local geometric features can be extracted (for example, based on the point cloud spatial position information, local neighborhoods are constructed through farthest point sampling and ball query, and geometric attributes such as curvature and normal vector are extracted), reflection intensity features (the reflection intensity values are concatenated with the geometric features to form a point cloud feature vector), etc.

[0042] Among them, the relevant introduction of the feature fusion module is as follows: The attention mechanism can be adopted to achieve efficient fusion of image and point cloud features: Spatial alignment: Project the point cloud onto the image plane to establish point-pixel correspondence.

[0043] Image-to-point cloud attention: Calculate the weight assignment of image features to point cloud features to enhance the color information related to the point cloud geometry.

[0044] Point cloud-to-image attention: Use the point cloud reflection intensity to screen the reliable regions in the image and suppress noise interference.

[0045] Feature concatenation and dimensionality reduction: The fused features are reduced in dimension through a fully connected layer, and a fused feature vector is output.

[0046] Among them, the relevant introduction of the color prediction and correction module is as follows: Color prediction head: Consists of N layers (such as 3 layers) of fully connected layers. The fused feature vector is input, and the original RGB color values (N×3) of each point cloud point are output.

[0047] Geometric constraint unit: Introduce spatial continuity loss to constrain the smooth transition of colors between adjacent point clouds and avoid color jumps.

[0048] Reflection intensity guidance unit: Dynamically adjust the color prediction weights through the reflection intensity values. For example, reduce the brightness gain in high reflection regions and enhance the color saturation in low reflection regions.

[0049] Among them, the relevant introduction of the output module is as follows: Convert the predicted original RGB color values into a standardized point cloud format and ensure the numerical validity.

[0050] In an optional embodiment, S101 obtains initial colored point cloud data, including: Obtain the point cloud data and the image data; perform fusion processing on the point cloud data and the image data to generate the initial colored point cloud data; wherein, the fusion processing includes data fusion and / or feature fusion.

[0051] For the relevant descriptions of the point cloud data and the image data, refer to the relevant descriptions in the previous embodiment, which will not be elaborated here.

[0052] In the embodiments of the present application, a data-level fusion method and a feature-level fusion method can be adopted to generate the initial colored point cloud data.

[0053] Among them, the data-level fusion method includes: through sensor calibration, align the coordinate systems of the point cloud data and the image data; map the pixel values in the image data to each point cloud point to generate the initial colored point cloud data. Specifically, through sensor calibration, align the point cloud data and the image data to the same coordinate system. Then, map the color information (i.e., RGB values) in the image data to each point of the point cloud. For example, according to the projection position of the point cloud point in the image, directly assign the corresponding RGB value. Each point cloud point not only contains the spatial position but also additional color information, forming a colored point cloud.

[0054] Among them, the feature-level fusion method includes: extract features from the point cloud data to obtain point cloud features; extract features from the image data to obtain image features; fuse the point cloud features and the image features to form a colored point cloud. Optionally, use a convolutional neural network to extract the features of the point cloud and the image, and calculate the weights between the features through an attention mechanism to achieve feature fusion and obtain a colored point cloud.

[0055] In an alternative embodiment, before data acquisition, sensors such as a fixedly installed lidar and a camera must be accurately calibrated. The purpose of calibration is to establish an accurate spatial correspondence between the point cloud data and the image data, so as to ensure that the point cloud can be correctly projected onto the image plane.

[0056] Among them, the internal parameter calibration of the camera can be completed by Zhang Zhengyou calibration method or a calibration method based on deep learning, etc., to accurately obtain the internal parameters of the camera, including focal length, optical center position, distortion coefficient, etc.

[0057] Among them, the internal parameter calibration of the lidar can be completed based on a calibration method of a target or a calibration method based on deep learning, etc., to accurately obtain the internal parameters of the lidar, including scanning angle deviation, distance deviation, point cloud reflection intensity calibration, etc.

[0058] Among them, for the extrinsic calibration between the camera and the lidar, the manual selection of control points calibration method or the checkerboard calibration method can be adopted. The above methods construct the spatial correspondence relationship between the point cloud data and the image data by determining the relative position and attitude relationship between the two sensors. The result of the extrinsic calibration enables the point cloud data of the lidar to be accurately projected onto the image plane of the camera, thus providing a basis for subsequent multi-sensor data fusion.

[0059] In an optional embodiment, the vehicle equipped with sensors travels in the scene at a low speed (such as within 40 km / h), and collects sensor data such as lidar and camera. During the data collection process, it is necessary to synchronize the data between different sensors to generate the matching relationship between different sensor data. Specifically, based on the time stamp of the lidar, search for the data frames of other sensors within the first half and the second half of the sampling period. Select the data frame with the closest time for synchronization. This can ensure that the time interval between the data frames of other sensors and the current lidar data frame is minimized, avoiding difficulties in data alignment caused by too large a time interval.

[0060] In an optional embodiment, after collecting the above data, it is also necessary to perform projection motion compensation. Specifically, obtain the motion information of the object; calculate the relative time difference between the point cloud data and the image data of each camera; calculate the offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference; adjust the point cloud data based on the offset to generate the adjusted point cloud data.

[0061] Among them, the motion information (including the speed, acceleration, and angle of the vehicle, etc.) is obtained from the Inertial Measurement Unit (IMU) and the Global Positioning System (GPS).

[0062] In the embodiments of the present application, the relative time difference between the current point cloud data and the image data of different cameras is calculated. Based on this time difference and the obtained motion information, the motion change of the vehicle during this period is deduced, and then the offset of the point cloud data relative to the image data of each camera is determined. In addition, in order to more accurately predict the motion speed and angle of the vehicle, the interpolation method is used to fill the time gap, improve the accuracy of motion compensation, and obtain more accurate motion information. For example, when the motion states of the vehicle at two time points are known, the vehicle state at any moment between these two time points can be estimated through interpolation. After that, according to the calculated offset and the accurate motion information obtained by the interpolation method, the position of the point cloud data is adjusted to compensate for the offset caused by the vehicle motion. This can ensure that the point cloud data and the image data are more accurately aligned in space, providing a reliable basis for subsequent multi-sensor data fusion. Through the above steps, the data alignment problem caused by sensor data asynchronization and vehicle motion can be effectively solved, improving the accuracy and reliability of environmental perception in the autonomous driving system.

[0063] After that, the motion-compensated point cloud data is projected onto the image plane through the internal and external parameters of the camera. The specific steps are as follows: Use the external parameters (rotation matrix and translation vector) to transform the point cloud data from the lidar coordinate system to the camera coordinate system. Use the internal parameters of the camera to project the point cloud data from the camera coordinate system onto the pixel coordinate system of the image plane. Establish the mapping relationship between each point in the point cloud data and different image pixel points. Extract the RGB values of the corresponding pixel points from the image data. Assign the extracted RGB values to the corresponding point cloud points, thereby adding initial color information to the point cloud, that is, obtaining the initial colored point cloud data.

[0064] It should be noted that in a multi-view camera system, each camera covers a different viewing range. To achieve comprehensive environmental perception, it is necessary to fuse the point cloud data with the image data of multiple cameras. The specific steps are as follows: Project the point cloud data onto the image plane of each camera respectively, and establish the mapping relationship between the point cloud points and the image pixel points of each camera. Extract the RGB values of the corresponding pixel points from the image data of each camera, and fuse the color information according to certain rules (such as average value, weighted average value, or select the clearest view) to assign more accurate color information to the point cloud points.

[0065] In an alternative embodiment, the training process of the color correction network includes: Step a1, obtain the test image data and the test colored point cloud data corresponding to the test image data.

[0066] Among them, the test image data refers to the image data that has been subjected to specific processing or not, and is used to evaluate and verify the performance of the color correction network. These data usually come from actual application scenarios or are generated by simulation to ensure that they can reflect various conditions and changes in the real environment.

[0067] In the embodiment of the present application, by obtaining the target image data, performing a simulation transformation operation on the target image data to generate the test image data; Among them, the simulation transformation operation includes at least one of the following: linear mapping operation, gamma transformation operation, light source simulation operation, frequency domain adjustment operation, and adversarial network generation operation. Among them, it is not limited to the above simulation transformation operations, and other related operations are also included in the protection scope of the present application.

[0068] Among them, the target image data refers to the image data collected under normal lighting conditions, and these data are used as preset label data. They provide a benchmark for the training and optimization of the color correction network, are used to evaluate the test results output by the network, and guide the adjustment and optimization of the network.

[0069] In the embodiment of the present application, through diverse image simulation transformation operations, different lighting conditions, environmental changes, or sensor characteristics, etc. are simulated, so as to provide more challenging and diverse data for the training of the color correction network. Specifically, the simulation transformation operation includes at least one of the following: Linear mapping operation, by adjusting the pixel value range of the image, changing the brightness and contrast of the image. This operation can simulate the image effects under different exposure conditions and enhance the network's adaptability to brightness changes.

[0070] Gamma transformation operation, used to change the brightness distribution of the image. By adjusting the gamma value of the image, the dark details or highlights of the image are enhanced. This operation can simulate the image effects under different light intensities and improve the network's robustness to light changes.

[0071] Light source simulation operation, by changing the lighting direction, lighting intensity, and light source color of the image, simulating the image effects under different lighting environments. For example, it can simulate the scenes under direct sunlight, overcast diffused reflection, or artificial light sources. This operation can not only change the brightness and contrast of the image, but also introduce shadow and reflection effects, enhancing the network's adaptability to complex lighting conditions.

[0072] Frequency domain adjustment operation, by processing the spectrum of the image, changing the frequency distribution of the image. For example, by high-pass filtering to enhance the edge details of the image, or by low-pass filtering to smooth the image. This operation can simulate the imaging characteristics of different sensors or the influence of environmental noise, and improve the network's robustness to different imaging conditions.

[0073] Adversarial network generation operation, using a generative adversarial network to generate realistic image data. Through the adversarial training of the generator and discriminator, synthetic images that are indistinguishable from real images are generated. This operation can simulate the image effects under extreme lighting conditions, rare scenarios, or sensor failures, enhancing the network's generalization ability for complex scenarios.

[0074] In the embodiments of the present application, the test image data generated through the above simulation transformation operation can provide diverse inputs for the training of the color correction network. These data not only cover images under normal lighting conditions but also include images in complex scenarios such as strong light, low light, and shadows. By training on these diverse data, the color correction network can learn a wider range of color correction strategies, thus better adapting to different lighting conditions and sensor characteristics.

[0075] Step a2, input the test image data and the test color point cloud data into the initial color correction network, and output a test result.

[0076] Among them, the test result refers to the corrected color point cloud data output after the initial color correction network processes the test image data and the test color point cloud data.

[0077] In the embodiments of the present application, after obtaining the test image data, perform data fusion in the above embodiments on the test image data to generate test color point cloud data. Input the test image data (including image data under different lighting conditions such as overexposure and low light from multiple perspectives) and the test color point cloud data (including point cloud spatial position information, point cloud reflection intensity, and initial color information) into the initial color correction network, and the network outputs a test result after feature extraction and fusion processing.

[0078] Step a3, optimize the initial color correction network based on the test result and preset label data to obtain the color correction network.

[0079] Among them, the preset label data refers to the image data collected under normal lighting conditions, and these data are used as real labels. They reflect the real color information of the scene and are the basis for evaluating and optimizing the color correction network. Optionally, the preset label data is the target image data mentioned above.

[0080] In the embodiments of the present application, by calculating the error (such as mean square error, structural similarity index, etc.) between the test result and the label data, evaluate the accuracy of the test result, optimize the initial color correction network, and after multiple iterations of optimization, finally obtain a color correction network that can accurately correct color deviations.

[0081] In the prior art, the following problems still exist in the point cloud color assignment technology: Lack of spatial information: The color correction algorithm only predicts the color under normal lighting at the image data level, but ignores the spatial information, resulting in poor prediction accuracy in some areas.

[0082] Insufficient input information: Existing color correction methods only rely on images of strong light and weak light for prediction, lacking other reliable information for assistance, resulting in inaccurate color prediction results.

[0083] To solve the above problems, an embodiment of the present application also provides a method for correcting color point cloud data. The initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information. The point cloud reflection intensity, the point cloud spatial position information, and the image data are input into the color correction network to output the target color point cloud data.

[0084] Among them, the point cloud reflection intensity is the reflectivity information of each point recorded in the point cloud data for the laser pulse. The point cloud reflection intensity has the following uses in the color correction network: When the point cloud data has no RGB color information, the point cloud reflection intensity can provide a visualization effect similar to a black-and-white photo for the point cloud. Based on the point cloud reflection intensity, the point cloud data can be classified or noise-filtered to remove points within a specific intensity range. The point cloud reflection intensity can be used as an auxiliary feature to help the network better understand the object materials and structures in the scene.

[0085] Among them, in the color correction network, the point cloud spatial position information has the following uses: It provides an accurate three-dimensional position for each point in space to help the network understand the shape and layout of the object. Through the point cloud spatial position information, the color information of the image data is projected onto the point cloud to generate a color point cloud. Combining the point cloud reflection intensity and the point cloud spatial position information can perform semantic segmentation and target detection more accurately.

[0086] In the embodiments of the present application, a multi-dimensional color correction network is utilized. This multi-dimensional color correction network takes into account three dimensions: the point cloud reflection intensity, the point cloud spatial position information, and the color information. It corrects the color information of the initial colored point cloud data in a high-dimensional space to generate accurate target colored point cloud data. Among them, by introducing the point cloud spatial position information, three-dimensional geometric constraints are provided for color assignment, making the color transition more natural. By introducing the geometric information of the point cloud data, the network can more stably output the corrected color information when dealing with lighting changes. Even under extreme lighting conditions (such as overexposure or low light), the network can utilize the geometric information to assist in color correction, enhancing the robustness of the system. For example, at the edges or shadow areas of an object, the change in color can be smoothed through the spatial position information, avoiding color jumps or discontinuities. In addition, under different poses and angles, the point cloud spatial position information can help the network predict more consistent colors. For example, when a vehicle observes the same object from different directions, the spatial position information can ensure that the color remains consistent under different viewpoints, avoiding color deviations caused by viewpoint changes. In addition, the point cloud spatial position information helps the network understand the shape and layout of the objects in the scene, thus more accurately performing color assignment. For example, the network can adjust the brightness and contrast of the color according to the surface curvature and direction of the object to make it more in line with the real scene. Through the point cloud spatial position information, shadow and reflection areas can be identified, and the color can be adjusted to simulate the real lighting effect. For example, the brightness of the color is reduced in the shadow area, and the contrast of the color is enhanced in the reflection area.

[0087] Among them, by introducing the point cloud reflection intensity, the point cloud reflection intensity provides the material characteristics of each point, helping the network more accurately predict the color. For example, points with a higher point cloud reflection intensity usually correspond to smooth or reflective surfaces, and their colors may be closer to the real colors; points with a lower point cloud reflection intensity may correspond to rough or highly absorptive surfaces, and their colors may need to be adjusted. The point cloud reflection intensity can reduce the influence of lighting conditions on color prediction. For example, under strong light or weak light conditions, the point cloud reflection intensity can help the network adjust the brightness and contrast of the color to make it closer to the real scene. The point cloud reflection intensity helps the network perceive the material characteristics of the object, thus more accurately predicting the color. For example, the point cloud reflection intensity can distinguish different materials such as metal, plastic, and vegetation, and the network can adjust the hue and saturation of the color according to this information. In addition, the point cloud reflection intensity combined with the spatial position information can enhance the network's semantic understanding of the scene. For example, the network can identify targets such as traffic signs, pedestrians, and vehicles based on the point cloud reflection intensity and spatial position information and assign more accurate colors to them.

[0088] By integrating the three-dimensional geometric information of point cloud data and the color information of image data, the color correction network can more comprehensively understand the physical characteristics of the scene. The network uses preset label data (true colors under normal lighting conditions) as the optimization target, and gradually corrects color deviations through error calculation and parameter adjustment.

[0089] For ease of understanding, a specific scenario is described here. The spatial position information (coordinates and normal vectors) of the point cloud data can be input into the network, enabling the network to perceive the local geometric structure (such as planes and curved surfaces), and avoiding color prediction errors caused by perspective occlusion in two-dimensional images. For example, when adjacent point clouds belong to the same object surface in space, their colors should transition smoothly (such as the body surface of a vehicle). The network achieves color consistency through spatial position constraints. In addition, in overexposed scenes, high-reflection intensity areas (such as metals and glass) are prone to "white overflow". The network combines the reflection intensity of the point cloud to suppress color distortion in bright areas. In low-reflection intensity areas (such as asphalt roads), details are lost in low light. By associating the reflection intensity of the point cloud with the spatial position, the true colors (such as the color of lane lines) are restored.

[0090] In summary, by introducing the spatial position information of the point cloud and the reflection intensity of the point cloud, the color correction network can significantly improve the effect of color assignment and correction. The spatial position information of the point cloud ensures the naturalness and perspective consistency of color transition, while the reflection intensity of the point cloud reduces the difficulty of color prediction and improves the accuracy of color prediction.

[0091] In an optional embodiment, the method further includes: performing semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result; and generating map data based on the target color point cloud data and the annotation result.

[0092] Among them, semantic annotation refers to assigning semantic category labels to each point or point cloud segment in the target color point cloud data. These labels can represent the categories of different objects in the point cloud data, such as roads, buildings, traffic signs, pedestrians, vehicles, etc. The main purpose of semantic annotation is to enable the machine to understand the semantic content of the point cloud data, so as to make more accurate decisions in applications such as autonomous driving and intelligent transportation.

[0093] Among them, structured annotation refers to performing more fine-grained annotation on the point cloud data, not only annotating the category of the object, but also annotating the structure and attributes of the object. For example: building annotation (annotating the height, number of floors, window and door positions of the building), road annotation (annotating the width, number of lanes, traffic sign positions of the road).

[0094] Among them, the annotation result usually exists in the form of labels, which can be the category labels of each point in the point cloud data, or the bounding box, polygon contour or other geometric attributes of the object.

[0095] In an embodiment of the present application, a high-precision map is generated based on target color point cloud data and annotation results. The high-precision map not only contains the three-dimensional geometric information of the point cloud, but also rich semantic information and structured data, which can provide more accurate environmental perception and decision-making support for autonomous driving vehicles.

[0096] In an embodiment of the present application, in order to construct a high-precision map, it is necessary to fuse color point cloud data from different times, different sensors (such as lidar, camera) or different perspectives. Specifically, it includes: aligning the data of different sensors to a unified coordinate system to ensure the consistency of the data in time and space. Using sensor calibration techniques (such as extrinsic calibration between lidar and camera) to establish the spatial relationship between sensors. Performing multi-sensor data fusion on the aligned data to generate color point cloud data. Merging the color point cloud data into a unified coordinate system and combining the annotation results to construct a high-precision map of the entire scene.

[0097] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0098] Corresponding to the color point cloud data correction method described in the above embodiments, Figure 2 The structural block diagram of the color point cloud data correction device provided by the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0099] Referring to Figure 2 , the color point cloud data correction device includes: An acquisition module, configured to acquire initial color point cloud data; wherein, the initial color point cloud data is obtained by fusing point cloud data and image data; A correction module, configured to input the image data and the initial color point cloud data into a preset color correction network and output target color point cloud data.

[0100] In a possible implementation manner, the acquisition module is configured to: Acquire the point cloud data and the image data; Perform fusion processing on the point cloud data and the image data to generate the initial color point cloud data; wherein, the fusion processing includes data fusion and / or feature fusion.

[0101] In a possible implementation manner, the color point cloud data correction device further includes a training module, configured to: Acquire test image data and corresponding test color point cloud data; Input the test image data and the test color point cloud data into the initial color correction network to output a test result; Optimize the initial color correction network based on the test result and preset label data to obtain the color correction network.

[0102] In a possible implementation, the color point cloud data correction device further includes a generation module for: Obtain target image data; Perform a simulation transformation operation on the target image data to generate the test image data; Wherein, the simulation transformation operation includes at least one of the following: linear mapping operation, gamma transformation operation, light source simulation operation, frequency domain adjustment operation, adversarial network generation operation.

[0103] In a possible implementation, the initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; the correction module is used for: Input the point cloud reflection intensity, the point cloud spatial position information, and the image data into the color correction network to output the target color point cloud data.

[0104] In a possible implementation, the color point cloud data correction device further includes a map generation module for: Perform semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result; Generate map data based on the target color point cloud data and the annotation result.

[0105] In a possible implementation, the color point cloud data correction device further includes an adjustment module for: Obtain the motion information of an object; Calculate the relative time difference between the point cloud data and the image data of each camera; Calculate the offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference; Adjust the point cloud data based on the offset to generate adjusted point cloud data.

[0106] It should be noted that the information interaction, execution process, etc. between the above modules, due to being based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, please refer to the method embodiment part specifically, and will not be elaborated here.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here.

[0108] An embodiment of this application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the foregoing method embodiments are implemented.

[0109] An embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each of the foregoing method embodiments can be implemented.

[0110] An embodiment of this application provides a computer program product. When the computer program product runs on a computer device, the computer device is caused to implement the steps in each of the foregoing method embodiments.

[0111] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0112] In the above embodiments, the descriptions of the various embodiments have their own focuses. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this document can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed device / computer device and method can be implemented in other ways. For example, the device / computer device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical, or other form.

[0115] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0117] Figure 3 The structural schematic diagram of a computer device provided in an embodiment of the present application. As Figure 3 shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 3 only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20. When the processor 20 executes the computer program 22, the steps in any of the above-described embodiments of the color point cloud data correction method are implemented.

[0118] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art can understand that Figure 3 this is only an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0119] The so-called processor 20 may be a central processing unit (CPU), and this processor 20 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0120] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 21 may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computer device. Further, the memory 21 may also include both the internal storage unit and the external storage device of the computer device. The memory 21 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of the computer program. The memory 21 may also be used to temporarily store data that has been output or will be output.

[0121] In each embodiment of the present application, the relevant user personal information that may be involved is all processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, and for reasonable purposes based on business scenarios, for the personal information actively provided by the user during the use of the product / service or generated due to the use of the product / service, as well as the personal information obtained with the user's authorization.

[0122] The user personal information processed by the applicant may vary depending on the specific product / service scenario, and it is subject to the specific scenario of the user's use of the product / service. It may involve the user's account information, device information, driving information, vehicle information, or other relevant information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0123] The applicant attaches great importance to the security of user personal information and has taken security protection measures that meet industry standards and are reasonable and feasible to protect the user's information and prevent the personal information from being accessed, publicly disclosed, used, modified, damaged, or lost without authorization.

Claims

1. A color point cloud data correction method, characterized in that: include: Acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; The image data and the initial color point cloud data are input into a preset color correction network, and target color point cloud data are output.

2. The color point cloud data correction method according to claim 1, characterized in that: The obtaining of initial color point cloud data comprises: Acquire the point cloud data and the image data; The point cloud data and the image data are fused to generate the initial color point cloud data; wherein the fusion process includes data fusion and / or feature fusion.

3. The color point cloud data correction method according to claim 1, characterized in that: The training process of the color correction network includes: Acquire test image data and test color point cloud data corresponding to the test image data; Input the test image data and the test color point cloud data into an initial color correction network, and output a test result; The initial color correction network is optimized based on the test results and preset label data to obtain the color correction network.

4. The color point cloud data correction method according to claim 3, characterized in that: The method further comprises: Acquire target image data; Performing a simulation transformation operation on the target image data to generate the test image data; Among them, the analog transformation operation includes at least one of the following: a linear mapping operation, a gamma transformation operation, a light source simulation operation, a frequency domain adjustment operation, and an adversarial network generation operation.

5. The color point cloud data correction method according to any one of claims 1 to 4, characterized in that: The initial color point cloud data includes point cloud reflection intensity and point cloud spatial position information; The step of inputting the image data and the initial color point cloud data into a preset color correction network and outputting target color point cloud data comprises: The point cloud reflection intensity, the point cloud spatial position information and the image data are input into the color correction network, and the target color point cloud data is output.

6. The color point cloud data correction method according to claim 5, characterized in that: The method further comprises: Performing semantic annotation and / or structured annotation on the target color point cloud data to generate an annotation result; Map data is generated based on the target color point cloud data and the annotation result.

7. The color point cloud data correction method according to claim 5, characterized in that: The method further comprises: Get the motion information of the object; Calculating the relative time difference between the point cloud data and the image data of each camera; Calculate the offset of the point cloud data relative to the image data of each camera based on the motion information and the relative time difference; The point cloud data is adjusted based on the offset to generate adjusted point cloud data.

8. A color point cloud data correction device, characterized in that: include: An acquisition module, used for acquiring initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; The correction module is used to input the image data and the initial color point cloud data into a preset color correction network and output target color point cloud data.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer program product, characterized in that When the computer program product is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 7.

Images