Enhanced low-illumination point cloud coloring method and device and electronic equipment

Abstract

The invention provides an enhanced low-illumination point cloud coloring method and device and electronic equipment. The method comprises the following steps: acquiring first image data and first point cloud data in a low-illumination scene, and matching the first image data and the first point cloud data to obtain second image data and second point cloud data; based on a target image enhancement model, performing image enhancement on the second image data to obtain target image data, and performing point cloud enhancement on the second point cloud data to obtain target point cloud data; fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data; and coloring the target point cloud data based on the color information. Therefore, according to the scheme, the quality and the accuracy of the point cloud data can be effectively improved, the application effect in various low-illumination scenes is ensured, and the safety and the reliability of operation in the low-illumination scenes are further ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an enhanced low-light point cloud coloring method, device and electronic device. Background Art

[0002] LiDAR is widely used because it can provide accurate, reliable and detailed spatial data. Its key advantage is that it can produce accurate distance measurements under various lighting conditions. However, a significant limitation of the point clouds generated by LiDAR is that they lack color information, which not only reduces the visual appeal, but also leads to its inadequate performance in perception applications that require texture information and complex scene interpretation, which greatly limits its usability in low-light scenarios, such as at night or in darker environments. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of the present application is to propose an enhanced low-light point cloud coloring method to improve the application effect of point cloud data in low-light scenes.

[0005] The second objective of this application is to propose an enhanced low-light point cloud coloring device.

[0006] The third objective of the present application is to provide an electronic device.

[0007] To achieve the above-mentioned objectives, the first aspect embodiment of the present application proposes an enhanced low-light point cloud coloring method, including: collecting first image data and first point cloud data in a low-light scene, and matching the first image data and the first point cloud data to obtain at least one matching data pair, the matching data pair including second image data and second point cloud data; based on a pre-trained target image enhancement model, performing image enhancement on the second image data to obtain target image data, and performing point cloud enhancement on the second point cloud data to obtain target point cloud data; fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data; and coloring the target point cloud data based on the color information.

[0008] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes an enhanced low-light point cloud coloring device, including: a matching module, used to collect first image data and first point cloud data in a low-light scene, and match the first image data and the first point cloud data to obtain at least one matching data pair, wherein the matching data pair includes second image data and second point cloud data; an enhancement module, used to perform image enhancement on the second image data based on a pre-trained target image enhancement model to obtain target image data, and to perform point cloud enhancement on the second point cloud data to obtain target point cloud data; a fusion module, used to fuse the target image data and the target point cloud data to obtain the color information of the target point cloud data from the target image data; and a coloring module, used to color the target point cloud data based on the color information.

[0009] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor can execute the enhanced low-light point cloud coloring method described in the first aspect embodiment above.

[0010] The enhanced low-light point cloud coloring method, device and electronic device provided in the present application achieve the acquisition of high-quality second image data and second point cloud data by acquiring first image data and first point cloud data, and matching the first image data and the first point cloud data. Furthermore, the second image data and the second point cloud data are enhanced respectively to obtain target image data and target point cloud data, and the target image enhancement data and the target point cloud enhancement data are fused to obtain the color information of the target point cloud data, and the target enhanced point cloud data is colored based on the color information. The method of the embodiment of the present application can effectively improve the quality and accuracy of point cloud data, ensure the application effect in a variety of low-light scenes, so that the colored point cloud data can be reliably applied under various dim conditions, thereby ensuring the safety and reliability of operations in low-light scenes.

[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A schematic diagram of a flow chart of an enhanced low-light point cloud coloring method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the layout of the image sensor and radar sensor provided in the embodiment of the present application;

[0015] Figure 3 A schematic diagram of a process of training a target image enhancement model in an enhanced low-light point cloud colorization method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of a flow chart of another enhanced low-light point cloud coloring method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of a flow chart of another enhanced low-light point cloud coloring method provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of a process for coloring low-light point cloud data provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of the structure of an enhanced low-light point cloud coloring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0021] The following describes the enhanced low-light point cloud coloring method and device of the embodiments of the present application with reference to the accompanying drawings.

[0022] The enhanced low-light point cloud coloring method proposed in the embodiment of the present application can be applied to application scenarios such as autonomous driving vehicles and industrial monitoring. It can also be applied to underground dim passage scenes, special and complex scenes in coal mines, geography or urban planning and other fields.

[0023] Figure 1 is a flowchart of an enhanced low-light point cloud coloring method according to an exemplary embodiment. Figure 1 As shown, the enhanced low-light point cloud coloring method of the embodiment of the present application includes but is not limited to the following steps:

[0024] S101, collecting first image data and first point cloud data in a low-light scene, and matching the first image data and the first point cloud data to obtain at least one matching data pair, where the matching data pair includes second image data and second point cloud data.

[0025] In some implementations, the first image data and the first point cloud data in a low-light scene may be collected based on an image sensor and a radar sensor. For example, the first image data may be collected using a visual camera, the first point cloud data may be collected using a mechanical laser radar, or the first point cloud data may be collected using a solid-state laser radar.

[0026] In some implementations, in order to ensure that the image sensor and the radar sensor have overlapping perception fields to focus on the expected perception target object, the image sensor can be placed above the radar sensor to collect the first image data and the first point cloud data of the overlapping perception fields. Figure 2 The layout diagram of the image sensor and radar sensor shown in the figure is as follows: Figure 2 The image sensor is a camera, and the radar sensor is a lidar. Figure 2 In the example, the camera is placed above the lidar, and both have fields of view facing outward toward the same perceived interest point target object and have overlapping perception fields of view.

[0027] Furthermore, the first image data and the first point cloud data can be matched based on the time information of collecting the first image data and the first point cloud data. That is, before collecting data, the clocks of the image sensor and the radar sensor need to be synchronized so that the image sensor and the radar sensor collect data at the same time point. Optionally, the internal clocks of the sensor components can be aligned so that the data of each sensor component is associated in terms of time flow rate to obtain an accurate timestamp of the sensor data.

[0028] Optionally, in order to avoid distortion of the first image data, the internal parameters of the image sensor may be calibrated in advance. Taking a visual camera as an example, the visual camera internal parameter matrix may be obtained to correct the visual camera data. The visual camera internal parameter matrix includes focal length, optical center, distortion coefficient, and slope coefficient.

[0029] In some implementations, the time difference between the first image data and the first point cloud data can be determined based on the time information of the first image data and the first point cloud data, and the first image data and the first point cloud data whose time difference is less than or equal to a set threshold are matched to obtain the second image data and the second point cloud data, and the first image data and the first point cloud data that cannot be matched are discarded.

[0030] S102, based on the pre-trained target image enhancement model, performing image enhancement on the second image data to obtain target image data, and performing point cloud enhancement on the second point cloud data to obtain target point cloud data.

[0031] In some implementations, the target image data may be output by inputting the second image data to be enhanced into the target image enhancement model. Optionally, the target image enhancement model may be constructed based on a deep learning model.

[0032] It is understandable that the deep learning model is mainly composed of an encoder and a decoder. The encoder is used to capture the contextual visual information of the input image and reduce the spatial size of the input image through a series of convolution and pooling operations.

[0033] The decoder, in response to the encoder, is used to reconstruct and output the rich visual features learned by the encoder. In actual operation, the decoder sequentially upsamples the compressed visual feature map and extracts the corresponding amplified visual features from a series of convolutional layers, thereby ensuring that the final output image can reflect the complex visual features in the input image, such as texture features.

[0034] At the same time, the deep learning model further improves the model performance by adding a series of operations such as cross-attention layers, self-attention layers, dense connections, residual connections, and regularization processing, thereby enhancing the ability to focus on and retain key features of the input visual image, ensuring that these visual information details can be fully utilized in the image enhancement process, thereby completing the generation of high-quality visually enhanced images.

[0035] That is to say, in the process of image enhancement by the target image enhancement model, the encoder can be used to extract and characterize the basic features and structural features of the second image data, and then the decoder can be used to generate target image data with more visual information details, thereby completing image enhancement processing of the second image data input in a low-light scene.

[0036] In some implementations, the noise of the second point cloud data may be processed to enhance the point cloud data. Optionally, the target point cloud data may be obtained by detecting the noise value in the second point cloud data and filtering the point cloud data with a noise value greater than a set noise threshold.

[0037] Optionally, in order to improve the overall detail feature information of the low-light scene, after filtering the noise, the area where the point cloud density is lower than the set density threshold may be filled to obtain the target point cloud data.

[0038] S103, fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data.

[0039] In some implementations, the target point cloud data may be fused with the pixels in the target image data, and the color information of the target point cloud data may be determined from the fused pixels. Alternatively, the target point cloud data may be converted into two-dimensional point cloud data and fused with the pixels.

[0040] Optionally, the target point cloud data is converted to the plane where the target image data is located to obtain two-dimensional point cloud data. The target point cloud data can be projected onto the plane where the target image data is located according to a conversion matrix between the target point cloud data and the target image data. The conversion matrix includes a relative rotation matrix and a translation matrix.

[0041] In some implementations, the color information of the target point cloud data can be obtained by obtaining pixel information of the pixel point and performing interpolation calculation on the color channel based on the pixel information. The color information can be a color target channel value. For example, if the color channel is an RGB channel, the color information is an R value, a G value, and a B value.

[0042] S104: Coloring the target point cloud data based on the color information.

[0043] In some implementations, the color channel value of the target point cloud data can be adjusted based on the color information to achieve coloring of the target point cloud data. The color target channel value is obtained from the color information, and the color channel value of the target point cloud data is adjusted to the color target channel value to obtain the colorized target point cloud data.

[0044] In the enhanced low-light point cloud coloring method provided in the embodiment of the present application, by acquiring the first image data and the first point cloud data, and matching the first image data and the first point cloud data, high-quality second image data and second point cloud data are acquired. Furthermore, the second image data and the second point cloud data are enhanced respectively to obtain the target image data and the target point cloud data, and the target image enhancement data and the target point cloud enhancement data are fused to obtain the color information of the target point cloud data, and the target enhanced point cloud data is colored based on the color information. The method of the embodiment of the present application can effectively improve the quality and accuracy of the point cloud data, ensure the application effect in a variety of low-light scenes, so that the colored point cloud data can be reliably applied under various dim conditions, thereby ensuring the safety and reliability of operations in low-light scenes.

[0045] Based on the above embodiments, the present application embodiment can explain the training process of the target image enhancement model, such as Figure 3 As shown, the training process of the target image enhancement model in the embodiment of the present application includes but is not limited to the following steps:

[0046] S301 , obtaining first sample image data, where the first sample image data includes a sample low-light image and a sample enhanced image corresponding to the sample low-light image.

[0047] In some implementations, any image and its corresponding enhanced image can be selected from an image library as first sample image data, and / or, images of the same scene under different lighting conditions can be captured by an image acquisition device as first sample image data, and / or, the sample low-light image can be manually enhanced to obtain a sample enhanced image.

[0048] S302 , normalizing the first sample image data, and padding the processed first sample image data to obtain second sample image data.

[0049] Optionally, an initial image enhancement model can be constructed based on a deep learning model.

[0050] In some implementations, in order to improve the learning efficiency of the image enhancement model during the training process and ensure the stability of the first sample image data, the first sample image data may be normalized. Optionally, the first sample image data may be normalized to obtain image data whose pixel values ​​fall within the range of 0-1.

[0051] Further, the processed first sample image data may be padded to ensure that the image enhancement model can process input images of any size. Optionally, the first sample image data may be padded by adding additional pixels around the edge or border of the processed first sample image data to obtain the second sample image data.

[0052] S303: Use the second sample image data to train the initial image enhancement model until the training is completed to obtain a target image enhancement model.

[0053] In some implementations, the second sample image data may be divided into a training set, a validation set, and a test set, wherein the training set is used for model training and learning; the validation set is used for parameter adjustment and preventing overfitting training; and the test set is used for evaluating the performance and generalization ability of the model.

[0054] Optionally, the training set is input into the initial image enhancement model, the corresponding predicted enhanced image is output, and the loss function of the image enhancement model is calculated based on the predicted enhanced image and the sample enhanced image in the validation set, and the model parameters of the image enhancement model are adjusted based on the loss function, and the adjusted image enhancement model is continuously trained until the training is completed to obtain the target image enhancement model. The model parameters include but are not limited to parameters such as the number of training iterations, the learning rate, and the batch size.

[0055] Optionally, mean square error (MSE), root mean square error (RMSE) and mean absolute error (MAE) can be used as loss functions. Optionally, a development loss function can also be defined to adapt to specific functional requirements.

[0056] Furthermore, after calculating the loss function, an optimizer can be used to adjust the parameters of the model to minimize the loss. For example, an Adam optimizer can be used for optimization adjustment.

[0057] Optionally, the training may end when the number of training times reaches a set number of training iterations, or the training may end when the accuracy of the model output prediction enhanced image reaches a set threshold.

[0058] In some implementations, after the image data in the training set is trained, the performance of the image enhancement model can be tested using a test set, and after the performance passes the test, a target image enhancement model is obtained to enhance the second image data according to the target image enhancement model.

[0059] In the enhanced low-light point cloud colorization method provided in the embodiment of the present application, an initial image enhancement model is constructed based on a deep learning model, and the model is trained using sample image data to obtain a target image enhancement model, so that the model can enhance its attention to and retention capabilities of key features of the input image data, thereby completing the generation of high-quality enhanced images.

[0060] Figure 4 is a flowchart of an enhanced low-light point cloud coloring method according to an exemplary embodiment. Figure 4 As shown, the enhanced low-light point cloud coloring method of the embodiment of the present application includes but is not limited to the following steps:

[0061] S401, collecting first image data and first point cloud data in a low-light scene, and matching the first image data and the first point cloud data to obtain at least one matching data pair, where the matching data pair includes second image data and second point cloud data.

[0062] In the embodiment of the present application, the method of collecting the first image data and the first point cloud data in step S401 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0063] In some implementations, before matching the first image data with the first point cloud data, the blurred image data in the first image data may be filtered to obtain clear focused image data. Optionally, blur detection may be performed on the first image data to obtain the variance values ​​of the first image data, and if the variance value is greater than or equal to the variance threshold, the first image data corresponding to the variance value is determined to be focused image data, so as to filter out the focused image data from the first image data, that is, the blurred image data in the first image data is discarded to filter out the focused image data.

[0064] Optionally, blur detection may be performed on the second image data based on Laplace operator variance. Blur detection may also be performed on the second image data using Fourier transform, edge detection, contrast measurement, or machine learning or deep learning methods.

[0065] Furthermore, the image data and the point cloud data can be matched according to the time information of the focused image data and the first point cloud data. The second image data and the second point cloud data are obtained by obtaining the first time information of the focused image data and the second time information of the first point cloud data, and performing time matching on the focused image data and the first point cloud data based on the first time information and the second time information.

[0066] Optionally, a first timestamp of the focused image data can be determined based on the first time information, and a second timestamp of the first point cloud data can be determined based on the second time information, and the time difference between the first timestamp and the second timestamp can be calculated, and the focused image data and the first point cloud data whose time difference is less than or equal to a set threshold are matched to obtain second image data and second point cloud data.

[0067] S402: Based on the size requirement of the target image enhancement model, perform image padding on the second image data to obtain third image data.

[0068] In some implementations, image filling can be achieved by adding additional pixels around the edge or border of the second image data until the image is filled to meet the size requirements of the target image enhancement model, and the second image data that meets the model size requirements is used as the third image data.

[0069] S403, inputting the third image data into the target image enhancement model, the target image enhancement model extracts features from the third image data, and downsamples the extracted feature data to obtain reduced-dimensional feature data.

[0070] In some implementations, by inputting the third image data into the target image enhancement model, the target image enhancement model can perform feature extraction on the third image data to extract feature data such as color, texture, shape, etc. Optionally, feature extraction can be performed by Convolutional Neural Networks (CNN).

[0071] Further, the extracted feature data is downsampled to obtain reduced-dimensional feature data. Optionally, the feature data may be downsampled by an encoder in the target image enhancement model to obtain reduced-dimensional feature data.

[0072] S404, reconstructing the dimension-reduced feature data to obtain image enhancement data, and performing inverse padding on the image enhancement data to obtain target image data having the same size as the second image data.

[0073] Optionally, the decoder in the target image enhancement model can reconstruct the reduced-dimensional feature data to obtain image enhancement data, that is, the decoder is responsible for restoring the features compressed by the encoder to the spatial resolution of the original image.

[0074] Optionally, the decoder can reconstruct the reduced-dimensional feature data by upsampling and feature fusion. For example, upsampling can be performed by deconvolution, bilinear interpolation, etc., and the feature data obtained by upsampling is fused with the reduced-dimensional feature data output by the encoder to reconstruct the reduced-dimensional feature data.

[0075] S405, identifying and removing abnormal fluctuating point cloud data in the second point cloud data to obtain third point cloud data.

[0076] In some implementations, the single point and cluster outliers in the second point cloud data can be removed, and the background information of the second point cloud data can be eliminated to remove abnormal fluctuating point cloud data in the second point cloud data, so as to obtain the third point cloud data, so as to ensure the accuracy and reliability of the point cloud data and avoid the appearance of objects in the point cloud data from being erroneously magnified.

[0077] S406, obtaining an abnormality threshold and a noise threshold, and in response to the abnormal value in the third point cloud data being greater than or equal to the abnormality threshold, and the noise value being greater than or equal to the noise threshold, performing secondary filtering on the third point cloud data to obtain fourth point cloud data.

[0078] In some implementations, the anomalies and noise in the third point cloud data can be filtered twice by threshold filtering to obtain more accurate and reliable fourth point cloud data. The third point cloud data can also be tested for abnormal features and noise features, and the abnormal features and noise features can be removed to filter the anomalies and noise twice.

[0079] Optionally, by setting an abnormality threshold and a noise threshold, and determining whether the abnormal value and the noise value in the third point cloud data are greater than or equal to the corresponding abnormality threshold and the noise threshold, if the abnormal value is greater than or equal to the abnormality threshold, and the noise value is greater than or equal to the noise threshold, the point cloud data greater than the threshold is removed to obtain the fourth point cloud data.

[0080] S407, determining an area where the point cloud density is lower than a density threshold from the fourth point cloud data, performing point cloud filling on the area, and obtaining target point cloud data.

[0081] In some implementations, in order to improve the overall detail feature information of the low-light scene while keeping the point cloud data structure unchanged, the sparse area of ​​the point cloud in the fourth point cloud data may be filled. Optionally, the sparse area may be determined based on a density threshold of the point cloud density. If there is an area in the fourth point cloud data where the point cloud density is lower than the density threshold, the area is a sparse area, and the point cloud in the area is filled to obtain the target point cloud data.

[0082] S408, fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data.

[0083] In the embodiment of the present application, the implementation method of step S408 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0084] S409, coloring the target point cloud data based on the color information.

[0085] In the embodiment of the present application, the implementation method of step S409 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0086] In the enhanced low-light point cloud coloring method provided in the embodiment of the present application, by acquiring the first image data and the first point cloud data, and matching the first image data and the first point cloud data, high-quality second image data and second point cloud data are acquired. Furthermore, the second image data is enhanced according to the target image enhancement model to obtain the target image data, and the second point cloud data is filtered to obtain the target point cloud data, and the target image enhancement data and the target point cloud enhancement data are fused to obtain the color information of the target point cloud data, and the target enhanced point cloud data is colored based on the color information. The method of the embodiment of the present application can effectively improve the quality and accuracy of the point cloud data, ensure the application effect in a variety of low-light scenes, so that the colored point cloud data can be reliably applied under various dim conditions, thereby ensuring the safety and reliability of operations in low-light scenes.

[0087] Figure 5 is a flowchart of an enhanced low-light point cloud coloring method according to an exemplary embodiment. Figure 5 As shown, the enhanced low-light point cloud coloring method of the embodiment of the present application includes but is not limited to the following steps:

[0088] S501, collecting first image data and first point cloud data in a low-light scene, and matching the first image data and the first point cloud data to obtain at least one matching data pair, where the matching data pair includes second image data and second point cloud data.

[0089] In the embodiment of the present application, the implementation method of step S501 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0090] S502, based on the pre-trained target image enhancement model, perform image enhancement on the second image data to obtain target image data, and perform point cloud enhancement on the second point cloud data to obtain target point cloud data.

[0091] In the embodiment of the present application, the implementation method of step S502 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0092] S503, obtaining a transformation matrix between the point cloud data and the image data, and based on the transformation matrix, mapping the target point cloud data to the plane where the target image data is located, so as to obtain coordinate information of the target point cloud data in the plane.

[0093] In some implementations, the image sensor and the radar sensor may be calibrated with external parameters to obtain a transformation matrix between the image sensor and the radar sensor, that is, between the point cloud data and the image data, wherein the transformation matrix may include a relative rotation matrix and a translation matrix.

[0094] Further, the target point cloud data is mapped to the plane where the target image data is located based on the transformation matrix, and the coordinate information of the target point cloud data in the plane is obtained. Optionally, during the mapping process, a specified right-handed or left-handed coordinate system convention can be used to eliminate invalid point cloud data in the image coordinate system.

[0095] S504, based on the size information and coordinate information of the target image data, the target point cloud data is screened to obtain valid target point cloud data.

[0096] In some implementations, in order to ensure that the point cloud data falls within the valid pixel size of the image data, a filtering condition may be determined based on the size information and coordinate information of the target image data, and the target point cloud data may be filtered based on the filtering condition to obtain valid target point cloud data. For example, the filtering condition may be that the x coordinate of the projection point of the point cloud data must be a positive number less than the width of the target image data, and the y coordinate must be a positive number less than the height of the target image data.

[0097] S505, obtaining pixel points of the target image data in the plane, and fusing the pixel points and the valid target point cloud data.

[0098] In some implementations, the pixel points of the target image data in the plane are determined, and the pixel points are matched with the valid target point cloud data, and then the successfully matched pixel points and point cloud data are fused. In other words, by determining whether the valid target point cloud data matches the pixel points, if they match, the pixel points are fused with the valid target point cloud data.

[0099] Optionally, it may be determined whether the pixel point coordinates and the coordinates of the valid target point cloud data match each other. If the coordinates of the two are the same, it is determined that the match is successful.

[0100] S506, determining pixel information of the matched pixel points, and interpolating the red R channel, the green G channel, and the blue B channel based on the pixel information to obtain color information of the target point cloud data.

[0101] Optionally, by determining pixel information of a pixel point that matches the valid target point cloud data, and performing interpolation calculation on the color channel based on the pixel information, the color information corresponding to the pixel point is obtained, and the color information is used as the color information of the target point cloud data.

[0102] Optionally, the red (Red, R) channel, the green (Green, G) channel, and the blue (Blue, B) channel may be interpolated to obtain R value, G value, and B value, respectively, and the values ​​may be used as color information to color the target point cloud data using the color information.

[0103] S507: Coloring the target point cloud data based on the color information.

[0104] In the embodiment of the present application, the implementation method of step S507 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0105] In the enhanced low-light point cloud coloring method provided in the embodiment of the present application, by acquiring the first image data and the first point cloud data, and matching the first image data and the first point cloud data, high-quality second image data and second point cloud data are acquired. Further, the second image data and the second point cloud data are enhanced respectively to obtain the target image data and the target point cloud data. By acquiring the transformation matrix, and fusing the target image enhancement data and the target point cloud enhancement data based on the transformation matrix to obtain the color information of the target point cloud data, and coloring the target enhanced point cloud data based on the color information. The method of the embodiment of the present application can effectively improve the quality and accuracy of the point cloud data, ensure the application effect in a variety of low-light scenes, so that the colored point cloud data can be reliably applied under various dim conditions, thereby ensuring the safety and reliability of operations in low-light scenes.

[0106] Figure 6 Schematic diagram of the process of colorizing low-light point cloud data. Figure 6 In the process of clock synchronization and calibration, the camera and lidar are used to collect image data and point cloud data respectively. In the clock synchronization and calibration, the camera intrinsic parameters are calibrated to avoid distortion of the collected image data; the lidar-camera extrinsic parameters are calibrated to ensure that the point cloud data is correctly mapped to the corresponding pixels in the image, and the point cloud depth information is aligned with the color and texture details of the image; the clock rates of the lidar and camera are synchronized so that the camera and lidar collect data at the same time point. Furthermore, the control synchronization of the camera and lidar is triggered to collect the first image data and the first point cloud data.

[0107] By matching and aligning the first image data and the first point cloud data in time, data registration is achieved to obtain the second image data and the second point cloud data. The second image data is then input into the trained target image enhancement model to enhance the second image data to obtain the target image data, and the second point cloud data is enhanced to obtain the target point cloud data. Furthermore, based on the conversion matrix obtained during the laser radar-camera extrinsic calibration process, the target image data and the target point cloud data are fused to obtain the color information corresponding to the target point cloud data from the target image data, and the target point cloud data is colored using the color information to obtain the enhanced colored point cloud data.

[0108] Among them, the training process of the target image enhancement model includes three stages: model structure design, model training and model use. In the model structure design stage, an initial image enhancement model can be constructed based on a deep learning model. In the model training stage, the initial image enhancement model is trained to obtain the target image enhancement model by a. selecting the loss function of the model, b. selecting the optimizer, c. establishing hyperparameters, d. building a data set, e. data normalization, f. image filling, g. data set classification, and h. model evaluation. In the model use stage, the target image enhancement model is used to enhance the input second image data to obtain the target image data.

[0109] In order to implement the above embodiments, the present application also proposes an enhanced low-light point cloud coloring device.

[0110] Figure 7 A schematic diagram of the structure of an enhanced low-light point cloud coloring device provided in an embodiment of the present application.

[0111] like Figure 7 As shown, the enhanced low-light point cloud coloring device 700 includes:

[0112] A matching module 701 is used to collect first image data and first point cloud data in a low-light scene, and match the first image data and the first point cloud data to obtain at least one matching data pair, where the matching data pair includes second image data and second point cloud data;

[0113] An enhancement module 702 is used to perform image enhancement on the second image data based on a pre-trained target image enhancement model to obtain target image data, and to perform point cloud enhancement on the second point cloud data to obtain target point cloud data;

[0114] A fusion module 703 is used to fuse the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data;

[0115] The coloring module 704 is used to color the target point cloud data based on the color information.

[0116] In a possible implementation manner of an embodiment of the present application, the enhancement module 702 is also used to: obtain first sample image data, the first sample image data including a sample low-light image, and a sample enhanced image corresponding to the sample low-light image; normalize the first sample image data, fill the processed first sample image data to obtain second sample image data; and use the second sample image data to train an initial image enhancement model until the training is completed to obtain the target image enhancement model.

[0117] In a possible implementation of an embodiment of the present application, the enhancement module 702 is also used to: based on the size requirement of the target image enhancement model, perform image padding on the second image data to obtain third image data; input the third image data into the target image enhancement model, and have the target image enhancement model perform feature extraction on the third image data, and downsample the extracted feature data to obtain reduced-dimensionality feature data; reconstruct the reduced-dimensionality feature data to obtain image enhancement data, and inversely pad the image enhancement data to obtain target image data of the same size as the second image data.

[0118] In a possible implementation of an embodiment of the present application, the enhancement module 702 is also used to: identify and remove abnormal fluctuating point cloud data in the second point cloud data to obtain third point cloud data; obtain an abnormal threshold and a noise threshold, and in response to the abnormal value in the third point cloud data being greater than or equal to the abnormal threshold, and the noise value being greater than or equal to the noise threshold, perform a secondary filtering on the third point cloud data to obtain fourth point cloud data; determine an area in the fourth point cloud data where the point cloud density is lower than the density threshold, perform point cloud filling on the area, and obtain the target point cloud data.

[0119] In a possible implementation method of an embodiment of the present application, the fusion module 703 is also used to: obtain a transformation matrix between point cloud data and image data, and based on the transformation matrix, map the target point cloud data to the plane where the target image data is located to obtain coordinate information of the target point cloud data in the plane; based on the size information and the coordinate information of the target image data, filter the target point cloud data to obtain valid target point cloud data; obtain pixel points of the target image data in the plane, and fuse based on the pixel points and the valid target point cloud data.

[0120] In a possible implementation of the embodiment of the present application, the fusion module 703 is also used to: determine whether the valid target point cloud data matches the pixel point, and if so, fuse the pixel point with the valid target point cloud data.

[0121] In a possible implementation of an embodiment of the present application, the fusion module 703 is also used to: determine pixel information of matching pixel points, and interpolate the red R channel, green G channel and blue B channel based on the pixel information to obtain color information of the target point cloud data.

[0122] In a possible implementation of an embodiment of the present application, the matching module 701 is further used to: perform blur detection on the first image data to obtain the variance value of each of the first image data, and if the variance value is greater than or equal to the variance threshold, determine that the first image data corresponding to the variance value is the focused image data, so as to filter out the focused image data from the first image data; obtain first time information of the focused image data and second time information of the first point cloud data, and based on the first time information and the second time information, time-match the focused image data with the first point cloud data to obtain the second image data and the second point cloud data.

[0123] In the enhanced low-light point cloud coloring device provided in the embodiment of the present application, by acquiring the first image data and the first point cloud data, and matching the first image data and the first point cloud data, high-quality second image data and second point cloud data are acquired. Furthermore, the second image data and the second point cloud data are enhanced respectively to obtain target image data and target point cloud data, and the target image enhancement data and the target point cloud enhancement data are fused to obtain the color information of the target point cloud data, and the target enhanced point cloud data is colored based on the color information. The method of the embodiment of the present application can effectively improve the quality and accuracy of the point cloud data, ensure the application effect in a variety of low-light scenes, so that the colored point cloud data can be reliably applied under various dim conditions, thereby ensuring the safety and reliability of operations in low-light scenes.

[0124] It should be noted that the aforementioned explanation of the embodiment of the enhanced low-light point cloud coloring method is also applicable to the enhanced low-light point cloud coloring device of this embodiment, and will not be repeated here.

[0125] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0126] The collection, storage, use, processing, transmission, provision and application of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0127] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0128] This application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.

[0129] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0130] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0131] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0133] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0136] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An enhanced low-light point cloud coloring method, characterized in that: The method comprises: Collecting first image data and first point cloud data in a low-light scene, and matching the first image data and the first point cloud data to obtain at least one matching data pair, wherein the matching data pair includes second image data and second point cloud data; Based on a pre-trained target image enhancement model, performing image enhancement on the second image data to obtain target image data, and performing point cloud enhancement on the second point cloud data to obtain target point cloud data; fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data; The target point cloud data is colored based on the color information.

2. The method according to claim 1, characterized in that: The training process of the target image enhancement model includes: Acquire first sample image data, where the first sample image data includes a sample low-light image and a sample enhanced image corresponding to the sample low-light image; Normalizing the first sample image data, and padding the processed first sample image data to obtain second sample image data; The initial image enhancement model is trained using the second sample image data until the training is completed to obtain the target image enhancement model.

3. The method according to claim 2, characterized in that The step of performing image enhancement on the second image data based on the pre-trained target image enhancement model to obtain target image data includes: Based on the size requirement of the target image enhancement model, performing image padding on the second image data to obtain third image data; Inputting the third image data into the target image enhancement model, extracting features from the third image data by the target image enhancement model, and downsampling the extracted feature data to obtain dimension-reduced feature data; The dimension-reduced feature data is reconstructed to obtain image enhancement data, and the image enhancement data is inversely filled to obtain target image data having the same size as the second image data.

4. The method according to claim 1, characterized in that: The step of performing point cloud enhancement on the second point cloud data to obtain target point cloud data includes: Identify and remove abnormal fluctuation point cloud data in the second point cloud data to obtain third point cloud data; Acquire an abnormality threshold and a noise threshold, and in response to an abnormal value in the third point cloud data being greater than or equal to the abnormality threshold, and a noise value being greater than or equal to the noise threshold, perform secondary filtering on the third point cloud data to obtain fourth point cloud data; Determine an area in which the point cloud density is lower than a density threshold from the fourth point cloud data, perform point cloud filling on the area, and obtain the target point cloud data.

5. The method according to claim 1, characterized in that The fusing the target image data and the target point cloud data comprises: Acquire a conversion matrix between the point cloud data and the image data, and based on the conversion matrix, map the target point cloud data to the plane where the target image data is located, so as to obtain coordinate information of the target point cloud data in the plane; Based on the size information and the coordinate information of the target image data, the target point cloud data is screened to obtain valid target point cloud data; Pixel points of the target image data in the plane are acquired, and fusion is performed based on the pixel points and the valid target point cloud data.

6. The method according to claim 5, characterized in that The fusing based on the pixel points and the valid target point cloud data includes: Determine whether the valid target point cloud data matches the pixel point, and if so, fuse the pixel point with the valid target point cloud data.

7. The method according to claim 6, characterized in that The acquiring color information of the target point cloud data from the target image data comprises: Pixel information of the matched pixel points is determined, and based on the pixel information, a red R channel, a green G channel, and a blue B channel are interpolated respectively to obtain color information of the target point cloud data.

8. The method according to claim 1, characterized in that The matching of the first image data and the first point cloud data includes: Performing blur detection on the first image data to obtain variance values ​​of the first image data, and if the variance value is greater than or equal to a variance threshold, determining that the first image data corresponding to the variance value is focused image data, so as to filter out the focused image data from the first image data; The first time information of the focused image data and the second time information of the first point cloud data are obtained, and based on the first time information and the second time information, the focused image data and the first point cloud data are time-matched to obtain the second image data and the second point cloud data.

9. An enhanced low-light point cloud coloring device, characterized in that: The device comprises: a matching module, configured to collect first image data and first point cloud data in a low-light scene, and match the first image data and the first point cloud data to obtain at least one matching data pair, wherein the matching data pair includes second image data and second point cloud data; an enhancement module, configured to perform image enhancement on the second image data based on a pre-trained target image enhancement model to obtain target image data, and to perform point cloud enhancement on the second point cloud data to obtain target point cloud data; A fusion module, used for fusing the target image data and the target point cloud data to obtain color information of the target point cloud data from the target image data; A coloring module is used to color the target point cloud data based on the color information.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

Images