Point cloud dyeing method, device and equipment and readable storage medium

Abstract

The invention discloses a point cloud dyeing method, apparatus and device, and a readable storage medium. The method comprises the steps of obtaining a point cloud frame and a plurality of image frames; aiming at each space point of each camera device, generating an observation transformation matrix used for representing the projection movement condition of the space points on an image plane in the exposure process of the camera devices; generating a relative timestamp for indicating the imaging moment of the spatial point; based on the relative timestamp and the observation transformation matrix, generating a pixel coordinate of the spatial point projected to an image frame of the camera device, and based on the pixel coordinate, performing pixel color picking on the spatial point to obtain a pixel vector; establishing a corresponding relationship between each spatial point of different camera devices and each point cloud in the point cloud frame; and determining a color vector of the corresponding point cloud based on the pixel vector of each spatial point corresponding to the same point cloud. Visibly, according to the method, space-time alignment of the three-dimensional space points and the pixel coordinates on the image frame can be realized by constructing the observation transformation matrix and the relative timestamp, and the point cloud dyeing precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and more particularly to a point cloud coloring method and device, equipment and readable storage medium. BACKGROUND

[0002] In three-dimensional reconstruction neighborhood, accurate scene initialization is crucial, and this is usually highly dependent on correct colored point clouds. The prior art often uses a rolling shutter camera to capture image sequences, and with the help of a laser radar to obtain point cloud information, the point cloud is projected onto the picture to realize coloring, thereby providing basic support for subsequent three-dimensional reconstruction work.

[0003] However, the line-by-line exposure characteristics of the rolling shutter camera are prone to introduce motion distortion. Dynamic objects have differences in position in multi-view images due to motion, and when the point cloud is projected onto the images of different views to take color, the projected pixel positions of the same spatial point in different images may correspond to different parts of dynamic objects or background areas. For example, a pedestrian moves during camera exposure, and the position of the pedestrian in the image shifts, and when the point cloud is projected onto the image, the point cloud may mistakenly take the background color before the pedestrian moves or the color of part of the body after the pedestrian moves, which is prone to cause errors in extracting point cloud color information.

[0004] Based on this, how to accurately extract point cloud color information has become the focus of the technical personnel in the neighborhood. SUMMARY

[0005] In view of this, the present application provides a point cloud coloring method, device, equipment and readable storage medium for extracting point cloud color information.

[0006] In order to achieve the above purpose, the present scheme is as follows:

[0007] A point cloud coloring method comprises:

[0008] acquiring a point cloud frame and a plurality of image frames collected by a laser radar and a plurality of camera devices;

[0009] determining each spatial point corresponding to each image frame collected by each camera device;

[0010] for each spatial point corresponding to each camera device, generating an observation transformation matrix for representing the projection movement of the spatial point on the image plane during the exposure process of the camera device; generating a relative time stamp for indicating the imaging time of the spatial point; based on the relative time stamp and the observation transformation matrix, generating a pixel coordinate of the spatial point projected onto the image frame of the camera device, and performing pixel color taking on the spatial point based on the pixel coordinate to obtain a pixel vector of the spatial point;

[0011] establish a correspondence between each spatial point of different camera devices and each point cloud in a point cloud frame;

[0012] determine a color vector of the corresponding point cloud based on the pixel vector of each spatial point corresponding to the same point cloud.

[0013] Optionally, the generating an observation transformation matrix for representing a projection movement of the spatial point on the image plane during the exposure process of the camera device comprises:

[0014] obtaining a first-row coordinate transformation matrix and a last-row coordinate transformation matrix for representing a transformation relationship between a target coordinate system and a camera coordinate system corresponding to a first-row exposure time and a last-row exposure time of each camera device;

[0015] based on the first-row coordinate transformation matrix and the last-row coordinate transformation matrix corresponding to the same camera device, constructing a rotation component function and a translation component function;

[0016] integrating the rotation component function and the translation component function to construct the observation transformation matrix of the spatial point.

[0017] Optionally, the generating a relative timestamp for indicating an imaging time of the spatial point comprises:

[0018] initializing a relative timestamp;

[0019] constructing a residual calculation function for quantifying an accuracy of the relative timestamp;

[0020] substituting the relative timestamp into the residual calculation function to obtain a residual value;

[0021] based on the residual value, updating the relative timestamp, and returning to the step of substituting the relative timestamp into the residual calculation function to obtain a residual value until the residual value is less than a preset residual threshold.

[0022] Optionally, based on the residual value, updating the relative timestamp comprises:

[0023] analyzing the residual calculation function, constructing a motion Jacobian matrix for representing a pose change rate, and constructing a projection Jacobian matrix for representing a normalized coordinate change rate, wherein the normalized coordinate change rate is a change rate of a longitudinal coordinate obtained by projecting the spatial point to a normalized plane based on the relative timestamp, and the pose change rate is a pose change of the camera device calculated based on the relative timestamp;

[0024] integrating the motion Jacobian matrix and the projection Jacobian matrix to obtain a total Jacobian matrix in combination with the chain rule;

[0025] constructing a relative timestamp update function based on the total Jacobian matrix and the residual calculation function;

[0026] updating the relative timestamp by substituting the residual value into the relative timestamp update function.

[0027] Optionally, the generating of the pixel coordinates of the spatial point projected onto the image frame of the camera device based on the relative timestamp and the observation transformation matrix comprises:

[0028] normalizing the relative timestamp to obtain a normalization coefficient;

[0029] substituting the normalization coefficient into the observation transformation matrix to obtain a rotation component and a translation component for representing the rotation and translation of the spatial point caused by the camera distortion during the exposure process;

[0030] obtaining the pixel coordinates of the spatial point projected onto the image frame of the camera device based on the rotation component and the translation component.

[0031] Optionally, the pixel color picking of the spatial point based on the pixel coordinates to obtain the pixel vector of the spatial point comprises:

[0032] performing neighborhood diffusion on the spatial point based on the pixel coordinates to determine the neighborhood of the spatial point;

[0033] searching the neighborhood to determine whether the spatial point corresponds to a neighborhood point;

[0034] if yes, determining whether the neighborhood point and the spatial point belong to the same object visible surface; if they belong to the same object visible surface, extracting pixel information matching the pixel coordinates from the image frame as a pixel vector; if they do not belong to the same object visible surface, selecting a point with a far depth value as a far point and another point as a near point from the spatial point and the neighborhood point, taking black as the pixel vector of the far point, and extracting pixel information matching the pixel coordinates of the near point from the image frame as the pixel vector of the near point;

[0035] if no, extracting pixel information matching the pixel coordinates from the image frame as the pixel vector of the spatial point.

[0036] Optionally, the determining of the color vector of the corresponding point cloud based on the pixel vectors of each spatial point corresponding to the same point cloud comprises:

[0037] determining the light saturation factor and the boundary distance factor of each spatial point corresponding to the same point cloud;

[0038] calculating the weight value corresponding to each spatial point based on the light saturation factor and the boundary distance factor of each spatial point.

[0039] A color vector of a corresponding point cloud is calculated based on the pixel vector and the weight value corresponding to each spatial point corresponding to the same point cloud.

[0040] A point cloud coloring device comprises:

[0041] An acquisition module is configured to acquire a point cloud frame and a plurality of image frames collected by a laser radar and a plurality of camera devices.

[0042] A determination module is configured to determine each spatial point corresponding to each image frame collected by each camera device.

[0043] A generation module is configured to generate, for each spatial point corresponding to each camera device, an observation transformation matrix for representing a projection movement of the spatial point on an image plane during an exposure process of the camera device; generate a relative time stamp for indicating an imaging moment of the spatial point; generate a pixel coordinate of the spatial point projected onto an image frame of the camera device based on the relative time stamp and the observation transformation matrix, and perform pixel color picking on the spatial point based on the pixel coordinate to obtain a pixel vector of the spatial point.

[0044] An establishment module is configured to establish a corresponding relationship between each spatial point of different camera devices and each point cloud in the point cloud frame.

[0045] A coloring module is configured to determine a color vector of a corresponding point cloud based on the pixel vector of each spatial point corresponding to the same point cloud.

[0046] A point cloud coloring device comprises a memory and a processor.

[0047] The memory is configured to store a program.

[0048] The processor is configured to execute the program to implement each step of the point cloud coloring method.

[0049] A readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement each step of the point cloud coloring method.

[0050] It can be seen from the technical solutions that the point cloud dyeing method provided by the application can obtain point cloud frames and multiple image frames collected by a laser radar and multiple camera devices. Based on this, the application can realize the original association of three-dimensional structure and two-dimensional color information through multi-sensor data collection. The application can determine each spatial point corresponding to the image frame collected by each camera device. For each spatial point corresponding to each camera device, an observation transformation matrix for representing the projection movement of the spatial point on the image plane during the exposure process of the camera device is generated. A relative timestamp for indicating the imaging moment of the spatial point is generated. Based on the relative timestamp and the observation transformation matrix, the pixel coordinates of the spatial point projected onto the image frame of the camera device are generated, and the pixel coordinates are used for pixel color picking of the spatial point to obtain a pixel vector of the spatial point. Based on this, the application can determine the projection position offset of each spatial point caused by camera motion or distortion during the exposure process of each camera device by constructing an observation transformation matrix, reflecting the projection influence of space-time changes on each spatial point in each camera device. At the same time, the application constructs an observation transformation matrix corresponding to each spatial point in each camera device, improves the pertinence of the observation transformation matrix, reduces error introduction as much as possible, and realizes accurate modeling of dynamic projection relationship. In addition, the application also constructs a relative timestamp representing the imaging moment of the spatial point during the exposure process of a specific camera device, accurately corresponds to the time offset characteristics of row-by-row exposure, provides a quantitative benchmark for space-time alignment of the spatial point and the camera exposure moment through the relative timestamp, and avoids projection errors caused by different time synchronization. Combining the time offset characteristics and the spatial offset characteristics, the spatial point is accurately mapped to the pixel position of the image frame, ensuring that the color of the spatial point is taken from the corresponding pixel at the real imaging moment, eliminating color mismatch caused by exposure time offset, and improving the accuracy of point cloud dyeing under single view. Subsequently, the application can establish the correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame. Based on the pixel vectors of each spatial point corresponding to the same point cloud, the color vector of the corresponding point cloud is determined. Based on this, the application can fuse the color observation values of the same point cloud by multiple cameras, suppress abnormal values through weighting and other strategies, reduce the influence of single-view observation error, and improve the robustness of point cloud dyeing. The dynamic object motion artifact and the low-quality color interference of the image edge and the overexposed area are suppressed, and finally the point cloud color closer to the real scene is obtained. It can be seen that the application can realize space-time alignment of three-dimensional spatial points and pixel coordinates on the image frame, accurate projection color picking and multi-view fusion by constructing an observation transformation matrix and a relative timestamp, systematically solves the problems of rolling shutter camera exposure distortion, time synchronization error, dynamic object interference, and finally realizes the improvement of point cloud dyeing precision and reliability, and provides high-quality color point cloud data for automatic driving environment perception, three-dimensional reconstruction and other applications. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only only the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0052] Figure 1 A point cloud dyeing method flow chart disclosed by the embodiments of the present application;

[0053] Figure 2 A pixel color picking schematic diagram provided by the embodiments of the present application;

[0054] Figure 3 A point cloud dyeing device structure block diagram disclosed by the embodiments of the present application;

[0055] Figure 4 A hardware structure block diagram of a point cloud dyeing device disclosed by the embodiments of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] The embodiments of the present application provide a point cloud dyeing method, which can be applied to various three-dimensional reconstruction systems or automatic driving systems, and can also be applied to various computer terminals or intelligent terminals. The execution subject can be a processor or server of a computer terminal or an intelligent terminal.

[0058] Next, the point cloud dyeing method of the present application will be described in detail, including the following steps: Figure 1

[0059] Step S1, acquiring point cloud frames and multiple image frames collected by a laser radar and multiple camera devices.

[0060] Specifically, the laser radar and each camera device can be deployed on the same moving object, such as on the same autonomous vehicle, or can be deployed on different moving objects, such as on different unmanned aerial vehicles.

[0061] ​The number of camera devices can be set according to actual needs, for example, one 128-line laser radar with a frequency of 10 Hz can be installed on an autonomous vehicle, and six 2 million-pixel rolling shutter camera devices are arranged in a circumferential direction with a frame rate of 10 Hz, and the optical center spacing of the camera devices is 60 degrees, covering a 360-degree field of view. The camera devices and the radar are synchronized by PTP protocol, and the timestamp alignment error is less than 1 ms.

[0062] The laser radar and each camera device can be used to collect point cloud frames and image frames of the same scene.

[0063] Step S2, determining each spatial point corresponding to the image frame collected by each camera device.

[0064] Specifically, the three-dimensional spatial points that the camera device effectively captures light and images can be determined.

[0065] Each camera device has a corresponding image frame.

[0066] The spatial points corresponding to different camera devices are not completely consistent, and there can be cases of partial spatial point overlap and partial spatial point non-overlap.

[0067] Step S3, for each spatial point corresponding to each camera device, generating an observation transformation matrix for representing the projection movement of the spatial point on the image plane during the exposure process of the camera device; generating a relative timestamp indicating the imaging time of the spatial point; based on the relative timestamp and the observation transformation matrix, generating a pixel coordinate of the spatial point projected onto the image frame of the camera device, and performing pixel color picking on the spatial point based on the pixel coordinate to obtain a pixel vector of the spatial point.

[0068] Specifically, for each spatial point corresponding to each camera device, the coordinate transformation matrix of the camera device can be constructed through external parameter calibration, laser odometry pose solution and IMU motion integration, and the observation transformation matrix of the projection movement of the spatial point on the image plane during the exposure process of the camera device can be constructed according to the coordinate transformation matrix, wherein the projection movement includes the rotation component variation with time and the translation component variation with time.

[0069] The gradient descent method can be used to determine the relative timestamp of each spatial point captured by the camera onto the image.

[0070] The relative timestamp and the observation transformation matrix of the spatial point on the same camera device are combined to project the spatial point from the target coordinate system to the camera coordinate system, and then project the spatial point from the camera coordinate system to the image frame to obtain the pixel coordinate, and perform pixel color picking on the spatial point based on the pixel coordinate to obtain the pixel vector of the spatial point on the image frame.

[0071] Step S4, correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame is established.

[0072] Specifically, the correspondence between each spatial point on different camera devices and different point clouds on the point cloud frame can be determined.

[0073] Step S5, based on the pixel vectors of each spatial point corresponding to the same point cloud, the color vector of the corresponding point cloud is determined.

[0074] Specifically, the pixel vectors of each spatial point of the same point cloud can be weighted and fused to determine the color vector of the corresponding point cloud.

[0075] It can be seen from the above technical solutions that the point cloud dyeing method provided by the application can obtain point cloud frames and multiple image frames collected by a laser radar and multiple camera devices. Based on this, the application can realize the original association of three-dimensional structure and two-dimensional color information through multi-sensor data collection. The application can determine each spatial point corresponding to the image frame collected by each camera device. For each spatial point corresponding to each camera device, an observation transformation matrix is generated to represent the projection movement of the spatial point on the image plane during the exposure process of the camera device. A relative timestamp is generated to indicate the imaging time of the spatial point. Based on the relative timestamp and the observation transformation matrix, the pixel coordinates of the spatial point projected onto the image frame of the camera device are generated, and the pixel coordinates are used to take color from the spatial point to obtain a pixel vector of the spatial point. Based on this, the application can determine the projection position offset of each spatial point caused by camera motion or distortion during the exposure process of each camera device by constructing an observation transformation matrix, reflecting the projection influence of space-time changes on each spatial point in each camera device. At the same time, the application constructs an observation transformation matrix corresponding to each spatial point in each camera device, improves the pertinence of the observation transformation matrix, reduces error introduction as much as possible, and realizes accurate modeling of dynamic projection relationship. In addition, the application also constructs a relative timestamp representing the imaging time of the spatial point during the exposure process of a specific camera device, accurately corresponds to the time offset characteristics of line-by-line exposure, provides a quantitative benchmark for space-time alignment of the spatial point and the camera exposure time through the relative timestamp, and avoids projection errors caused by different time synchronization. Combining the time offset characteristics and the spatial offset characteristics, the spatial point is accurately mapped to the pixel position of the image frame, ensuring that the color of the spatial point is taken from the corresponding pixel at its real imaging time, eliminating color mismatch caused by exposure time offset, and improving the accuracy of point cloud dyeing under single view. Subsequently, the application can establish the correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame. Based on the pixel vectors of each spatial point corresponding to the same point cloud, the color vector of the corresponding point cloud is determined. Based on this, the application can fuse the color observation values of multiple cameras for the same point cloud, suppress abnormal values through weighting and other strategies, reduce the influence of single-view observation error, and improve the robustness of point cloud dyeing. The dynamic object motion artifact and the low-quality color interference of the image edge and the overexposed area are suppressed, and finally the point cloud color closer to the real scene is obtained. It can be seen that the application can realize space-time alignment of three-dimensional spatial points and pixel coordinates on the image frame, accurate projection color taking and multi-view fusion by constructing an observation transformation matrix and a relative timestamp, systematically solves the problems of rolling shutter camera exposure distortion, time synchronization error, dynamic object interference, and finally realizes the improvement of point cloud dyeing precision and reliability, and provides high-quality color point cloud data for automatic driving environment perception, three-dimensional reconstruction and other applications.

[0076] In some embodiments of the present application, the process of generating the observation transformation matrix for characterizing the movement of the spatial point projected on the image plane during the exposure process of the camera device is described in detail as follows:

[0077] S30, obtaining a first row coordinate transformation matrix and a last row coordinate transformation matrix for characterizing the transformation relationship between the target coordinate system and the camera coordinate system corresponding to the first row exposure time and the last row exposure time of each camera device.

[0078] Specifically, since the camera device has a pose transformation caused by motion during the exposure process, the camera device pose at the first row exposure time and the camera device pose at the last row exposure time included in the same exposure process are different.

[0079] Wherein, the first row exposure time can be the start time of the exposure process, and the last row exposure time can be the end time of the exposure process.

[0080] The first row coordinate transformation matrix of each camera device can represent the transformation relationship between the camera coordinate system and the target coordinate system at the first row exposure time of the corresponding camera device.

[0081] The last row coordinate transformation matrix of each camera device can represent the transformation relationship between the camera coordinate system and the target coordinate system at the last row exposure time of the corresponding camera device.

[0082] Wherein, the target coordinate system can be a unified coordinate system, such as a world coordinate system.

[0083] Through external parameter calibration, the coordinate system transformation matrix between the camera device and the laser radar can be obtained Through laser odometry calculation, the global pose transformation matrix of the laser radar when the camera device collects an image frame can be obtained Through IMU integration, the time compensation matrix between the start time of the image frame and the laser radar timestamp time of the image frame can be obtained , as shown in the following formula:

[0084]

[0085]

[0086] = -

[0087]

[0088]

[0089] Wherein, For camera equipment rotation increment; The difference is the timestamp value. This is the Rodriguez formula; For camera equipment angular velocity; This is the translation vector in the camera coordinate system; Total exposure time; For camera equipment linear velocity; for The rotation matrix that maps the camera coordinate system to the world coordinate system; for The rotation matrix that maps the world coordinate system to the camera coordinate system; The translational pose components of the camera in the world coordinate system.

[0090] Based on this, the first row coordinate transformation matrix in the image frame can be obtained. The calculation function:

[0091]

[0092] for The rotation matrix that maps the world coordinate system to the camera coordinate system; for The translational pose components of the camera in the world coordinate system.

[0093] The translation increment of the camera equipment is affected by velocity and acceleration, where the acceleration 'a' needs to be adjusted to account for the influence of gravity and transferred to the world coordinate system:

[0094]

[0095] These are the raw measurements from the IMU; To achieve zero bias in the accelerometer; This is the gravity vector.

[0096] Based on this, construct a matrix for calculating the first row coordinate transformation of each camera device. The calculation function:

[0097]

[0098] This is the rotation matrix that maps the world coordinate system to the camera coordinate system at the first exposure time; The translation pose component of the camera in the world coordinate system at the first exposure time.

[0099] Similarly, by obtaining the readout time k per line and the total number of lines N from the camera parameters, the exposure times of the first and last lines of the rolling shutter camera in this frame are as follows, based on the camera's exposure mechanism:

[0100]

[0101] Total exposure time is:

[0102]

[0103] wherein, is the first row exposure time; is the last row exposure time.

[0104] On this basis, the time compensation matrix between the first and last row exposure time of the image frame camera device can be obtained , and the last row coordinate transformation matrix of the image frame is obtained accordingly :

[0105]

[0106] In the formula, is the rotation matrix of the world coordinate system mapped to the camera coordinate system at the last row exposure time; is the translation pose component of the camera in the world coordinate system at the last row exposure time.

[0107] S31, based on the first row coordinate transformation matrix and the last row coordinate transformation matrix corresponding to the same camera device, a rotation component function and a translation component function are constructed.

[0108] Specifically, the rotation component function and the translation component function of the space point can be constructed by interpolating the first row coordinate transformation matrix and the last row coordinate transformation matrix of the same camera device.

[0109] The rotation component function can be obtained by combining the spherical linear interpolation Slerp method as follows:

[0110]

[0111]

[0112] The translation component function can be obtained by linear interpolation as follows: :

[0113]

[0114] In the formula, denotes the trace of the matrix.

[0115] S33, the observation transformation matrix of the space point is constructed by integrating the rotation component function and the translation component function.

[0116] Specifically, the observation transformation matrix of the spatial point can be constructed according to the rotation component function and the translation component function :

[0117]

[0118] From the above technical solution, it can be seen that the embodiment provides an optional way of constructing an observation transformation matrix of a spatial point. Through the above way, the observation transformation matrix of each spatial point corresponding to the camera device can be calculated according to the pose transformation of the camera device during exposure. From the whole to the local, accurate derivation is realized, and fine derivation from the overall pose of the camera to the individual projection of the spatial point is realized.

[0119] In some embodiments of the present application, the process of generating the relative timestamp indicating the imaging moment of the spatial point in step S3 is described in detail, and the steps are as follows:

[0120] S30, initializing the relative timestamp.

[0121] Specifically, a variable corresponding to the relative timestamp can be constructed, and the variable can be initialized.

[0122] S31, constructing a residual calculation function for quantifying the accuracy of the relative timestamp.

[0123] Specifically, assuming that the camera device has no distortion, the spatial point P i can be projected on the pixel plane through a series of transformations, and the coordinates of the point on the pixel plane are (u, v) at this time. The relative timestamp t h of the point can be obtained by directly multiplying the row number u by the reading time k of each row. h = uk, u= .

[0124] However, there is distortion on the camera device, so u is the theoretical row.

[0125] And the spatial point P i According to the projection behavior of the actual projection on the pixel plane calculated by the relative timestamp .

[0126] In order to improve the accuracy of the projection row, the relative timestamp needs to be optimized to minimize the difference between the projection row and the theoretical row.

[0127] The spatial point P i According to the projection row actually projected on the pixel plane calculated by the relative timestamp, the steps are as follows:

[0128] The spatial point P i is projected into the camera coordinate system through the observation transformation matrix, and the point in the camera coordinate system at the corresponding moment is obtained :

[0129]

[0130]

[0131] Normalize it to the Z=1 plane to obtain normalized plane coordinates. :

[0132]

[0133] The pixel coordinates are obtained by projecting the intrinsic parameter matrix K of the camera device onto the pixel coordinate system:

[0134]

[0135] Considering that the ultimate goal is to obtain the normalization coefficients obtained by normalizing relative timestamps. To minimize the difference between the projected row and the theoretical row, we directly optimize the relative timestamp t, which is equivalent to the row residual. h It is more convenient, therefore, a residual calculation function for quantifying relative timestamp accuracy can be constructed:

[0136]

[0137] The residual value; Principal point exposure time offset, which is principal point time minus camera pose time; exposure time at the center of the image frame, which is generally 0.5 times the total exposure time.

[0138] S32. Substitute the relative timestamp into the residual calculation function to obtain the residual value.

[0139] Specifically, the current value of the variable can be substituted into the residual calculation function to calculate the residual value.

[0140] S33. Based on the residual value, update the relative timestamp and return to step S32 until the residual value is less than a preset residual threshold.

[0141] Specifically, the variables can be updated using gradient descent based on the residual values ​​until the residual values ​​are less than a preset residual threshold or the maximum number of iterations is reached. The latest variable value is then used as a relative timestamp.

[0142] As can be seen from the above technical solution, this embodiment provides an optional method for generating relative timestamps. By comparing the gap between theoretical and actual situations, the reliability of the updated relative timestamps can be measured, and based on this, the determination of relative timestamps can be gradually completed, making the reliability of relative timestamps higher.

[0143] In some embodiments of the present application, the step S33 is described in detail as follows: updating the relative timestamp based on the residual value.

[0144] S330, analyzing the residual calculation function, constructing a motion Jacobian matrix for representing the pose change rate, and constructing a projection Jacobian matrix for representing the normalized coordinate change rate, wherein the normalized coordinate change rate is the change rate of the longitudinal coordinate obtained by projecting the space point to the normalized plane based on the relative timestamp, and the pose change rate is the pose change of the camera device calculated based on the relative timestamp.

[0145] Specifically, the influence of the change of the relative timestamp on the residual value can be evaluated by constructing the Jacobian matrix.

[0146] Specifically, the motion Jacobian matrix can be constructed as The pose change rate is described as

[0147]

[0148] wherein the first term represents the inter-frame angular velocity using the first-order derivative of the rotation part , and the second term represents the inter-frame linear velocity using the first-order derivative of the translation part :

[0149]

[0150] wherein Log(·) in the rotation representation refers to the mapping from the Lie group SO(3) to the Lie algebra so(3) by the anti-symmetric matrix ln(R) to convert the rotation matrix into the angular velocity vector:

[0151]

[0152] The projection Jacobian matrix can be constructed as to represent the normalized coordinate change rate, and to explicitly analyze the camera projection model to avoid the automatic differentiation overhead:

[0153]

[0154] wherein the space point P i The coordinates in the world coordinate system are p i (x, y, z).

[0155] S331, integrating the motion Jacobian matrix and the projection Jacobian matrix to obtain a total Jacobian matrix according to the chain rule.

[0156] Specifically, by the chain rule, the motion Jacobian matrix and the projection Jacobian matrix can be integrated to obtain the total Jacobian matrix as shown below :

[0157]

[0158] S332, based on the total Jacobian matrix and the residual calculation function, constructing a relative timestamp update function.

[0159] Specifically, the learning rate can be set as , and the relative timestamp update function is constructed as

[0160]

[0161] In the formula, is the relative timestamp at the n+1th iteration; is the relative timestamp at the nth iteration.

[0162] Generally, the learning rate can be 0.7.

[0163] S3323, substituting the residual value into the relative timestamp update function to update the relative timestamp.

[0164] Specifically, the latest relative timestamp can be substituted into the total Jacobian matrix to calculate the Jacobian value, the residual value can be calculated by substituting the relative timestamp into the residual calculation function, and the relative timestamp can be updated by substituting the residual value and the Jacobian value into the relative timestamp update function.

[0165] As can be seen from the above technical solutions, the embodiment provides an optional way of updating the relative timestamp based on the residual value. Through the above way, the influence of the relative timestamp on the residual value can be decomposed by constructing the Jacobian matrix, and the gradient update of the relative timestamp can be completed based on this, the influence weight of the decomposed relative timestamp on the residual value (error term) is quantified, and the gradient update direction of the relative timestamp is derived based on the decomposition result, the timestamp parameter is iteratively optimized to improve the spatio-temporal alignment accuracy of multiple spatial points under multiple sensors, thereby improving the point cloud coloring accuracy.

[0166] In some embodiments of the present application, the process of generating the pixel coordinates of the spatial point projected onto the image frame of the camera device based on the relative timestamp and the observation transformation matrix in step S3 is described in detail as follows:

[0167] S30, normalizing the relative timestamp to obtain a normalization coefficient.

[0168] Specifically, the relative timestamp can be normalized to obtain a normalization coefficient a specific value of the normalization coefficient.

[0169] S31, the normalization coefficient is substituted into the observation transformation matrix to calculate the rotation component and the translation component of the space point caused by the camera distortion during the exposure process.

[0170] Specifically, the specific value of the normalization coefficient can be substituted into the rotation component function and the translation component function of the observation transformation matrix to obtain the rotation component and the translation component of the space point.

[0171] The rotation component and the translation component replace the rotation component function and the translation component function in the observation transformation matrix to obtain the observation matrix corresponding to the space point.

[0172] S32, based on the rotation component and the translation component, the pixel coordinates of the space point projected onto the image frame of the camera device are obtained.

[0173] Specifically, the space point can be projected into the camera coordinate system according to the observation matrix of the space point to obtain the camera coordinates.

[0174] The camera coordinates are normalized to the Z=1 plane to obtain the normalized plane coordinates of the space point.

[0175] The normalized plane coordinates are projected into the pixel coordinate system through the intrinsic matrix of the camera device to obtain the pixel coordinates of the space point.

[0176] As can be seen from the above technical solutions, the embodiment provides an alternative way of projecting a space point from a world coordinate system to a pixel coordinate system. Through the above way, the pixel vector of the space point under the camera device can be better found.

[0177] In some embodiments of the present application, the process of pixel color picking of the space point based on the pixel coordinates in step S3 to obtain the pixel vector of the space point is described in detail as follows:

[0178] S30, based on the pixel coordinates, the space point is diffused in the neighborhood to determine the neighborhood of the space point.

[0179] Specifically, for each space point corresponding to the same camera device, the depth value between each space point and the camera device is determined, and the space points of the camera device are sorted according to the size of the depth value to obtain a space point sequence. The space points of the camera device can be sorted by using the argsort() function.

[0180] A hash table is constructed and initialized.

[0181] Select each spatial point from the spatial point sequence in the order from front to back.

[0182] According to the pixel coordinates of the selected spatial point, perform neighborhood diffusion on the selected spatial point to determine the neighborhood corresponding to the selected spatial point.

[0183] S31, search the neighborhood to determine whether the spatial point corresponds to a neighborhood point; if yes, perform step S32; if no, perform step S33.

[0184] Specifically, search the neighborhood to determine whether there is a spatial point in the neighborhood; if yes, it is determined that the selected spatial point corresponds to a neighborhood point, and then step S32 is performed.

[0185] If no, step S33 is performed.

[0186] S32, determine whether the neighborhood point and the spatial point belong to the same object visible surface; if they belong to the same object visible surface, extract pixel information matching the pixel coordinates from the image frame as a pixel vector; if they do not belong to the same object visible surface, select a point with a far depth value from the spatial point and the neighborhood point as a far point, select another point as a near point, take black as the pixel vector of the far point, and extract pixel information matching the pixel coordinates of the near point from the image frame as the pixel vector of the near point.

[0187] Specifically, the object visible surface refers to the surface of the same object and is located in the viewing cone of the camera device.

[0188] The following function can be used to determine whether the neighborhood point and the spatial point belong to the same object visible surface:

[0189]

[0190] In the formula, is the rectangular neighborhood . is a preset rectangular region size; is a depth competition tolerance, which can be determined according to the maximum allowed depth variation of the same object visible surface; is the pixel coordinates of the neighborhood point ; d j is the depth value of the neighborhood point; d i is the depth value of the spatial point p i .

[0191] Generally, may be 2 and 3, respectively. The depth competition tolerance can be 1 m.

[0192] If the result is True, it is determined whether a hash table corresponding to the neighborhood point exists, and if yes, the space point is written into the hash table, and if no, the neighborhood point and the space point are written into a newly initialized hash table.

[0193] If the result is False, a black vector is determined, a point with a far depth value is selected as a far point from the space point and the neighborhood point, and another point is selected as a near point.

[0194] Since the neighborhood point and the space point do not belong to the same object visible surface, the far point can be occluded by the near point and belong to an invisible region of the camera device, and thus the black vector can be used as a pixel vector of the far point, and pixel information matching the pixel coordinates of the near point is extracted from the image frame as a pixel vector of the near point, as shown in the black box. Figure 2

[0195] After all the space points in the space point sequence are selected, pixel information matching each pixel coordinate is extracted from the image frame according to the pixel coordinates of each space point corresponding to each hash table, each pixel information is processed to balance detail preservation and noise suppression, and a pixel vector of each space point is obtained.

[0196] S33, pixel information matching the pixel coordinates is extracted from the image frame as a pixel vector of the space point.

[0197] Specifically, if there is no neighborhood point, pixel information matching the pixel coordinates can be directly extracted from the image frame as a pixel vector of the space point.

[0198] As can be seen from the above technical solutions, the embodiment provides an optional way of pixel color picking of the space point based on the pixel coordinates to obtain a pixel vector of the space point. Through the above way, the same object visible surface evaluation of multiple space points can be performed to avoid the case that a far background point covers a near target, further avoid color layering, and improve the reliability of the present application.

[0199] In some embodiments of the present application, the process of step S5 of determining a color vector of a corresponding point cloud based on pixel vectors of each space point corresponding to the same point cloud is described in detail, and the steps are as follows:

[0200] S50, determining an illumination saturation factor and a boundary distance factor of each space point corresponding to the same point cloud.

[0201] Specifically, the illumination saturation factor and the boundary distance factor of each space point corresponding to each camera device can be determined.

[0202] ​S51. Calculate the weight value corresponding to each spatial point based on the illumination saturation factor and boundary distance factor of each spatial point.

[0203] Specifically, each spatial point can be... Light saturation factor Sub- and boundary distance factors Substituting these values ​​into the weight calculation function, we can calculate the weights for each spatial point. The corresponding weight value .

[0204] The weight calculation function can be shown below:

[0205]

[0206] in, For spatial points Projected coordinates in the m-th image frame The pixel brightness value at that location is simultaneously normalized to [0,1]. The saturation brightness threshold; For spatial points Projected to pixel coordinates in pixel coordinate system to image boundary The minimum Euclidean distance, in pixels; is the boundary attenuation coefficient. || ||norm is the norm operation.

[0207] Generally, the saturation brightness threshold can be 0.95; the boundary attenuation coefficient can be 0.1.

[0208] S52. Based on the pixel vector and weight value corresponding to each spatial point of the same point cloud, calculate the color vector of the corresponding point cloud.

[0209] Specifically, the color vector of the corresponding point cloud can be calculated by substituting the pixel vector and weight value corresponding to each spatial point of the same point cloud into the weighted fusion function.

[0210] The weighted fusion function is shown below:

[0211]

[0212] In the formula, For spatial points The pixel vector of the m-th image frame; M is a vector containing spatial points. The total number of image frames; This is the error parameter.

[0213] In addition, to improve the confidence of the point cloud color vector, the color vector of each spatial point can also be calculated. Valid number of observations :

[0214]

[0215] wherein, is an indicator function, which is 1 if the condition is true, and 0 otherwise; is a weight threshold.

[0216] When the number of valid observations is less than the preset minimum number of observations, the color vector of the corresponding point cloud is discarded.

[0217] The minimum number of observations can be 2.

[0218] As can be seen from the above technical solutions, the present application can comprehensively color point clouds from pixel vectors of multiple perspectives, avoid ghosting artifacts, and avoid the influence of low-confidence image regions on coloring, thereby reducing the influence of overexposed and image edge distortion regions.

[0219] Further, the weight value can be optimized in combination with the signal-to-noise ratio of the camera device to further improve the reliability of point cloud coloring.

[0220] Next, the point cloud coloring device provided by the present application will be described in detail. Figure 3 The point cloud coloring device provided by the present application will be described in detail below, which can be compared with the point cloud coloring method provided above.

[0221] Referring to Figure 3 It can be found that the point cloud coloring device can include:

[0222] The acquisition module 10 is configured to acquire point cloud frames and multiple image frames collected by a laser radar and multiple camera devices.

[0223] The determination module 20 is configured to determine each spatial point corresponding to each image frame collected by each camera device.

[0224] The generation module 30 is configured to, for each spatial point corresponding to each camera device, generate an observation transformation matrix for representing the projection movement of the spatial point on an image plane during the exposure process of the camera device; generate a relative timestamp for indicating the imaging moment of the spatial point; generate a pixel coordinate of the spatial point projected onto the image frame of the camera device based on the relative timestamp and the observation transformation matrix, and perform pixel color picking on the spatial point based on the pixel coordinate to obtain a pixel vector of the spatial point.

[0225] The establishment module 40 is configured to establish a corresponding relationship between each spatial point of different camera devices and each point cloud in the point cloud frame.

[0226] A color module 50 is configured to determine a color vector of a corresponding point cloud based on pixel vectors of respective spatial points corresponding to the point cloud.

[0227] Further, the generating module 30 can comprise:

[0228] A coordinate transformation matrix obtaining unit is configured to obtain a first-row coordinate transformation matrix and a last-row coordinate transformation matrix corresponding to a transformation relationship between a target coordinate system and a camera coordinate system at a first-row exposure time and a last-row exposure time of each camera device;

[0229] A function constructing unit is configured to construct a rotation component function and a translation component function based on the first-row coordinate transformation matrix and the last-row coordinate transformation matrix corresponding to the same camera device;

[0230] An observation transformation matrix constructing unit is configured to construct the observation transformation matrix of the spatial point by integrating the rotation component function and the translation component function.

[0231] Further, the generating module 30 can further comprise:

[0232] A relative timestamp initialization unit is configured to initialize a relative timestamp;

[0233] A residual error calculation function constructing unit is configured to construct a residual error calculation function for quantifying accuracy of the relative timestamp;

[0234] A residual error value calculating unit is configured to substitute the relative timestamp into the residual error calculation function to obtain a residual error value;

[0235] A relative timestamp updating unit is configured to update the relative timestamp based on the residual error value and return to the step of substituting the relative timestamp into the residual error calculation function to obtain a residual error value until the residual error value is less than a preset residual error threshold.

[0236] Further, the relative timestamp updating unit can comprise:

[0237] A first relative timestamp updating subunit is configured to analyze the residual error calculation function, construct a motion Jacobian matrix for representing a pose change rate, and construct a projection Jacobian matrix for representing a normalized coordinate change rate, wherein the normalized coordinate change rate is a change rate of a longitudinal coordinate obtained by projecting the spatial point to a normalized plane based on the relative timestamp, and the pose change rate is a pose change of the camera device calculated based on the relative timestamp;

[0238] A second relative timestamp updating subunit is configured to integrate the motion Jacobian matrix and the projection Jacobian matrix to obtain a total Jacobian matrix in combination with a chain rule.

[0239] A third relative timestamp updating subunit is configured to construct a relative timestamp updating function based on the total Jacobian matrix and the residual calculation function;

[0240] A fourth relative timestamp updating subunit is configured to update a relative timestamp by substituting the residual value into the relative timestamp updating function.

[0241] Further, the generating module 30 can comprise:

[0242] A normalization coefficient calculation unit is configured to normalize the relative timestamp to obtain a normalization coefficient;

[0243] A component calculation unit is configured to substitute the normalization coefficient into the observation transformation matrix to calculate a rotation component and a translation component for representing the rotation of the spatial point caused by the camera distortion during the exposure process;

[0244] A pixel coordinate determination unit is configured to obtain a pixel coordinate of the spatial point projected onto the image frame of the camera device based on the rotation component and the translation component.

[0245] Further, the generating module 30 can comprise:

[0246] A neighborhood determination unit is configured to perform neighborhood diffusion on the spatial point based on the pixel coordinate to determine a neighborhood of the spatial point;

[0247] A neighborhood point searching unit is configured to search the neighborhood to determine whether the spatial point corresponds to a neighborhood point; if yes, an object visible surface judgment unit is invoked; if no, a pixel vector determination unit is invoked;

[0248] The object visible surface judgment unit is configured to determine whether the neighborhood point and the spatial point belong to the same object visible surface; if yes, pixel information matching the pixel coordinate is extracted from the image frame as a pixel vector; if no, a point with a far depth value is selected from the spatial point and the neighborhood point as a far point, and another point is selected as a near point, black is taken as the pixel vector of the far point, and pixel information matching the pixel coordinate of the near point is extracted from the image frame as the pixel vector of the near point;

[0249] The pixel vector determination unit is configured to extract pixel information matching the pixel coordinate from the image frame as the pixel vector of the spatial point.

[0250] Further, the dyeing module 50 can comprise:

[0251] A light saturation factor determination unit is configured to determine a light saturation factor and a boundary distance factor corresponding to each spatial point of the same point cloud;

[0252] a weight value calculation unit configured to calculate a weight value corresponding to each spatial point based on an illumination saturation factor and a boundary distance factor of each spatial point;

[0253] a color vector calculation unit configured to calculate a color vector of a point cloud based on a pixel vector corresponding to each spatial point of the point cloud and the weight value corresponding to each spatial point.

[0254] The point cloud coloring device provided by the embodiments of the present application can be applied to a point cloud coloring device, such as a PC terminal, a cloud platform, a server, a server cluster, and the like. Optionally, Figure 4 A hardware structure block diagram of the point cloud coloring device is shown, and referring to Figure 4 The hardware structure of the point cloud coloring device can include at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.

[0255] In the embodiments of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete communication with each other through the communication bus 4.

[0256] The processor 1 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0257] The memory 3 can include a high-speed RAM memory, and can also include a non-volatile memory, etc., such as at least one disk memory.

[0258] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to:

[0259] acquire a point cloud frame and a plurality of image frames collected by a laser radar and a plurality of camera devices;

[0260] determine each spatial point corresponding to an image frame collected by each camera device;

[0261] For each spatial point corresponding to each camera device, an observation transformation matrix is generated to represent the projection movement of the spatial point on an image plane during the exposure process of the camera device; a relative timestamp is generated to indicate the imaging time of the spatial point; based on the relative timestamp and the observation transformation matrix, a pixel coordinate of the spatial point projected onto the image frame of the camera device is generated, and pixel color of the spatial point is taken based on the pixel coordinate to obtain a pixel vector of the spatial point;

[0262] establish a correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame;

[0263] determine a color vector of the corresponding point cloud based on the pixel vectors of the spatial points corresponding to the same point cloud.

[0264] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0265] The embodiment of the application further provides a readable storage medium which can store a program suitable for processor execution, and the program is used for:

[0266] acquiring a point cloud frame and a plurality of image frames collected by a laser radar and a plurality of camera devices;

[0267] determining each spatial point corresponding to each image frame collected by each camera device;

[0268] generating, for each spatial point corresponding to each camera device, an observation transformation matrix used for representing a projection movement of the spatial point on an image plane during an exposure process of the camera device; generating a relative time stamp used for indicating an imaging moment of the spatial point; generating a pixel coordinate of the spatial point projected on an image frame of the camera device based on the relative time stamp and the observation transformation matrix, and performing pixel color picking on the spatial point based on the pixel coordinate to obtain a pixel vector of the spatial point;

[0269] establishing a correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame;

[0270] determining a color vector of the corresponding point cloud based on the pixel vectors of the spatial points corresponding to the same point cloud.

[0271] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0272] Finally, it should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0273] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0274] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Various embodiments of this application can be combined with each other. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of point cloud coloring, the method comprising: The method comprises the following steps: acquiring point cloud frames and image frames collected by a laser radar and a plurality of camera devices; determining each spatial point corresponding to the image frames collected by each camera device; generating, for each spatial point corresponding to each camera device, an observation transformation matrix for representing the projection movement of the spatial point on the image plane during the exposure of the camera device; generating a relative timestamp for indicating the imaging time of the spatial point; based on the relative timestamp and the observation transformation matrix, generating the pixel coordinates of the spatial point projected onto the image frame of the camera device, and performing pixel color picking on the spatial point based on the pixel coordinates to obtain a pixel vector of the spatial point; establishing a correspondence between each spatial point of different camera devices and each point cloud in the point cloud frames; based on the pixel vectors of each spatial point corresponding to the same point cloud, determining the color vector of the corresponding point cloud.

2. The method of claim 1, wherein, The method comprises the following steps: acquiring the first-row coordinate transformation matrix and the last-row coordinate transformation matrix corresponding to the first-row exposure time and the last-row exposure time of each camera device, which are used to represent the transformation relationship between the target coordinate system and the camera coordinate system; based on the first-row coordinate transformation matrix and the last-row coordinate transformation matrix corresponding to the same camera device, constructing a rotation component function and a translation component function; integrating the rotation component function and the translation component function to construct the observation transformation matrix of the spatial point.

3. The method of claim 1, wherein, The method comprises the following steps: initializing the relative timestamp; constructing a residual calculation function for quantifying the accuracy of the relative timestamp; substituting the relative timestamp into the residual calculation function to obtain a residual value; based on the residual value, updating the relative timestamp, and returning to the step of substituting the relative timestamp into the residual calculation function to obtain a residual value until the residual value is less than a preset residual threshold.

4. The method of claim 3, wherein, Based on the residual value, updating the relative timestamp comprises the following steps: analyzing the residual calculation function to construct a motion Jacobian matrix for representing the pose change rate and a projection Jacobian matrix for representing the normalized coordinate change rate, wherein the normalized coordinate change rate is the change rate of the longitudinal coordinate obtained by projecting the spatial point onto the normalized plane based on the relative timestamp, and the pose change rate is the pose change of the camera device calculated based on the relative timestamp; combining the chain rule to integrate the motion Jacobian matrix and the projection Jacobian matrix to obtain a total Jacobian matrix; based on the total Jacobian matrix and the residual calculation function, constructing a relative timestamp update function; substituting the residual value into the relative timestamp update function to update the relative timestamp.

5. The method of claim 1, wherein, The method comprises the following steps: normalizing the relative timestamp to obtain a normalization coefficient; The normalized coefficient is substituted into the observation transformation matrix to calculate a rotation component and a translation component of the spatial point caused by camera distortion during the exposure process; Based on the rotation component and the translation component, pixel coordinates of the spatial point projected onto an image frame of the camera device are obtained.

6. The method of claim 1, wherein, The pixel taking color of the spatial point based on the pixel coordinates includes: Based on the pixel coordinates, neighborhood diffusion is performed on the spatial point to determine a neighborhood of the spatial point; The neighborhood is searched to determine whether the spatial point corresponds to a neighborhood point; If yes, it is determined whether the neighborhood point and the spatial point belong to the same visible surface of an object; if they belong to the same visible surface of an object, pixel information matching the pixel coordinates is extracted from the image frame as a pixel vector; if they do not belong to the same visible surface of an object, a point with a far depth value is selected from the spatial point and the neighborhood point as a far point, another point is selected as a near point, black is taken as the pixel vector of the far point, and pixel information matching the pixel coordinates of the near point is extracted from the image frame as the pixel vector of the near point; If no, pixel information matching the pixel coordinates is extracted from the image frame as the pixel vector of the spatial point.

7. The method of claim 1, wherein, The color vector of the corresponding point cloud is determined based on the pixel vectors of each spatial point corresponding to the same point cloud, and includes: The illumination saturation factor and the boundary distance factor corresponding to each spatial point are determined; The weight value corresponding to each spatial point is calculated based on the illumination saturation factor and the boundary distance factor of each spatial point; The color vector of the corresponding point cloud is calculated based on the pixel vector and the weight value corresponding to each spatial point corresponding to the same point cloud.

8. A point cloud coloring device, characterized by, It includes: An acquisition module is configured to acquire a point cloud frame and a plurality of image frames collected by a laser radar and a plurality of camera devices; A determination module is configured to determine each spatial point corresponding to each image frame collected by each camera device; A generation module is configured to, for each spatial point corresponding to each camera device, generate an observation transformation matrix for representing a projection movement of the spatial point on an image plane during an exposure process of the camera device, and generate a relative time stamp indicating an imaging moment of the spatial point; Based on the relative time stamp and the observation transformation matrix, a pixel coordinate of the spatial point projected onto an image frame of the camera device is generated, and a pixel taking color of the spatial point is performed based on the pixel coordinate to obtain a pixel vector of the spatial point; An establishment module is configured to establish a correspondence between each spatial point of different camera devices and each point cloud in the point cloud frame; A dyeing module is configured to determine a color vector of a corresponding point cloud based on pixel vectors of each spatial point corresponding to the same point cloud.

9. A point cloud coloring device, characterized by, It includes a memory and a processor; The memory is configured to store a program; The processor is configured to execute the program to implement each step of the point cloud dyeing method according to any one of claims 1-7.

10. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement each step of the point cloud dyeing method according to any one of claims 1-7.

Images