Method, device and electronic device for spatiotemporal alignment of point cloud and image pixels

Abstract

The present disclosure provides a method, device, electronic device and storage medium for the spatiotemporal alignment of point clouds and image pixels. The method includes: collecting the timestamp of the point cloud obtained by scanning the surrounding environment with a laser radar; calling the underlying software interface that controls the camera to capture the image, so as to capture the camera shutter triggering time and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of the image; wherein, after the exposure time ends, the camera collects the original image data; according to the timestamp of the point cloud, the camera shutter triggering time and the exposure time, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the moment when the camera exposure is completed; the interpolated point cloud and the image pixels are merged to obtain the spatiotemporal aligned point cloud and image pixels. Accordingly, the spatiotemporal alignment of the point cloud and the image pixels using the real and accurate shutter triggering time and exposure time data can significantly improve the quality of the final output results.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the field of data processing technology, and in particular, to a method, device, electronic device, and storage medium for spatiotemporal alignment of point clouds and image pixels. Background Art

[0002] The spatiotemporal alignment technology of point cloud and image pixels can ensure that the point cloud data collected by the lidar and the image data taken by the camera are accurately matched in space and time. This technology can make full use of the precise three-dimensional geometric information provided by the point cloud data and the rich texture and color information provided by the image data, enhance the complementarity of sensor data, and improve the accuracy and robustness of scene understanding and the joint use of lasers and images.

[0003] Traditionally, spatiotemporal alignment relies on hardware clock synchronization, which triggers the simultaneous acquisition of point clouds and images to ensure that the acquisition time of the two sensors is consistent. However, since the acquisition frequencies of point clouds and images may be different, researchers have developed a post-processing soft synchronization method that uses software algorithms to find the closest image frame based on the timestamp of the point cloud, and uses interpolation and other methods to achieve frame alignment, thereby achieving spatiotemporal alignment.

[0004] However, the soft synchronization method relies on estimated data provided by third-party chip suppliers, such as the camera shutter trigger time and exposure time. When the data provided by third-party chip suppliers are estimates rather than precise measurements, time deviations will occur in actual applications. In addition, different types of lidars have different laser scanning characteristics, and the timestamps of the scanned point clouds are different, which will cause limitations in the accuracy of the final spatiotemporal alignment and make it difficult to fully meet the needs of high-precision applications. Summary of the invention

[0005] The present disclosure provides a method for spatiotemporal alignment of a point cloud and image pixels, the method comprising:

[0006] Collect the timestamp of the point cloud obtained by the lidar scanning the surrounding environment;

[0007] Call the underlying software interface that controls the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data;

[0008] According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time;

[0009] The interpolated point cloud and image pixels are fused to obtain the spatiotemporally aligned point cloud and image pixels.

[0010] Optionally, the method further includes:

[0011] The time period for the laser radar to complete a complete scan is divided into a plurality of sub-time periods, each sub-time period corresponding to the time interval between two adjacent beams of point clouds with different laser radar scanning timings;

[0012] The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including:

[0013] Determining a camera exposure completion time according to the camera shutter triggering time and the exposure duration;

[0014] In a sub-time period covering the moment when the camera exposure is completed, the point cloud and the image pixels are interpolated according to the timestamps of the two adjacent point clouds corresponding to the sub-time period and the moment when the camera exposure is completed, so as to estimate the point cloud position corresponding to the moment when the camera exposure is completed.

[0015] Optionally, the laser radar is a mechanical rotating laser radar, and its scanning direction is longitudinal array scanning. The time period for the laser radar to complete a complete scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including:

[0016] The mechanical rotating laser radar completes a complete scan of the surrounding environment, obtains a time period for completing the scan and a plurality of longitudinal arrays obtained by the scan, each longitudinal array comprising a plurality of point clouds collected by the mechanical rotating laser radar;

[0017] According to the obtained multiple longitudinal arrays, the time period for completing the scan is divided into multiple sub-time periods, each sub-time period corresponds to the time interval between point clouds collected by two adjacent longitudinal arrays with different scanning timings of the mechanical rotating laser radar;

[0018] The point clouds contained in different longitudinal arrays have different scanning timings, and all the point clouds contained in one longitudinal array have the same scanning timing.

[0019] Optionally, the laser radar is a hybrid solid-state laser radar, and its scanning direction is a transverse array scan. The time period for the laser radar to complete a complete scan is divided into a plurality of sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including:

[0020] Performing a complete scan of the surrounding environment by the hybrid solid-state laser radar to obtain a time period for completing the scan and a plurality of transverse arrays obtained by the scan, each transverse array comprising a plurality of point clouds collected by the hybrid solid-state laser radar;

[0021] According to the obtained multiple transverse arrays, the time period for completing the scan is divided into multiple sub-time periods, each sub-time period corresponds to the time interval between point clouds collected by two adjacent transverse arrays with different hybrid solid-state laser radar scanning timings;

[0022] The point clouds contained in different transverse arrays have different scanning timings, and all the point clouds contained in one transverse array have the same scanning timing.

[0023] Optionally, the method further includes:

[0024] According to the setting positions, setting angles, field angles of view and the time when the camera shutter is triggered to start collecting images of the laser radar and the camera, the point clouds obtained by scanning the surrounding environment by the laser radar are screened out, which are located within the coverage of the camera collected images and whose timestamps are not earlier than the time when the camera shutter is triggered to start collecting images;

[0025] The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including:

[0026] According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the screened point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time.

[0027] Optionally, the method further includes:

[0028] Detecting a target region of interest in an original image captured by a camera, and determining a center line in the target region;

[0029] Collecting the timestamp when the laser radar scans to the physical position corresponding to the center line;

[0030] Determine the timestamp of the laser radar scanning the point cloud within the target area according to the scanning frequency of the laser radar, the scanning line beam, and the timestamp when the laser radar scans the physical position corresponding to the center line;

[0031] According to the timestamp of the point cloud scanned by the laser radar to the target area, the camera shutter triggering time and the exposure duration, the point cloud and image pixels in the target area are interpolated to estimate the position of the point cloud in the target area corresponding to the moment when the camera exposure is completed.

[0032] Optionally, the method further includes:

[0033] Analyze the image captured by the camera through the large model to obtain the target of interest in the image and the motion attributes of the target, wherein the motion attributes include size, motion speed, motion acceleration and motion direction;

[0034] According to the motion properties of the target, a velocity interpolation compensation parameter is calculated to adjust the point cloud position obtained by scanning the surrounding environment by the laser radar;

[0035] The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including:

[0036] According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the adjusted point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time.

[0037] The present disclosure also provides a device for spatiotemporal alignment of point clouds and image pixels, the device comprising:

[0038] An acquisition unit, used to acquire the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment;

[0039] A capture unit is used to call the underlying software interface that controls the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data;

[0040] An interpolation unit, configured to perform interpolation calculation on the point cloud and image pixels according to the timestamp of the point cloud, the camera shutter triggering moment and the exposure duration, so as to estimate the point cloud position corresponding to the moment when the camera exposure is completed;

[0041] The fusion unit is used to fuse the point cloud and image pixels after interpolation calculation to obtain the point cloud and image pixels after time and space alignment.

[0042] The present disclosure also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0043] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0044] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0045] Through the embodiments of the present disclosure, firstly, the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment is collected; the underlying software interface for controlling the camera to capture images is called to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data; further, according to the timestamp of the point cloud, the camera shutter triggering moment and the actual exposure time, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the moment when the camera exposure is completed; finally, the interpolated point cloud and the image pixels are fused to obtain the point cloud and image pixels aligned in time and space.

[0046] In the above manner, the technical solution of the present disclosure can accurately collect the exact moment of camera shutter triggering and the actual duration of the exposure process by directly accessing the underlying software interface, thus breaking the situation dominated by third-party chip suppliers in the past. These suppliers usually only provide estimated shutter start time and exposure cycle information for commercial confidentiality considerations. After adopting the technical solution of the present disclosure, more accurate time data can be obtained, which is particularly important for application scenarios that require high-precision time synchronization, such as advanced image processing fields such as autonomous driving, three-dimensional reconstruction or augmented reality. Especially when the point cloud is fused with two-dimensional image pixels, ensuring the high consistency of the timestamps between the two is one of the key factors to achieve high-quality results. Using real and accurate shutter trigger moment and exposure duration data to align the point cloud and image pixels in time and space can significantly improve the quality of the final output results. Compared with the traditional approach that relies on estimated values, this method has higher accuracy and can significantly reduce or even eliminate various errors caused by time deviation, so that the generated three-dimensional model or augmented reality picture looks more natural and smooth, and the details are also richer and more delicate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a schematic diagram of a laser radar and camera observation area shown in an exemplary embodiment;

[0049] Figure 2is a flow chart of a method for spatiotemporal alignment of point clouds and image pixels shown in an exemplary embodiment;

[0050] Figure 3 is a flow chart of another method for spatiotemporal alignment of point cloud and image pixels shown in an exemplary embodiment;

[0051] Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment;

[0052] Figure 5 It is a block diagram of a device for spatiotemporal alignment of point clouds and image pixels, shown in an exemplary embodiment. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present disclosure.

[0054] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this disclosure. In some other embodiments, the steps included in the method may be more or less than those described in this disclosure. In addition, a single step described in this disclosure may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this disclosure may also be combined into a single step for description in other embodiments.

[0055] In applications such as autonomous driving, 3D reconstruction, and augmented reality, the spatiotemporal alignment of point clouds and image pixels is crucial. It ensures that the point cloud data collected by the radar and the image data taken by the camera can be accurately matched in space and time, so that the 3D points in the point cloud and the 2D pixels in the image are captured synchronously at the same time point, and their spatial positions are consistent. This technology is crucial to improving the accuracy and robustness of scene understanding because it can make full use of the precise 3D geometric information provided by the point cloud data and the rich texture and color information provided by the image data.

[0056] For example, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing a laser radar and a camera observation area according to an exemplary embodiment. Figure 1As shown in the figure, the LiDAR is installed in the center of the roof, which can obtain a wider field of view to facilitate environmental scanning and thus have a comprehensive understanding of the surrounding environment. The on-board camera can be installed in different positions of the vehicle. Figure 1 There are six on-board cameras, which are installed in front of the car, front left, front right, rear left, rear right and rear. They collect visual information from different perspectives for functions such as road monitoring around the car and traffic sign recognition. The area scanned by the laser radar is 360° around, and the area it covers can be called observation area 1. The area scanned by the on-board camera is Figure 1 There are six areas between the two solid lines extending from the on-board camera in the figure, which can be called observation area 2. The image pixels taken by the on-board camera and the point cloud in the lidar scanning area are temporally and spatially aligned, that is, the image pixels and point cloud in the overlapping area of ​​observation area 1 and observation area 2 are aligned.

[0057] Traditionally, spatiotemporal alignment relies on hardware clock synchronization, which ensures that the acquisition time of the two sensors is consistent by triggering the acquisition of point clouds and images at the same time. However, since the acquisition frequencies of point clouds and images may be different, such as point clouds acquired at 10Hz and images acquired at 30Hz, they cannot always be perfectly aligned. To solve this problem, researchers have developed a post-processing soft synchronization method that uses software algorithms to find the closest image frame based on the timestamp of the point cloud, and uses frame interpolation and other methods to achieve frame alignment, thereby achieving spatiotemporal alignment. Although the accuracy of this method may be slightly lower than that of hardware synchronization, it provides a flexible and cost-effective solution, especially when hardware synchronization equipment is not feasible or too expensive.

[0058] In addition, the soft synchronization method relies on estimated data provided by third-party chip suppliers, such as the camera shutter trigger time and exposure time. When the data provided by third-party chip suppliers is an estimate rather than an accurate measurement, it will lead to time deviations in actual applications. Technicians try to correct these deviations through algorithms, such as using motion information in image sequences or machine learning methods to predict and correct time errors. However, due to the inaccuracy of the original time data, these methods are still limited in accuracy and it is difficult to fully meet the needs of high-precision applications. As a result, in the process of fusion of point clouds and two-dimensional image pixels, the inconsistency of timestamps will cause the generated three-dimensional model or augmented reality screen to appear unnatural and unsmooth, and the details are not rich and delicate enough. In addition, different types of lidar have different laser scanning characteristics, and the timestamps of the scanned point clouds are different, which will cause limitations in the accuracy of the final spatiotemporal alignment, making it difficult to fully meet the needs of high-precision applications.

[0059] In view of this, the present disclosure aims to propose a technical solution that can perform spatiotemporal alignment of point clouds and image pixels based on precise shutter start time and exposure period.

[0060] The technical solution first collects the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment; then, calls the underlying software interface that controls the camera to capture images to capture the camera shutter triggering time and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data; further, according to the timestamp of the point cloud, the camera shutter triggering time and the exposure time, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the moment when the camera exposure is completed; finally, the interpolated point cloud and the image pixels are fused to obtain the point cloud and image pixels aligned in time and space.

[0061] For example, an autonomous vehicle is equipped with a 360° scanning LiDAR and a forward-facing camera on the roof. Our goal is to align the point cloud data acquired by the LiDAR with the image taken by the forward-facing camera in terms of timestamps. This can be achieved through the following steps: Step 1: Collect the timestamp of the LiDAR point cloud: The LiDAR starts a full 360° scan and records the timestamp for each point cloud during the scan. Start scanning, until the time End, each beam of point cloud in the entire scanning process has its own accurate timestamp, for example, the timestamp of the first beam of point cloud is , the timestamp of the last point cloud is Step 2: Control the camera to shoot and record the time information: The system sends a command to the forward camera to trigger the shutter, and then captures the time when the camera shutter is triggered from the underlying software interface that controls the camera to shoot images. , the exposure time of the shot is 0.05 seconds (50 milliseconds). This means that Start to + 0.05 seconds (i.e. ) During this time, the camera's sensor is collecting light. After the exposure is over, the camera processes the raw data and generates the final image. This process is usually very fast and can be ignored. The timestamp corresponding to the final image can be considered to be Step 3: Calculate the interpolated point cloud position: Use linear interpolation or other interpolation methods to estimate the position of the point cloud based on the timestamp of the lidar point cloud, the camera shutter trigger time, and the camera exposure time. The position of each point cloud at the time (i.e. the time when the camera exposure is completed). For example, if a point cloud is at The position at the moment is P1, The position at the moment is P9, then the point cloud of the beam can be calculated by interpolation. Step 4: Fusion of point cloud and image pixels: The interpolated point cloud data is transformed into the same coordinate system as the camera image. This usually involves using a pre-calibrated extrinsic matrix (parameters describing the relative position relationship between the lidar and the camera) to align the transformed point cloud data with the camera image to ensure that they are consistent in space and time.

[0062] It can be seen that in the technical solution of the present invention, the timestamp of the point cloud obtained by scanning the surrounding environment by the laser radar is first collected; the underlying software interface for controlling the camera to capture images is called to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data; further, according to the timestamp of the point cloud, the camera shutter triggering moment and the actual exposure time, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the moment when the camera exposure is completed; finally, the interpolated point cloud and the image pixels are fused to obtain the point cloud and image pixels aligned in time and space.

[0063] In the above manner, the technical solution of the present disclosure can accurately collect the exact moment of camera shutter triggering and the actual duration of the exposure process by directly accessing the underlying software interface, thus breaking the situation dominated by third-party chip suppliers in the past. These suppliers usually only provide estimated shutter start time and exposure cycle information for commercial confidentiality considerations. After adopting the technical solution of the present disclosure, more accurate time data can be obtained, which is particularly important for application scenarios that require high-precision time synchronization, such as advanced image processing fields such as autonomous driving, three-dimensional reconstruction or augmented reality. Especially when the point cloud is fused with two-dimensional image pixels, ensuring the high consistency of the timestamps between the two is one of the key factors to achieve high-quality results. Using real and accurate shutter trigger moment and exposure duration data to align the point cloud and image pixels in time and space can significantly improve the quality of the final output results. Compared with the traditional approach that relies on estimated values, this method has higher accuracy and can significantly reduce or even eliminate various errors caused by time deviation, so that the generated three-dimensional model or augmented reality picture looks more natural and smooth, and the details are also richer and more delicate.

[0064] The present disclosure is described below through specific embodiments in combination with specific application scenarios.

[0065] See also Figure 2 , Figure 2 The flowchart of a method for spatiotemporal alignment of a point cloud and image pixels is shown in an exemplary embodiment. The method may perform the following steps:

[0066] Step 202: Collect the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment.

[0067] For example, the LiDAR starts a full 360° scan and records a timestamp for each point cloud during the scan. Start scanning, until the time End, each beam of point cloud in the entire scanning process has its own accurate timestamp, for example, the timestamp of the first beam of point cloud is , the timestamp of the last point cloud is .

[0068] Among them, LiDAR is a sensor that determines the distance of a target by emitting a laser beam and measuring the time it takes to reflect back. It can generate high-precision three-dimensional point cloud data. A point cloud is a collection of many discrete points, each of which contains spatial coordinates and possible color or intensity information. In autonomous driving, point clouds are often used to represent the environment scanned by LiDAR. LiDAR devices usually have a built-in high-precision time synchronization mechanism that can record an accurate timestamp for each point cloud during each scan, which can be achieved through hardware timestamps or software timestamps. The present disclosure does not limit the area scanned by the LiDAR, the specific angle of the scan, and the specific method of the scan.

[0069] Step 204: Call the underlying software interface that controls the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of the image; wherein, after the exposure time ends, the camera collects the original image data.

[0070] For example, the system sends a command to the forward-facing camera to trigger the shutter, and then captures the camera shutter triggering time from the underlying software interface that controls the camera to capture images. , the exposure time of the shot is 0.05 seconds (50 milliseconds). This means that Start to + 0.05 seconds (i.e. ) During this time, the camera's sensor is collecting light. After the exposure is over, the camera processes the raw data and generates the final image. This process is usually very fast and can be ignored. The timestamp corresponding to the final image can be considered to be .

[0071] The shutter is the trigger mechanism in the camera that controls the light entering the camera's photosensitive element (such as a CCD or CMOS sensor). Triggering the shutter means starting the process, allowing light to enter the camera and starting the image exposure process. The underlying software interface is a set of APIs (Application Programming Interface) provided in the operating system or camera firmware that allows developers or system software to interact directly with the hardware. In this case, the underlying software interface allows the system to obtain detailed information about the shooting process, such as the exact moment the camera triggers the shutter and the actual exposure time. Exposure time refers to the length of time the camera shutter is open, allowing light to shine on the photosensitive element. arrive This period of time refers to the time from the shutter opening ( ) to shutter closed ( ), which is the time period during which the camera sensor collects light. This period of time determines how much light the sensor collects, which affects the brightness and quality of the image. Here, the exposure time is 0.05 seconds (50 milliseconds). After the exposure is over, the camera's processor processes the collected raw image data. This includes steps such as demosaicing, white balance adjustment, color correction, and noise reduction to produce a high-quality image. Since the camera processes the raw data very quickly and can be ignored, the time it takes to generate the final image can be approximated as the time the shutter is closed. .

[0072] Step 206: performing interpolation calculation on the point cloud and the image pixels according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, so as to estimate the point cloud position corresponding to the camera exposure completion time.

[0073] For example, linear interpolation or other interpolation methods can be used to estimate the time stamp of the lidar point cloud, the camera shutter trigger time, and the camera exposure time. The position of each point cloud at the time (i.e. the time when the camera exposure is completed). For example, if a beam of point clouds is at The position at the moment is P1, The position at the moment is P10, then the interpolation can be used to calculate Point cloud bundle position P4 at time instant.

[0074] Among them, interpolation is a mathematical method used to estimate the value of the middle point between two known data points. Linear interpolation is a simple interpolation method. Its idea is to assume that the change between two data points is linear. If the known point P1 is at time The position and point P10 at time The position of the linear interpolation can be estimated by the equation of the straight line at time Position P4. In addition to linear interpolation, there are many other interpolation methods, such as polynomial interpolation, spline interpolation, etc. These methods can be more complex, but can also more accurately estimate the value of the intermediate point, especially in the case of nonlinear changes. Before interpolation, it may be necessary to pre-process the point cloud data, such as denoising, filtering, etc., to improve the accuracy of interpolation. Implementing the interpolation algorithm in software may involve writing or using an existing mathematical library to perform interpolation calculations. The present disclosure does not limit how to perform the interpolation calculations.

[0075] Step 208: Fusing the interpolated point cloud with the image pixels to obtain the time-space aligned point cloud and image pixels.

[0076] For example, the interpolated point cloud data is converted to the same coordinate system as the camera image. This process may involve using a pre-calibrated extrinsic parameter matrix (parameters that describe the relative position relationship between the lidar and the camera) to align the transformed point cloud data with the camera image to ensure that they are consistent in space and time.

[0077] Among them, spatiotemporal alignment refers to ensuring that data from different sensors (such as point cloud data from lidar and image pixel data from cameras) are consistent in space and time, which is usually achieved through precise time synchronization and spatial registration. Point cloud data is usually generated in the lidar coordinate system and needs to be converted to the camera coordinate system in order to be fused with image data. This process involves the use of extrinsic matrix, which is a parameter that describes the relative position relationship between the lidar and the camera. The extrinsic matrix is ​​a matrix that describes the spatial position and orientation between sensors, which includes rotation matrices and translation vectors. Through this matrix, point cloud data can be converted from the lidar coordinate system to the camera coordinate system. Image registration is to align two or more images in space so that they represent different perspectives of the same scene. In point cloud and image fusion, registration involves matching point cloud data with image data at the pixel level. Data fusion is the merging of data from different sensors into a unified representation. When point cloud and image data need to be fused, the fusion may involve extracting color information of point cloud data from image data, or mapping texture information of image data to point cloud data. The present disclosure does not limit the specific fusion method.

[0078] In one embodiment shown, the method also includes: dividing the time period for the laser radar to complete a complete scan into multiple sub-time periods, each sub-time period corresponding to the time interval between two adjacent beams of point clouds with different laser radar scanning timings; interpolating the point cloud and the image pixels according to the timestamp of the point cloud, the camera shutter trigger moment and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion moment, including: determining the camera exposure completion moment according to the camera shutter trigger moment and the exposure duration; within the sub-time period covering the camera exposure completion moment, interpolating the point cloud and the image pixels according to the timestamps of the two adjacent beams of point clouds corresponding to the sub-time period and the camera exposure completion moment to estimate the point cloud position corresponding to the camera exposure completion moment.

[0079] For example, the LiDAR scans a full 360° in 0.1 seconds and continuously records point cloud data during this process. The LiDAR can collect 1,000 point cloud data points on the same horizontal line in one scan. The time interval between each point cloud data point is 0.1 seconds divided by 1000, which is 0.0001 seconds (0.1 milliseconds). Start scanning, such as , the timestamp of the first point cloud may be , the timestamp of the second point cloud is , and so on, until the timestamp of the 999th point cloud is , the timestamp of the last point cloud is , marking the end of a scanning cycle. While the LiDAR is scanning, the system Trigger the shutter of the forward-facing camera, such as , the exposure time is 0.02 seconds (20 milliseconds). This means that from Start to Seconds (i.e. ) During this time, the camera’s sensor is collecting light. In order to time-align the lidar point cloud data with the image captured by the camera, we need to find the time that coincides with the end of the camera’s exposure. The closest point cloud data. Specifically, we need to find the point cloud data with the closest timestamp. Assume that the timestamps of these two point clouds are and , the locations are and The linear interpolation method can be used to estimate the time stamps and positions of the two point clouds. The point cloud position at the moment, the specific formula is as follows:

[0080]

[0081] Among them, each pixel in the image captured by the camera has the same timestamp, but the timestamps of different point clouds are affected by the laser radar scanning method. In addition to the laser point cloud scanning method obtained by a one-time complete scan, the scanning method in which a bunch of laser beams are emitted in a time sequence will bring time sequence differences to the collected point clouds. In this embodiment, This is the moment when the LiDAR starts scanning, and no point cloud data has been collected yet. It is the timestamp of the first point cloud data point after the LiDAR starts scanning, indicating 0.0001 seconds (0.1 milliseconds) have passed since the start. The LiDAR scanning frequency refers to the number of times the LiDAR completes a full scan per second. For example, 10 Hz means 10 full scans per second. A point cloud is a collection of many discrete points, each of which contains spatial coordinates. And possible color or intensity information. In autonomous driving, point clouds are often used to represent the environment scanned by LiDAR. In the above process, find the time when the camera exposure ends. The closest two point cloud data points is one of the key steps. In the specific implementation, first obtain all point cloud data and their corresponding timestamps from the lidar. Traverse the point cloud data list and find the closest timestamp. For two point cloud data points, a binary search algorithm can be used to efficiently find the closest timestamp. The present disclosure does not limit the specific search method.

[0082] In one embodiment shown, the laser radar is a mechanical rotating laser radar, and its scanning direction is longitudinal array scanning. The time period for completing a complete scan of the laser radar is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including: the mechanical rotating laser radar completes a complete scan of the surrounding environment to obtain the time period for completing the scan and multiple longitudinal arrays obtained by the scan, each longitudinal array containing a number of point clouds collected by the mechanical rotating laser radar; according to the multiple longitudinal arrays obtained, the time period for completing the scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between the point clouds collected by two adjacent longitudinal arrays with different mechanical rotating laser radar scanning timings; wherein, the point clouds contained in different longitudinal arrays have different scanning timings, and all point clouds contained in one longitudinal array have the same scanning timing.

[0083] For example, an autonomous vehicle is equipped with a mechanical rotating laser radar that can perform a 360° full environment scan and generate high-precision 3D point cloud data. The laser radar has a scanning frequency of 10Hz, which means that it completes a full scanning cycle every 0.1 seconds. In this scan, the laser radar generates 64 longitudinal arrays, each of which contains a number of (such as 100) point cloud data points. In a complete scanning cycle, the laser radar collects a total of 6400 point clouds (64 arrays * 100 point clouds / array). Since the time period to complete a full scan is 0.1 seconds, the 64 longitudinal arrays of the laser radar are collected in sequence in this 0.1 second. Assuming that the rotation speed of the laser radar is uniform, the collection time interval of each longitudinal array is 0.1 seconds / 64 = 0.0015625 seconds (about 1.5625 milliseconds). According to the 64 longitudinal arrays, the 0.1 second time period can be divided into 64 sub-time periods, each of which corresponds to the collection time interval of two adjacent longitudinal arrays. For example, the first sub-time period is from 0 seconds to 0.0015625 seconds, the second sub-time period is from 0.0015625 seconds to 0.003125 seconds, and so on, until the 64th sub-time period ends at 0.1 seconds. Point clouds on different longitudinal arrays have different scanning timings. For example, the point cloud on the first longitudinal array is collected between 0 seconds and 0.0015625 seconds, while the point cloud on the second longitudinal array is collected between 0.0015625 seconds and 0.003125 seconds. All point clouds on a longitudinal array have the same scanning timing, that is, they are all collected in the same sub-time period.

[0084] Among them, mechanical rotating laser radar is a common type of laser radar, which performs 360° environmental scanning by physical rotation. This type of radar is usually mounted on a rotating platform, and the laser transmitter and receiver on the rotating platform will scan the surrounding environment as the platform rotates. The working principle of mechanical rotating laser radar is to calculate the distance by emitting laser pulses and measuring the time for these pulses to be reflected back from the target, thereby generating three-dimensional point cloud data. Mechanical rotating laser radar usually has a multi-beam design, such as 16 lines, 32 lines, 64 lines, etc. Each beam represents a laser beam in the vertical direction, and these beams are distributed in the vertical direction to form multiple longitudinal scanning planes. Each beam scans in the horizontal direction as the rotating platform rotates, thereby forming several columns of data in the vertical direction. For example, if the laser radar has 32 beams, then in one complete rotation, each beam will scan a line in the vertical direction, and there will be a total of 32 lines of data. Due to the multi-beam design, mechanical rotating laser radar can provide higher resolution in the vertical direction, which helps to capture details in the environment in more detail.

[0085] In one embodiment shown, the laser radar is a hybrid solid-state laser radar, and its scanning direction is transverse array scanning. The time period for the laser radar to complete a complete scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including: performing a complete scan of the surrounding environment with the hybrid solid-state laser radar to obtain the time period for completing the scan and multiple transverse arrays obtained by the scan, each transverse array containing a number of point clouds collected by the hybrid solid-state laser radar; according to the multiple transverse arrays obtained, the time period for completing the scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between the point clouds collected by two adjacent transverse arrays with different hybrid solid-state laser radar scanning timings; wherein, the point clouds contained in different transverse arrays have different scanning timings, and all point clouds contained in one transverse array have the same scanning timing.

[0086] For example, a self-driving car is equipped with a hybrid solid-state lidar designed for 120° field of view coverage in the forward direction. This lidar uses electronic scanning technology, such as micro-electromechanical systems or optical phased arrays, to achieve fast lateral array scanning. The lidar begins to scan 120° forward, completing a full 120° environment scan every 0.05 seconds. In this scan, the lidar generates 16 lateral arrays, each of which contains several beams (for example, 60 beams) of point cloud data points. In a complete scanning cycle, the lidar collects a total of 960 beams of point clouds (16 arrays * 60 beams of point clouds / array). The time cycle for the hybrid solid-state lidar to complete a complete scan is 0.05 seconds. In this 0.05 second, the 16 lateral arrays of the lidar are collected sequentially. Assuming that the scanning speed of the lidar is uniform, the collection time interval of each lateral array is 0.05 seconds / 16 = 0.003125 seconds (about 3.125 milliseconds). According to the 16 transverse arrays, the 0.05-second time period can be divided into 16 sub-time periods, each of which corresponds to the acquisition time interval between two adjacent transverse arrays. For example, the first sub-time period is from 0 seconds to 0.003125 seconds, the second sub-time period is from 0.003125 seconds to 0.006250 seconds, and so on, until the 16th sub-time period ends at 0.05 seconds. Point clouds on different transverse arrays have different scanning timings. For example, the point cloud on the first transverse array is collected between 0 seconds and 0.003125 seconds, while the point cloud on the second transverse array is collected between 0.003125 seconds and 0.006250 seconds, and all point clouds on a transverse array have the same scanning timing, that is, they are all collected in the same sub-time period.

[0087] Among them, hybrid solid-state LiDAR is a technology between traditional mechanical rotating LiDAR and pure solid-state LiDAR. It combines the advantages of mechanical scanning and solid-state scanning, aiming to improve reliability and reduce costs while maintaining high performance. Hybrid solid-state LiDAR usually uses mirrors or other small mechanical components to achieve scanning. These mirrors or small mechanical components can move quickly in one dimension to scan out rows of data. In the vertical direction, the laser beam can cover different height ranges through multiple beams or by adjusting the angle of the mirror.

[0088] In this embodiment, such subdivided sub-time periods can be used to accurately synchronize the lidar point cloud data and the camera image pixels. For example, if the camera triggers the shutter at 0.015 seconds and the exposure time is 10 milliseconds, then we can determine the lidar point cloud data corresponding to the camera exposure moment by looking for the 5th sub-time period (assuming that the exposure completion moment 0.025 seconds falls within the 5th sub-time period).

[0089] In one embodiment shown, the method also includes: based on the setting positions, setting angles, field of view angles of the laser radar and the camera, and the time when the camera shutter is triggered to start collecting images, screening out point clouds obtained by the laser radar scanning the surrounding environment, which are located within the coverage of the camera collected image and whose timestamps are no earlier than the time when the camera shutter is triggered to start collecting images; interpolating the point cloud with the image pixels according to the timestamp of the point cloud, the camera shutter trigger time and the exposure time to estimate the point cloud position corresponding to the time when the camera exposure is completed, including: interpolating the screened point cloud with the image pixels according to the timestamp of the point cloud, the camera shutter trigger time and the exposure time to estimate the point cloud position corresponding to the time when the camera exposure is completed.

[0090] For example, a lidar is installed at the front of a vehicle with a horizontal field of view of 120°, a vertical field of view of 30°, and a scanning frequency of 10Hz. A camera is installed below the lidar with a horizontal field of view of 100° and a vertical field of view of 25°. Assume that at 10:30:00.150 AM, the system triggers the shutter of the forward camera with an exposure time of 50 milliseconds (0.05 seconds). At 10:30:00.150 AM, the lidar is scanning, and it can be assumed that it has just started a new scanning cycle. From the point cloud data generated by the lidar, filter out the point clouds within the camera field of view of 100°×25°. Further filter out the point cloud data with timestamps no earlier than 10:30:00.150 AM, which correspond to the images collected during the camera exposure. For the filtered point cloud data, calculate the timestamps of these point clouds according to the lidar scanning frequency (such as the timestamp of point cloud P5 ), and then calculate the camera exposure completion time (such as ), using an interpolation method (e.g. linear interpolation), based on the timestamp of the point cloud P5 and the camera exposure ends , estimated at the time when exposure is completed The corresponding point cloud position P5'.

[0091] Among them, the setting position refers to the installation position of the lidar and camera on the vehicle. The proper position is crucial to ensure the overlap of the sensor coverage area and the synchronization of data. The setting angle refers to the installation angle of the lidar and camera, which determines the direction of their field of view. For lidar and cameras, this usually includes horizontal and vertical field of view angles. The field of view angle refers to the stereoscopic angle range that the lidar and camera can detect. The horizontal field of view angle and the vertical field of view angle together define the detection range of the sensor.

[0092] In this embodiment, through the above method, it can be ensured that only those point cloud data that are within the coverage of the camera image and whose timestamp is not earlier than the start time of camera exposure are used for interpolation calculation with image pixels. This helps to improve the efficiency and accuracy of data fusion and provide more accurate environmental perception information for the autonomous driving system.

[0093] In one embodiment shown, the method also includes: detecting a target area of ​​interest in the original image captured by the camera, and determining a center line in the target area; collecting a timestamp when the laser radar scans to a physical position corresponding to the center line; determining a timestamp when the laser radar scans to a point cloud within the target area based on a scanning frequency of the laser radar, a scanning line beam, and a timestamp when the laser radar scans to a physical position corresponding to the center line; interpolating the point cloud within the target area with the image pixels based on the timestamp when the laser radar scans to the point cloud within the target area, the camera shutter trigger moment, and the exposure duration, so as to estimate the position of the point cloud within the target area corresponding to the moment when the camera exposure is completed.

[0094] For example, the camera captures an original image containing a pedestrian in front at 10:30:00.150 AM. The target detection algorithm is used to detect the pedestrian in the image, and the pedestrian is used as the region of interest. A rectangular frame is generated to enclose the area where the pedestrian is located as the target area. The vertical center line of the rectangular frame is calculated, and this line will be used as the basis for the alignment calculation. The timestamp when the lidar scans to the physical position corresponding to the vertical center line is collected. The lidar scans the physical position corresponding to the vertical center line of the area where the pedestrian is located at 10:30:00.155 AM, and records the timestamp Tc at this time.

[0095] Then, according to the laser radar scanning frequency of 10Hz, it is determined that the laser radar scans a circle in 0.1 seconds. Assuming that the laser radar is a longitudinal array scanning (forming several columns in the vertical direction), there are 16 beams in one scanning circle. It can be known that there are 16 beams of point clouds corresponding to one horizontal plane. These 16 beams of point clouds are collected in sequence within 0.1 seconds. All the point clouds on the beams where each beam of point cloud is located have the same timestamp, and then combined with the timestamp As a benchmark, more accurate timestamps of other point clouds except those on the vertical center line can be calculated in turn. Then, according to the laser radar scanning range, the size of the target area, etc., the timestamp range of the point cloud scanned by the laser radar in the target area can be determined, for example -0.002 seconds and +0.002 seconds; Finally, the laser radar -0.002 seconds to The timestamp of the point cloud collected at +0.002 seconds is time-aligned with the end time of camera exposure to estimate the position of the point cloud in the target area corresponding to the end time of camera exposure.

[0096] Among them, whether the center line is a horizontal center line or a vertical center line depends on the scanning method of the laser radar. If it is scanned from left to right (such as a mechanical rotating laser radar), then the center line will be the left and right center line of the target object. If it is scanned up and down (such as a hybrid solid-state laser radar), then the center line will be the upper and lower center line of the target object.

[0097] It should be noted that in a complex dynamic environment, the rapid movement of objects may make it difficult for traditional methods to accurately estimate their positions. Moreover, for a spatial moving object based on the time axis as the baseline (such as the pedestrian in this embodiment), the longer the time required to compensate for the timestamps of each point cloud in the target area where it is located, the greater the error between the calculated point cloud timestamp and the real point cloud timestamp. This embodiment can relatively reduce the cumulative error in the calculation of point cloud timestamps based on the compensation before and after the center line moment. In this way, the point cloud in the target area can be more accurately aligned with the camera image pixels, thereby improving the perception accuracy and robustness of the system, which is suitable for application scenarios that require high-precision alignment, such as target tracking, obstacle detection, etc.

[0098] In one embodiment shown, the method also includes: analyzing an image captured by a camera through a large model to obtain a target of interest in the image and motion attributes of the target, the motion attributes including size, motion speed, motion acceleration and motion direction; calculating a speed interpolation compensation parameter according to the motion attributes of the target to adjust the point cloud position obtained by scanning the surrounding environment by a laser radar; interpolating the point cloud with image pixels according to the timestamp of the point cloud, the camera shutter trigger moment and the exposure duration to estimate the point cloud position corresponding to the moment when the camera exposure is completed, including: interpolating the adjusted point cloud with image pixels according to the timestamp of the point cloud, the camera shutter trigger moment and the exposure duration to estimate the point cloud position corresponding to the moment when the camera exposure is completed.

[0099] For example, a self-driving car is equipped with a high-speed camera and a high-performance lidar. The camera is used to capture images of road conditions, while the lidar is used to obtain 3D point cloud data of the surrounding environment. The camera captures a raw image containing the vehicle in front at 10:30:00.150 AM. A deep learning model is used to detect the target of interest in the image as the vehicle in front and track its movement. By analyzing the motion properties of the vehicle in consecutive frames, the model calculates that the vehicle is moving at a constant speed of 60 km / h to the north and the body length is 4 meters. Then, based on the motion properties of the vehicle, a speed interpolation compensation parameter is calculated. This parameter will be used to adjust the point cloud position obtained by the lidar scan to reflect the movement of the vehicle during the camera exposure. Assume that the vehicle moves about 0.83 meters during the 50 milliseconds of camera exposure. The point cloud position obtained by the lidar scan is adjusted to move 0.83 meters north to reflect the actual movement of the vehicle during the exposure. Finally, the adjusted point cloud is interpolated with the image pixels to estimate the point cloud position corresponding to the time when the camera exposure is completed (10:30:00.200 AM).

[0100] Among them, the specific category of the target of interest can also be identified, such as the target of interest is a pedestrian, a vehicle, a tree, etc., so as to better perform speed interpolation compensation according to the category. For example, if the target of interest is identified as a tree, then its movement speed and movement acceleration are both 0, thereby reducing the error in the model's calculation of the speed of the target of interest. In terms of implementation means, a pre-trained deep learning model, such as a convolutional neural network, can be used to analyze images and detect objects in images. The present disclosure does not limit the specific type of the deep learning model.

[0101] In order to help those skilled in the art better understand the present method as a whole, Figure 3 This method is described.

[0102] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of another method for spatiotemporal alignment of point cloud and image pixels, shown in an exemplary embodiment. Figure 3 As shown in the figure, firstly, the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment is collected, and the camera shutter triggering time and actual exposure time, as well as the motion attribute information such as the speed, acceleration and movement direction of the camera shooting target are obtained by calling the underlying software interface that controls the camera to shoot images; then, the camera exposure completion time is determined according to the camera shutter triggering time and the actual exposure time, so as to align the time axis of the point cloud and the image pixel; then, the point cloud motion compensation method is determined based on the laser scanning method (such as longitudinal array scanning or transverse array scanning), and based on the setting position, setting angle, field angle of the laser radar and the camera and the time when the camera shutter is triggered to start collecting images, the point cloud obtained by the laser radar scanning the surrounding environment, which is located within the coverage of the camera collection image and the time stamp is not earlier than the time when the camera shutter is triggered to start collecting images, is screened out for subsequent interpolation calculation (i.e., determining the starting time of point cloud compensation), and adjusting the point cloud position based on the target movement speed and direction, etc.; finally, the above methods are superimposed, the point cloud and image pixels are interpolated, and the position of the point cloud corresponding to the time when the camera exposure is completed is estimated, and finally the result of spatiotemporal alignment of the point cloud and the image pixel is obtained.

[0103] Corresponding to the above-mentioned embodiment of the method for spatiotemporal alignment of point cloud and image pixels, the present disclosure also provides an embodiment of a device for spatiotemporal alignment of point cloud and image pixels.

[0104] See also Figure 4 , Figure 4 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present disclosure can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present disclosure do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0105] See also Figure 5 , Figure 5 FIG. 5 is a block diagram of a device for spatiotemporal alignment of a point cloud and image pixels, shown in an exemplary embodiment. The device 500 for spatiotemporal alignment of a point cloud and image pixels can be applied to Figure 4 In the electronic device shown, the technical solution of the present disclosure is implemented. The device includes:

[0106] The acquisition unit 502 is used to acquire the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment;

[0107] The capture unit 504 is used to call the underlying software interface for controlling the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data;

[0108] An interpolation unit 506 is used to perform interpolation calculation on the point cloud and the image pixels according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, so as to estimate the point cloud position corresponding to the camera exposure completion time;

[0109] The fusion unit 508 is used to fuse the point cloud and image pixels after interpolation calculation to obtain the point cloud and image pixels after time and space alignment.

[0110] In some embodiments, the apparatus further comprises:

[0111] A separation unit, used to divide the time period for the laser radar to complete a complete scan into a plurality of sub-time periods, each sub-time period corresponding to the time interval between two adjacent beams of point clouds with different laser radar scanning timings;

[0112] The interpolation unit comprises:

[0113] A first determining subunit, configured to determine a camera exposure completion time according to the camera shutter triggering time and the exposure duration;

[0114] The first estimation subunit is used to interpolate the point cloud and the image pixels within a sub-time period covering the moment when the camera exposure is completed, according to the timestamps of two adjacent point clouds corresponding to the sub-time period and the moment when the camera exposure is completed, so as to estimate the point cloud position corresponding to the moment when the camera exposure is completed.

[0115] In some embodiments, the laser radar is a mechanical rotating laser radar, and its scanning direction is longitudinal array scanning. The separation unit includes:

[0116] A first scanning subunit is used to complete a complete scan of the surrounding environment by the mechanical rotating laser radar, obtain a time period for completing the scan and a plurality of longitudinal arrays obtained by the scan, each longitudinal array containing a plurality of point clouds collected by the mechanical rotating laser radar;

[0117] A first dividing subunit is used to divide the time period for completing the scanning into a plurality of sub-time periods according to the obtained plurality of longitudinal arrays, each sub-time period corresponding to a time interval between point clouds collected by two adjacent longitudinal arrays with different scanning timings of the mechanical rotating laser radar;

[0118] The point clouds contained in different longitudinal arrays have different scanning timings, and all the point clouds contained in one longitudinal array have the same scanning timing.

[0119] In some embodiments, the laser radar is a hybrid solid-state laser radar, and its scanning direction is a transverse array scan. The separation unit includes:

[0120] A second scanning subunit is used to perform a complete scan of the surrounding environment of the hybrid solid-state laser radar, obtain a time period for completing the scan and a plurality of transverse arrays obtained by the scan, each transverse array containing a plurality of point clouds collected by the hybrid solid-state laser radar;

[0121] A second dividing sub-unit is used to divide the time period for completing the scanning into a plurality of sub-time periods according to the obtained plurality of transverse arrays, each sub-time period corresponding to a time interval between point clouds collected by two adjacent transverse arrays with different hybrid solid-state laser radar scanning timings;

[0122] The point clouds contained in different transverse arrays have different scanning timings, and all the point clouds contained in one transverse array have the same scanning timing.

[0123] In some embodiments, the apparatus further comprises:

[0124] A screening unit is used to screen out point clouds obtained by scanning the surrounding environment by the laser radar, which are located within the coverage of the camera captured image and whose timestamps are not earlier than the time when the camera shutter is triggered to start collecting images, according to the setting positions, setting angles, field angles of the laser radar and the camera and the time when the camera shutter is triggered to start collecting images;

[0125] The interpolation unit comprises:

[0126] The first interpolation subunit is used to perform interpolation calculation on the screened point cloud and the image pixels according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, so as to estimate the point cloud position corresponding to the camera exposure completion time.

[0127] In some embodiments, the apparatus further comprises:

[0128] A first determining unit, configured to detect a target region of interest in an original image captured by a camera, and determine a center line in the target region;

[0129] A second acquisition unit is used to acquire a timestamp when the laser radar scans to a physical position corresponding to the center line;

[0130] A second determination unit is used to determine the timestamp when the laser radar scans the point cloud in the target area according to the scanning frequency of the laser radar, the scanning line beam and the timestamp when the laser radar scans the physical position corresponding to the center line;

[0131] The estimation unit is used to interpolate the point cloud in the target area with the image pixels according to the timestamp of the point cloud in the target area scanned by the laser radar, the camera shutter triggering time and the exposure time, so as to estimate the position of the point cloud in the target area corresponding to the moment when the camera exposure is completed.

[0132] In some embodiments, the apparatus further comprises:

[0133] An analysis unit, used to analyze the image captured by the camera through the large model to obtain the object of interest in the image and the motion attributes of the object, wherein the motion attributes include size, motion speed and motion direction;

[0134] An adjustment unit, used to calculate a velocity interpolation compensation parameter according to the motion attribute of the target, so as to adjust the point cloud position obtained by the laser radar scanning the surrounding environment;

[0135] The interpolation unit comprises:

[0136] The interpolation subunit is used to interpolate the adjusted point cloud and the image pixels according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, so as to estimate the point cloud position corresponding to the camera exposure completion time.

[0137] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0138] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0140] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0146] The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0147] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0148] The above description is only a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A method for spatiotemporal alignment of point clouds and image pixels, characterized in that: The method comprises: Collect the timestamp of the point cloud obtained by the lidar scanning the surrounding environment; Call the underlying software interface that controls the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data; Detecting a target region of interest in an original image captured by a camera, and determining a center line in the target region; Collecting the timestamp when the laser radar scans to the physical position corresponding to the center line; Determine the timestamp of the laser radar scanning the point cloud within the target area according to the scanning frequency of the laser radar, the scanning line beam, and the timestamp when the laser radar scans the physical position corresponding to the center line; According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time; The interpolated point cloud and image pixels are fused to obtain the spatiotemporally aligned point cloud and image pixels; The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including: According to the timestamp of the point cloud scanned by the laser radar to the target area, the camera shutter triggering time and the exposure duration, the point cloud and image pixels in the target area are interpolated to estimate the point cloud position in the target area corresponding to the moment when the camera exposure is completed.

2. The method according to claim 1, characterized in that The method further comprises: The time period for the laser radar to complete a complete scan is divided into a plurality of sub-time periods, each sub-time period corresponding to the time interval between two adjacent beams of point clouds with different laser radar scanning timings; The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including: Determining a camera exposure completion time according to the camera shutter triggering time and the exposure duration; In a sub-time period covering the moment when the camera exposure is completed, the point cloud and the image pixels are interpolated according to the timestamps of the two adjacent point clouds corresponding to the sub-time period and the moment when the camera exposure is completed, so as to estimate the point cloud position corresponding to the moment when the camera exposure is completed.

3. The method according to claim 2, characterized in that The laser radar is a mechanical rotating laser radar, and its scanning direction is longitudinal array scanning. The time period for the laser radar to complete a complete scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including: The mechanical rotating laser radar completes a complete scan of the surrounding environment, obtains a time period for completing the scan and a plurality of longitudinal arrays obtained by the scan, each longitudinal array comprising a plurality of point clouds collected by the mechanical rotating laser radar; According to the obtained multiple longitudinal arrays, the time period for completing the scan is divided into multiple sub-time periods, each sub-time period corresponds to the time interval between point clouds collected by two adjacent longitudinal arrays with different scanning timings of the mechanical rotating laser radar; The point clouds contained in different longitudinal arrays have different scanning timings, and all the point clouds contained in one longitudinal array have the same scanning timing.

4. The method according to claim 2, characterized in that: The laser radar is a hybrid solid-state laser radar, and its scanning direction is a transverse array scan. The time period for the laser radar to complete a complete scan is divided into multiple sub-time periods, and each sub-time period corresponds to the time interval between two adjacent beams of point clouds with different laser radar scanning timings, including: Performing a complete scan of the surrounding environment by the hybrid solid-state laser radar to obtain a time period for completing the scan and a plurality of transverse arrays obtained by the scan, each transverse array comprising a plurality of point clouds collected by the hybrid solid-state laser radar; According to the obtained multiple transverse arrays, the time period for completing the scan is divided into multiple sub-time periods, each sub-time period corresponds to the time interval between point clouds collected by two adjacent transverse arrays with different hybrid solid-state laser radar scanning timings; The point clouds contained in different transverse arrays have different scanning timings, and all the point clouds contained in one transverse array have the same scanning timing.

5. The method according to claim 1, characterized in that: The method further comprises: According to the setting positions, setting angles, field angles of the laser radar and the camera, and the time when the camera shutter is triggered to start collecting images, the point clouds obtained by scanning the surrounding environment by the laser radar are screened out, which are located within the coverage of the camera collected images and whose timestamps are not earlier than the time when the camera shutter is triggered to start collecting images; The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including: According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the screened point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time.

6. The method according to claim 1, characterized in that The method further comprises: Analyze the image captured by the camera through the large model to obtain the target of interest in the image and the motion attributes of the target, wherein the motion attributes include size, motion speed, motion acceleration and motion direction; According to the motion properties of the target, a velocity interpolation compensation parameter is calculated to adjust the point cloud position obtained by scanning the surrounding environment by the laser radar; The interpolation calculation of the point cloud and the image pixels is performed according to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration to estimate the point cloud position corresponding to the camera exposure completion time, including: According to the timestamp of the point cloud, the camera shutter triggering time and the exposure duration, the adjusted point cloud and the image pixels are interpolated to estimate the point cloud position corresponding to the camera exposure completion time.

7. A device for spatiotemporal alignment of point clouds and image pixels, characterized in that: The device comprises: The first acquisition unit is used to acquire the timestamp of the point cloud obtained by the laser radar scanning the surrounding environment; A capture unit is used to call the underlying software interface that controls the camera to capture images, so as to capture the camera shutter triggering moment and the actual exposure time in the process from the camera shutter triggering to start collecting images to the final generation of images; wherein, after the exposure time ends, the camera collects the original image data; A first determining unit, configured to detect a target region of interest in an original image captured by a camera, and determine a center line in the target region; A second acquisition unit is used to acquire a timestamp when the laser radar scans to a physical position corresponding to the center line; A second determination unit is used to determine the timestamp when the laser radar scans the point cloud in the target area according to the scanning frequency of the laser radar, the scanning line beam and the timestamp when the laser radar scans the physical position corresponding to the center line; An interpolation unit, configured to perform interpolation calculation on the point cloud and image pixels according to the timestamp of the point cloud, the camera shutter triggering moment and the exposure duration, so as to estimate the point cloud position corresponding to the moment when the camera exposure is completed; The interpolation unit is further used to perform interpolation calculation on the point cloud and image pixels in the target area according to the timestamp of the point cloud scanned by the laser radar in the target area, the camera shutter triggering time and the exposure duration, so as to estimate the point cloud position in the target area corresponding to the camera exposure completion time; The fusion unit is used to fuse the point cloud and image pixels after interpolation calculation to obtain the point cloud and image pixels after time and space alignment.

8. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 6 by calling the machine-readable instructions.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images