Data collection processing method and device, storage medium and electronic equipment

Abstract

The application provides a processing method and device for collecting data, a storage medium and an electronic device. First, initial point cloud data output by a laser radar in a rotating motion scanning period, image data output by an image collection device and first vehicle pose data of a target vehicle output by a pose detection device in the rotating motion scanning period are collected. Then, a synchronization time of the initial point cloud data and the image data is determined based on a shooting time of the image data, and second vehicle pose data of the target vehicle at the synchronization time is determined based on the first vehicle pose data. Finally, the initial point cloud data is converted into target point cloud data at the synchronization time according to the second vehicle pose data. The coordinate conversion processing is performed on the initial point cloud data, so that each initial point cloud data is reflected in the coordinate system at the synchronization time. Since the span between the synchronization time and the initial time or the last time of the rotating motion scanning period is small, the time span of the motion compensation is small, thereby improving the accuracy of the compensation.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a data processing method, apparatus, storage medium and electronic device. Background Technology

[0002] With the development of intelligent driving technology, various sensor devices (such as lidar, cameras, inertial navigation, etc.) are often used as vehicle state perception elements to acquire environmental data and vehicle state data during the driving process.

[0003] Because the vehicle's pose changes significantly during movement, and each frame of point cloud generated by the lidar takes time, the resulting point cloud differs greatly from real-world objects, leading to distortion. Currently, motion compensation is typically performed on the point cloud data to represent all point cloud data as if they were collected at the same time corresponding to the same reference vehicle's pose, thereby mitigating the distortion. This usually requires motion compensation of the point cloud data in a frame based on the pose difference, bringing it back to the point cloud data at the beginning or end of the lidar's scanning cycle. However, each compensation requires moving the point cloud data from the initial moment to the final moment (or vice versa), resulting in a large time span for motion compensation, which can easily affect the accuracy of the compensation. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, and electronic device to alleviate the technical problem of low current vehicle status perception capability.

[0005] To address the aforementioned technical problems, this application provides the following technical solution:

[0006] This application provides a method for processing collected data, including:

[0007] The system collects initial point cloud data output by the lidar during the rotational motion scanning cycle, image data output by the image acquisition device, and first vehicle pose data of the target vehicle output by the pose detection device during the rotational motion scanning cycle.

[0008] Based on the capture time of the image data, determine the synchronization time of the initial point cloud data and the image data;

[0009] Based on the first vehicle pose data, the second vehicle pose data of the target vehicle at the synchronization time are determined;

[0010] The initial point cloud data is processed by coordinate transformation based on the second vehicle pose data to obtain the target point cloud data at the synchronization time.

[0011] The step of determining the synchronization time of the initial point cloud data and the image data includes:

[0012] Determine the time when the image data was captured;

[0013] When the shooting time meets the synchronization condition, the shooting time is determined to be the synchronization time of the initial point cloud data and the image data.

[0014] The method further includes, before determining the shooting time as the synchronization time of the initial point cloud data and the image data:

[0015] When the shooting time is within the target scanning time interval of the rotational motion scanning cycle, the shooting time is determined to satisfy the synchronization condition; wherein, the target scanning time interval includes the time interval corresponding to when the lidar scans into the shooting angle range of the image acquisition device.

[0016] The step of determining the shooting time as the synchronization time of the initial point cloud data and the image data includes:

[0017] If the number of image data is equal to 1, the shooting time is determined to be the synchronization time of the initial point cloud data and the image data;

[0018] If the number of image data is greater than 1, calculate the difference between the shooting time corresponding to each image data and the midpoint time within the target scanning time interval;

[0019] The shooting time corresponding to the smaller difference value is taken as the synchronization time of the initial point cloud data and the image data.

[0020] The step of determining the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data includes:

[0021] Determine the initial adjacent vehicle pose data and the final adjacent vehicle pose data in the first vehicle pose data; wherein, the initial adjacent vehicle pose data is the first vehicle pose data corresponding to the moment adjacent to the initial moment of the target vehicle in the rotational motion scanning cycle, and the final adjacent vehicle pose data is the first vehicle pose data corresponding to the moment adjacent to the end moment of the target vehicle in the rotational motion scanning cycle.

[0022] Based on the initial adjacent vehicle pose data and the final adjacent vehicle pose data, the vehicle pose of the target vehicle is linearly interpolated to obtain the second vehicle pose data.

[0023] The step of performing coordinate transformation processing on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time includes:

[0024] Based on the second vehicle pose data, determine the target coordinates in the target radar coordinate system corresponding to each initial point cloud data at the synchronization time;

[0025] The coordinates of each initial point cloud data are converted into the target coordinates to obtain the target point cloud data.

[0026] The step of performing coordinate transformation on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time further includes:

[0027] The pixels in the image data at the synchronization time are fused with the target point cloud data and presented in the same coordinate system.

[0028] This application embodiment also provides a data processing device, including:

[0029] The acquisition module is used to acquire the initial point cloud data output by the lidar during the rotational motion scanning cycle, the image data output by the image acquisition device, and the first vehicle pose data of the target vehicle output by the pose detection device during the rotational motion scanning cycle.

[0030] The synchronization time determination module is used to determine the synchronization time of the initial point cloud data and the image data;

[0031] The second vehicle pose data determination module is used to determine the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data.

[0032] The compensation processing module is used to perform coordinate transformation processing on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time.

[0033] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described data acquisition processing methods.

[0034] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in any of the above-described data acquisition processing methods.

[0035] This application provides a data processing method, apparatus, storage medium, and electronic device. First, it acquires initial point cloud data output by a lidar during its rotational scanning cycle, image data output by an image acquisition device, and first vehicle pose data of the target vehicle during its rotational scanning cycle, output by a pose detection device. Then, it determines the synchronization time of the initial point cloud data and the image data based on the image data capture time, and determines the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data. Finally, it performs coordinate transformation processing on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time. By pre-determining the synchronization time of the initial point cloud data and the image data, and performing coordinate transformation processing on the initial point cloud data based on the vehicle pose data of the target vehicle at that synchronization time, all initial point cloud data are reflected in the coordinate system at the synchronization time. Since the span between the synchronization time and the initial or final moment of the lidar's rotational scanning cycle is small, the time span of the motion compensation is small, thereby improving the accuracy of the compensation. Attached Figure Description

[0036] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating the data processing method provided in the embodiments of this application.

[0038] Figure 2 This is a schematic diagram of a scenario for the data processing method provided in the embodiments of this application.

[0039] Figure 3 This is another scenario diagram illustrating the data processing method provided in the embodiments of this application.

[0040] Figure 4 This is a schematic diagram of the data processing device provided in the embodiments of this application.

[0041] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0042] Figure 6 This is another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] This application provides a data acquisition processing method, apparatus, storage medium, and electronic device.

[0045] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The specific process can be as follows:

[0046] S101. Collect the initial point cloud data output by the lidar during the rotational motion scanning cycle, the image data output by the image acquisition device, and the first vehicle pose data of the target vehicle during the rotational motion scanning cycle output by the pose detection device.

[0047] The lidar, image acquisition equipment, and pose detection equipment are all installed on the target vehicle. LiDAR, short for laser detection and ranging system, measures the propagation distance between the sensor transmitter and the target object, analyzes the magnitude of reflected energy, amplitude, frequency, and phase of the reflected spectrum on the target object's surface, and thus presents corresponding initial point cloud data to reflect the precise three-dimensional structural information of the target object. The initial point cloud data is a dataset of spatial points scanned by the lidar equipment. Each point contains three-dimensional coordinate information, reflection intensity information, and echo count information. The first vehicle pose data is used to characterize the pose changes of the target vehicle at various times during the rotational motion scanning cycle (e.g., the time it takes to scan one revolution of the vehicle body).

[0048] Specifically, in this embodiment, the lidar is a moving mechanical radar, which scans the environment around the target vehicle in a rotating scanning manner in the working mode, and outputs initial point cloud data at the end of each rotating scanning cycle. At the same time, the image acquisition device (e.g., vehicle-mounted camera) exposes at the exposure time in each rotating scanning cycle to output image data. The pose detection device on the target vehicle (e.g., pose sensor) also outputs multiple pose data of the target vehicle at each time in the rotating scanning cycle as the first vehicle pose data.

[0049] S102. Based on the shooting time of the image data, determine the synchronization time of the initial point cloud data and the image data.

[0050] The synchronization time is used to characterize the consistency between the initial point cloud data and the image data. Specifically, in practical applications, point cloud data is usually combined with image data to analyze obstacle information in the vehicle's surrounding environment. To ensure the effectiveness of the analysis, it is necessary to ensure that the point cloud data and the image data represent the same environmental factors (e.g., obstacles).

[0051] In this embodiment, the image data capture time is first determined. When the capture time meets the synchronization condition, the capture time is determined as the synchronization time of the initial point cloud data and the image data. The synchronization condition is used as the basis for determining whether the initial point cloud data (i.e., one frame of point cloud) and the image data generated within one rotational motion scanning cycle are consistent.

[0052] Optionally, before step S102 above, the shooting angle range of the image acquisition device is first determined, and the time interval corresponding to when the LiDAR scans within the shooting angle range (a certain time interval within the rotational motion scanning cycle) is determined. This time interval is taken as the target scanning time interval. When the shooting time is within the target scanning time interval, it indicates that the image data is captured within the overlapping angle of the LiDAR scanning angle and the shooting angle of the image acquisition device. That is, it indicates that the point cloud and image data of the frame are consistent at the shooting time. Therefore, it is determined that the shooting time meets the synchronization condition.

[0053] For example, such as Figure 2 As shown, camera 201 is installed directly in front of the target vehicle. LiDAR 202 starts scanning clockwise from the starting point 2021 directly behind the target. The rotational scanning cycle is 0-100ms. When LiDAR 202 scans into the field of view 2011 of camera 201, the target scanning time interval is 33.3-66.6ms. The image data A is captured at 43.3ms. Since its capture time is within 33.3-66.6ms, it is determined that the synchronization condition is met.

[0054] Furthermore, considering that in practical applications, multiple image data capture times may fall within the target scanning time interval, this embodiment continues to determine the number of image data to identify the synchronization time, thereby ensuring that the initial point cloud data and image data at the selected synchronization time are most consistent.

[0055] If the number of image data is equal to 1, then the shooting time is determined as the synchronization time of the initial point cloud data and the image data. For example, if the target scanning time interval is 33.3-66.6ms, and the image data with the shooting time within 33.3-66.6ms is only image data A (shooting time is 43.3ms), then 43.3ms is determined as the synchronization time of the initial point cloud data and the image data.

[0056] If the number of image data is greater than 1, calculate the difference between the capture time of each image data and the midpoint of the target scanning time interval (the midpoint of the rotational motion scanning cycle), and take the capture time corresponding to the smaller difference as the synchronization time of the initial point cloud data and image data. For example, the rotational motion scanning cycle is 0-100ms, the target scanning time interval is 33.3-66.6ms, and the image data captured within the 33.3-66.6ms range are image data A (captured at 43.3ms) and image data B (captured at 35ms). Since the difference between 43.3ms and the midpoint of the target scanning time interval (i.e., 50ms) is 6.7ms, and the difference between 35ms and the midpoint of the target scanning time interval is 15ms, 43.3ms is determined as the synchronization time of the initial point cloud data and image data.

[0057] It should be noted that, as Figure 2 As shown, since it is only necessary to determine the angle range of the viewing angle 2011 and the corresponding scanning time when the LiDAR 202 scans into the viewing angle 2011, the target scanning time interval can be determined. Subsequently, the captured image data can be directly compared with the target scanning time interval to determine the synchronization time. This process does not require limiting the trigger time of the camera 201, thus avoiding the phenomenon of low accuracy of the synchronization time caused by inaccurate trigger time of the camera device (for example, the camera 201 cannot immediately perform image data acquisition when it receives the trigger request, or the LiDAR 202 cannot accurately determine the time when the trigger request is sent to the camera 201 due to uneven movement speed).

[0058] S103. Based on the first vehicle pose data, determine the second vehicle pose data of the target vehicle at the synchronization time.

[0059] Since the vehicle pose changes continuously during the target vehicle's movement, and it takes a long time for the lidar to form each frame of point cloud, the initial point cloud data generated in this process is distorted (i.e., the initial point cloud data is very different from the object in the real world). Therefore, in this embodiment, the vehicle pose of the target vehicle at a certain preset time is first determined and used as a reference pose, so that all the initial point cloud data in a subsequent frame of point cloud are represented as the state when the target vehicle is in the reference pose, thereby achieving the purpose of correcting the distortion of the initial point cloud data.

[0060] Optionally, the synchronization time is selected as the preset time mentioned above. Specifically, in this embodiment, the initial adjacent vehicle pose data (the first vehicle pose data corresponding to the adjacent time of the target vehicle at the beginning of the rotational motion scanning cycle) and the ending adjacent vehicle pose data (the first vehicle pose data corresponding to the adjacent time of the target vehicle at the end of the rotational motion scanning cycle) in the first vehicle pose data are first determined. Then, linear interpolation processing is performed on the initial adjacent vehicle pose data and the ending adjacent vehicle pose data to obtain all vehicle pose data corresponding to the target vehicle at each scanning time. Then, the second vehicle pose data corresponding to the synchronization time is retrieved from all vehicle pose data. The significance of the linear interpolation processing method is that it uses a straight line passing through two points to approximate the linear relationship between time and vehicle pose data. Based on this linear relationship, the vehicle pose data at each time can be determined.

[0061] Specifically, the adjacent times of the initial / end time can be within the rotational motion scanning cycle (i.e., after the rotational motion scan starts and before it ends). For example, if the rotational motion scanning cycle is 50-100ms, 50.1ms is the adjacent time of the initial time, and 99.9ms is the adjacent time of the end time. Alternatively, they can be outside the rotational motion scanning cycle (i.e., before the rotational motion scan starts and after it ends). For example, if the rotational motion scanning cycle is 50-100ms, 49.9ms is the adjacent time of the initial time, and 100.1ms is the adjacent time of the end time.

[0062] S104. Perform coordinate transformation on the initial point cloud data based on the pose data of the second vehicle to obtain the target point cloud data at the synchronization time.

[0063] The purpose of coordinate transformation is to project all point cloud data to the same time corresponding to the same vehicle pose, so that all point cloud data in a frame of point cloud appear as if they were collected at the same time, thus achieving the effect of point cloud distortion correction.

[0064] In existing technologies, point cloud data is typically motion-compensated to the point cloud data at the beginning or end of the radar motion scanning cycle. In other words, each compensation requires motion compensation from the initial point cloud data to the final point (or vice versa). As such, the time span of this motion compensation is relatively large, making it impossible to treat the vehicle's trajectory as linear motion during the motion compensation process. This means that it is impossible to guarantee the accuracy of the difference information between the vehicle pose at each point cloud data acquisition time and the reference vehicle pose. In addition, since it is impossible to guarantee that the point cloud data acquired at the initial or final moment is synchronized with the image data of the vehicle camera at the exposure time, it is easy to affect the accuracy of the compensation.

[0065] To avoid the above situation, in this embodiment, the target coordinates in the target radar coordinate system corresponding to each initial point cloud data at the synchronization time are determined based on the second vehicle pose data, and the coordinates of each initial point cloud data are converted into the target coordinates to obtain the target point cloud data.

[0066] Optionally, during the process of determining the target coordinates, the difference between the second vehicle pose data and all vehicle pose data is calculated respectively, and the target radar coordinate system corresponding to each initial point cloud data and the synchronization time is determined based on the difference (e.g., Figure 3 The projection distance of the X'-Y'-Z' coordinate system is used to determine the target coordinates of each initial point cloud data in the target radar coordinate system. The coordinates of the initial point cloud data are then determined from the initial coordinates (the initial coordinates are the coordinates of each initial point cloud data in the initial radar coordinate system at each scanning time, such as...). Figure 3 The coordinates in the XYZ coordinate system are converted to target coordinates to obtain target point cloud data. At this time, each target point cloud data represents the target vehicle in the state of synchronization. Since the span between the synchronization time and the initial or final time of the LiDAR rotation scanning cycle is relatively small, the time span of motion compensation is small. During the motion compensation process, the driving trajectory of the target vehicle can be regarded as linear motion, thereby ensuring that the difference information between the vehicle pose corresponding to the acquisition time of each initial point cloud data (i.e., all vehicle pose data) and the vehicle pose data at the synchronization time (i.e., the second vehicle pose data) is accurate. In addition, the target point cloud data and image data at the synchronization time are synchronized, thereby effectively improving the accuracy of compensation.

[0067] Furthermore, following step S104 above, the method further includes:

[0068] The pixels in the image data at the synchronization time are fused with the target point cloud data and presented in the same coordinate system.

[0069] Specifically, LiDAR and image acquisition devices (such as cameras) each have their own advantages and disadvantages. Image acquisition devices can provide rich color information, and computer vision algorithms are very mature, but they cannot provide three-dimensional information about the vehicle's surrounding environment. LiDAR can provide three-dimensional information about the vehicle's surrounding environment, but it cannot provide color information about the vehicle's surrounding environment. However, in practical applications, in order to improve the vehicle's navigation and obstacle avoidance performance, it is necessary to ensure that the collected data about the vehicle's surrounding environment can simultaneously reflect both three-dimensional and color information. Therefore, it is crucial to fuse the information provided by LiDAR and image acquisition devices to ensure the safe driving of the vehicle.

[0070] In this embodiment, the color parameters of each pixel in the image data at the synchronization time are fused with the corresponding target point cloud data into the target radar coordinate system. This allows the target point cloud data presented in the target radar coordinate system to reflect both the three-dimensional information of the vehicle's surrounding environment and the color information of the surrounding environment. Based on the fused target point cloud data, the vehicle's surrounding environment information can be quickly and comprehensively understood, thereby effectively improving the safety of vehicle driving.

[0071] Optionally, the target point cloud data obtained after the above distortion correction also has a variety of application scenarios, such as: judging the accuracy of camera and lidar parameters based on the overlap between image data and target point cloud data, scene modeling, etc.

[0072] For example, the same coordinate system can refer to the two-dimensional coordinate system of the image data. Then, the target point cloud data can be projected onto the two-dimensional coordinate system to obtain several target projection points. Then, the pixel points and target projection points in the two-dimensional coordinate system can be obtained, which can be used as or to generate the final fusion result.

[0073] Based on this, target objects (such as people, vehicles, road signs, etc.) can be identified in pixels to obtain the region of each target object (which can be denoted as the first region); the same target object can also be identified in the target projection point to obtain the region of each target object (which can be denoted as the second region). Then, for the same target object, the area ratio of the overlapping area of ​​the corresponding first region and second region to the first region (or second region) can be calculated, and then the average or weighted average of all area ratios can be used as the evaluation information of the fusion result. This evaluation information can be used to evaluate the quality of the fusion effect. The higher the evaluation information, the better the fusion result.

[0074] For point cloud data and image data, during the processing of steps S101 to S104 above, it is actually assumed that the time axes of point cloud data and image data are aligned during synchronization. That is, the image acquired by the image acquisition device at a certain time A and the point cloud data acquired by the radar at a certain time A should be data at the same objective time A. However, due to the influence of factors such as delays in the acquisition and processing process and the clock source used, the time axes may not be aligned. This may result in poor fusion effect even if the calibration parameters (such as intrinsic and extrinsic parameters) of the image acquisition device for acquiring image data and the calibration parameters (such as intrinsic and extrinsic parameters) of the radar for acquiring point cloud data are relatively accurate. For example, at least one of the positions, sizes, shapes, etc. of the first region and the second region may deviate significantly.

[0075] If the vehicle is stationary, the fusion effect generally cannot reflect the accuracy of the time axis alignment. However, in steps S101 to S104 above, since the vehicle is in motion, and the accuracy of the calibration parameters has been verified, especially when the vehicle's speed is higher than a certain first threshold and the vehicle's angular velocity is higher than a certain second threshold, the relationship between the fusion effect and the time axis alignment can be accurately and fully reflected.

[0076] Therefore, after repeatedly executing steps S101 to S104 and performing fusion, the fusion result at each time point in multiple time points can be obtained (e.g., the pixel points and target projection points in the two-dimensional coordinate system at each time point in multiple time points). Then, at the multiple time points, the first time point where the vehicle's speed is higher than the first threshold and the second time point where the vehicle's angular velocity is higher than the second threshold can be selected. The evaluation information at the first time point and the evaluation information at the second time point can be calculated. The statistical value (e.g., the average value) of all the evaluation information at the first and second times points can be counted as the global evaluation information. The alignment degree of the time axis can then be reflected by the global evaluation information. If the global evaluation information is higher than the specified threshold, the alignment degree of the time axis is determined to meet the requirements.

[0077] As can be seen, the above scheme effectively evaluates the alignment of the time axes of the two types of detection data (image data and point cloud data) by evaluating the fusion effect at a certain speed and angular velocity.

[0078] As described above, the data processing method provided in this application first acquires initial point cloud data output by the lidar during its rotational scanning cycle, image data output by the image acquisition device, and first vehicle pose data of the target vehicle output by the pose detection device during its rotational scanning cycle. Then, based on the image data capture time, the synchronization time of the initial point cloud data and the image data is determined. Based on the first vehicle pose data, the second vehicle pose data of the target vehicle at the synchronization time is determined. Finally, coordinate transformation processing is performed on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time. By pre-determining the synchronization time of the initial point cloud data and the image data, and performing coordinate transformation processing on the initial point cloud data based on the target vehicle's pose data at that synchronization time, all initial point cloud data are reflected in the coordinate system at the synchronization time. Since the span between the synchronization time and the initial or final moment of the lidar's rotational scanning cycle is small, the time span of the motion compensation is small, thereby improving the accuracy of the compensation.

[0079] Based on the methods described in the above embodiments, this embodiment will be further described from the perspective of a data processing device.

[0080] Please see Figure 4 , Figure 4The present application provides a specific description of a data processing device for data acquisition, which may include: a data acquisition module 10, a synchronization time determination module 20, a second vehicle pose data determination module 30, and a compensation processing module 40, wherein:

[0081] (1) Acquisition Module 10

[0082] The acquisition module 10 is used to acquire the initial point cloud data output by the lidar during the rotational motion scanning cycle, the image data output by the image acquisition device, and the first vehicle pose data of the target vehicle during the rotational motion scanning cycle output by the pose detection device.

[0083] (2) Synchronization Timing Determination Module 20

[0084] The synchronization time determination module 20 is used to determine the synchronization time of the initial point cloud data and the image data based on the shooting time of the image data.

[0085] Specifically, the synchronization timing determination module 20 is used for:

[0086] Determine the time when the image data was captured;

[0087] When the shooting time meets the synchronization conditions, the shooting time is determined as the synchronization time of the initial point cloud data and image data.

[0088] Specifically, the synchronization timing determination module 20 is also used for:

[0089] When the shooting time is within the target scanning time interval of the rotational motion scanning cycle, the shooting time is determined to meet the synchronization condition; wherein, the target scanning time interval includes the time interval corresponding to when the lidar scans into the shooting angle range of the image acquisition device.

[0090] Furthermore, the synchronization timing determination module 20 is also used for:

[0091] If the number of image data is equal to 1, the shooting time is determined as the synchronization time of the initial point cloud data and image data;

[0092] If the number of image data is greater than 1, calculate the difference between the shooting time corresponding to each image data and the midpoint of the target scanning time interval;

[0093] The shooting time corresponding to the smaller numerical difference is taken as the synchronization time of the initial point cloud data and image data.

[0094] (3) Second vehicle pose data determination module 30

[0095] The second vehicle pose data determination module 30 is used to determine the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data.

[0096] Specifically, the second vehicle pose data determination module 30 is used for:

[0097] Determine the initial adjacent vehicle pose data and the final adjacent vehicle pose data in the first vehicle pose data; wherein, the initial adjacent vehicle pose data is the first vehicle pose data corresponding to the adjacent time of the target vehicle at the beginning of the rotational motion scanning cycle, and the final adjacent vehicle pose data is the first vehicle pose data corresponding to the adjacent time of the target vehicle at the end of the rotational motion scanning cycle.

[0098] Based on the initial and final adjacent vehicle pose data, the target vehicle's pose is linearly interpolated to obtain the second vehicle pose data.

[0099] (4) Compensation processing module 40

[0100] The compensation processing module 40 is used to perform coordinate transformation processing on the initial point cloud data based on the pose data of the second vehicle to obtain the target point cloud data at the synchronization time.

[0101] Specifically, the compensation processing module 40 is used for:

[0102] Based on the second vehicle pose data, determine the target coordinates in the target radar coordinate system corresponding to each initial point cloud data at the synchronization time;

[0103] The coordinates of each initial point cloud data are converted into target coordinates to obtain the target point cloud data.

[0104] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0105] As described above, the data processing device provided in this application first acquires initial point cloud data output by the lidar during its rotational scanning cycle, image data output by the image acquisition device, and first vehicle pose data of the target vehicle during its rotational scanning cycle, all output by the pose detection device, through the acquisition module 10. Then, the synchronization time determination module 20 determines the synchronization time of the initial point cloud data and the image data based on the image data capture time. The second vehicle pose data determination module 30 determines the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data. Finally, the compensation processing module 40 performs coordinate transformation processing on the initial point cloud data according to the second vehicle pose data to obtain the target point cloud data at the synchronization time. By pre-determining the synchronization time of the initial point cloud data and the image data, and performing coordinate transformation processing on the initial point cloud data based on the vehicle pose data of the target vehicle at that synchronization time, all initial point cloud data are reflected in the coordinate system at the synchronization time. Since the span between the synchronization time and the initial or final moment of the lidar's rotational scanning cycle is small, the time span of the motion compensation is small, thereby improving the accuracy of the compensation.

[0106] Accordingly, embodiments of the present invention also provide a data processing system, including any of the data processing devices provided in the embodiments of the present invention, which can be integrated into an electronic device.

[0107] The process involves collecting initial point cloud data output by the lidar during its rotational scanning cycle, image data output by the image acquisition device, and first vehicle pose data of the target vehicle output by the pose detection device during its rotational scanning cycle. Based on the image data capture time, the synchronization time of the initial point cloud data and the image data is determined. Based on the first vehicle pose data, the second vehicle pose data of the target vehicle at the synchronization time is determined. The initial point cloud data is then subjected to coordinate transformation processing based on the second vehicle pose data to obtain the target point cloud data at the synchronization time.

[0108] The specific implementation details of each of the above devices can be found in the preceding embodiments, and will not be repeated here.

[0109] Since the data processing system can include any of the data processing devices provided in the embodiments of the present invention, it can achieve the beneficial effects that any of the data processing devices provided in the embodiments of the present invention can achieve. For details, please refer to the previous embodiments, which will not be repeated here.

[0110] In addition, embodiments of this application also provide an electronic device. For example... Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502. The processor 501 and the memory 502 are electrically connected.

[0111] The processor 501 is the control center of the electronic device 500. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 502 and calling the data stored in the memory 502, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0112] In this embodiment, the processor 501 in the electronic device 500 loads the instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 runs the application programs stored in the memory 502 to realize various functions:

[0113] The system collects initial point cloud data output by the lidar during the rotational motion scanning cycle, image data output by the image acquisition device, and the first vehicle pose data of the target vehicle during the rotational motion scanning cycle output by the pose detection device.

[0114] Based on the capture time of the image data, determine the synchronization time between the initial point cloud data and the image data;

[0115] Based on the first vehicle pose data, determine the second vehicle pose data of the target vehicle at the synchronization time;

[0116] Based on the pose data of the second vehicle, the initial point cloud data is processed by coordinate transformation to obtain the target point cloud data at the synchronization time.

[0117] Figure 6 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the data processing method provided in the above embodiments.

[0118] RF circuit 610 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 610 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 610 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0119] The memory 620 can be used to store software programs and modules. The processor 680 executes various functional applications and data processing by running the software programs and modules stored in the memory 620, thereby realizing the function of storing 5G capability information. The memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 620 may further include memory remotely located relative to the processor 680, and these remote memories can be connected to the electronic device 600 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The input unit 630 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 630 may include a touch-sensitive surface 631 and other input devices 632. The touch-sensitive surface 631, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 631), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 631 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 680, and can receive and execute commands sent by the processor 680. In addition, the touch-sensitive surface 631 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 631, the input unit 630 may also include other input devices 632. Specifically, other input devices 632 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0121] Display unit 640 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic device 600. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 640 may include display panel 641, optionally configured as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, touch-sensitive surface 631 may cover display panel 641. When touch-sensitive surface 631 detects a touch operation on or near it, it transmits the information to processor 680 to determine the type of touch event. Subsequently, processor 680 provides corresponding visual output on display panel 641 according to the type of touch event. Although in Figure 6 In this embodiment, the touch-sensitive surface 631 and the display panel 641 are implemented as two separate components to realize the input and output functions. However, in some embodiments, the touch-sensitive surface 631 and the display panel 641 can be integrated to realize the input and output functions.

[0122] The electronic device 600 may also include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 641 according to the ambient light level, and the proximity sensor can turn off the display panel 641 and / or backlight when the electronic device 600 is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that the electronic device 600 may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0123] Audio circuitry 660, speaker 661, and microphone 662 provide an audio interface between the user and electronic device 600. Audio circuitry 660 converts received audio data into electrical signals, which are then transmitted to speaker 661, where they are converted into sound signals for output. Conversely, microphone 662 converts collected sound signals into electrical signals, which are received by audio circuitry 660, converted back into audio data, and then processed by processor 680 before being transmitted via RF circuitry 610 to, for example, another terminal, or output to memory 620 for further processing. Audio circuitry 660 may also include an earphone jack to facilitate communication between external headphones and electronic device 600.

[0124] Electronic device 600, through transmission module 670 (e.g., Wi-Fi module), enables users to send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 6 The transmission module 670 is shown, but it is understood that it is not a necessary component of the electronic device 600 and can be omitted as needed without changing the nature of the invention.

[0125] Processor 680 is the control center of electronic device 600. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of electronic device 600 by running or executing software programs and / or modules stored in memory 620, and by calling data stored in memory 620. Optionally, processor 680 may include one or more processing cores; in some embodiments, processor 680 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 680.

[0126] The electronic device 600 also includes a power supply 690 (such as a battery) that supplies power to the various components. In some embodiments, the power supply may be logically connected to the processor 680 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 690 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0127] Although not shown, the electronic device 600 may also include a camera (such as a front-facing camera and a rear-facing camera), a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the electronic device also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations:

[0128] The system collects initial point cloud data output by the lidar during the rotational motion scanning cycle, image data output by the image acquisition device, and the first vehicle pose data of the target vehicle during the rotational motion scanning cycle output by the pose detection device.

[0129] Based on the capture time of the image data, determine the synchronization time between the initial point cloud data and the image data;

[0130] Based on the first vehicle pose data, determine the second vehicle pose data of the target vehicle at the synchronization time;

[0131] Based on the pose data of the second vehicle, the initial point cloud data is processed by coordinate transformation to obtain the target point cloud data at the synchronization time.

[0132] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0133] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps in any of the data acquisition and processing methods provided in the embodiments of the present invention.

[0134] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0135] Since the instructions stored in the storage medium can execute the steps in any of the data acquisition processing methods provided in the embodiments of the present invention, the beneficial effects that any of the data acquisition processing methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0136] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0137] In summary, although the present application has disclosed the preferred embodiments as described above, the above preferred embodiments are not intended to limit the present application. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be determined by the scope defined in the claims.

Claims

1. A method for processing collected data, characterized in that, include: The system collects initial point cloud data output by the lidar during the rotational motion scanning cycle, image data output by the image acquisition device, and first vehicle pose data of the target vehicle output by the pose detection device during the rotational motion scanning cycle. Based on the capture time of the image data, determine the synchronization time of the initial point cloud data and the image data; Based on the first vehicle pose data, the second vehicle pose data of the target vehicle at the synchronization time are determined; Based on the second vehicle pose data, the initial point cloud data is subjected to coordinate transformation to obtain the target point cloud data at the synchronization time. The process of determining the synchronization time between the initial point cloud data and the image data based on the capture time of the image data includes: If the number of image data is equal to 1, the shooting time is determined to be the synchronization time of the initial point cloud data and the image data; If the number of image data is greater than 1, calculate the difference between the shooting time corresponding to each image data and the midpoint of the target scanning time interval; The shooting time corresponding to the smaller difference value is taken as the synchronization time of the initial point cloud data and the image data; The method further includes: fusing each pixel in the image data at the synchronization time with the target point cloud data in the same coordinate system for presentation.

2. The data processing method according to claim 1, characterized in that, The step of determining the synchronization time of the initial point cloud data and the image data based on the shooting time of the image data includes: Determine the time when the image data was captured; When the shooting time meets the synchronization condition, the shooting time is determined to be the synchronization time of the initial point cloud data and the image data.

3. The data processing method according to claim 2, characterized in that, Before determining the shooting time as the synchronization time of the initial point cloud data and the image data based on the image data, the method further includes: When the shooting time is within the target scanning time interval of the rotational motion scanning cycle, the shooting time is determined to satisfy the synchronization condition; wherein, the target scanning time interval includes the time interval corresponding to when the lidar scans into the shooting angle range of the image acquisition device.

4. The data processing method according to claim 2, characterized in that, The step of determining the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data includes: Determine the initial adjacent vehicle pose data and the final adjacent vehicle pose data in the first vehicle pose data; wherein, the initial adjacent vehicle pose data is the first vehicle pose data corresponding to the moment adjacent to the initial moment of the target vehicle in the rotational motion scanning cycle, and the final adjacent vehicle pose data is the first vehicle pose data corresponding to the moment adjacent to the end moment of the target vehicle in the rotational motion scanning cycle. Based on the initial adjacent vehicle pose data and the final adjacent vehicle pose data, the vehicle pose of the target vehicle is linearly interpolated to obtain the second vehicle pose data.

5. The data processing method according to claim 4, characterized in that, The step of performing coordinate transformation processing on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time includes: Based on the second vehicle pose data, determine the target coordinates in the target radar coordinate system corresponding to each initial point cloud data at the synchronization time; The coordinates of each initial point cloud data are converted into the target coordinates to obtain the target point cloud data.

6. A data acquisition and processing device, characterized in that, include: The acquisition module is used to acquire the initial point cloud data output by the lidar during the rotational motion scanning cycle, the image data output by the image acquisition device, and the first vehicle pose data of the target vehicle output by the pose detection device during the rotational motion scanning cycle. The synchronization time determination module is used to determine the synchronization time of the initial point cloud data and the image data based on the shooting time of the image data; The second vehicle pose data determination module is used to determine the second vehicle pose data of the target vehicle at the synchronization time based on the first vehicle pose data. The compensation processing module is used to perform coordinate transformation processing on the initial point cloud data based on the second vehicle pose data to obtain the target point cloud data at the synchronization time. The synchronization time determination module is further configured to: if the number of image data is equal to 1, determine the shooting time as the synchronization time of the initial point cloud data and the image data; if the number of image data is greater than 1, calculate the difference between the shooting time corresponding to each image data and the middle time within the target scanning time interval; and take the shooting time corresponding to the smaller difference as the synchronization time of the initial point cloud data and the image data. The compensation processing module is also used to fuse each pixel in the image data at the synchronization time with the target point cloud data in the same coordinate system for presentation.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data acquisition processing method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps of the data acquisition processing method according to any one of claims 1 to 5.

Images