Space-time synchronization method and device for radar point cloud and image, vehicle and product

Abstract

This application relates to the field of multimodal data processing technology, and provides a method, apparatus, vehicle, and product for spatiotemporal synchronization of radar point clouds and images. The method includes: acquiring a first image set and a first radar point cloud at the current moment during vehicle movement; the first image set includes multiple first images corresponding to different first moments; performing inter-frame prediction on the first image set to obtain a second image set containing the first image set; the second image set includes multiple second images corresponding to different second moments, where the time interval between two adjacent second moments is less than the time interval between two adjacent first moments; and acquiring a third image from the second image set that is spatiotemporally synchronized with the first radar point cloud based on the current moment. Using this method, radar point clouds and images with high spatiotemporal synchronization can be obtained.

Description

Technical Field

[0001] This application belongs to the field of multimodal data processing technology, and in particular relates to a method, device, vehicle and product for spatiotemporal synchronization of radar point clouds and images. Background Technology

[0002] In the process of intelligent driving, vehicles typically perceive the driving environment based on multimodal perception data. To improve the accuracy of driving environment perception, it is usually necessary to fuse radar point clouds and camera images taken at the same time to obtain the driving environment.

[0003] Currently, due to the differences in the frequency at which various sensors collect corresponding sensing data and the differences in the start time of sensor operation, radar point clouds and images acquired at the most recent moment are not spatiotemporally synchronized data. In other words, radar point clouds and images obtained by existing technologies have low spatiotemporal synchronization. Summary of the Invention

[0004] This application provides a method, apparatus, vehicle, and product for spatiotemporal synchronization of radar point clouds and images, which can solve the problem of low spatiotemporal synchronization between radar point clouds and images in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for spatiotemporal synchronization of radar point clouds and images, the method comprising:

[0006] During vehicle movement, a first image set and a first radar point cloud at the current moment are acquired; the first image set includes multiple first images corresponding to the first moment respectively.

[0007] Inter-frame prediction is performed on the first image set to obtain a second image set containing the first image set; the second image set contains multiple second images corresponding to the second time points respectively, and the time interval between two adjacent second time points is less than the time interval between two adjacent first time points;

[0008] Based on the current moment, acquire a third image from the second image set that is spatiotemporally synchronized with the first radar point cloud.

[0009] In one embodiment, after acquiring a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time, the method further includes:

[0010] Obtain the dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system; the dynamic joint calibration parameters are used to represent the spatial mapping relationship between the camera coordinate system and the radar coordinate system during vehicle movement;

[0011] Based on dynamic joint calibration parameters, the third image and the first radar point cloud are calibrated to obtain a fourth image and the second radar point cloud that are spatiotemporally synchronized after calibration.

[0012] In one embodiment, obtaining the dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system includes:

[0013] The first location information corresponding to the first radar point cloud and the second location information corresponding to the third image are obtained respectively.

[0014] Based on the first and second position information, dynamic joint calibration parameters are generated.

[0015] In one embodiment, obtaining first location information corresponding to the first radar point cloud and second location information corresponding to the third image includes:

[0016] During the vehicle's movement, acquire third position information at multiple third moments;

[0017] From multiple third location information sources, the third location information corresponding to the third time closest to the current time is determined as the first location information; and,

[0018] The third position information corresponding to the third time closest to the target time is determined as the second position information; the target time is the second time corresponding to the third image.

[0019] In one embodiment, the acquisition frequency of the third location information is greater than the acquisition frequency of the first image and greater than the acquisition frequency of the first radar point cloud.

[0020] In one embodiment, dynamic joint calibration parameters are generated based on first location information and second location information, including:

[0021] Calculate the spatial transformation matrix between the first position information and the second position information;

[0022] Based on the spatial transformation matrix, the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system are corrected to obtain the dynamic joint calibration parameters.

[0023] In one embodiment, acquiring a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time includes:

[0024] From the second image set, the second image corresponding to the second time closest to the current time is determined as the third image.

[0025] Secondly, embodiments of this application provide a spatiotemporal synchronization device for radar point clouds and images, the device comprising:

[0026] The first acquisition module is used to acquire a first image set and a first radar point cloud at the current moment during the vehicle's movement; the first image set includes multiple first images corresponding to the first moment respectively;

[0027] The prediction module is used to perform inter-frame prediction on the first image set to obtain a second image set containing the first image set; the second image set contains multiple second images corresponding to the second time moments respectively, and the time interval between two adjacent second time moments is less than the time interval between two adjacent first time moments;

[0028] The second acquisition module is used to acquire a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time.

[0029] Thirdly, embodiments of this application provide a vehicle including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0031] Fifthly, embodiments of this application provide a computer program product that, when run on a vehicle, causes the vehicle to perform the method described in the first aspect.

[0032] The beneficial effects of this application embodiment compared to the prior art are as follows: During vehicle movement, a first image set containing multiple first images corresponding to different first moments, and a first radar point cloud at the current moment, can be acquired. Since the frequency of acquiring the first images may differ from the frequency of acquiring the first radar point cloud, there may not be a moment among the multiple first moments that completely corresponds to the current moment. In this case, if the first image corresponding to the most recent first moment is directly used as the image spatiotemporally synchronized with the first radar point cloud, the spatiotemporal synchronization between the most recent first moment and the radar point cloud remains low due to the time interval between the current moment and the most recent first moment. Therefore, the vehicle can first perform inter-frame prediction on the first image set to obtain a second image set containing the first image set, and then, based on the current moment, acquire a third image from the second image set that is spatiotemporally synchronized with the first radar point cloud. Since the second image set generated after inter-frame prediction contains multiple second images corresponding to different second moments, and the time interval between two adjacent second moments is less than the time interval between two adjacent first moments, it can be considered that the acquisition frequency of the second image set generated after inter-frame prediction is much higher than the acquisition frequency of the first image set. Based on this, it can be assumed that even if multiple second moments do not have moments exactly the same as the current moment, the spatiotemporal synchronization between the third image acquired from the second image set and the first radar point cloud is at least greater than or equal to the spatiotemporal synchronization between the first image acquired from the first image set and the first radar point cloud. Therefore, the above method can improve the spatiotemporal synchronization between the first radar point cloud and the third image. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the implementation of a spatiotemporal synchronization method for radar point clouds and images according to an embodiment of this application.

[0035] Figure 2 This is a flowchart illustrating the implementation of a spatiotemporal synchronization method for radar point clouds and images, provided in another embodiment of this application.

[0036] Figure 3 This is a schematic diagram illustrating an implementation method for generating dynamic joint calibration parameters in a spatiotemporal synchronization method for radar point clouds and images provided in an embodiment of this application.

[0037] Figure 4This is a flowchart illustrating the implementation of a spatiotemporal synchronization method for radar point clouds and images, provided in another embodiment of this application.

[0038] Figure 5 This is a schematic diagram of the structure of a spatiotemporal synchronization device for radar point clouds and images provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of the structure of a vehicle provided in one embodiment of this application. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0042] It should be noted that the information collection process (such as the facial image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0043] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0044] In the process of intelligent driving, vehicles typically perceive the driving environment based on multimodal perception data. To improve the accuracy of driving environment perception, it is usually necessary to fuse radar point clouds and images at the same time to obtain the driving environment.

[0045] Currently, due to the differences in the frequency at which various sensors collect corresponding sensing data and the differences in the start time of sensor operation, radar point clouds and images acquired at the most recent moment are not spatiotemporally synchronized data. In other words, radar point clouds and images obtained by existing technologies have low spatiotemporal synchronization.

[0046] Furthermore, to improve the spatiotemporal synchronization of radar point clouds and images, existing technologies also employ pulse triggers in each sensor to send pulse signals based on the satellite time corresponding to a unified satellite system, instructing each sensor to collect sensing data at the same time. In other words, each sensor collects sensing data at the same frequency.

[0047] However, this approach is limited by the accuracy of the triggers in each sensor. As the accuracy decreases, the spatiotemporal synchronization of the multimodal sensing data will also decrease.

[0048] Based on this, in order to improve the spatiotemporal synchronization of radar point clouds and images, one embodiment of this application provides a method for spatiotemporal synchronization of radar point clouds and images, which can be applied to vehicles. For example, it can be applied to electronic devices such as intelligent driving controllers and central controllers in vehicles. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0049] Please see Figure 1 , Figure 1 The following is a flowchart illustrating the implementation of a spatiotemporal synchronization method for radar point clouds and images provided in an embodiment of this application. The method includes the following steps:

[0050] S101. During the vehicle's movement, acquire a first image set and a first radar point cloud at the current moment; the first image set includes multiple first images corresponding to the first moment.

[0051] In one embodiment, the vehicle is typically equipped with radar sensors and image sensors to collect perception data of the vehicle's surrounding environment, thereby enabling driving environment perception.

[0052] The radar sensors include, but are not limited to, lidar sensors, millimeter-wave radar sensors, ultrasonic radar sensors, and infrared radar sensors, etc., and there is no limitation on the type. Furthermore, the number of cameras can be one or more, and there is no limitation on the number. For example, when there is one camera, it can be a surround-view camera to collect data on the vehicle's surrounding environment from all directions. Alternatively, there can be four cameras, respectively positioned at the front, rear, left, and right sides of the vehicle, to collect data on the driving environment from the corresponding directions.

[0053] It should be noted that the frequency at which the radar sensor acquires the first radar point cloud is usually different from the frequency at which the camera acquires the first image, resulting in low spatiotemporal synchronization between the most recently acquired first radar point cloud and the first image.

[0054] For example, the camera's acquisition frequency can be 25Hz, and the radar sensor's acquisition frequency can be 10Hz. It is understood that if the camera's acquisition frequency is higher than the radar sensor's acquisition frequency, then the number of first images will be greater than the number of first radar point clouds within the same time period. Furthermore, among the large number of first images and first radar point clouds, some may be acquired at exactly the same time, while others may be acquired at time intervals.

[0055] The above-mentioned acquisition frequency is only one example. In another embodiment, the acquisition frequency of the camera can be lower than the acquisition frequency of the radar sensor.

[0056] It should be added that even if the camera's acquisition frequency is the same as the radar's acquisition frequency, the spatiotemporal synchronization of the first image and the first radar point cloud acquired at the same time may still be low due to the accuracy of the triggers in each sensor (camera and radar sensor).

[0057] In one embodiment, the first radar point cloud at the current moment can be considered as the radar point cloud acquired at the most recent moment, and it needs to be spatiotemporally synchronized with the first image.

[0058] The number of multiple first moments can be set according to the actual situation and is not limited. For example, multiple first moments can be considered to include the current moment and a preset number of first moments before the current moment.

[0059] Based on the above explanation, it can be assumed that each first image and each first radar point cloud has its own corresponding timestamp of being acquired. For example, if the first image uses the first moment as its timestamp, the first radar point cloud acquired at this time can also use the current moment as its timestamp.

[0060] It should be noted that the camera's acquisition frequency differs from that of the radar sensor. Therefore, relative to the current moment when the first radar point cloud is acquired, the earliest moment in the first image acquired by the camera at this time may be earlier than the current moment (e.g., a difference of milliseconds or seconds), later than the current moment, or equal to the current moment; no limitation is made in this regard.

[0061] S102. Perform inter-frame prediction on the first image set to obtain a second image set containing the first image set.

[0062] In one embodiment, the above-mentioned inter-frame prediction is to predict the images that the camera may capture at more times, and the set of the predicted images and the original first image set is the above-mentioned second image set.

[0063] As an example, let's assume the camera captures the first image once per second, and the first image set contains four frames of the first image. The first image set can include the first images corresponding to the first moments, such as 10:30:01, 10:30:02, 10:30:03, and 10:30:04.

[0064] It should be noted that this example is only for the purpose of understanding how to generate a second image set. In real-world scenarios, the frequency at which the camera captures the first image is usually much higher than the frequency in the example above, and the number of first images contained in the first image set is also much greater than 4 frames.

[0065] The vehicle can perform inter-frame prediction on the first image set, obtaining predicted images corresponding to the second time points 10:30:01:20, 10:30:01:40, 10:30:02:20, 10:30:02:40, 10:30:03:20, 10:30:03:40, 10:30:04:20, and 10:30:04:40, respectively. That is, two additional predicted images are generated between two adjacent first time points. Furthermore, the latest time point corresponding to the predicted image (10:30:04:40) is later than the latest first time point in the first image set (10:30:04).

[0066] At this point, when both the predicted image and the first image in the first image set are used as the second image set, the time interval between two adjacent frames at the second time point is 20ms, which is less than the time interval between two adjacent first time points (1s).

[0067] In one embodiment, inter-frame prediction utilizes the correlation (temporal correlation) between image frames to achieve image prediction. The vehicle can perform inter-frame prediction on the first image set to obtain a second image set using image prediction models, video compression coding techniques, etc.

[0068] The neural network structure of the image prediction model can be any of the various convolutional neural network structures capable of image processing, which will not be described in detail here. Furthermore, video compression coding techniques include, but are not limited to, those corresponding to video coding standards such as H.263 and H.264.

[0069] S103. Based on the current time, acquire a third image from the second image set that is spatiotemporally synchronized with the first radar point cloud.

[0070] In one embodiment, spatiotemporal synchronization includes temporal synchronization and spatial synchronization. Furthermore, it can be assumed that when environmental data is acquired in the same space at the same time, after the radar point cloud and image are transformed to the same coordinate system (e.g., transforming the radar point cloud to the camera coordinate system, or the camera coordinate system to the radar coordinate system), the positions representing the same object in the radar point cloud and image should be the same.

[0071] Based on this, it can be considered that the second image in the second image set, which has the highest degree of fusion with the first radar point cloud in the same coordinate system, is the third image with the highest spatiotemporal synchronization with the first radar point cloud.

[0072] The vehicle can process the first radar point cloud and each second image according to a preset registration model of point cloud and image to obtain a third image that is spatiotemporally synchronized with the first radar point cloud.

[0073] In another embodiment, since the camera and radar sensors are both located in the same space and collect sensing data in the same space, the second image in the second image set that corresponds to the second moment closest to the current moment can also be considered the third image with the highest spatiotemporal synchronization with the first radar point cloud.

[0074] In this embodiment, the method of obtaining the third image from the second image set is not limited.

[0075] It is understandable that, since the time interval between the second moments of two adjacent frames is less than the time interval between the first moments of two adjacent frames, it can be assumed that even if there are multiple second moments that are not exactly the same as the current moment, the spatiotemporal synchronization between the third image obtained from the second image set and the first radar point cloud is at least greater than or equal to the spatiotemporal synchronization between the first image obtained from the first image set and the first radar point cloud.

[0076] In this embodiment, during vehicle movement, a first image set containing multiple first images corresponding to different first moments, and a first radar point cloud at the current moment, can be acquired. Since the frequency of acquiring the first images may differ from the frequency of acquiring the first radar point cloud, there may not be a single moment among the multiple first moments that perfectly corresponds to the current moment. In this case, if the first image corresponding to the most recent first moment is directly used as the image spatiotemporally synchronized with the first radar point cloud, the spatiotemporal synchronization between the most recent first moment and the radar point cloud remains low due to the time interval between the current moment and the most recent first moment. Therefore, the vehicle can first perform inter-frame prediction on the first image set to obtain a second image set containing the first image set, and then, based on the current moment, acquire a third image from the second image set that is spatiotemporally synchronized with the first radar point cloud. Since the second image set generated after inter-frame prediction contains multiple second images corresponding to different second moments, and the time interval between two adjacent second moments is less than the time interval between two adjacent first moments, it can be considered that the acquisition frequency of the second image set generated after inter-frame prediction is much higher than the acquisition frequency of the first image set. Based on this, it can be assumed that even if multiple second moments do not completely match the current moment, the spatiotemporal synchronization between the third image acquired from the second image set and the first radar point cloud is at least greater than or equal to the spatiotemporal synchronization between the first image acquired from the first image set and the first radar point cloud. Therefore, the above method can improve the spatiotemporal synchronization between the first radar point cloud and the third image. Furthermore, the above spatiotemporal synchronization method does not rely on pulse triggers set in each sensor, thus possessing universality.

[0077] In another embodiment, since the vehicle's movement will cause the radar sensors and cameras to move, the radar sensors and cameras, while collecting perception data during their movement, may affect the collected perception data. Consequently, even if images and radar point clouds are collected at the same time, there will still be some spatial alignment error. That is, the first radar point cloud and the third image are closer to being synchronized in time, but spatial synchronization errors still exist.

[0078] Based on this, in order to further improve the spatiotemporal synchronization of the first radar point cloud and the third image, the vehicle can also, according to, such as Figure 2 Steps S201-S202, as shown, process the first radar point cloud and the third image to improve their spatiotemporal synchronization. Details are as follows:

[0079] S201. Obtain the dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system; the dynamic joint calibration parameters are used to calibrate the spatial mapping relationship between the camera coordinate system and the radar coordinate system during vehicle movement.

[0080] In one embodiment, joint calibration refers to using multiple sensors (e.g., radar and cameras) to perceive the surrounding environment and solving the relative positions and attitude relationships between the various sensors through a certain algorithm, thereby realizing the fusion and utilization of the perceived data.

[0081] Based on this, it can be assumed that the dynamic joint calibration parameters can be used to represent the spatial mapping relationship between the camera coordinate system and the radar coordinate system during vehicle motion. This mapping relationship can be considered as the aforementioned relative position and attitude relationship.

[0082] In one embodiment, the aforementioned dynamic joint calibration parameters include, but are not limited to, the joint calibration parameters when the camera coordinate system is mapped to the radar coordinate system, and the joint calibration parameters when the radar coordinate system is mapped to the camera coordinate system.

[0083] In one embodiment, the aforementioned dynamic joint calibration parameters can be obtained by modifying preset static joint calibration parameters based on vehicle driving environment information. For example, the corresponding correction weights are determined based on driving information, and then the static joint calibration correction parameters are modified based on the correction weights to obtain the dynamic joint calibration correction parameters.

[0084] In one embodiment, the aforementioned driving environment information includes, but is not limited to, vehicle speed, acceleration, and weather type (e.g., rain, snow, fog, etc.), and is not limited thereto. The aforementioned static joint calibration parameters can be considered as the spatial mapping relationship between the camera coordinate system and the radar coordinate system when the vehicle is stationary, and will not be described in detail.

[0085] Understandably, since the vehicle is stationary, the surrounding environment captured by the radar sensor is completely consistent with that captured by the camera. Therefore, it can be assumed that the spatiotemporal synchronization of the image and radar point cloud corresponding to the static joint calibration parameters is usually the highest, and there is no spatial error caused by vehicle movement.

[0086] For example, a vehicle can pre-set a mapping relationship between driving environment information and correction weights to determine the correction weights corresponding to the current driving environment information based on the mapping relationship.

[0087] In another embodiment, the vehicle may also have a pre-set prediction model for dynamic joint calibration parameters, which can predict based on driving environment information and static joint calibration parameters to obtain dynamic joint calibration parameters. In this embodiment, the method for obtaining the dynamic joint calibration parameters is not limited.

[0088] As an example, a vehicle can be based on, for example Figure 3 Steps S301-S302, as shown, generate the aforementioned dynamic joint calibration parameters. Details are as follows:

[0089] S301. Obtain the first position information corresponding to the first radar point cloud and the second position information corresponding to the third image.

[0090] In one embodiment, the first location information can be obtained by mapping a first radar point cloud to a preset vehicle coordinate system based on calibration parameters between a preset radar coordinate system and a vehicle coordinate system. Similarly, the second location information can also be obtained by mapping objects contained in a third image to the vehicle coordinate system based on calibration parameters between a preset camera coordinate system and a vehicle coordinate system.

[0091] In another embodiment, the vehicle can also acquire third position information at multiple third moments during its movement. Then, from the multiple third position information, the third position information corresponding to the third moment closest to the current moment is determined as the first position information; and the third position information corresponding to the third moment closest to the target moment is determined as the second position information; the target moment is the second moment corresponding to the third image.

[0092] The vehicle may be equipped with a positioning device to obtain third-party location information in multiple spatiotemporal dimensions. Examples of such positioning devices include, but are not limited to, GPS (Global Positioning System) devices, BeiDou satellite navigation devices, and vehicle integrated navigation and positioning devices; no specific limitation is imposed.

[0093] It should be noted that the acquisition frequency of the positioning device for collecting the third location information can be set according to the actual situation. However, in order to accurately represent the dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system during motion, the acquisition frequency of the positioning device must be at least greater than the acquisition frequency of the camera and greater than the acquisition frequency of the radar sensor. That is, the acquisition frequency of the third location information is greater than the acquisition frequency of the first image and greater than the acquisition frequency of the first radar point cloud. For example, the acquisition frequency of the positioning device can be 100Hz.

[0094] Based on this, when the third position information corresponding to the third moment closest to the current moment is determined as the first position information, it can be considered that the first radar point cloud collected by the radar sensor at the current moment is actually closer to the point cloud obtained by the radar sensor from collecting the surrounding environment of the vehicle under the first position information.

[0095] Furthermore, when the third location information corresponding to the third time closest to the target time is determined to be the second location information, it can be assumed that the third image captured by the camera at the target time is actually closer to the image obtained by the camera from the surrounding environment of the vehicle under the second location information.

[0096] However, as explained in S103 above, the first radar point cloud and the third image are more closely synchronized in time, while spatial synchronization still has errors. That is, when the first radar point cloud and the third image are synchronized in time (i.e., the current time is the same as the target time), the surrounding environment collected by the radar sensor should be completely consistent with the surrounding environment collected by the camera, and the first position information corresponding to the first radar point cloud should also be completely consistent with the second position information corresponding to the third image.

[0097] However, due to vehicle movement, spatial errors will occur between the first radar point cloud acquired by the radar sensor and the third image acquired by the camera at the same time. This spatial error can be considered the error between the first and second position information. Therefore, the vehicle needs to perform step S302 to generate dynamic joint calibration parameters capable of correcting the spatial error.

[0098] S302. Based on the first position information and the second position information, generate dynamic joint calibration parameters.

[0099] As an example, the vehicle can calculate the spatial transformation matrix between the first and second position information. Then, based on the spatial transformation matrix, the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system are corrected to obtain the dynamic joint calibration parameters.

[0100] In one embodiment, the aforementioned spatial transformation matrix includes, but is not limited to, a rotation matrix R and a translation matrix T, and is not limited thereto. The calculation of the spatial transformation matrix between two positional information points is an existing method and will not be described in detail. Furthermore, based on the spatial transformation matrix, the method for correcting the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system can be to multiply the static joint calibration parameters by the spatial transformation matrix to obtain the dynamic joint calibration parameters.

[0101] S202. Based on the dynamic joint calibration parameters, the third image and the first radar point cloud are calibrated to obtain the calibrated spatiotemporally synchronized fourth image and second radar point cloud.

[0102] In one embodiment, calibrating the third image and the first radar point cloud based on dynamic joint calibration parameters can be achieved by: transforming the third image into the radar coordinate system based on the dynamic joint calibration parameters, and / or transforming the second radar point cloud into the camera coordinate system based on the dynamic joint calibration parameters, without limitation.

[0103] Based on the above explanation, when only the third image is transformed to the radar coordinate system, the transformed fourth image and the first radar point cloud (in this case, the first radar point cloud is also the second radar point cloud) can be considered as a spatiotemporally synchronized fourth image and second radar point cloud. Furthermore, when only the first radar point cloud is transformed to the camera coordinate system, the transformed second radar point cloud and the third image (in this case, the third image is also the fourth image) can be considered as a spatiotemporally synchronized second radar point cloud and fourth image. Finally, when both the third image and the first radar point cloud are transformed to the camera coordinate system, a calibrated spatiotemporally synchronized fourth image and second radar point cloud can be directly obtained.

[0104] In this embodiment, the spatial transformation matrix generated based on the first position information corresponding to the time-synchronized first radar point cloud and the second position information corresponding to the third image can be used to characterize the spatial error between the camera coordinate system and the radar coordinate system during motion. Based on this, correcting the static joint calibration parameters corresponding to the camera coordinate system and the radar coordinate system based on the spatial transformation matrix can enable the final dynamic joint calibration parameters to compensate for the subtle spatiotemporal errors between the camera coordinate system and the radar coordinate system. Furthermore, the spatiotemporal synchronization of the fourth image and the second radar point cloud can be further improved.

[0105] It should be added that the above embodiments are merely examples of spatiotemporal synchronization between an image captured by a single camera and a radar point cloud acquired by a radar sensor. When there are multiple cameras, the spatiotemporal synchronization method between the first image captured by each camera and the radar point cloud acquired by the radar sensor can be described with reference to the above examples, and will not be explained further.

[0106] In another embodiment, reference Figure 4 , Figure 4 This is a flowchart illustrating the implementation of a spatiotemporal synchronization method for radar point clouds and images, provided in another embodiment of this application. During vehicle movement, a first image set containing multiple first images at multiple first moments can be acquired using a camera, a first radar point cloud at the current moment can be acquired using a radar sensor, and third position information at multiple third moments can be acquired using a positioning device.

[0107] The vehicle can then perform inter-frame prediction on the first image set to obtain a second image set containing the first image set. Furthermore, from the second image set, the second image corresponding to the closest second moment to the current moment is determined as the third image, thus initially improving the spatiotemporal synchronization between the first radar point cloud and the third image.

[0108] Subsequently, the vehicle selects the third position information corresponding to the most recent third time from multiple third position information sources as the first position information; and selects the third position information corresponding to the most recent third time from the target time as the second position information. Then, it calculates the spatial transformation matrix between the first and second position information, and based on this spatial transformation matrix, corrects the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system to obtain dynamic joint calibration parameters.

[0109] Finally, the vehicle calibrates the third image and the first radar point cloud based on the dynamic joint calibration parameters, and obtains the calibrated spatiotemporally synchronized fourth image and second radar point cloud.

[0110] In this embodiment, after obtaining the spatiotemporally synchronized fourth image and second radar point cloud, the fourth image and the second radar point cloud can be fused to perceive the surrounding environment. Alternatively, roadside target detection and tracking can be performed based on the fourth image and the second radar point cloud. In this embodiment, the application scenarios of the spatiotemporally aligned fourth image and second radar point cloud are not limited.

[0111] In this embodiment, the above method is used to perform inter-frame prediction on the first image set with a low acquisition frequency to generate a second image set, which can initially obtain a first radar point cloud and a third image with spatiotemporal synchronization. Then, based on the correspondence between the first radar point cloud and the third image during motion and multiple third position information acquired by the same positioning device, a spatial variation matrix that can characterize the spatial error between the camera coordinate system and the radar coordinate system during spatial motion is generated. Based on this, the static joint calibration parameters corresponding to the camera coordinate system and the radar coordinate system are corrected based on the spatial variation matrix, so that the final dynamic joint calibration parameters can compensate for the subtle spatiotemporal errors between the camera coordinate system and the radar coordinate system. Furthermore, the spatiotemporal synchronization of the fourth image and the second radar point cloud can be further improved.

[0112] Please see Figure 5 , Figure 5 This is a structural block diagram of a spatiotemporal synchronization device for radar point clouds and images provided in an embodiment of this application. The spatiotemporal synchronization device for radar point clouds and images in this embodiment includes modules for performing... Figures 1 to 4 The steps in the corresponding embodiments. Please refer to the details. Figures 1 to 4 as well as Figures 1 to 4 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 5 The spatiotemporal synchronization device 500 for radar point clouds and images may include: a first acquisition module 510, a prediction module, and a second acquisition module 530, wherein:

[0113] The first acquisition module 510 is used to acquire a first image set and a first radar point cloud at the current moment during the vehicle's movement; the first image set includes multiple first images corresponding to the first moment.

[0114] The prediction module 520 is used to perform inter-frame prediction on the first image set to obtain a second image set containing the first image set; the second image set contains multiple second images corresponding to the second time moments respectively, and the time interval between two adjacent second time moments is less than the time interval between two adjacent first time moments.

[0115] The second acquisition module 530 is used to acquire a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time.

[0116] In one embodiment, the spatiotemporal synchronization device 500 for radar point clouds and images further includes:

[0117] The third acquisition module is used to acquire the dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system; the dynamic joint calibration parameters are used to represent the spatial mapping relationship between the camera coordinate system and the radar coordinate system during vehicle movement.

[0118] The calibration module is used to calibrate the third image and the first radar point cloud based on dynamic joint calibration parameters, so as to obtain a fourth image and the second radar point cloud that are spatiotemporally synchronized after calibration.

[0119] In one embodiment, the third acquisition module is further configured to:

[0120] First location information corresponding to the first radar point cloud and second location information corresponding to the third image are acquired respectively; dynamic joint calibration parameters are generated based on the first and second location information.

[0121] In one embodiment, the third acquisition module is further configured to:

[0122] During vehicle movement, third position information at multiple third moments is acquired; from the multiple third position information, the third position information corresponding to the third moment closest to the current moment is determined as the first position information; and the third position information corresponding to the third moment closest to the target moment is determined as the second position information; the target moment is the second moment corresponding to the third image.

[0123] In one embodiment, the acquisition frequency of the third location information is greater than the acquisition frequency of the first image and greater than the acquisition frequency of the first radar point cloud.

[0124] In one embodiment, the third acquisition module is further configured to:

[0125] Calculate the spatial transformation matrix between the first position information and the second position information; based on the spatial transformation matrix, correct the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system to obtain the dynamic joint calibration parameters.

[0126] In one embodiment, the second acquisition module 530 is further configured to:

[0127] From the second image set, the second image corresponding to the second time closest to the current time is determined as the third image.

[0128] When it is understood that, Figure 5 The block diagram of the spatiotemporal synchronization device for radar point clouds and images shown illustrates how each module performs [the necessary functions]. Figures 1 to 4 The steps in the corresponding embodiments, and for Figures 1 to 4 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figures 1 to 4 as well as Figures 1 to 4 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0129] Figure 6 This is a structural block diagram of a vehicle provided in one embodiment of this application. For example... Figure 6 As shown, the vehicle 600 in this embodiment includes a processor 610, a memory 620, and a computer program 630 stored in the memory 620 and executable on the processor 610, such as a program for a spatiotemporal synchronization method of radar point clouds and images. When the processor 610 executes the computer program 630, it implements the steps in the various embodiments of the spatiotemporal synchronization methods of radar point clouds and images described above, for example... Figure 1 S101 to S104 are shown. Alternatively, the processor 610 implements the above when executing the computer program 630. Figure 5 The functions of each module in the corresponding embodiments, for example, Figure 5 For details on the functions of each module shown, please refer to [link / reference]. Figure 5 The relevant descriptions in the corresponding embodiments.

[0130] For example, the computer program 630 can be divided into one or more modules, one or more of which are stored in the memory 620 and executed by the processor 610 to implement the spatiotemporal synchronization method of radar point cloud and image provided in the embodiments of this application. One or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 630 in the vehicle 600. For example, the computer program 630 can implement the spatiotemporal synchronization method of radar point cloud and image provided in the embodiments of this application.

[0131] Vehicle 600 may include, but is not limited to, processor 610 and memory 620. Those skilled in the art will understand that... Figure 6 This is merely an example of vehicle 600 and does not constitute a limitation on vehicle 600. It may include more or fewer components than shown, or combine certain components, or different components. For example, a vehicle may also include input / output devices, network access devices, buses, etc.

[0132] The processor 610 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0133] The memory 620 can be an internal storage unit of the vehicle 600, such as a hard drive or memory of the vehicle 600. The memory 620 can also be an external storage device of the vehicle 600, such as a plug-in hard drive, smart memory card, flash memory card, etc., installed on the vehicle 600. Furthermore, the memory 620 can include both internal storage units and external storage devices of the vehicle 600.

[0134] This application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the spatiotemporal synchronization method of radar point cloud and image as described in the above embodiments.

[0135] This application provides a computer program product that, when run on a vehicle, causes the vehicle to execute the spatiotemporal synchronization method of radar point cloud and image in the above embodiments.

[0136] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for spatiotemporal synchronization of radar point clouds and images, characterized in that, The method includes: During vehicle movement, a first image set and a first radar point cloud at the current moment are acquired; the first image set includes multiple first images corresponding to the first moment respectively. Inter-frame prediction is performed on the first image set to obtain a second image set containing the first image set; the second image set contains multiple second images corresponding to the second time points respectively, and the time interval between two adjacent second time points is less than the time interval between two adjacent first time points; Based on the current moment, a third image that is spatiotemporally synchronized with the first radar point cloud is acquired from the second image set.

2. The method according to claim 1, characterized in that, After acquiring a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time, the method further includes: The dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system are obtained; the dynamic joint calibration parameters are used to represent the spatial mapping relationship between the camera coordinate system and the radar coordinate system during the vehicle's movement. Based on the dynamic joint calibration parameters, the third image and the first radar point cloud are calibrated to obtain a calibrated spatiotemporally synchronized fourth image and second radar point cloud.

3. The method according to claim 2, characterized in that, The acquisition of dynamic joint calibration parameters between the camera coordinate system and the radar coordinate system includes: The first location information corresponding to the first radar point cloud and the second location information corresponding to the third image are obtained respectively. The dynamic joint calibration parameters are generated based on the first location information and the second location information.

4. The method according to claim 3, characterized in that, Obtaining the first location information corresponding to the first radar point cloud and the second location information corresponding to the third image includes: During the movement of the vehicle, third position information at multiple third moments is acquired; From the plurality of third location information, the third location information corresponding to the third time closest to the current time is determined as the first location information; and, The third location information corresponding to the third time closest to the target time is determined as the second location information; the target time is the second time corresponding to the third image.

5. The method according to claim 4, characterized in that, The acquisition frequency of the third location information is greater than the acquisition frequency of the first image and greater than the acquisition frequency of the first radar point cloud.

6. The method according to claim 3, characterized in that, The step of generating the dynamic joint calibration parameters based on the first location information and the second location information includes: Calculate the spatial transformation matrix between the first location information and the second location information; Based on the spatial transformation matrix, the preset static joint calibration parameters between the camera coordinate system and the radar coordinate system are corrected to obtain the dynamic joint calibration parameters.

7. The method according to any one of claims 1-6, characterized in that, The step of acquiring a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time includes: From the second image set, the second image corresponding to the second time closest to the current time is determined as the third image.

8. A spatiotemporal synchronization device for radar point clouds and images, characterized in that, The device includes: The first acquisition module is used to acquire a first image set and a first radar point cloud at the current moment during the vehicle's movement; the first image set includes multiple first images corresponding to the first moment respectively; The prediction module is used to perform inter-frame prediction on the first image set to obtain a second image set containing the first image set; the second image set contains multiple second images corresponding to second time points respectively, and the time interval between two adjacent second time points is less than the time interval between two adjacent first time points; The second acquisition module is used to acquire a third image that is spatiotemporally synchronized with the first radar point cloud from the second image set based on the current time.

9. A vehicle, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it causes the vehicle to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-7 to be performed.

Images