Point cloud data and image alignment method, device and equipment, medium and vehicle

Abstract

The invention discloses a point cloud data and image alignment method, device and equipment, a medium and a vehicle, and relates to the technical field of data processing. The method comprises the following steps: determining N target moments; obtaining a target image corresponding to each target moment to obtain N target images; obtaining current point cloud data; acquiring at least one item of translation information and rotation information of the target object at the end moment of the first acquisition period as first pose information; acquiring at least one item of translation information and rotation information of the target object at each target moment as second pose information; obtaining the variable quantity of the second pose information corresponding to each target moment relative to the first pose information and the point cloud data variable quantity corresponding to the variable quantity; and superposing the point cloud data variable quantity corresponding to each target moment on the current point cloud data to obtain target point cloud data corresponding to each target moment. According to the embodiment of the invention, the consistency of the point cloud data acquisition time and the image acquisition time can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a method, device, equipment, medium and vehicle for aligning point cloud data and images. Background Art

[0002] At present, radar can be used to scan around objects to form point cloud data to identify the surrounding environment. At the same time, the moment when the image acquisition device such as the camera is triggered to collect 2D images is used as the acquisition moment of the 2D image, and the environment is perceived through the 2D image. The collected point cloud data can be used in the cloud to detect various obstacles (such as vehicles, pedestrians, and cyclists, etc.) through the 3D object detection model to obtain 3D detection results. The collected 2D images can also detect various obstacles through the corresponding 2D image physical detection model to obtain 2D detection results. In order to provide true value data for the vehicle-side algorithm, the 2D detection results are usually matched with the 3D detection results. When the two are matched successfully, it is considered that the radar and the image acquisition device detect the same object at the same time and position. Then the probability of the existence of this object is about 100%, and the 2D detection results and the 3D detection results can be used as the true value data input of the vehicle-side algorithm.

[0003] In the above process, it is necessary to ensure the accuracy of the true value data by ensuring that the image acquisition time and the point cloud data acquisition time are at the same time. However, there is a problem in the related art that the point cloud data acquisition time is inconsistent with the image acquisition time. Summary of the invention

[0004] The embodiment of the present application provides a method for aligning point cloud data and images, which can improve or solve the problem of inconsistency between the acquisition time of point cloud data and the acquisition time of image in the related art, and is conducive to improving the consistency of the acquisition time of point cloud data and the acquisition time of image.

[0005] In a first aspect, an embodiment of the present application provides a method for aligning point cloud data and an image, and the method for aligning point cloud data and an image includes:

[0006] Determine N target moments, where the target moment is the moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, N≥2, and N is an integer;

[0007] Acquire a target image corresponding to each target moment to obtain N target images, where the target image is acquired by an image acquisition device at the target moment;

[0008] Obtain the current point cloud data collected by the radar at the end of the first acquisition cycle, where the first acquisition cycle covers N target moments;

[0009] Acquire at least one of the translation information and the rotation information of the target object at the end of the first acquisition cycle as the first pose information; acquire at least one of the translation information and the rotation information of the target object at each target moment as the second pose information; the target object includes an image acquisition device and a radar;

[0010] Obtain the change amount of the second pose information corresponding to each target moment relative to the first pose information and the change amount of the point cloud data corresponding to the change amount;

[0011] The target point cloud data corresponding to each target moment is obtained by superimposing the current point cloud data with the change in the point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

[0012] In some optional implementations of the first aspect, determining N target moments includes:

[0013] Obtain a first acquisition frequency, a second acquisition frequency and an initial moment, wherein the first acquisition frequency is the acquisition frequency of the radar, the second acquisition frequency is the acquisition frequency of the image acquisition device, and the initial moment is the moment after the first acquisition trigger moment of the image acquisition device is delayed by the exposure time;

[0014] Determine the reciprocal of the first acquisition frequency as a first acquisition period;

[0015] Determine the reciprocal of the second acquisition frequency as the second acquisition period;

[0016] The first acquisition cycle is divided into N acquisition sub-cycles, the initial time is the initial time of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial time of each acquisition sub-cycle is the target time.

[0017] In some optional implementations of the first aspect, the first acquisition cycle is divided into N acquisition sub-cycles, the initial moment is the initial moment of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial moment of each acquisition sub-cycle is the target moment:

[0018] The start time of the Mth acquisition sub-cycle is the end time of the M-1th acquisition sub-cycle, 1≤M≤N, and M is an integer.

[0019] In some optional implementations of the first aspect, the acquisition triggering moment is the moment when the image acquisition device is started by a hard triggering method.

[0020] In some optional implementations of the first aspect, a change amount of the second pose information corresponding to each target moment relative to the first pose information and a change amount of the point cloud data corresponding to the change amount are obtained:

[0021] Get the translation change corresponding to each target moment;

[0022] Obtain the rotation matrix corresponding to each target moment as the rotation change corresponding to each target moment;

[0023] The variation includes the translation variation and rotation variation corresponding to each target moment.

[0024] In some optional implementations of the first aspect, a change amount of the second pose information corresponding to each target moment relative to the first pose information and a change amount of the point cloud data corresponding to the change amount are obtained:

[0025] The difference between the target product and the translation change corresponding to each target moment is taken as the target point cloud data corresponding to each target moment. The target product is the product of the inverse matrix of the rotation matrix corresponding to each target moment and the current point cloud data.

[0026] Based on the same inventive concept, in a second aspect, an embodiment of the present application provides a device for aligning point cloud data and an image, and the device for aligning point cloud data and an image comprises:

[0027] A determination module, used to determine N target moments, where the target moment is a moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, N ≥ 2, and N is an integer;

[0028] A first acquisition module is used to acquire a target image corresponding to each target moment to obtain N target images, where the target image is acquired by an image acquisition device at the target moment;

[0029] The second acquisition module is used to acquire the current point cloud data collected by the radar at the end of the first acquisition cycle, where the first acquisition cycle covers N target moments;

[0030] A third acquisition module is used to acquire at least one of the translation information and the rotation information of the target object at the end of the first acquisition cycle as the first pose information; and is also used to acquire at least one of the translation information and the rotation information of the target object at each target moment as the second pose information; the target object includes an image acquisition device and a radar;

[0031] A fourth acquisition module is used to obtain a change amount of the second posture information corresponding to each target moment relative to the first posture information and a change amount of the point cloud data corresponding to the change amount;

[0032] The superposition module is used to superimpose the current point cloud data with the point cloud data change corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

[0033] Based on the same inventive concept, in a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory storing computer program instructions;

[0034] The processor reads and executes the computer program instructions to implement the method for aligning point cloud data and images according to the first aspect.

[0035] Based on the same inventive concept, in a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the point cloud data and image alignment method of the first aspect is implemented.

[0036] Based on the same inventive concept, in a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product includes computer program instructions, and when the computer program instructions are executed by a processor, the method for aligning point cloud data and images of the first aspect is implemented.

[0037] Based on the same inventive concept, in a sixth aspect, an embodiment of the present application provides a vehicle, including at least one of the following:

[0038] The point cloud data and image alignment device of the second aspect; or,

[0039] The electronic device according to the third aspect; or

[0040] As the computer storage medium of the fourth aspect.

[0041] According to the point cloud data and image alignment method, device, equipment, medium and vehicle provided in the embodiments of the present application, firstly, N target moments are determined, then the target image corresponding to each target moment is obtained to obtain N target images, and then the current point cloud data collected by the radar at the end moment of the first acquisition cycle, at least one of the translation information and rotation information of the target object at the end moment of the first acquisition cycle is obtained as the first pose information, and at least one of the translation information and rotation information of the target object at each target moment is obtained as the second pose information, and then the change of the second pose information corresponding to each target moment relative to the first pose information and the change of the point cloud data corresponding to the change are obtained, and then the current point cloud data is superimposed with the change of the point cloud data corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image. That is to say, in the embodiment of the present application, the target moment is the moment after the exposure time of the image acquisition device is delayed from the acquisition trigger moment of the image acquisition device, that is, the exposure time is taken into account when compensating the N target images. Therefore, the target point cloud image corresponding to each target moment obtained by superimposing the current point cloud data on the point cloud data change corresponding to each target moment is precisely synchronized with the acquisition time of the image. In summary, according to the embodiment of the present application, it is conducive to achieving precise synchronization between the acquisition time of the point cloud data and the acquisition time of the image, that is, it is conducive to improving the consistency of the acquisition time of the point cloud data and the acquisition time of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 It is a flow chart of a method for aligning point cloud data and images provided in an embodiment of the present application;

[0044] Figure 2 This is a schematic diagram of the target time provided in the embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of a principle of compensating N target images provided by an embodiment of the present application;

[0046] Figure 4 It is a structural schematic diagram of a device for aligning point cloud data and images provided in an embodiment of the present application;

[0047] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0050] In order to solve the problem of inconsistency between the acquisition time of point cloud data and the acquisition time of image in the related art, the embodiment of the present application provides a method, device, equipment, medium and vehicle for aligning point cloud data and image. The following first introduces the method for aligning point cloud data and image provided by the embodiment of the present application.

[0051] Figure 1 It is a flow chart of a method for aligning point cloud data and images provided in an embodiment of the present application.

[0052] like Figure 1 As shown, the point cloud data and image alignment method provided in the embodiment of the present application can be applied to an electronic device or a point cloud data and image alignment device, and the following description is based on the application to an electronic device. The point cloud data and image alignment method provided in the embodiment of the present application can include the following S110 to S160.

[0053] S110, determining N target moments, where the target moment is the moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, N≥2, and N is an integer.

[0054] S120, acquiring a target image corresponding to each target moment to obtain N target images, wherein the target image is acquired by an image acquisition device at the target moment.

[0055] S130, obtaining current point cloud data collected by the radar at the end of a first collection cycle, where the first collection cycle covers N target moments;

[0056] S140, obtaining at least one of the translation information and the rotation information of the target object at the end time of the first acquisition cycle as the first pose information; obtaining at least one of the translation information and the rotation information of the target object at each target moment as the second pose information; the target object includes an image acquisition device and a radar;

[0057] S150, obtaining a change in the second pose information corresponding to each target moment relative to the first pose information and a change in the point cloud data corresponding to the change.

[0058] S160, superimposing the current point cloud data with the change amount of the point cloud data corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

[0059] According to the alignment method of point cloud data and image provided by the embodiment of the present application, firstly, N target moments are determined, then the target image corresponding to each target moment is obtained to obtain N target images, then the current point cloud data collected by the radar at the end of the first acquisition cycle, at least one of the translation information and rotation information of the target object at the end of the first acquisition cycle is obtained as the first pose information, and at least one of the translation information and rotation information of the target object at each target moment is obtained as the second pose information, then the change amount of the second pose information corresponding to each target moment relative to the first pose information and the change amount of the point cloud data corresponding to the change amount are obtained, and then the current point cloud data is superimposed with the change amount of the point cloud data corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image. That is, in the embodiment of the present application, the target moment is the moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, that is, the exposure time is taken into account when compensating the N target images, therefore, the target point cloud image corresponding to each target moment obtained by superimposing the current point cloud data with the change amount of the point cloud data corresponding to each target moment is precisely synchronized with the acquisition time of the image. In summary, according to the embodiments of the present application, it is helpful to achieve accurate synchronization between the point cloud data acquisition time and the image acquisition time, that is, it is helpful to improve the consistency between the point cloud data acquisition time and the image acquisition time.

[0060] The specific implementation of S110 to S160 will be described in detail below.

[0061] In S110, the image acquisition device may be a device capable of acquiring images. The image acquisition device may be a vehicle-mounted camera, a vehicle-mounted camera, etc. In actual implementation, the specific components included in the image acquisition device and the specific number of components may be set according to actual conditions, and are not limited here. For example, the image acquisition device may include 6 cameras, and the installation positions of the 6 cameras may be different, and they may be exposed at the same time and acquire images in different directions at the same time.

[0062] The exposure time corresponding to the image acquisition device may be stored in the electronic device, and the exposure time may be directly obtained from the electronic device itself to calculate the target time.

[0063] It is understandable that, since the exposure of the image acquisition device requires a period of time, the image can only be acquired after the exposure of the image acquisition device is completed.

[0064] In some optional implementations, the acquisition triggering moment is the moment when the image acquisition device is started by a hard triggering method. By controlling the image acquisition device to acquire images at the acquisition triggering moment by a hard triggering method, the acquisition triggering moment of the image acquisition device can be accurately controlled, ensuring that the current point cloud data and image data are acquired at the same time.

[0065] The hard triggering method may be a method of performing exposure by triggering an image acquisition device through hardware. In other words, the hard triggering method may be a method of performing acquisition by an image acquisition device such as a camera under the action of an external trigger signal.

[0066] The acquisition triggering moment may be the moment when the image acquisition device is triggered to start when an external triggering signal is received.

[0067] In some optional implementations, determining N target moments may include:

[0068] Obtain a first acquisition frequency, a second acquisition frequency and an initial moment, wherein the first acquisition frequency is the acquisition frequency of the radar, the second acquisition frequency is the acquisition frequency of the image acquisition device, and the initial moment is the moment after the first acquisition trigger moment of the image acquisition device is delayed by the exposure time;

[0069] Determine the reciprocal of the first acquisition frequency as a first acquisition period;

[0070] Determine the reciprocal of the second acquisition frequency as the second acquisition period;

[0071] The first acquisition cycle is divided into N acquisition sub-cycles, the initial time is the initial time of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial time of each acquisition sub-cycle is the target time.

[0072] In this embodiment, the first acquisition period is divided according to the second acquisition period and the initial moment, so that N target moments can be quickly determined, providing a basis for the subsequent generation of N target images.

[0073] The electronic device may store the first acquisition trigger time, the first acquisition frequency, the second acquisition frequency and the initial time, and may directly call the first acquisition trigger time, the first acquisition frequency, the second acquisition frequency and the initial time from itself. The first acquisition trigger time may be set according to actual conditions and is not limited here.

[0074] It should be noted that the first acquisition frequency is greater than the second acquisition frequency. Under this premise, the first acquisition frequency and the second acquisition frequency can be set according to actual conditions and are not limited here. For example, the first acquisition frequency can be 30 Hz and the second acquisition frequency can be 10 Hz. For another example, the first acquisition frequency can be 30 Hz and the second acquisition frequency can be 20 Hz.

[0075] The value of N is less than or equal to the ratio of the first acquisition frequency to the second acquisition frequency. Under this premise, the value of N can be set according to actual conditions and is not limited here. For example, if the first acquisition frequency is 30 Hz and the second acquisition frequency is 8 Hz, the value of N can be 2, 3, etc.

[0076] The duration of the acquisition sub-cycle is greater than or equal to the duration of the second acquisition cycle.

[0077] In some optional implementations, the first acquisition cycle is divided into N acquisition sub-cycles, the initial moment is the initial moment of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial moment of each acquisition sub-cycle is the target moment:

[0078] The start time of the Mth acquisition sub-cycle is the end time of the M-1th acquisition sub-cycle, 1≤M≤N, and M is an integer.

[0079] In this embodiment, the first acquisition cycle is divided according to the start time of the Mth acquisition sub-cycle as the end time of the M-1th acquisition sub-cycle, which is conducive to quickly and accurately determining the target time and providing a basis for the subsequent generation of N target images.

[0080] It can be understood that the start time of the first acquisition sub-cycle is the initial time. The start time of the second acquisition sub-cycle can be the time after the initial time is delayed by the second acquisition cycle, the start time of the third acquisition sub-cycle can be the time after the start time of the second acquisition sub-cycle is delayed by the second acquisition cycle, and so on. The start time of the Nth acquisition sub-cycle can be the time after the start time of the N-1th acquisition sub-cycle is delayed by the second acquisition cycle.

[0081] like Figure 2 As shown, T represents the frame end time of each frame in the point cloud data scanned by the radar; t1 represents the first target moment (that is, the initial moment) of the six cameras, which can be used as a sample of the left rear camera; t2 represents the second target moment of the six cameras, which can be used as a sample of the front camera; t3 represents the third target moment of the six cameras, which can be used as a sample of the right rear camera; t1' represents the first acquisition trigger moment of the image acquisition device, that is, the moment when the radar triggers the left rear position camera to expose; t2' represents the second acquisition trigger moment of the image acquisition device, that is, the moment when the radar triggers the front position camera to expose; t3' represents the third acquisition trigger moment of the image acquisition device, that is, the moment when the radar triggers the right rear position camera to expose. Figure 2 The horizontal dotted line in represents the auxiliary line, the arrow in the circle with an arrow represents the direction of radar scanning, and the circle represents the radar acquisition cycle. Among them, the target time includes t1, t2 and t3, and the acquisition trigger time includes t1', t2' and t3'.

[0082] Exemplarily, the first acquisition frequency is 30 Hz, the second acquisition frequency is 10 Hz, the first acquisition period is 1 / 30 s, the second acquisition period is 1 / 10 s, the N target times include t1, t2 and t3, t1 = (T-83) / 1000s, t2 = (T-50) / 1000s, t3 = (T-17) / 1000ms.

[0083] In S120, after determining the N target moments, the electronic device may also acquire a target image corresponding to each target moment to obtain N target images.

[0084] For example, the electronic device can control the image acquisition device to acquire images at each target moment by hard triggering to obtain a target image corresponding to each target moment. The target moment and the target image can be in a one-to-one correspondence, that is, one target moment corresponds to one target image.

[0085] In S130, after the electronic device acquires the target image corresponding to each target moment to obtain N target images, it can also acquire the current point cloud data acquired by the radar at the end moment of the first acquisition cycle.

[0086] The current point cloud data is the point cloud data at the end of the first acquisition cycle, and is acquired by the radar at the end of the first acquisition cycle. The first acquisition cycle covers N target moments.

[0087] In S140, after the electronic device obtains the current point cloud data collected by the radar at the end of the first acquisition cycle, it can also obtain at least one of the translation information and rotation information of the target object at the end of the first acquisition cycle as the first pose information; and obtain at least one of the translation information and rotation information of the target object at each target moment as the second pose information.

[0088] Exemplarily, the first pose information may be the first translation information and the first rotation information of the target object in the target coordinate system at the end moment of the first acquisition cycle, and the second pose information may be the second translation information and the second rotation information of the target object in the target coordinate system at each target moment.

[0089] As an example, the first translation information may be the translation information of the target object at the end of the first acquisition period relative to the start of the first acquisition period. The first rotation information may be the rotation information of the target object at the end of the first acquisition period relative to the start of the first acquisition period. The second translation information may be the translation information of the target object at each target moment relative to the start of the first acquisition period. The second rotation information may be the rotation information of the target object at each target moment relative to the start of the first acquisition period.

[0090] As another example, the first translation information may be the translation information of the target object at the end moment of the first acquisition period relative to the start moment of the first acquisition period. The first rotation information may be the rotation information of the target object at the end moment of the first acquisition period relative to the start moment of the first acquisition period. The second translation information may be the translation information of the target object at each target moment relative to the previous target moment of the target object at that target moment. The second rotation information may be the rotation information of the target object at each target moment relative to the previous target moment of the target object at that target moment. For example, the second translation information of the Pth target moment may be the translation information of the Pth target moment relative to the P-1th target moment, where P is a positive integer and P is less than or equal to N.

[0091] The target coordinate system can be an odometer coordinate system. The odometer coordinate system is a coordinate system used to represent the position and posture of a moving object during motion. It is usually defined relative to a reference point or reference coordinate system. The origin of the odometer coordinate system is usually the initial position of the object, while the coordinate axes are defined relative to the direction of motion and rotation of the object. The odometer coordinate system has strong real-time performance and can improve positioning accuracy.

[0092] The inertial measurement unit carried by the target object can measure the angular velocity and acceleration of the target object at the end of the first acquisition cycle, and the first translation information and the first rotation information of the target object can be obtained by integrating the acceleration and the angular velocity. The inertial measurement unit carried by the target object can measure the angular velocity and acceleration of the target object at each target moment, and the second translation information and the second rotation information of the target object can be obtained by integrating the acceleration and the angular velocity.

[0093] Alternatively, the position information of the target object may be provided by a global positioning system (GPS), and the first translation information and first rotation information of the target object at the end of the first acquisition cycle, as well as the second translation information and second rotation information of the target object at each target moment, may be calculated by using the position information at different moments. Alternatively, the first translation information and first rotation information of the vehicle at the end of the first acquisition cycle, as well as the second translation information and second rotation information of the vehicle at each target moment, may be obtained by detecting and tracking feature points or objects in the environment through visual sensors such as cameras and laser radars installed on the target object. For example, the first translation information and first rotation information of the vehicle at the end of the first acquisition cycle, as well as the second translation information and second rotation information of the vehicle at each target moment, may be determined using a visual odometer algorithm. Alternatively, the first translation information and first rotation information of the vehicle at the end of the first acquisition cycle, as well as the second translation information and second rotation information of the vehicle at each target moment, may be obtained through sensors for measuring translation information and rotation information, such as steering sensors and wheel encoders on the target object.

[0094] The target object includes an image acquisition device and a radar. The target object may include a vehicle equipped with the image acquisition device and the radar.

[0095] In S150, after acquiring the first pose information and the second pose information, the electronic device may also acquire a change in the second pose information corresponding to each target moment relative to the first pose information and a change in the point cloud data corresponding to the change.

[0096] In some optional implementations, the change amount of the second pose information corresponding to each target moment relative to the first pose information and the change amount of the point cloud data corresponding to the change amount are obtained:

[0097] Get the translation change corresponding to each target moment;

[0098] Obtain the rotation matrix corresponding to each target moment as the rotation change corresponding to each target moment;

[0099] The variation includes the translation variation and rotation variation corresponding to each target moment.

[0100] In this embodiment, by sequentially obtaining the translation variation corresponding to the no-target moment and the rotation matrix corresponding to the no-target moment as the rotation variation corresponding to each target moment, a basis is provided for the subsequent alignment of the current point cloud data and the target image.

[0101] As an example, when the reference times of the first pose information and the second pose information are both the starting time of the first acquisition cycle, the difference between the translation information in the second pose information corresponding to each target moment and the translation information in the first pose information can be used as the translation change corresponding to each target moment; the difference between the rotation information in the second pose information corresponding to each target moment and the rotation information in the first pose information can be used as the rotation change corresponding to each target moment.

[0102] As another example, when the reference moment of the second pose information is the previous target moment of the target moment, the sum of the translation information in the second pose information corresponding to the 1st to the Pth target moments and the difference between the translation information in the first pose information can be used as the translation change corresponding to the Pth target moment; the sum of the rotation information in the second pose information corresponding to the 1st to the Pth target moments and the difference between the rotation information in the first pose information can be used as the rotation change corresponding to the Pth target moment.

[0103] In one example, the rotation change corresponding to each target moment may include a rotation matrix corresponding to each target moment. The translation information may include at least one of coordinates and longitude and latitude.

[0104] In some optional implementations, the change amount of the second pose information corresponding to each target moment relative to the first pose information and the change amount of the point cloud data corresponding to the change amount are obtained:

[0105] The target point cloud data corresponding to each target moment is taken as the target product and the difference between the translation change amount corresponding to each target moment. The target product is the product of the inverse matrix of the rotation matrix corresponding to each target moment and the current point cloud data. In this embodiment, the target point cloud data corresponding to each target moment can be accurately and quickly determined by taking the difference between the target product and the translation change amount corresponding to each target moment as the target point cloud data corresponding to each target moment.

[0106] Exemplarily, the target time includes the first target time t1, the translation change corresponding to t1 is H1, the rotation matrix corresponding to t1 is R1, the target point cloud data corresponding to t1 is Gt1, and the current point cloud data is GT1, then Gt1=R1 -1 ×GT1-H1.

[0107] In order to facilitate understanding of the point cloud data and image alignment method provided in the embodiment of the present application, a specific example is used below to illustrate.

[0108] like Figure 3 As shown, the T moment includes the T1 moment, the T2 moment, the T3 moment and the T4 moment. Figure 3The current point cloud data collected at time T1, the current point cloud data collected at time T2, the current point cloud data collected at time T3, and the current point cloud data collected at time T4 are shown. The images below t1 to t3 (i.e., the target time) respectively represent the target images generated by triggering 6 cameras to be exposed by hard triggering 3 times in the current point cloud data acquisition cycle at each time from T1 to T4, and the exposure at the three times from t1 to t3. That is, the image below t1 represents the image obtained by triggering 6 cameras to be exposed by the first hard trigger at time t1 in the first acquisition cycle of the current point cloud data, the image below t2 represents the image obtained by triggering 6 cameras to be exposed by the first hard trigger at time t2 in the first acquisition cycle of the current point cloud data, and the image below t3 represents the image obtained by triggering 6 cameras to be exposed by the first hard trigger at time t3 in the current point cloud data acquisition cycle. Among them, the first acquisition frequency is 30Hz, the second acquisition frequency is 10Hz, the trigger period is 1 / 30s, the acquisition period is 1 / 10s, t1 = (T-83) / 1000s, t2 = (T-50) / 1000s, t3 = (T-17) / 1000s. T represents the frame end time of each frame in the point cloud data scanned by the radar.

[0109] Because the acquisition time of point cloud data is the frame tail time of point cloud data, the acquisition time of point cloud data and camera (i.e. image acquisition device) can be precisely synchronized by compensating point cloud data to the corresponding three camera exposure times. The essence of point cloud compensation is to generate target point cloud data corresponding to other times (i.e. t1, t2 and t3) based on the current point cloud data, through the pose information of the ego vehicle (i.e. target object) at each camera exposure time and the pose information at the acquisition time of point cloud data in the odometer coordinate system (i.e. target coordinate system). In other words, if there is point cloud data at time T and ego vehicle information (i.e. ego vehicle pose information, etc.) at times T / 1000s, (T-83) / 1000s, (T-50) / 1000s and (T-17) / 1000s, target point cloud data at other times can be generated, and the acquisition time of point cloud data and image can be precisely synchronized, which is conducive to improving the consistency of point cloud data acquisition time and image acquisition time.

[0110] It can be seen that in the embodiments of the present application, the image acquisition device and radar built into the vehicle can be used, and the modification of the vehicle is relatively simple and has a certain degree of versatility.

[0111] Based on the same inventive concept as the above-mentioned method for aligning point cloud data and images, the embodiment of the present application further provides a device for aligning point cloud data and images. The device for aligning point cloud data and images is described in detail below.

[0112] Figure 4It is a structural schematic diagram of a point cloud data and image alignment device provided in an embodiment of the present application.

[0113] like Figure 4 As shown, the point cloud data and image alignment device 400 provided in the embodiment of the present application can be applied to electronic devices. The point cloud data and image alignment device 400 can include a determination module 410, a first acquisition module 420, a second acquisition module 430, a third acquisition module 440, a fourth acquisition module 450, and an overlay module 460.

[0114] A determination module 410 is used to determine N target moments, where the target moment is a moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, where N≥2, and N is an integer;

[0115] A first acquisition module 420 is used to acquire a target image corresponding to each target moment to obtain N target images, where the target image is acquired by an image acquisition device at the target moment;

[0116] The second acquisition module 430 is used to acquire the current point cloud data collected by the radar at the end of the first acquisition cycle, where the first acquisition cycle covers N target moments;

[0117] The third acquisition module 440 is used to acquire at least one of the translation information and the rotation information of the target object at the end of the first acquisition cycle as the first pose information; and is also used to acquire at least one of the translation information and the rotation information of the target object at each target moment as the second pose information; the target object includes an image acquisition device and a radar;

[0118] A fourth acquisition module 450 is used to obtain a change amount of the second pose information corresponding to each target moment relative to the first pose information and a change amount of the point cloud data corresponding to the change amount;

[0119] The superposition module 460 is used to superimpose the current point cloud data with the point cloud data change corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

[0120] According to the device for aligning point cloud data and image provided in the embodiment of the present application, firstly, N target moments are determined, then the target image corresponding to each target moment is obtained to obtain N target images, then the current point cloud data collected by the radar at the end of the first acquisition cycle, at least one of the translation information and rotation information of the target object at the end of the first acquisition cycle as the first pose information, and at least one of the translation information and rotation information of the target object at each target moment as the second pose information, then the change amount of the second pose information corresponding to each target moment relative to the first pose information and the change amount of the point cloud data corresponding to the change amount are obtained, and then the current point cloud data is superimposed with the change amount of the point cloud data corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image. That is, in the embodiment of the present application, the target moment is the moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, that is, the exposure time is taken into account when compensating the N target images, therefore, the target point cloud image corresponding to each target moment obtained by superimposing the current point cloud data with the change amount of the point cloud data corresponding to each target moment is precisely synchronized with the acquisition time of the image. In summary, according to the embodiments of the present application, it is helpful to achieve accurate synchronization between the point cloud data acquisition time and the image acquisition time, that is, it is helpful to improve the consistency between the point cloud data acquisition time and the image acquisition time.

[0121] In some optional implementations, the determination module 410 may include:

[0122] A first acquisition submodule is used to acquire a first acquisition frequency, a second acquisition frequency and an initial time, wherein the first acquisition frequency is the acquisition frequency of the radar, the second acquisition frequency is the acquisition frequency of the image acquisition device, and the initial time is the time after the first acquisition trigger time of the image acquisition device is delayed by the exposure time;

[0123] A first determining submodule, configured to determine the inverse of the first acquisition frequency as a first acquisition period;

[0124] A second determining submodule, configured to determine the inverse of the second acquisition frequency as a second acquisition period;

[0125] The division submodule is used to divide the first acquisition cycle into N acquisition sub-cycles, the initial time is the initial time of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial time of each acquisition sub-cycle is the target time.

[0126] In some optional implementations, the first acquisition cycle is divided into N acquisition sub-cycles, the initial moment is the initial moment of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial moment of each acquisition sub-cycle is the target moment:

[0127] The start time of the Mth acquisition sub-cycle is the end time of the M-1th acquisition sub-cycle, 1≤M≤N, and M is an integer.

[0128] In some optional implementations, the acquisition triggering moment is the moment when the image acquisition device is started by a hard triggering method.

[0129] In some optional implementations, the fourth acquisition module 450 may include:

[0130] The second acquisition submodule is used to obtain the translation change corresponding to each target moment;

[0131] The third acquisition submodule is used to obtain the rotation matrix corresponding to each target moment as the rotation change corresponding to each target moment;

[0132] The variation includes the translation variation and rotation variation corresponding to each target moment.

[0133] In some optional implementations, the fourth acquisition module 450 may be specifically configured to:

[0134] The difference between the target product and the translation change corresponding to each target moment is taken as the target point cloud data corresponding to each target moment. The target product is the product of the inverse matrix of the rotation matrix corresponding to each target moment and the current point cloud data.

[0135] Regarding the point cloud data and image alignment device in the above embodiment, the specific manner in which each module performs operations and the beneficial effects thereof have been described in detail in the embodiment of the method, and will not be elaborated on in detail here.

[0136] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0137] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0138] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0139] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0140] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0141] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement any one of the point cloud data and image alignment methods in the above embodiments.

[0142] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0143] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0144] Bus 510 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0145] In addition, in combination with the method for aligning point cloud data and images in the above embodiment, the embodiment of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the method for aligning point cloud data and images provided in the embodiment of the present application is implemented.

[0146] An embodiment of the present application also provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for aligning point cloud data and images as provided in the embodiment of the present application.

[0147] The present application also provides a vehicle, which may include at least one of the following:

[0148] The point cloud data and image alignment device in the above embodiment; or,

[0149] The electronic device in the above embodiment, or

[0150] The computer storage medium in the above embodiments.

[0151] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0152] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to execute the required commands. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0153] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0154] Aspects of the present application are described above with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable point cloud data and image alignment device to produce a machine so that these instructions executed by the processor of the computer or other programmable point cloud data and image alignment device enable the implementation of the functions / actions specified in one or more boxes in the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0155] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

[0156] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for aligning point cloud data and images, It is characterized in that include: Determine N target moments, where the target moment is a moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, N≥2, and N is an integer; Acquire a target image corresponding to each of the target moments to obtain N target images, wherein the target images are acquired by the image acquisition device at the target moments; Acquire current point cloud data collected by the radar at the end of a first collection cycle, where the first collection cycle covers the N target moments; Acquire at least one of translation information and rotation information of the target object at the end of the first acquisition cycle as the first pose information; Acquire at least one of the translation information and the rotation information of the target object at each target moment as the second posture information; the target object includes the image acquisition device and the radar; Obtaining a change amount of the second posture information corresponding to each target moment relative to the first posture information and a change amount of the point cloud data corresponding to the change amount; The target point cloud data corresponding to each target moment is obtained by superimposing the current point cloud data with the change amount of the point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

2. The method according to claim 1, It is characterized in that The determining of N target moments comprises: Acquire a first acquisition frequency, a second acquisition frequency and an initial time, wherein the first acquisition frequency is the acquisition frequency of the radar, the second acquisition frequency is the acquisition frequency of the image acquisition device, and the initial time is the time after the first acquisition triggering time of the image acquisition device is delayed by the exposure time; Determine the reciprocal of the first acquisition frequency as the first acquisition period; Determine the reciprocal of the second acquisition frequency as a second acquisition period; The first acquisition cycle is divided into N acquisition sub-cycles, the initial moment is the initial moment of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial moment of each acquisition sub-cycle is the target moment.

3. The method according to claim 2, It is characterized in that The first acquisition cycle is divided into N acquisition sub-cycles, the initial time is the initial time of the first acquisition sub-cycle, each acquisition sub-cycle has the same time length as the second acquisition cycle, and the initial time of each acquisition sub-cycle is the target time: The start time of the Mth acquisition sub-cycle is the end time of the M-1th acquisition sub-cycle, 1≤M≤N, and M is an integer.

4. The method according to claim 1, It is characterized in that The acquisition triggering moment is the moment when the image acquisition device is started by using a hard triggering method.

5. The method according to claim 1, It is characterized in that In the step of obtaining the change amount of the second posture information corresponding to each target moment relative to the first posture information and the change amount of the point cloud data corresponding to the change amount: Obtaining the translation change corresponding to each target moment; Obtaining a rotation matrix corresponding to each target moment as a rotation change corresponding to each target moment; The variation includes the translation variation and the rotation variation corresponding to each target moment.

6. The method according to claim 5, It is characterized in that In the step of obtaining the change amount of the second posture information corresponding to each target moment relative to the first posture information and the change amount of the point cloud data corresponding to the change amount: The difference between the target product and the translation change corresponding to each target moment is used as the target point cloud data corresponding to each target moment, and the target product is the product of the inverse matrix of the rotation matrix corresponding to each target moment and the current point cloud data.

7. A device for aligning point cloud data and images, It is characterized in that include: A determination module, used to determine N target moments, where the target moment is a moment after the acquisition trigger moment of the image acquisition device is delayed by the exposure time of the image acquisition device, N≥2, and N is an integer; A first acquisition module, used for acquiring a target image corresponding to each target moment to obtain N target images, wherein the target image is acquired by the image acquisition device at the target moment; A second acquisition module is used to acquire the current point cloud data collected by the radar at the end of a first acquisition cycle, where the first acquisition cycle covers the N target moments; a third acquisition module, used to acquire at least one of the translation information and the rotation information of the target object at the end time of the first acquisition period as the first pose information; and also used to acquire at least one of the translation information and the rotation information of the target object at each target moment as the second pose information; the target object includes the image acquisition device and the radar; A fourth acquisition module, used to acquire a change amount of the second posture information corresponding to each target moment relative to the first posture information and a change amount of the point cloud data corresponding to the change amount; The superposition module is used to superimpose the current point cloud data with the point cloud data change amount corresponding to each target moment to obtain the target point cloud data corresponding to each target moment, so as to complete the alignment of the current point cloud data and the target image.

8. An electronic device, It is characterized in that The electronic device comprises a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the point cloud data and image alignment method as described in any one of claims 1-6.

9. A computer storage medium, It is characterized in that The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the point cloud data and image alignment method described in any one of claims 1 to 6 is implemented.

10. A vehicle, It is characterized in that Include at least one of the following: The point cloud data and image alignment device as claimed in claim 7; or, The electronic device as claimed in claim 8, or The computer storage medium of claim 9.

Images