A time and space co-synchronization method for a camera system and a lidar

Abstract

The present application relates to the technical field of multi-sensor data fusion, in particular to a camera system and laser radar time and space synchronization method. The present application first selects one image frame from each image frame collected by the camera, and then selects one point cloud frame from each point cloud frame collected by the laser radar. Since the time difference of the selected image frame and point cloud frame is less than the matching threshold, the selected image frame and point cloud frame form a pre-matching pair. Then, the position difference information between the two image frames and point cloud frames in the pre-matching pair is calculated. Finally, the timestamps and external parameter calibration of the camera and laser radar are looped and synchronized according to the required synchronization value. From the above analysis, since the present application does not rely on artificial markers and loop synchronization, it can automatically optimize the synchronization of the camera and laser radar in the image process, thereby realizing space and time synchronization.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor data fusion technology, specifically to a method for joint temporal and spatial synchronization of a camera system and a lidar. Background Technology

[0002] Data fusion of multiple sensors (sensors in LiDAR systems that acquire point cloud images and sensors in camera systems that acquire video images) aims to leverage the strengths of each sensor to improve the system's detection capabilities. LiDAR can accurately obtain 3D coordinate information of the surrounding environment, unaffected by changes in lighting; however, its point cloud data lacks color texture information and is affected by smooth or penetrable surfaces. Compared to LiDAR, cameras easily acquire rich color semantic information but struggle to obtain depth information and are severely affected by lighting conditions. Data fusion of cameras and LiDAR can be applied to fields such as mobile mapping and autonomous driving.

[0003] Data fusion hinges on two key aspects: time synchronization and spatial synchronization (timestamp alignment and extrinsic parameter calibration), namely, the time synchronization and acquisition location synchronization of images acquired by the camera system and LiDAR. A common method for timestamp alignment is to simultaneously synchronize multiple sensors using a GPS clock. However, accurate GPS timing requires expensive hardware, and its accuracy is affected by obstacles. Traditional extrinsic parameter calibration for cameras and LiDAR typically involves manual calibration. Manual calibration usually requires creating a manual calibration board and then automatically or manually matching 2D feature points in the image and 3D feature points in the point cloud. This method is cumbersome, time-consuming, and inflexible in scene setup, making it unsuitable for real-time calibration.

[0004] In summary, existing technologies for camera and lidar have poor real-time synchronization.

[0005] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method for joint temporal and spatial synchronization of a camera system and a lidar, which solves the problem of poor real-time synchronization between cameras and lidar in existing technologies.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for joint temporal and spatial synchronization of a camera system and a lidar, comprising:

[0009] A pre-matching pair is determined, consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the pre-matching pair is less than a matching threshold.

[0010] Determine the required synchronization value corresponding to the pre-matching pair, wherein the required synchronization value characterizes the positional difference information between the corresponding points in the image frame and the point cloud frame;

[0011] Based on the required synchronization value, the camera system and the lidar are cyclically synchronized.

[0012] In one implementation, determining a pre-matching pair consisting of image frames acquired by a camera system and point cloud frames acquired by a lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the pre-matching pair is less than a matching threshold, includes:

[0013] Synchronize the first frame of the image captured by the camera system and the first frame of the point cloud captured by the lidar in time;

[0014] Determine each image frame acquired by the camera system after the first image frame;

[0015] Determine each point cloud frame acquired by the lidar after the first frame of the point cloud;

[0016] The image frames are sorted according to their acquisition time;

[0017] The point cloud frames are sorted according to their acquisition time;

[0018] From the sorted image frames and the sorted point cloud frames, image frames and point cloud frames with acquisition time differences less than a matching threshold are selected to form pre-matching pairs.

[0019] In one implementation, determining the required synchronization value corresponding to the pre-matching pair, wherein the required synchronization value characterizes the positional difference information between corresponding points in the image frame and the point cloud frame, includes:

[0020] Each adjacent matching pair corresponding to each of the pre-matching pairs is determined, and the image subframe of the adjacent matching pair is the image frame of the pre-matching pair. The acquisition time of each point cloud subframe of the adjacent matching pair is adjacent to the acquisition time of the point cloud frame of the pre-matching pair.

[0021] Determine the required sub-synchronization value corresponding to each of the adjacent matching pairs of each group of pre-matching pairs;

[0022] Determine the smallest of the required sub-synchronization values ​​among the various required sub-synchronization values;

[0023] The required synchronization value for each pre-matched pair is determined based on the minimum required sub-synchronization value for each group.

[0024] In one implementation, determining the required sub-synchronization values ​​corresponding to each adjacent matching pair of each pre-matching pair in each group includes:

[0025] Extract depth discontinuous feature points from each of the point cloud subframes of the adjacent matching pairs;

[0026] Extract the edge features of the image subframes of the adjacent matching pairs;

[0027] Based on the edge features, the image subframe is converted into an edge image;

[0028] The distance image is obtained by determining the difference between the non-edge pixels of the image subframe and the pixel values ​​of the edge image;

[0029] Project the depth-discontinuous feature points onto the distance image;

[0030] Based on the depth discontinuity feature points projected onto the distance image, each required sub-synchronization value is determined.

[0031] In one implementation, projecting the depth-discontinuous feature points onto the distance image includes:

[0032] Determine the distance between the depth discontinuous feature point and the origin of the point cloud coordinate system, which is located in the lidar;

[0033] The ratio of the three-dimensional abscissa to the three-dimensional ordinate of the feature point with discontinuous depth is determined and denoted as the first ratio.

[0034] Based on the arctangent function corresponding to the first ratio;

[0035] Based on the arctangent function and the pixel width of the distance image, the two-dimensional abscissa of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image is obtained;

[0036] The distance between the three-dimensional vertical coordinates of the discontinuous depth feature points and the distance image is determined and denoted as the second ratio.

[0037] Determine the arcsine function corresponding to the second ratio;

[0038] Based on the arcsine function and the pixel height of the distance image, the two-dimensional ordinate of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image is obtained;

[0039] Based on the two-dimensional horizontal coordinate and the two-dimensional vertical coordinate, the two-dimensional coordinates of the depth discontinuity feature points in the two-dimensional coordinate system of the distance image are obtained, and the depth discontinuity feature points are projected onto the distance image.

[0040] In one implementation, determining each desired sub-synchronization value based on the depth discontinuity feature points projected onto the distance image includes:

[0041] The two-dimensional coordinates of each feature point whose depth is discontinuous are summed to obtain the sum;

[0042] Divide the sum by the total number of feature point clouds of each of the discontinuous feature points at each depth to obtain the desired sub-synchronization value.

[0043] In one implementation, the step of cyclically synchronizing the camera system and the lidar based on the required synchronization value includes:

[0044] Based on the required synchronization value and the total number of matching pairs corresponding to the pre-matching pair, the acquisition time difference of the acquired images between the camera system and the lidar is calculated;

[0045] Based on the acquisition time difference, the acquisition time and external parameter calibration of subsequent images acquired by the camera system and the lidar are cyclically synchronized.

[0046] In one implementation, the step of cyclically synchronizing the acquisition time and extrinsic parameter calibration of subsequent images acquired by the camera system and the lidar based on the acquisition time difference includes:

[0047] The acquisition time of subsequent images acquired by the camera system and the lidar is fixed;

[0048] After fixing the acquisition time, update each group of pre-matched pairs based on the acquisition time difference;

[0049] The sum of the required synchronization values ​​for each pre-matched pair after the update is recorded as the total required synchronization value.

[0050] When the required total synchronization value is determined to be at its minimum, the rotation angle and translation distance of the camera system relative to the lidar are determined.

[0051] Based on the rotation angle and the translation distance, the subsequent image acquisition positions in the extrinsic parameter calibration of the camera system and the lidar are synchronized.

[0052] Secondly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a time and space joint synchronization program for a camera system and a lidar stored in the memory and executable on the processor. When the processor executes the time and space joint synchronization program for the camera system and the lidar, it implements the steps of the time and space joint synchronization method for the camera system and the lidar described above.

[0053] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a time-space joint synchronization program for a camera system and a lidar, wherein when the time-space joint synchronization program for the camera system and the lidar is executed by a processor, the steps of the time-space joint synchronization method for the camera system and the lidar described above are implemented.

[0054] Beneficial Effects: This invention first selects one image frame from each image frame acquired by the camera, and then selects one point cloud frame from each point cloud frame acquired by the LiDAR. Since the acquisition parallax between the selected image frame and the point cloud frame is less than the matching threshold, the selected image frame and point cloud frame are combined into a pre-matching pair. Then, the positional difference information (required synchronization value) between the two images within the pre-matching pair is calculated. Finally, the camera and LiDAR are cyclically synchronized according to the required synchronization value. From the above analysis, because this invention uses cyclic synchronization, if the camera and LiDAR become out of sync during image acquisition, they can be immediately synchronized, thus achieving real-time synchronization. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the time and space cyclic synchronization in an embodiment of the present invention;

[0056] Figure 2 This is an overall flowchart of the present invention;

[0057] Figure 3 This is a single-frame point cloud image in an embodiment of the present invention;

[0058] Figure 4 This refers to the distance image in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of spatially discontinuous feature points projected onto a distance image in an embodiment of the present invention;

[0060] Figure 6 This is a distribution map of the synchronization timestamps calculated for the matching pairs of N point cloud frames and image frames in this embodiment of the invention.

[0061] Figure 7 This is an optimization graph of the matching values ​​for four iterations in an embodiment of the present invention;

[0062] Figure 8 This is a point cloud diagram from an embodiment of the present invention;

[0063] Figure 9 Image diagrams of embodiments of the present invention;

[0064] Figure 10 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0066] Research has shown that data fusion of multiple sensors (sensors in LiDAR that acquire point cloud images and sensors in camera systems that acquire images) aims to leverage the strengths of each sensor to improve the system's detection capabilities. LiDAR can accurately obtain 3D coordinate information of the surrounding environment, unaffected by changes in lighting; however, its point cloud data lacks color texture information and is affected by smooth or penetrable surfaces. Compared to LiDAR, cameras easily acquire rich color semantic information but struggle to obtain depth information and are severely affected by lighting conditions. Data fusion of cameras and LiDAR can be applied to mobile mapping, autonomous driving, and other fields. The key to data fusion involves two aspects: time synchronization and spatial synchronization (timestamp alignment and extrinsic parameter calibration), namely, the time synchronization and acquisition location synchronization of images acquired by the camera system and LiDAR. A common method for timestamp alignment is to simultaneously synchronize multiple sensors using a GPS clock. However, accurate GPS timing requires expensive hardware, and its accuracy is affected by obstacles. Traditional extrinsic parameter calibration for cameras and LiDAR is typically done manually. Manual calibration usually requires the creation of a manual calibration board, and then the matching of 2D feature points of the image and 3D feature points of the point cloud using automatic or manual methods. This method is cumbersome and time-consuming, and the scene layout is inflexible, which is not conducive to real-time calibration.

[0067] To address the aforementioned technical problems, this invention provides a joint temporal and spatial synchronization method for a camera system and a lidar system, solving the problem of poor real-time synchronization between cameras and lidar systems in existing technologies. Specifically, a pre-matching pair is first determined, consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar system. The acquisition time difference between the image frames and point cloud frames in the pre-matching pair is less than a matching threshold. Then, a required synchronization value corresponding to the pre-matching pair is determined. This required synchronization value characterizes the positional difference information between corresponding points in the image frames and point cloud frames. Finally, based on the required synchronization value, the camera system and lidar system are cyclically synchronized.

[0068] For example, a camera system and a LiDAR system each acquire images of a building. The camera system acquires image frames C1, C2, and C3 at times t1, t2, and t3, respectively, while the LiDAR acquires point cloud frames M1, M2, and M3 at times t1′, t′2, and t3′, respectively. The difference between t2 and t1′ is less than 0.01 (the matching threshold), the difference between t3 and t′2 is also less than 0.01 (the matching threshold), and the difference between t1 and any one of t1′, t′2, or t3′ is greater than 0.01. Therefore, C2 and M1 form a pre-matching pair, and C3 and M2 also form a pre-matching pair. Then, the position difference S1 (required synchronization value) of the same point on the building in the coordinate system between C2 and M1, and the position difference S2 (required synchronization value) of the same point on the building in the coordinate system between C3 and M2 are calculated. If S1 is less than S2, then the difference between t2 and t1′ is... Figure 1 The acquisition time difference between the sensors (the image sensor in the camera and the point cloud image sensor in the LiDAR) is used to adjust the acquisition time of the camera and LiDAR to achieve time synchronization. Then, the acquisition time of both is fixed, and their extrinsic parameters (the camera's shooting angle relative to the building and the LiDAR's shooting angle relative to the building, such as rotation parameters and translation distance) are synchronized. Synchronizing their extrinsic parameters essentially means spatially synchronizing the camera and LiDAR.

[0069] Exemplary methods

[0070] The temporal and spatial joint synchronization method of the camera system and LiDAR in this embodiment can be applied to a terminal device, which can be a terminal product with image acquisition capabilities, such as an image sensor. In this embodiment, as... Figure 2 As shown, the joint temporal and spatial synchronization method of the camera system and lidar specifically includes the following steps:

[0071] S100, determine a preset matching pair consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the preset matching pair is less than a matching threshold.

[0072] If the difference in acquisition time between the two frames in a pre-matching pair is less than the matching threshold φ, then the Nth pre-matching pair PA... N The image frames and point cloud frames in the data satisfy the following conditions:

[0073]

[0074] For the Nth group of pre-matching centering lidar, the tth... i Point cloud frames collected in real time, For the Nth pre-matching camera system, the tth... j Image frames captured at different times, ||t i ,t j || represents t i With t j The absolute value of the difference.

[0075] In one embodiment, step S100 includes the following steps S101 to S106:

[0076] S101, synchronize the first frame of the image acquired by the camera system and the first frame of the point cloud acquired by the lidar in time.

[0077] Assume that the camera system and the lidar acquire the first frame of the image and the first frame of the point cloud at the same time.

[0078] S102, determine the various image frames acquired by the camera system after the first image frame.

[0079] After capturing the first image frame, the camera system will continue to capture each image frame at a fixed acquisition frequency.

[0080] S103, determine the point cloud frames acquired by the lidar that are located after the first frame of the point cloud.

[0081] After acquiring the first frame of the point cloud, the lidar will continue to acquire each point cloud frame at a fixed acquisition frequency.

[0082] S104, sort the image frames according to their acquisition time.

[0083] S105, sort the point cloud frames according to the acquisition time.

[0084] S106, from the sorted image frames and the sorted point cloud frames, select image frames and point cloud frames with acquisition time difference less than the matching threshold to form a pre-matching pair.

[0085] For example, LiDAR assigns a Unix time to each frame of point cloud data. Assuming the camera and LiDAR acquire data simultaneously, based on the camera's fixed acquisition frequency, the Unix (timestamp) time of each image frame can be obtained. By setting a synchronization threshold, time synchronization matching pairs between point cloud and image can be obtained. Table 1 lists ten pre-matching pairs.

[0086] Table 1

[0087]

[0088] For example, the acquisition time difference between the third image frame and the second point cloud frame in Table 1 is 1664181985.250260-1664181985.249470=0.000790, which is less than the matching threshold φ (0.01). Therefore, the third image frame and the second point cloud frame constitute a pre-matching pair. The same applies to other pre-matching pairs.

[0089] S200, determine the required synchronization value corresponding to the preset matching pair, wherein the required synchronization value represents the positional difference information between the corresponding points in the image frame and the point cloud frame.

[0090] In one embodiment, step S200 includes the following steps S201 to S2017:

[0091] S201, determine each adjacent matching pair corresponding to each of the pre-matching pairs, wherein the image subframe of the adjacent matching pair is the image frame of the pre-matching pair, and the acquisition time of each point cloud subframe of the adjacent matching pair is adjacent to the acquisition time of the point cloud frame of the pre-matching pair.

[0092] Pre-matched pairs PA N Including image frames and point cloud frames Then adjacent matching pairs are For example, if a is 2, then the adjacent matching pairs are in, Acquisition time and point cloud frames acquired by the lidar system The collection times are adjacent, and The acquisition times are set sequentially. The reason for using this method to create adjacent matching pairs is that the camera and LiDAR cannot be controlled to acquire data simultaneously, so single-frame images are used. The required synchronization value S is calculated by computing multiple points a few seconds before and after the matching point cloud frame.

[0093] S202, extract depth discontinuous feature points from each of the point cloud subframes of the adjacent matching pairs.

[0094] That is, extract separately The depth discontinuity feature points of these four point cloud subframes (e.g., from...) (Find depth discontinuity feature points). If there is a point in a point cloud subframe whose depth difference with its surrounding points is large, then that point is a depth discontinuity feature point.

[0095] The principle of extracting depth discontinuity feature points is as follows:

[0096] Feature point extraction largely depends on sorting the unordered point cloud into an ordered state, thus facilitating the use of subsequent extraction algorithms. Figure 3 Taking the isometric projection of panoramic images as an example, a new angle index value is assigned to the point cloud of each line, ranging from -π to π, and the calculation formula is as follows:

[0097]

[0098] The 3D point obtained by scanning the b-th beamline is represented as follows: This is the angle index value.

[0099] Then, all point clouds are reordered according to angle indices from -π to π, and depth discontinuity feature points are obtained using the following feature extraction algorithm:

[0100] γ(p b [j])=max(||p b [j-1].dp b [j].d||,||p b [j+1].dp b [j].d||)

[0101] Here, γ(p) b [j]) indicates that in p b The depth discontinuity value at point [j], p b [j-1]·d is p b [j-1] is the distance from point [j-1] to the center of the LiDAR sensor, and p is also a constant. b [j]·d is p b [j] Distance from point [j] to the center of the LiDAR sensor. A threshold of 0.5 meters is set when γ(p) b If the value of [j]) is greater than 0.5 meters, then p is determined. b [j] represents the feature points of depth discontinuity in the point cloud.

[0102] S203, extract the edge features of the image subframes of the adjacent matching pairs.

[0103] S204, Based on the edge features, the image subframe is converted into an edge image.

[0104] S205, determine the difference between the non-edge pixels of the image subframe and the pixel value of the edge image to obtain the distance image.

[0105] Distance image is the distance from the non-edge pixels surrounding the edge to that edge.

[0106] Steps S203, S204, and S205 obtain the distance image based on the following principle:

[0107] The original image is an RGB image and needs to be converted to a single-channel grayscale image, followed by edge feature extraction and conversion to... Figure 4 The edge image shown. The panoramic image's extremely wide field of view gives it rich edge features, requiring a gradient threshold for filtering. The gradient value of each pixel is calculated as follows:

[0108]

[0109] Among them, G x G y These represent the gradient values ​​in the X and Y directions, respectively, and there are no restrictions on the chosen gradient operator. After obtaining the edge feature image, distance calculation begins. Figure 4 All edge pixels in the image are set to 0. Then, the distance from the remaining pixels to the nearest edge is calculated using L2 distance. In other words, the closer a non-edge pixel is to its nearest edge, the smaller its pixel value.

[0110] S206, determine the distance d between the depth discontinuous feature point (x,y,z) and the origin (0,0,0) of the point cloud coordinate system, wherein the point cloud coordinate system is located in the lidar.

[0111]

[0112] S207, determine the ratio of the three-dimensional abscissa x to the three-dimensional ordinate y of the feature point with discontinuous depth, and denot it as the first ratio x / y.

[0113] S208, based on the arctangent function arctan(x / y) corresponding to the first ratio.

[0114] S209, based on the arctangent function and the pixel width W of the distance image, obtain the two-dimensional abscissa u of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image:

[0115] u=(int)((0.5-arctan(x / y) / 2π)×W+0.5)

[0116] S2010, determine the three-dimensional vertical coordinate z of the discontinuous feature point at the depth and the distance d, and denot it as the second ratio z / d.

[0117] S2011, Determine the arcsine function arcsin(z / d) corresponding to the second ratio z / d.

[0118] S2012, based on the arcsine function arcsin(z / d) and the pixel height H of the distance image, obtain the two-dimensional ordinate v of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image:

[0119] v=(int)((0.5-arcsin(z / d) / π)×H+0.5)

[0120] S2013, based on the two-dimensional horizontal coordinate and the two-dimensional vertical coordinate, obtain the two-dimensional coordinate D of the depth discontinuity feature point in the two-dimensional coordinate system of the distance image. u,v The process involves projecting the depth-discontinuous feature points onto the distance image.

[0121] The projection method described above is used to obtain the following: Figure 5 The projection diagram shown.

[0122] S2014, sum the two-dimensional coordinates of each of the depth-discontinuous feature points to obtain the sum.

[0123] F i .D u,v For the i-th depth discontinuous feature point F i Two-dimensional coordinates D u,v D u,v It is the pixel value at a distance of (u,v) from the image.

[0124] S2015, divide the sum by the total number of feature point clouds of each of the depth-discontinuous feature points to obtain each required sub-synchronization value S.

[0125]

[0126] S2016, determine the smallest of the required sub-synchronization values ​​among the various required sub-synchronization values.

[0127] S2017, Based on the minimum required sub-synchronization value of each group, determine the required synchronization value of the pre-matching pair in each group.

[0128] For example, the above has four adjacent matching pairs: PA' N1 The corresponding required sub-synchronization values ​​are S1 and PA′. N2 The corresponding required sub-synchronization values ​​are S2 and PA′. N3 The corresponding required sub-synchronization values ​​are S3 and PA′. N4 The corresponding required sub-synchronization value is S4. If S3 is the smallest among S1, S2, S3, and S4, then PA′ N1 PA′ N2 PA′ N3 PA′ N4 The pre-matching pair The required synchronization value S is S3.

[0129] S300, based on the required synchronization value, cyclically synchronize the camera system and the lidar.

[0130] In this embodiment, synchronization can be time synchronization and spatial synchronization, or only one of the two. When time synchronization and spatial synchronization are performed alternately, step S300 includes the following steps S301 to S306:

[0131] S301, based on the required synchronization value and the total number of matching pairs n corresponding to the pre-matching pair, calculate the acquisition time difference T between the camera system and the lidar for the acquired images. Interval (Synchronize timestamps).

[0132]

[0133] The above pre-matched pairs Adjacent matching pairs The required sub-synchronization value S3 is minimized, where Δ is the value acquired by the lidar. Time and camera capture The time difference, Δ, is the difference T between the acquisition time of the image subframe and the point cloud subframe corresponding to the minimum required sub-synchronization value in each pre-matching pair. The sum of all the T values ​​for pre-matching pairs and the average value is T. Interval .

[0134] To reduce the impact of specific environments on the registration algorithm, this embodiment performs synchronous calculations on N sets of matching pairs at fixed intervals of n, ultimately obtaining N / n synchronization time differences. Outliers are filtered out by setting high and low thresholds, and the remaining data is averaged to obtain the following... Figure 6 The final synchronization time difference (synchronization timestamp) is shown.

[0135] S302, fix the acquisition time of subsequent images acquired by the camera system and the lidar.

[0136] S303, after fixing the acquisition time, update each group of pre-matched pairs according to the acquisition time difference.

[0137] S304, determine the updated pre-matched pairs (M) of each group. f C f The sum of the required synchronization values ​​for each group corresponding to ) is denoted as the total required synchronization value.

[0138] S305, when the required total synchronization value is determined to be the minimum, the rotation angle R and translation distance t of the camera system relative to the lidar.

[0139] Steps S304 and S305 calculate R and t based on the following formulas:

[0140]

[0141] The above equation is solved using least squares optimization, the core of which lies in finding the derivative of the Jacobian matrix. First, according to the chain rule, it can be decomposed into:

[0142]

[0143] Where Loss is the synchronization value of all matched pairs, p is the pixel coordinate of the distance from the image, and q is the 3D coordinate relative to the camera coordinate system. It is the pixel gradient of the distance image. It is the derivative of 2D pixel coordinates and 3D spatial coordinates. It is the derivative of 3D coordinates with extrinsic parameters. Specifically, a certain term is expressed as follows:

[0144]

[0145]

[0146] S306, based on the rotation angle and the translation distance, cyclically update the subsequent image acquisition position in the extrinsic parameter calibration of the camera system and the lidar.

[0147] In one embodiment, the loop iterated four times, resulting in the following: Figure 7 The results of the matching value optimization are shown.

[0148] pass Figure 8 and Figure 9 This verifies that the point cloud frames and image frames of the present invention have a good fusion effect.

[0149] In summary, this invention first selects one image frame from each image frame acquired by the camera, and then selects one point cloud frame from each point cloud frame acquired by the LiDAR. Since the acquisition parallax between the selected image frame and the point cloud frame is less than the matching threshold, the selected image frame and point cloud frame are combined into a pre-matching pair. Next, the positional difference information (required synchronization value) between the two images within the pre-matching pair is calculated. Finally, the camera and LiDAR are cyclically synchronized according to the required synchronization value. From the above analysis, because this invention uses cyclic synchronization, if the camera and LiDAR become out of sync during image acquisition, they can be immediately synchronized, thus achieving real-time synchronization.

[0150] This invention first achieves time synchronization through single-frame image and point cloud registration. Based on this, it selects the time-registered images and point clouds for automatic extrinsic parameter calibration. The iterative network ensures that the calibration results are unaffected by single environmental changes, allowing for continuous global optimization and thus improving the accuracy of both time and spatial synchronization.

[0151] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 10 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for joint temporal and spatial synchronization of a camera system and a lidar system. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0152] Those skilled in the art will understand that Figure 10 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a time-space joint synchronization program for a camera system and a lidar stored in the memory and executable on the processor. When the processor executes the time-space joint synchronization program for the camera system and the lidar, it implements the following operation instructions:

[0154] A pre-matching pair is determined, consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the pre-matching pair is less than a matching threshold.

[0155] Determine the required synchronization value corresponding to the pre-matching pair, wherein the required synchronization value characterizes the positional difference information between the corresponding points in the image frame and the point cloud frame;

[0156] Based on the required synchronization value, the camera system and the lidar are cyclically synchronized.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of time and space co-synchronization of a camera system and a lidar, characterized in that, include: A pre-matching pair is determined, consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the pre-matching pair is less than a matching threshold. Determine the required synchronization value corresponding to the pre-matching pair, wherein the required synchronization value characterizes the positional difference information between the corresponding points in the image frame and the point cloud frame; Based on the required synchronization value, the camera system and the lidar are cyclically synchronized; Determining the required synchronization value corresponding to the pre-matched pair includes: Each adjacent matching pair corresponding to each of the pre-matching pairs is determined, and the image subframe of the adjacent matching pair is the image frame of the pre-matching pair. The acquisition time of each point cloud subframe of the adjacent matching pair is adjacent to the acquisition time of the point cloud frame of the pre-matching pair. Extract depth discontinuous feature points from each of the point cloud subframes of the adjacent matching pairs; Extract the edge features of the image subframes of the adjacent matching pairs; Based on the edge features, the image subframe is converted into an edge image; The distance image is obtained by determining the difference between the non-edge pixels of the image subframe and the pixel values ​​of the edge image; Project the depth-discontinuous feature points onto the distance image; Based on the depth discontinuity feature points projected onto the distance image, determine each required sub-synchronization value; Determine the smallest of the required sub-synchronization values ​​among the various required sub-synchronization values; The required synchronization value for each pre-matched pair is determined based on the minimum required sub-synchronization value for each group.

2. The camera system and lidar time and space co-synchronization method of claim 1, wherein, The step of determining a pre-matching pair consisting of image frames acquired by the camera system and point cloud frames acquired by the lidar, wherein the acquisition time difference between the image frames and the point cloud frames in the pre-matching pair is less than a matching threshold, includes: Synchronize the first frame of the image captured by the camera system and the first frame of the point cloud captured by the lidar in time; Determine each image frame acquired by the camera system after the first image frame; Determine each point cloud frame acquired by the lidar after the first frame of the point cloud; The image frames are sorted according to their acquisition time; The point cloud frames are sorted according to their acquisition time; From the sorted image frames and the sorted point cloud frames, image frames and point cloud frames with acquisition time differences less than a matching threshold are selected to form pre-matching pairs.

3. The camera system and lidar time and space co-synchronization method of claim 1, wherein, The step of projecting the depth-discontinuous feature points onto the distance image includes: Determine the distance between the depth discontinuous feature point and the origin of the point cloud coordinate system, which is located in the lidar; The ratio of the three-dimensional abscissa to the three-dimensional ordinate of the feature point with discontinuous depth is determined and denoted as the first ratio. Based on the arctangent function corresponding to the first ratio; Based on the arctangent function and the pixel width of the distance image, the two-dimensional abscissa of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image is obtained; Determine the three-dimensional vertical coordinates of the discontinuous depth feature points and the distance thereto, and denot this as the second ratio. Determine the arcsine function corresponding to the second ratio; Based on the arcsine function and the pixel height of the distance image, the two-dimensional ordinate of the depth discontinuous feature point in the two-dimensional coordinate system of the distance image is obtained; Based on the two-dimensional horizontal coordinate and the two-dimensional vertical coordinate, the two-dimensional coordinates of the depth discontinuity feature points in the two-dimensional coordinate system of the distance image are obtained, and the depth discontinuity feature points are projected onto the distance image.

4. The camera system and lidar time and space co-synchronization method of claim 3, wherein, The determination of each required sub-synchronization value based on the depth discontinuity feature points projected onto the distance image includes: The two-dimensional coordinates of each feature point whose depth is discontinuous are summed to obtain the sum; Divide the sum by the total number of feature point clouds of each of the discontinuous feature points at each depth to obtain the desired sub-synchronization value.

5. The camera system and lidar time and space co-synchronization method of claim 1, wherein, The step of cyclically synchronizing the camera system and the lidar based on the required synchronization value includes: Based on the required synchronization value and the total number of matching pairs corresponding to the pre-matching pair, the acquisition time difference of the acquired images between the camera system and the lidar is calculated; Based on the acquisition time difference, the acquisition time and external parameter calibration of subsequent images acquired by the camera system and the lidar are cyclically synchronized.

6. The camera system and lidar time and space co-synchronization method of claim 5, wherein, The step of cyclically synchronizing the acquisition time and extrinsic parameter calibration of subsequent images acquired by the camera system and the lidar based on the acquisition time difference includes: The acquisition time of subsequent images acquired by the camera system and the lidar is fixed; After fixing the acquisition time, update each group of pre-matched pairs based on the acquisition time difference; The sum of the required synchronization values ​​for each pre-matched pair after the update is recorded as the total required synchronization value. When the required total synchronization value is determined to be at its minimum, the rotation angle and translation distance of the camera system relative to the lidar are determined. Based on the rotation angle and the translation distance, the subsequent image acquisition positions in the extrinsic parameter calibration of the camera system and the lidar are synchronized.

7. A terminal device, characterized by comprising: The terminal device includes a memory, a processor, and a time and space joint synchronization program for the camera system and the lidar stored in the memory and executable on the processor. When the processor executes the time and space joint synchronization program for the camera system and the lidar, it implements the steps of the time and space joint synchronization method for the camera system and the lidar as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a time and space joint synchronization program for the camera system and the lidar. When the time and space joint synchronization program for the camera system and the lidar is executed by a processor, it implements the steps of the time and space joint synchronization method for the camera system and the lidar as described in any one of claims 1-6.

Images