A method and device for synchronizing point cloud and image based on semantic point matching

Abstract

The application discloses a point cloud and image synchronization method and device based on semantic point matching, which is characterized in that: by inputting the continuous image frames shot by a camera into a frame interpolation model, the adjacent frame images are supplemented to obtain the interpolated frame images, and then the obtained images are subjected to image semantic segmentation; by inputting the three-dimensional point cloud data scanned by a laser radar of the same frequency into a point cloud semantic segmentation model, the point cloud semantic segmentation is carried out; finally, the point cloud segmentation result is mapped to an image coordinate system, and a synchronization image which is consistent with a synchronization moment and has the best synchronization is found through a semantic point matching algorithm. The application realizes the millisecond-level synchronization of the point cloud and the image data, reduces the synchronization error of data fusion, thereby improving the environment perception capability of the downstream automatic driving vehicle under complex road conditions, and has wide practical value.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and device for synchronizing point clouds and images based on semantic point matching. Background Art

[0002] Autonomous driving technology uses a variety of sensing devices to acquire environmental information, such as cameras, millimeter-wave radar, ultrasonic sensors, and GPS positioning. These diverse sensing devices provide comprehensive environmental information, enabling the vehicle to more accurately understand its surroundings. To effectively utilize the sensory information acquired by these multiple sensors, data fusion is often employed. This data fusion typically employs multiple algorithms to fuse data from different sensor types, ensuring high accuracy while minimizing errors.

[0003] During the data fusion stage, fusing the point cloud data scanned by the LiDAR (Lidar) with the image data collected by the camera is a key task in multi-sensor fusion. Cameras and LiDAR are commonly used sensors in autonomous driving systems, but they differ in how they collect data. Cameras collect image data based on instantaneous exposure, while LiDAR acquires point cloud data through periodic scanning. Therefore, even if the LiDAR and camera collect data at the same frequency, the vehicle's movement can still cause the collected data to become out of sync, affecting the precision and accuracy of subsequent processing. To address this issue, it is necessary to effectively fuse the data generated by the two sensors and use more effective algorithms to improve real-time synchronization and eliminate errors in the fused data. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention proposes a point cloud and image synchronization method and device based on semantic point matching. It adopts a semantic segmentation model to process point cloud and image data, uses semantic information to construct a matching degree association between point cloud and image information, and obtains the highest level of synchronization between point cloud and image by calculating the matching error of semantic points, thus solving the problem of point cloud and image asynchrony in the multi-sensor fusion process in the existing technology.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for synchronizing point clouds and images based on semantic point matching, the method comprising the following steps:

[0007] Step 1: The on-board LiDAR and camera are fixed to the vehicle to form a rigid structure and remain relatively stationary. During the vehicle's driving process, the on-board LiDAR and camera scan and shoot at the same frequency within the acquisition cycle to collect point cloud and image data of the surrounding environment. Continuous point cloud and image acquisition data are generated within the acquisition cycle based on the timestamp of the acquisition moment.

[0008] Step 2: Using an image interpolation algorithm, a number of interpolated images between two adjacent frames captured by the camera are obtained, thereby forming an image set within the time period from the moment before the current moment to the moment after the current moment;

[0009] Step 3: Use the trained image semantic segmentation model to perform semantic segmentation on the images in the image set obtained in step 2 to obtain a set of segmentation maps;

[0010] Step 4: Use the trained point cloud semantic segmentation model to segment the point cloud dataset obtained by the vehicle-mounted lidar scan to obtain the spatial coordinates and category labels of all semantic points;

[0011] Step 5: Use the semantic point matching obtained in step 4 to obtain the segmentation map of the two-dimensional image at the synchronization moment, and use the source image corresponding to the segmentation map with the highest matching accuracy as the final synchronization image.

[0012] Furthermore, in step one, when collecting data, the hardware clock is used to synchronize time so that the laser radar scanning time and the camera shooting time are close to the same moment.

[0013] Furthermore, the image interpolation algorithm in step 2 is a deep learning frame interpolation algorithm based on motion information and appearance features.

[0014] Furthermore, the step 2 specifically includes:

[0015] For t i Moment Image Take the adjacent image of the previous moment Perform interpolation, the interpolation multiple is n, and get [t i-1 ,t i ]Interval interpolation set Contains n generated interpolated images;

[0016] For t i Moment Image Take the adjacent image at the next moment Perform interpolation, the interpolation multiple is n, and get [t i ,t i+1 ]Interval interpolation set Contains n generated interpolated images;

[0017] For t i Moment Image From this we can get that [t i-1 ,t i+1 ] interval image collection gather Contains a total of 2*n+1 images.

[0018] Furthermore, the image semantic segmentation model in step three is a codec method based on deep learning.

[0019] Furthermore, the step three specifically includes the following sub-steps:

[0020] (3.1) Yes The encoder takes the input image img i Perform feature extraction on each label channel to obtain the feature map feature i ;

[0021] (3.2) The decoder converts the semantic feature i Upsample to the size of the source image to obtain the corresponding segmentation result map seg i ; (3.3) Each image sample outputs the corresponding segmentation map, and each semantic label object on the segmentation map is drawn with different pixel values. Get t i A collection of segmentation graphs at a given moment

[0022] Furthermore, the point cloud semantic segmentation model in step 4 is a projection-based deep learning codec method.

[0023] Furthermore, the step 4 includes the following sub-steps:

[0024] (4.1) t i Time point cloud Input the three-dimensional space to two-dimensional space mapping module in the projection-based deep school codec method to obtain the corresponding two-dimensional mapping map pcd proj ;

[0025] (4.2) The two-dimensional mapping pcd obtained in step (4.1) proj Input decoder module to extract semantic features of two-dimensional mapping proj ;

[0026] (4.3) Semantic features proj Input decoder module for decoding, upsampling to two-dimensional map pcd proj The size of the segmentation map is obtained, and the segmentation map is used to distinguish the labels by pixel values;

[0027] (4.4) Decoded feature proj Input the mapping module from two-dimensional space to three-dimensional space. When mapping each point, the semantics are distinguished according to the pixel value of the segmentation map to obtain the semantic label set of all points.

[0028] Furthermore, the step five includes the following sub-steps:

[0029] (5.1) In the camera's external parameter matrix Mat ext Add a row of (0, 0, 0, 1) vectors at the end of the row to form a 4x4 square matrix; in the camera's 3x3 size Mat iner Add a column of zero vectors at the end of the column to form a 3x4 matrix; where Mat ext =[rotat 3D |trans 3D ], Mat iner =trans 2D ×scaling 2D , trans 2D is the two-dimensional translation matrix of the camera, scaling 2D is the two-dimensional scaling matrix of the camera, rotat 3D is the three-dimensional rotation matrix of the camera, trans 3D is the three-dimensional translation column vector of the camera;

[0030] (5.2) at t i moment, for point cloud According to the mapping rules from point cloud to image and the internal and external parameter matrix Mat of the camera iner and Mat ext , get the two-dimensional mapping proj of the point cloud ti ,

[0031] (5.3) For the segmentation graph seg i Mapping proj with point cloud ti , in seg i The width and height range w×h of proj ti Filter: u=0 <u<w,v=0<v<h,得到与seg i The two-dimensional mapping proj′ of the filtered point cloud of the same size ti ;

[0032] (5.4) Traversal and seg i The two-dimensional mapping proj′ of the filtered point cloud of the same size ti N on p points, for any point (x c ,y c )exist The pixels of the segmentation map are judged on the top. If the mapping coordinates of the c-type points (x c ,y c ) in seg i There is (x c ,yc )=pix c The match is correct, and the matching error is match c =0, otherwise it is a matching error, match c =1; for all i After matching the points within the range, calculate seg i The matching accuracy

[0033] (5.5) For the set of segmentation graphs All elements of proj′ ti The matching accuracy is calculated to get the highest matching accuracy Sync max , whose segmentation map is seg max , its corresponding source image imgs ync This is the desired synchronization image

[0034] A point cloud and image synchronization device based on semantic point matching includes one or more processors for implementing a point cloud and image synchronization method based on semantic point matching.

[0035] The beneficial effects of the present invention are as follows:

[0036] The present invention's semantic point matching-based point cloud and image synchronization method and device utilize continuously acquired point cloud and image data, employing a semantic segmentation model and interpolation model to synchronize point cloud and image data. In vehicle-driven scenarios, this method can synchronize point cloud and image data within milliseconds without the need for additional auxiliary equipment, reducing synchronization errors caused by sensor device disparity and improving the accuracy of downstream decision-making tasks such as autonomous driving detection and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flowchart of a method for synchronizing point clouds and images based on semantic point matching provided by an embodiment of the present invention;

[0039] Figure 2 This is an example diagram of frame insertion according to an embodiment of the present invention.

[0040] Figure 3 FIG. 4 is an example diagram of image segmentation according to an embodiment.

[0041] Figure 4 A point cloud image fusion synchronization effect comparison chart for an embodiment.

[0042] Figure 5 A schematic diagram of a point cloud and image synchronization device based on semantic point matching for an embodiment. DETAILED DESCRIPTION

[0043] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. The following detailed description includes specific details for the purpose of providing a thorough understanding of the exemplary embodiments. However, it will be apparent to those skilled in the art that the exemplary embodiments can be practiced without these specific details. In some instances, well-known structures and components are not described in detail in order to avoid obscuring the understanding of the exemplary embodiments.

[0044] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0045] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "and / or" as used herein refer to and encompasses any and all possible combinations of one or more of the associated listed items.

[0046] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "and / or" as used herein refer to and encompasses any and all possible combinations of one or more of the associated listed items.

[0047] As shown in FIG. 1, an embodiment of the present application provides a point cloud and image synchronization method based on semantic point matching, which includes the following steps: Figure 1

[0048] Step 1: Data acquisition. The vehicle-mounted laser radar and vehicle-mounted camera are fixed on the vehicle to form a rigid structure and remain relatively static. During the vehicle driving process, point cloud and image data of the surrounding environment are collected. The driving speed of the vehicle on urban roads is usually not higher than 60 km / s. ​

[0049] (1.1) During the acquisition cycle, the camera and LiDAR capture and scan at the same frequency, and the captured data carries the timestamp of the current acquisition moment. Based on the timestamp of the acquisition moment, continuous point cloud and image acquisition data are generated within the acquisition cycle. During acquisition, the hardware clock is used to synchronize the time, so that the LiDAR scanning time and the camera capture time are close to the same moment.

[0050] This example captures point cloud and image data from a standard road section in Beijing, including information about vehicles, pedestrians, cyclists, roads, and green belts. The LiDAR scanning frequency and camera capture frequency are both 10 Hz, and the camera is fixed in a forward-facing perspective. The LiDAR scanner is mounted on the vehicle's roof.

[0051] (1.2) Assume that the continuous collection moments in the collection period constitute the time set {t0, t1,…, t n}, denoted as T, for any time t i ∈T, at t i There are always two-dimensional images captured by the camera and 3D point cloud data collected by LiDAR t i When the time is not the boundary of the T set, there is t i The previous moment is t i-1 ,t i The next moment is t i+1 , the images and point cloud data corresponding to the moments are respectively available at the adjacent moments.

[0052] Step 2: Use the interpolation algorithm to interpolate frames and obtain several interpolated frame images between two frames.

[0053] After capturing consecutive frames of camera images, an interpolation algorithm is used to interpolate adjacent frames. This interpolation algorithm can be based on optical flow information or deep learning. This invention uses a deep learning-based frame interpolation algorithm to extract the motion features of pixels in two adjacent frames and, using motion inertia inference, generate images between the two frames not captured by the camera. The interpolation factor is the number of images generated between the two frames.

[0054] For t i Moment Image Take the adjacent image of the previous moment Perform interpolation, the interpolation multiple is n, and get [t i-1 ,t i ]Interval interpolation set Contains n generated interpolated images;

[0055] For t i Moment Image Take the adjacent image at the next moment Perform interpolation, the interpolation multiple is n, and get [t i ,t i+1 ]Interval interpolation set Contains n generated interpolated images.

[0056] For t i Moment Image From this we can get that [t i-1 ,t i+1 ] interval image collection gather Contains a total of 2*n+1 images.

[0057] In this embodiment, the interpolation algorithm adopts a deep learning frame interpolation algorithm based on motion information and appearance features (Extracting Motion and Appearance via Inter-Frame Attention for EfficientVideo Frame Interpolation (CVPR2023) Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, Limin Wang.), and extracts the aggregated features of the appearance information of the same area at different positions in two frames and the motion features between two adjacent frames through the inter-frame attention mechanism designed by Transformer to generate the missing image between the two frames.

[0058] The interpolation multiple n is 12, such as Figure 2 As shown, the upper left picture is the image at the previous moment when the interpolation algorithm was input, and the lower left picture is the multiple interpolation images obtained; the upper right picture is the image at the next moment when the interpolation algorithm was input, and the lower right picture is the multiple interpolation images obtained. The final set Contains a total of 25 images.

[0059] Step 3: Use the image semantic segmentation model to perform image segmentation.

[0060] A codec method based on deep learning is used to i Image of the moment and the image set of adjacent frames before and after Segmentation is performed to achieve more accurate segmentation results. However, the image semantic segmentation model here is not limited to the codec method based on deep learning. Other existing image semantic segmentation models can also be used. The specific segmentation sub-steps include:

[0061] (3.1) The network structure of the segmentation model is the Encoder-Decoder structure. The encoder takes the input image img i Perform feature extraction on each label channel to obtain the feature map feature i ;

[0062] (3.2) The decoder converts the semantic feature i Upsample to the size of the source image to obtain the corresponding segmentation result map seg i ;

[0063] (3.3) Each image sample outputs a corresponding segmentation map, on which each semantic label object is drawn with a different pixel value. Get t i A collection of segmentation graphs at a given moment

[0064] In this embodiment, the SegFormer image segmentation method based on Transformer design (SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. EnzeXie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and PingLuo. NeurIPS 2021.) is used to perform t i Image of the moment and the image set of adjacent frames before and after This method is applicable to various resolutions and has a lighter decoder module, which makes the image segmentation process faster and improves the efficiency of the overall synchronization algorithm. At the same time, the source image size obtained by the decoder after upsampling is 1920×1080. Figure 3 As shown in Figure 1, the corresponding segmentation map outputted by one of the image samples is shown.

[0065] Step 4: Use the point cloud semantic segmentation model to perform point cloud segmentation.

[0066] In the field of point cloud segmentation, deep learning methods are also the main method. Deep learning methods can be divided into point-based segmentation, voxel-based segmentation, and projection-based segmentation. The present invention uses a projection-based deep learning codec method to i Time point cloud Semantic segmentation is performed. The projection-based point cloud segmentation method (SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation. (ECCV 2020) By Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.) includes a step of mapping the point cloud from three-dimensional space to two-dimensional space, and the segmentation implemented based on the two-dimensional space features has higher two-dimensional space accuracy for the point cloud and image fusion synchronization problem targeted by the present application, which is helpful for subsequent step synchronization calculation. The segmentation model also takes the Encoder-Decoder network structure as the main backbone, and the projection-based method adds a mapping module to project the point cloud to two dimensions, and adds a two-dimensional to three-dimensional mapping to return the segmentation result. The two-dimensional mapping of the point cloud data is obtained proj .

[0067] The specific point cloud segmentation process includes the following sub-steps:

[0068] (4.1) Input the point cloud i at time t into the three-dimensional space to two-dimensional space mapping module in the projection-based deep school encoder-decoder method, to obtain the corresponding two-dimensional mapping graph pcd proj ;

[0069] (4.2) Input the two-dimensional mapping graph pcd proj obtained in step (4.1) into the decoder module to extract the semantic features feature proj of the two-dimensional mapping;

[0070] (4.3) Input the semantic features feature proj into the decoder module for decoding, upsample to the size of the two-dimensional mapping graph pcd proj to obtain the segmentation graph, and distinguish the labels of the segmentation graph by pixel value;

[0071] (4.4) Input the decoded feature proj into the two-dimensional space to three-dimensional space mapping module, and when mapping each point, distinguish the semantics according to the pixel value of the segmentation graph to obtain a set of semantic labels of all points

[0072] In this embodiment, the data structure of the segmentation result is shown in Table 1, and the two-dimensional space coordinates of each point are one-to-one corresponding to the label of the category to which the point belongs. ​

[0073] Table 1 Two-dimensional space coordinates of each point and its category label

[0074]

[0075] Step 5: Use semantic point matching to obtain the 2D image at the synchronization moment.

[0076] Since the camera and lidar are fixed on the vehicle, the relative coordinates of the camera with respect to the lidar can be considered stable and unchanged, so the projection calculation coefficient matrix from the three-dimensional point cloud coordinates to the two-dimensional camera coordinates remains unchanged.

[0077] The data collection process of this embodiment is performed by a lidar scanner fixed on the roof and a camera fixed at the front view of the vehicle. The camera parameters such as two-dimensional translation, zoom, and three-dimensional translation and rotation are obtained through camera calibration.

[0078] Step 5 specifically includes the following sub-steps:

[0079] (5.1) Let the two-dimensional translation matrix of the camera be trans 2D , the two-dimensional scaling matrix is ​​scaling 2D Matrix, then the intrinsic parameter matrix of the camera is Mat iner =trans 2D ×scaling 2D ; The three-dimensional rotation matrix of the camera is rotat 3D , the three-dimensional translation column vector is trans 3D , the external parameter matrix is ​​Mat ext =[rotat 3D |trans 3D ], in order to make Mat ext Form a square matrix, add a row of (0,0,0,1) vectors at the end of the row to form a 4x4 square matrix; in order to make the 3x3 size Mat iner and Mat ext Can be multiplied, in Mat iner Add a column of zero vectors at the end of the column to form a 3x4 matrix.

[0080] (5.2) at t i moment, for point cloud According to the mapping rule from point cloud to image, the camera's internal and external parameter matrix Mat is known iner and Mat ext , we can get the two-dimensional mapping proj of the point cloud ti , proj ti It is composed of a series of points (u, v, z), where u and v represent the X-axis and Y-axis coordinates of the point on the two-dimensional image, and z represents the depth information of the point.

[0081] From step (3.3), we can see that for A collection of segmentation maps after interpolation Among them are In seg i Regarding the category label c, There is a corresponding pixel value pix c , to distinguish the colors of different semantic category targets on the segmentation map. From step (4.4), we can see that for pcd ti After completing the point cloud segmentation, the semantic label set of each point can be obtained In the two-dimensional mapping process of point cloud to image, each point is one-to-one corresponding, so for the two-dimensional point set proj mapped to the image ti Any point (x c ,y c ), we can see that its category label is c,

[0082] (5.3) For the segmentation graph seg i Mapping proj with point cloud ti , in seg i The width and height range w×h of proj ti Filter: u=0<u<w,v=0<v<h, get the same as seg i proj′ of the same size ti .

[0083] In this embodiment, w = 1920, h = 1080. In this embodiment, the number of sample point clouds before data filtering is about 200,000 to 300,000, and the number of sample point clouds after filtering is about 40,000 to 80,000.

[0084] (5.4) in proj′ ti There are N p points, traverse N p points, for any point (x c ,y c )exist The pixels of the segmentation map are judged on the top. If the mapping coordinates of the c-type points (x c ,y c ) in seg i There is (x c ,y c )=pix c The match is correct, and the matching error is match c =0, otherwise it is a matching error, match c = 1. For all i After matching the points within the range, calculate seg i The matching accuracy

[0085] (5.5) For the set of segmentation graphs All elements of proj′ ti The matching accuracy is calculated to get the highest matching accuracy Sync max , whose segmentation map is seg max , its corresponding source image img sync This is the desired synchronized image.

[0086] like Figure 4 As shown in the figure, this embodiment synchronizes and aligns the collected camera and radar data, and fuses the synchronized data at the synchronization moment, and compares it with the data fusion result of the original collection moment. One of the frames of data selected in the comparison figure is used to achieve the effect visualization. Figure 4 As can be seen, the first row shows the mapping results of the point cloud obtained by hardware clock timestamp matching and the camera source image before synchronization. Among them, ① represents the road sign point cloud obtained by the lidar, ② represents the road sign image captured by the camera, ③ represents the vehicle point cloud, ④ represents the vehicle image, ⑤ represents the street lamp point cloud, and ⑥ represents the street lamp image. As can be seen from this figure, the point clouds and corresponding images of each object do not completely overlap, and there is a large mapping error. The second row shows the mapping results of the point cloud obtained by semantic point matching and the interpolated image after synchronization. By zooming in on the local part of the two-dimensional mapping image, it can be seen that the local zooming in after synchronization greatly corrects the mapping error of the vehicle. After the synchronization process, the image coordinates of objects including vehicles, road signs and street lamps on both sides of the road are well aligned with each other.

[0087] Corresponding to the aforementioned embodiment of the method for synchronizing point cloud and image based on semantic point matching, the present invention also provides an embodiment of an apparatus for synchronizing point cloud and image based on semantic point matching.

[0088] See also Figure 5 An embodiment of the present invention provides a point cloud and image synchronization device based on semantic point matching, which includes one or more processors for implementing the point cloud and image synchronization method based on semantic point matching in the above embodiment.

[0089] The embodiment of the point cloud and image synchronization device based on semantic point matching of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5 As shown, this is a hardware structure diagram of any device with data processing capability where the point cloud and image synchronization device based on semantic point matching of the present invention is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0090] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0091] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0092] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the point cloud and image synchronization method based on semantic point matching in the above embodiment is implemented.

[0093] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (SmartMedia card, SMC), an SD card, a flash card (Flash card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0094] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0095] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for synchronizing point clouds and images based on semantic point matching, characterized in that: The method comprises the following steps: Step 1: The on-board LiDAR and camera are fixed to the vehicle to form a rigid structure and remain relatively stationary. During the vehicle's driving process, the on-board LiDAR and camera scan and shoot at the same frequency within the acquisition cycle to collect point cloud and image data of the surrounding environment. Continuous point cloud and image acquisition data are generated within the acquisition cycle based on the timestamp of the acquisition moment. Step 2: Using an image interpolation algorithm, a number of interpolated images between two adjacent frames captured by the camera are obtained, thereby forming an image set within the time period from the moment before the current moment to the moment after the current moment; Step 3: Use the trained image semantic segmentation model to perform semantic segmentation on the images in the image set obtained in step 2 to obtain a set of segmentation maps; Step 4: Use the trained point cloud semantic segmentation model to segment the point cloud dataset obtained by the vehicle-mounted lidar scan to obtain the spatial coordinates and category labels of all semantic points; Step 5: Use the semantic point matching obtained in step 4 to obtain the segmentation map of the two-dimensional image at the synchronization moment, and use the source image corresponding to the segmentation map with the highest matching accuracy as the final synchronization image; The step five includes the following sub-steps: (5.1) In the camera's external parameter matrix Mat ext Add a row of (0,0,0,1) vectors at the end of the row to form a 4x4 square matrix; in the camera's 3x3 size Mat iner Add a column of zero vectors at the end of the column to form a 3x4 matrix; where Mat ext =[rotat 3D |trans 3D ], Mat iner =trans 2D ×scaling 2D , trans 2D is the two-dimensional translation matrix of the camera, scaling 2D is the two-dimensional scaling matrix of the camera, rotat 3D is the three-dimensional rotation matrix of the camera, trans 3D is the three-dimensional translation column vector of the camera; (5.2) at t i moment, for point cloud According to the mapping rules from point cloud to image and the internal and external parameter matrix Mat of the camera iner and Mat ext , get the two-dimensional mapping proj of the point cloud ti , (5.3) For the segmentation map seg i and the point cloud mapping proj ti , within the width-height range w×h of seg i , filter proj ti : u = 0 < u < w, v = 0 < v < h, to obtain the two-dimensional mapping proj′ i of the filtered point cloud with the same size as seg ti ; (5.4) Traversal and seg i The two-dimensional mapping proj′ of the filtered point cloud of the same size ti N on p points, for any point (x c ,y c )exist The pixels of the segmentation map are judged on the top. If the mapping coordinates of the c-type points (x c ,y c ) in seg i There is (x c ,y c )=pix c The match is correct, and the matching error is match c =0, otherwise it is a matching error, match c =1; for all i After matching the points within the range, calculate seg i The matching accuracy (5.5) For the set of segmentation graphs All elements of proj′ ti The matching accuracy is calculated to get the highest matching accuracy Sync max , whose segmentation map is seg max , its corresponding source image img sync This is the desired synchronized image.

2. The method for synchronizing point cloud and image based on semantic point matching according to claim 1, characterized in that: In step 1, when collecting data, the hardware clock is used to synchronize time so that the laser radar scanning time and the camera shooting time are close to the same moment.

3. The method for synchronizing point cloud and image based on semantic point matching according to claim 1, characterized in that: The image interpolation algorithm in step 2 is a deep learning frame interpolation algorithm based on motion information and appearance features.

4. The method for synchronizing point cloud and image based on semantic point matching according to claim 3, characterized in that: The second step specifically includes: For t i Moment Image Take the adjacent previous moment image Perform interpolation, the interpolation multiple is n, and get [t i-1 ,t i ]Interval interpolation set Contains n generated interpolated images; For t i Moment Image Take the adjacent image at the next moment Perform interpolation, the interpolation multiple is n, and get [t i ,t i+1 ]Interval interpolation set Contains n generated interpolated images; For t i Moment Image From this we can get that [t i-1 ,t i+1 ] interval image collection gather Contains a total of 2*n+1 images.

5. The method for synchronizing point cloud and image based on semantic point matching according to claim 1, characterized in that: The image semantic segmentation model in step three is a codec method based on deep learning.

6. The method for synchronizing point cloud and image based on semantic point matching according to claim 5, characterized in that: The step three specifically includes the following sub-steps: (3.1) Yes The encoder takes the input image img i Perform feature extraction on each label channel to obtain the feature map feature i ; (3.2) The decoder converts the semantic feature i Upsample to the size of the source image to obtain the corresponding segmentation result map seg i ; (3.3) Each image sample outputs a corresponding segmentation map, on which each semantic label object is drawn with a different pixel value. Get t i A collection of segmentation graphs at a given moment 7. The method for synchronizing point cloud and image based on semantic point matching according to claim 1, characterized in that: The point cloud semantic segmentation model in step 4 is a projection-based deep learning codec method.

8. The method for synchronizing point cloud and image based on semantic point matching according to claim 7, characterized in that: The step 4 includes the following sub-steps: (4.1) t i Time point cloud Input the three-dimensional space to two-dimensional space mapping module in the projection-based deep school codec method to obtain the corresponding two-dimensional mapping map pcd proj ; (4.2) The two-dimensional mapping pcd obtained in step (4.1) proj Input decoder module to extract semantic features of two-dimensional mapping proj ; (4.3) Semantic features proj Input decoder module for decoding, upsampling to two-dimensional map pcd proj The size of the segmentation map is obtained, and the segmentation map is used to distinguish the labels by pixel values; (4.4) Decoded feature proj Input the mapping module from two-dimensional space to three-dimensional space. When mapping each point, the semantics are distinguished according to the pixel value of the segmentation map to obtain the semantic label set of all points.

9. A point cloud and image synchronization device based on semantic point matching, characterized in that: The method comprises one or more processors for implementing the point cloud and image synchronization method based on semantic point matching according to any one of claims 1 to 8.

Images