A point cloud and image registration method and device, electronic equipment and storage medium

Abstract

The application provides a point cloud and image registration method and device, electronic equipment and storage medium, and relates to the technical field of intelligent driving. In the application, a point cloud depth image and a camera panorama image are obtained, a first contour feature of a target object in the point cloud depth image and a second contour feature of the target object in the camera panorama image are extracted, a position mapping relationship between the point cloud depth image and the camera panorama image is determined based on a position relationship between the first contour feature and the second contour feature, pixel values of the camera panorama image are mapped into the point cloud depth image based on a coordinate mapping relationship between point cloud coordinates of the point cloud depth image and pixel coordinates of the camera panorama image and a target image is obtained. In this way, features of all target objects in the scanning point cloud depth image or the camera panorama image can be avoided, the complexity of calculation is reduced, and the rate of point cloud and image registration is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a point cloud and image registration method, apparatus, electronic device and storage medium. Background Technology

[0002] Image registration refers to the process of matching and superimposing two or more images acquired by different sensors or under different conditions (illuminance, camera position and angle).

[0003] In intelligent driving technology, radar can be used to detect 3D targets and obtain positional information such as distance, azimuth, and altitude, but it cannot obtain the target's texture features, such as color. Image sensors can obtain the target's texture features, but cannot determine the target's depth (distance). Therefore, in order to obtain an accurate target image, it is necessary to register two different data sources: point clouds and images.

[0004] Existing 2D-to-2D point cloud and image registration methods typically convert point clouds into depth or intensity images based on their height or intensity values, thus achieving image registration between two different data sources: point clouds and images. These methods often employ mutual information image registration algorithms; however, these algorithms suffer from high computational complexity and poor real-time performance. Summary of the Invention

[0005] This invention provides a point cloud and image registration method to improve the registration speed. The specific technical solution is as follows:

[0006] Firstly, this application provides a point cloud and image registration method, including:

[0007] Acquire point cloud depth images and camera panoramic images;

[0008] Extract the first contour features of the target object from the point cloud depth image and the second contour features of the target object from the camera panoramic image;

[0009] Based on the positional relationship between the first contour feature and the second contour feature, the positional mapping relationship between the point cloud depth image and the camera panoramic image is determined;

[0010] Based on the position mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, the pixel values ​​of the camera panoramic image are mapped to the point cloud depth image to obtain the target image.

[0011] Based on the above method, the depth point cloud image and the camera panoramic image are registered according to the obvious contour features of the edge information of the target object in the road detected by the edge detection algorithm. This avoids global scanning of the features of the target object in the point cloud depth image or the camera panoramic image, reduces the computational complexity, and improves the registration speed of the point cloud depth image and the camera panoramic image.

[0012] In one possible implementation, acquiring the panoramic image from the camera includes:

[0013] Acquire a first image and a second image captured by an image acquisition device, wherein the first image and the second image are captured at the same camera moment from different shooting angles or at two adjacent camera moments;

[0014] Select a common target interest point from the first interest point set corresponding to the first image and the second interest point set corresponding to the second image;

[0015] Based on the first location information of the target interest point in the first image and the second location information of the target interest point in the second image, the positional mapping relationship between the first image and the second image is determined;

[0016] Based on the location mapping relationship, the panoramic image of the camera is generated.

[0017] Based on the above method, it is possible to extract interest points in the first and second images that do not change with rotation, scaling, or brightness, i.e., interest points with feature invariance. According to the positional mapping relationship of the target interest points in the first and second images, a camera panoramic image containing various target objects can be obtained, making the information of the target objects contained in the camera panoramic image richer.

[0018] In one possible implementation, prior to acquiring the point cloud depth image, the process further includes:

[0019] Acquire a first point cloud set and a second point cloud set collected by a radar detection device, wherein the first point cloud set and the second point cloud set are obtained by scanning from different scanning angles at the same laser moment or at two adjacent laser moments;

[0020] Select a candidate point cloud subset from the first point cloud set, and determine the target point cloud subset in the second point cloud set that corresponds to the candidate point cloud subset;

[0021] Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix between the candidate point cloud subset and the target point cloud subset is constructed;

[0022] Based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position of the target point cloud subset;

[0023] Calculate the positional deviation between the candidate point cloud subset and the target point cloud subset;

[0024] Determine whether the position deviation value is greater than a set threshold;

[0025] If so, the transformation matrix is ​​iteratively updated until the position deviation value is less than or equal to the threshold, or the number of iterations is equal to the set maximum number of iterations;

[0026] If not, then a complete set of scenic spots cloud data is generated based on the transformation matrix.

[0027] Based on the above method, it is possible to quickly synthesize a panoramic view cloud set from the point cloud data sets scanned by the radar detection equipment from various angles or at various laser moments, thereby improving the rate of acquiring a panoramic view cloud set corresponding to the scene of the camera panoramic image.

[0028] In one possible implementation, acquiring the point cloud depth image includes:

[0029] Select each candidate panoramic cloud located in the set of panoramic cloud spaces from the panoramic cloud set, and take each candidate panoramic cloud as the target panoramic cloud subset;

[0030] The target panoramic point cloud subset is mapped to the pixel space corresponding to the point cloud space, and the coordinate mapping relationship between the point cloud space and the pixel space is recorded.

[0031] Determine the vertical coordinates of each panoramic cloud in the target panoramic cloud subset;

[0032] Calculate the average of the sums of all vertical coordinates to obtain the elevation value, and use the elevation value as the depth value of the point cloud depth image to generate the point cloud depth image.

[0033] Based on the above method, by converting the 3D panoramic view cloud collection into a point cloud depth image, it is easier to register the point cloud depth image and the camera panoramic image in the future.

[0034] Secondly, this application provides a point cloud and image registration apparatus, comprising:

[0035] The data acquisition module is used to acquire point cloud depth images and camera panoramic images;

[0036] The feature extraction module is used to extract the first contour features of the target object in the point cloud depth image and the second contour features of the target object in the camera panoramic image;

[0037] The mapping module is used to determine the positional mapping relationship between the point cloud depth image and the camera panoramic image based on the positional relationship between the first contour feature and the second contour feature;

[0038] The point cloud and image registration module is used to map the pixel values ​​of the camera panoramic image to the point cloud depth image based on the position mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, so as to obtain the target image.

[0039] In one possible implementation, the data acquisition module is specifically used for:

[0040] Acquire a first image and a second image captured by an image acquisition device, wherein the first image and the second image are captured at the same camera moment from different shooting angles or at two adjacent camera moments;

[0041] Select a common target interest point from the first interest point set corresponding to the first image and the second interest point set corresponding to the second image;

[0042] Based on the first location information of the target interest point in the first image and the second location information of the target interest point in the second image, the positional mapping relationship between the first image and the second image is determined;

[0043] Based on the location mapping relationship, the panoramic image of the camera is generated.

[0044] In one possible implementation, the data acquisition module is further configured to:

[0045] Acquire a first point cloud set and a second point cloud set collected by a radar detection device, wherein the first point cloud set and the second point cloud set are obtained by scanning from different scanning angles at the same laser moment or at two adjacent laser moments;

[0046] Select a candidate point cloud subset from the first point cloud set, and determine the target point cloud subset in the second point cloud set that corresponds to the candidate point cloud subset;

[0047] Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix between the candidate point cloud subset and the target point cloud subset is constructed;

[0048] Based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position of the target point cloud subset;

[0049] Calculate the positional deviation between the candidate point cloud subset and the target point cloud subset;

[0050] Determine whether the position deviation value is greater than a set threshold;

[0051] If so, the transformation matrix is ​​iteratively updated until the position deviation value is less than or equal to the threshold, or the number of iterations is equal to the set maximum number of iterations;

[0052] If not, then a complete set of scenic spots cloud data is generated based on the transformation matrix.

[0053] In one possible implementation, the data acquisition module is specifically used for:

[0054] Select each candidate panoramic cloud located in the set of panoramic cloud spaces from the panoramic cloud set, and take each candidate panoramic cloud as the target panoramic cloud subset;

[0055] The target panoramic point cloud subset is mapped to the pixel space corresponding to the point cloud space, and the coordinate mapping relationship between the point cloud space and the pixel space is recorded.

[0056] Determine the vertical coordinates of each panoramic cloud in the target panoramic cloud subset;

[0057] Calculate the average of the sums of all vertical coordinates to obtain the elevation value, and use the elevation value as the depth value of the point cloud depth image to generate the point cloud depth image.

[0058] Thirdly, this application provides an electronic device, comprising:

[0059] Memory, used to store computer programs;

[0060] When the processor executes the computer program stored in the memory, it implements the steps of the point cloud and image registration method described above.

[0061] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the point cloud and image registration method described above.

[0062] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0063] Figure 1 A flowchart of a point cloud and image registration method provided in this application;

[0064] Figure 2A schematic diagram of the point cloud and image registration system architecture provided for this application;

[0065] Figure 3 This is a schematic diagram showing the location of the camera provided in this application;

[0066] Figure 4 The location diagrams of the first and second candidate panoramic view clouds provided for this application;

[0067] Figure 5 A schematic diagram illustrating the coordinate mapping between point cloud space and pixel space provided in this application;

[0068] Figure 6 A schematic diagram of a point cloud and image registration device provided in this application;

[0069] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0071] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0072] Image registration refers to the process of matching and superimposing two or more images acquired by different sensors or under different conditions (illuminance, camera position and angle).

[0073] In intelligent driving technology, radar can be used to detect 3D targets and obtain positional information such as distance, azimuth, and altitude, but it cannot obtain the target's texture features, such as color. Image sensors can obtain the target's texture features, but cannot determine the target's depth (distance). Therefore, in order to obtain an accurate target image, it is necessary to register two different data sources: point clouds and images.

[0074] Existing 2D-to-2D point cloud and image registration methods typically convert point clouds into depth or intensity images based on their height or intensity values, thus achieving image registration between two different data sources: point clouds and images. These methods often employ mutual information image registration algorithms; however, these algorithms suffer from high computational complexity and poor real-time performance.

[0075] Therefore, in order to achieve image registration between two different data sources, point clouds and images, reduce computational complexity, and improve registration speed, this application provides a point cloud and image registration method, specifically including: first, acquiring a point cloud depth image and a panoramic image from a camera; then, extracting the first contour feature of the target object from the point cloud depth image and the second contour feature of the target object from the panoramic image; then, determining the positional mapping relationship between the point cloud depth image and the panoramic image based on the positional mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the panoramic image; finally, mapping the pixel values ​​of the panoramic image to the point cloud depth image based on the positional mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the panoramic image, obtaining the target image.

[0076] The above method allows us to first acquire a point cloud depth image transformed from a point cloud coordinate system to a pixel coordinate system. Then, by extracting the first contour features of the target object from the point cloud depth image and the second contour features of the target object from the camera panoramic image, we can avoid global scanning of features in both the point cloud depth image and the camera panoramic image, thus reducing computational complexity. Finally, by using the positional relationship between the first and second contour features and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, we can map the pixel values ​​of the camera panoramic image to the point cloud depth image to obtain the target image, thereby improving the speed of point cloud and image registration.

[0077] Reference Figure 1 The diagram shown is a flowchart of a point cloud and image registration method provided in an embodiment of this application. The method includes:

[0078] S1 acquires point cloud depth images and camera panoramic images.

[0079] Firstly, the method provided in this application can be applied to Figure 2 The system architecture shown includes: a vehicle, image acquisition equipment, radar detection equipment, and image processing equipment.

[0080] This application does not impose a specific limit on the number of the aforementioned devices, and the aforementioned devices can be physically deployed in vehicles. The method provided in this application can be run in an image processing device. The following is a brief introduction to the aforementioned devices and their respective functions.

[0081] Image acquisition devices are a type of physical sensor with environmental perception capabilities, capable of acquiring the texture features of target objects in real time. Target objects can be lane lines, streetlights, vehicles, pedestrians, or any entity in the scene. Image acquisition devices include any type of vehicle-mounted camera, such as front-view cameras, rear-view cameras, side-view cameras, surround-view cameras, or built-in cameras. Image acquisition devices are used to acquire different image data from multiple perspectives of the vehicle in real time. It should be noted that at the same camera moment, there is an overlap between two images acquired from two adjacent perspectives, which facilitates the later synthesis of panoramic images from multiple perspectives.

[0082] Radar detection equipment is another type of physical sensor with environmental perception capabilities, capable of acquiring positional information such as distance and height of target objects. Radar detection equipment can be any one of Laser Radar, Millimeter-Wave Radar, Ultrasound, or a combination thereof, and this application does not impose specific limitations on this. Radar detection equipment can also collect different point cloud data from multiple angles of the vehicle in real time at the same laser moment, so as to synthesize the point cloud data collected from multiple perspectives into a global point cloud set in the later stage.

[0083] The image processing equipment is used to receive image data acquired by the image acquisition equipment and point cloud data acquired by the radar detection equipment, and to process the two different data sources according to the set data processing rules.

[0084] The vehicle can be any type of vehicle with autonomous driving capabilities. In addition to the aforementioned equipment, the vehicle is also equipped with a positioning system, a control system, etc. The positioning system can be an inertial measurement unit (IMU), real-time kinematic (RTK) technology, or a global positioning system (GPS). This application does not impose any specific restrictions on this.

[0085] In this application embodiment, a single camera can only capture scene images within a certain field of view. That is, the images captured by a single camera have certain limitations. In order to obtain a panoramic image that includes various target objects from all directions, it is necessary to install corresponding cameras at multiple angles of the vehicle, for example, one camera at every 90° interval. Then, the image processing device can receive the image data captured by each camera through wireless or wired transmission. This application does not impose specific restrictions on the communication method between the image acquisition device and the image processing device.

[0086] The image processing device receives the first and second images acquired by the image acquisition device, such as... Figure 3 As shown, with the driver's position as the origin of the coordinate system, the first image captured by camera 1 in the positive x-axis direction and the second image captured by camera 2 in the positive y-axis direction are shown.

[0087] After receiving the first image and the second image, the image processing device needs to extract interest points (feature points) in the first and second images that do not change with factors such as rotation, scaling, and brightness variations. In this embodiment, the Scale Invariant Feature Transform (SIFT) algorithm can be used to extract feature points, as follows:

[0088] Step 1, Scale-space extremum detection: Search all images in the Gaussian pyramid scale space and identify potential scale- and rotation-invariant feature points using the difference of Gaussians function.

[0089] Step 2, Feature point localization: At each candidate feature point location, a refined model is simulated to determine the feature point location and scale.

[0090] Step 3, feature point orientation determination: Based on the local gradient direction of the image, each feature point is assigned one or more orientations. All subsequent processing of image data is transformed relative to the orientation, scale, and position of the feature points, thereby ensuring the invariance of the features.

[0091] Step 4, Feature Point Description: Within the neighborhood of each feature point, measure the local gradient of the image at a selected scale. These gradients serve as descriptors for the feature points, allowing for relatively large local deformations and illumination intensities.

[0092] It should be noted here that scale space refers to the set of scales formed by repeatedly applying Gaussian filtering to any image, resulting in images of different scales.

[0093] After extracting points of interest from a first image and a second image, the image processing device can select a common target point of interest from the first set of target points corresponding to the first image and the second set of target points corresponding to the second image. The target point of interest can be a lane line, green light, vehicle, or pedestrian in the image; this application does not impose specific limitations on this. The target point of interest can be represented by a 128*1 feature matrix, which can be used to describe the color, dimension, size, position, etc. of the target point of interest.

[0094] Based on the first position information of the target interest point in the first image and the second position information of the target interest point in the second image, the image processing device can determine the positional mapping relationship between the first image and the second image. This positional mapping relationship can be described by a rotation and translation matrix. According to the rotation and translation matrix, the first image can be transformed to the corresponding position in the second image, or the second image can be transformed to the position in the first image. Finally, the first image and the translated first image are added together, and the overlapping part between the first image and the translated first image is removed. This yields a camera image synthesized from two images (the first image and the second image) taken from two shooting angles at the same camera moment.

[0095] In one possible implementation, following the steps described above, it is also possible to synthesize all images acquired from multiple angles at the same time to obtain a camera panorama at the same camera moment, or to synthesize two images taken at two adjacent camera moments to obtain a camera panorama image.

[0096] By using the SIFT algorithm, interest points that do not change with rotation, scaling, or brightness in the first and second images can be extracted, i.e., interest points with feature invariance, thus improving the feature extraction rate. Based on the positional mapping relationship of the target interest points in the first and second images, a camera panoramic image containing various target objects can be obtained, making the information of the target objects contained in the camera panoramic image richer.

[0097] In this embodiment of the application, after the image processing device acquires the panoramic image from the camera, in order to ensure that the scene of the point cloud data collected by the radar detection device is consistent with the scene of the panoramic image from the camera, it is necessary to synthesize a set of panoramic point cloud data corresponding to the scene of the panoramic image from the camera.

[0098] Specifically, the image processing device can use the Iterative ClosestPoint (ICP) algorithm to synthesize a panorama cloud set. The specific steps are as follows:

[0099] The image processing device first acquires the first point cloud set and the second point cloud set collected by the radar detection device. The first point cloud set and the second point cloud set can be obtained by the radar detection device from different scanning angles or from two adjacent laser moments at the same laser moment. The radar detection device can be arranged with reference to the above-mentioned camera setup in the vehicle, which will not be elaborated here.

[0100] Select a subset P of candidate point clouds from the first set of point clouds M. For example, P = {P1, P2, ..., P...} n}

[0101] Find the target point cloud subset Q from the second point cloud set N that corresponds to the candidate point cloud subset P. For example, Q = {Q1, Q2, ..., Q...} n In the target point cloud subset, the target point cloud Q1 can be the point cloud that is closest to the candidate point cloud P1 in the candidate point cloud subset. The other target point clouds in the target point cloud subset are set in the same way as described above, and will not be repeated here.

[0102] Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix is ​​constructed between the candidate point cloud subset P and the target point cloud subset Q; and based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position Q of the target point cloud subset. * For example, Q * =R×P+T, where R is the rotation matrix and T is the translation matrix.

[0103] Calculate the positional deviation between the candidate point cloud subset P and the target point cloud subset Q. The positional deviation can be calculated in the following way:

[0104]

[0105] Determine whether the positional deviation between the candidate point cloud subset P and the target point cloud subset Q is greater than a set threshold. If so, iteratively update the transformation matrix until the positional deviation is less than or equal to the set threshold, or the number of iterations of the transformation matrix is ​​equal to the set maximum number of iterations; if not, based on the transformation matrix, generate a panoramic point cloud set synthesized from the first point cloud set and the second point cloud set at the same laser moment.

[0106] Similar to the steps described above for generating a panoramic image from a camera, after generating a panoramic view cloud set synthesized from the first and second point cloud sets at the same laser moment, the image processing device can also synthesize all point cloud sets from multiple angles collected at the same laser moment to obtain a panoramic view cloud set at the same laser moment, or synthesize the third and fourth point cloud sets scanned at two adjacent laser moments to obtain a panoramic view cloud set corresponding to the scene of the panoramic image from the camera.

[0107] By using the ICP algorithm, it is possible to quickly synthesize a panoramic view cloud set from the point cloud data sets scanned by the radar detection device from various angles or at various laser moments, thereby improving the speed of acquiring a panoramic view cloud set corresponding to the scene of the camera panoramic image.

[0108] After acquiring the panoramic view cloud, the image processing device first needs to convert the 3D panoramic view cloud into a point cloud depth image in order to register the obtained 3D panoramic view cloud with the 2D camera panoramic image.

[0109] In this embodiment of the application, the method for converting the entire viewpoint cloud collection into a point cloud depth image is as follows:

[0110] The image processing device first selects candidate panoramic view clouds located within a defined point cloud space from the panoramic view cloud set, and then uses each candidate panoramic view cloud as a subset of the target point cloud. For example... Figure 4 As shown, a first candidate panoramic cloud A with point cloud coordinates of (0.2m, 0.4m, 0.5m) and a second candidate panoramic cloud B with point cloud coordinates of (0.3m, 0.5m, 0.9m) are selected from the panoramic cloud set. The first candidate panoramic cloud A and the second candidate panoramic cloud are used as the target point cloud subset. It should be noted that there can be multiple candidate panoramic clouds. This application only uses the first candidate panoramic cloud A and the second candidate panoramic cloud B as examples to illustrate the inventive point of this application.

[0111] Then, the target panoramic point cloud subset is mapped to the pixel space (grid) corresponding to the point cloud space, and the coordinate mapping relationship (index relationship) between the point cloud space and the pixel space is recorded. The mapping process can be found in [reference needed]. Figure 5 As shown.

[0112] Based on the vertical coordinates of each panoramic cloud in the target panoramic cloud subset, such as the first vertical coordinate of the first candidate panoramic cloud being 0.5m and the second vertical coordinate of the second candidate panoramic cloud being 0.9m.

[0113] The elevation value is obtained by calculating the average of the sum of the vertical coordinates of all panoramic clouds in the target panoramic cloud subset. The elevation value is then used as the depth value of the point cloud depth image to generate the point cloud depth image. It should be noted that the average of the sum of the intensity values ​​of all panoramic clouds in the panoramic cloud subset can also be used as the depth value of the point cloud depth image. The intensity value refers to the laser energy information obtained from the reflection back from the surface of the target object. This application does not impose specific restrictions on the type of depth value of the point cloud depth image.

[0114] It should be noted that if there is no candidate panoramic cloud in a certain pixel space, then the empty pixel corresponding to that pixel space will be removed.

[0115] By using the above method, the 3D panoramic view cloud collection can be converted into a point cloud depth image, which facilitates the subsequent registration of the point cloud depth image and the camera panoramic image.

[0116] S2 extracts the first contour features of the target object in the point cloud depth image and the second contour features of the target object in the camera panoramic image.

[0117] In this embodiment, after acquiring a point cloud depth image and a camera panoramic image, the image detection device can use an edge detection algorithm to extract the contour features of each candidate target object in the point cloud depth image and the contour features of each candidate target object in the camera panoramic image. Then, feature matching is performed on the contour features in the point cloud depth image and the contour features in the camera panoramic image to determine a common target object in the point cloud depth image and the camera panoramic image. The first contour feature of the target object in the point cloud depth image and the second contour feature of the target object in the camera panoramic image are obtained. The first contour feature and the second contour feature can be the structural information of the edge of the lane, such as the curb, boundary line or lane line. This application does not make specific limitations on this.

[0118] By extracting the first contour features of the target object in the point cloud depth image and the second contour features of the target object in the camera panoramic image using the edge detection algorithm, the global scanning of the target object features in the point cloud depth image or the camera panoramic image is avoided, which reduces the computational complexity and improves the registration speed of the point cloud depth image and the camera panoramic image.

[0119] S3, Based on the positional relationship between the first contour feature and the second contour feature, determine the positional mapping relationship between the point cloud depth image and the camera panoramic image;

[0120] In this embodiment of the application, after the image processing device acquires the first contour feature of the target object in the point cloud depth image and the second contour feature of the target object in the camera panoramic image, it can acquire the first contour position information corresponding to the first contour feature and the second contour position information corresponding to the second contour feature. Based on the first contour position information and the second contour position information, the image processing device can establish a positional mapping relationship between the point cloud depth image and the camera panoramic image. This positional mapping relationship can be a rotation and translation matrix between the point cloud depth image and the camera panoramic image. Its function is the same as the positional mapping relationship (rotation and translation matrix) between the first image and the second image mentioned above, and will not be described again here.

[0121] By constructing a rotation and translation matrix between the point cloud depth image and the camera panoramic image, the point cloud depth image can be transformed to the corresponding position in the camera panoramic image, thus achieving position registration between the point cloud depth image and the camera panoramic image.

[0122] S4. Based on the position mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, the pixel values ​​of the camera panoramic image are mapped to the point cloud depth image to obtain the target image.

[0123] In this embodiment, the image processing device, based on the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, can first transform the point cloud depth image to the corresponding position in the camera panoramic image. After registering the positions of the point cloud depth image and the camera panoramic image, it can, based on the coordinate mapping (indexing) relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, reverse the indexing of all pixel values ​​in the camera panoramic image, so that the pixel values ​​of the camera panoramic image are mapped to the point cloud depth image, generating a color point cloud, and using the color point cloud as the target image after the point cloud and image registration.

[0124] It should be noted here that the coordinate mapping (indexing) relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image has been recorded when mapping the target panoramic point cloud subset to the corresponding pixel space (grid) of the point cloud space.

[0125] In summary, the point cloud and matching method provided in this application, through the SIFT algorithm, can extract feature-invariant interest points from the first and second images. Based on the positional mapping relationship of the target interest points in the first and second images, it can register images acquired from various angles or adjacent camera moments to synthesize a camera panoramic image containing various target objects, thus enriching the information of target objects contained in the camera panoramic image. Through the ICP algorithm, it can quickly register various point cloud data sets scanned by radar detection equipment from various angles or adjacent laser moments to synthesize a panoramic point cloud set corresponding to the scene of the camera panoramic image. Then, through elevation mapping, the panoramic point cloud set is converted into a point cloud depth image, realizing the unification of multiple LiDAR and image spaces. Finally, through the edge detection algorithm, the depth point cloud image and the camera panoramic image are registered based on the obvious contour features of the edge information of the target objects in the detected road, avoiding global scanning of the features of the target objects in the point cloud depth image or the camera panoramic image, reducing computational complexity, and improving the registration speed of the point cloud depth image and the camera panoramic image.

[0126] Based on the methods provided in the above embodiments, this application also provides a point cloud and image registration device, such as... Figure 6 The diagram shown is a schematic representation of a point cloud and image registration device according to an embodiment of this application. The device includes:

[0127] Data acquisition module 601 is used to acquire point cloud depth images and camera panoramic images;

[0128] Feature extraction module 602 is used to extract the first contour features of the target object in the point cloud depth image and the second contour features of the target object in the camera panoramic image;

[0129] The mapping module 603 is used to determine the positional mapping relationship between the point cloud depth image and the camera panoramic image based on the positional relationship between the first contour feature and the second contour feature;

[0130] The point cloud and image registration module 604 is used to map the pixel values ​​of the camera panoramic image to the point cloud depth image based on the position mapping relationship and the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, so as to obtain the target image.

[0131] In one possible implementation, the data acquisition module 601 is specifically used for:

[0132] Acquire a first image and a second image captured by an image acquisition device, wherein the first image and the second image are captured at the same camera moment from different shooting angles or at two adjacent camera moments;

[0133] Select a common target interest point from the first interest point set corresponding to the first image and the second interest point set corresponding to the second image;

[0134] Based on the first location information of the target interest point in the first image and the second location information of the target interest point in the second image, the positional mapping relationship between the first image and the second image is determined;

[0135] Based on the location mapping relationship, the panoramic image of the camera is generated.

[0136] In one possible implementation, the data acquisition module 601 is further configured to:

[0137] Acquire a first point cloud set and a second point cloud set collected by a radar detection device, wherein the first point cloud set and the second point cloud set are obtained by scanning from different scanning angles at the same laser moment or at two adjacent laser moments;

[0138] Select a candidate point cloud subset from the first point cloud set, and determine the target point cloud subset in the second point cloud set that corresponds to the candidate point cloud subset;

[0139] Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix between the candidate point cloud subset and the target point cloud subset is constructed;

[0140] Based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position of the target point cloud subset;

[0141] Calculate the positional deviation between the candidate point cloud subset and the target point cloud subset;

[0142] Determine whether the position deviation value is greater than a set threshold;

[0143] If so, the transformation matrix is ​​iteratively updated until the position deviation value is less than or equal to the threshold, or the number of iterations is equal to the set maximum number of iterations;

[0144] If not, then a complete set of scenic spots cloud data is generated based on the transformation matrix.

[0145] In one possible implementation, the data acquisition module 601 is specifically used for:

[0146] Select each candidate panoramic cloud located in the set of panoramic cloud spaces from the panoramic cloud set, and take each candidate panoramic cloud as the target panoramic cloud subset;

[0147] The target panoramic point cloud subset is mapped to the pixel space corresponding to the point cloud space, and the coordinate mapping relationship between the point cloud space and the pixel space is recorded.

[0148] Determine the vertical coordinates of each panoramic cloud in the target panoramic cloud subset;

[0149] Calculate the average of the sums of all vertical coordinates to obtain the elevation value, and use the elevation value as the depth value of the point cloud depth image to generate the point cloud depth image.

[0150] Based on the same inventive concept, this application also provides an electronic device that can realize the functions of the aforementioned point cloud and image registration device. (Refer to...) Figure 7 The electronic device includes:

[0151] At least one processor 701 and a memory 702 connected to at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 The example shown is the connection between processor 701 and memory 702 via bus 700. Bus 700 is... Figure 7 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 700 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 7 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller; there is no restriction on the name.

[0152] In this embodiment, memory 702 stores instructions executable by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can perform the point cloud and image registration method discussed above. Processor 401 can implement... Figure 6 The functions of each module in the device shown.

[0153] The processor 701 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 702 and calling data stored in memory 702, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0154] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0155] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the point cloud and image registration method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0156] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 702 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0157] By designing and programming the processor 701, the code corresponding to the point cloud and image registration methods described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute these methods during operation. Figure 1 The steps of the point cloud and image registration method in the illustrated embodiment are described below. How to design and program the processor 701 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0158] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the point cloud and image registration method described above.

[0159] In some possible implementations, various aspects of the point cloud and image registration method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the point cloud and image registration method according to the various exemplary embodiments of this application described above.

[0160] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A point cloud and image registration method, characterized in that, The method includes: Acquire point cloud depth images and camera panoramic images; Extract the first contour features of the target object from the point cloud depth image and the second contour features of the target object from the camera panoramic image; Based on the positional relationship between the first contour feature and the second contour feature, the positional mapping relationship between the point cloud depth image and the camera panoramic image is determined; Based on the position mapping relationship, the point cloud depth image and the camera panoramic image are positionally registered. Based on the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, the pixel values ​​in the camera panoramic image are reversed and assigned, so that the pixel values ​​of the camera panoramic image are mapped to the point cloud depth image to generate a color point cloud. The color point cloud is then used as the target image after the point cloud and image are registered.

2. The method as described in claim 1, characterized in that, The acquisition of panoramic images from the camera includes: Acquire a first image and a second image captured by an image acquisition device, wherein the first image and the second image are captured at the same camera moment from different shooting angles or at two adjacent camera moments; Select a common target interest point from the first interest point set corresponding to the first image and the second interest point set corresponding to the second image; Based on the first location information of the target interest point in the first image and the second location information of the target interest point in the second image, the positional mapping relationship between the first image and the second image is determined; Based on the location mapping relationship, the panoramic image of the camera is generated.

3. The method as described in claim 1, characterized in that, Before acquiring the point cloud depth image, the process also includes: Acquire a first point cloud set and a second point cloud set collected by a radar detection device, wherein the first point cloud set and the second point cloud set are obtained by scanning from different scanning angles at the same laser moment or at two adjacent laser moments; Select a candidate point cloud subset from the first point cloud set, and determine the target point cloud subset in the second point cloud set that corresponds to the candidate point cloud subset; Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix between the candidate point cloud subset and the target point cloud subset is constructed; Based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position of the target point cloud subset; Calculate the positional deviation between the candidate point cloud subset and the target point cloud subset; Determine whether the position deviation value is greater than a set threshold; If so, the transformation matrix is ​​iteratively updated until the position deviation value is less than or equal to the threshold, or the number of iterations is equal to the set maximum number of iterations; If not, then a complete set of scenic spots cloud collections is generated based on the transformation matrix.

4. The method as described in claim 3, characterized in that, The acquisition of the point cloud depth image includes: Select each candidate panoramic cloud located in the set of panoramic cloud spaces from the panoramic cloud set, and take each candidate panoramic cloud as the target panoramic cloud subset; The target panoramic point cloud subset is mapped to the pixel space corresponding to the point cloud space, and the coordinate mapping relationship between the point cloud space and the pixel space is recorded. Determine the vertical coordinates of each panoramic cloud in the target panoramic cloud subset; Calculate the average of the sums of all vertical coordinates to obtain the elevation value, and use the elevation value as the depth value of the point cloud depth image to generate the point cloud depth image.

5. A point cloud and image registration device, characterized in that, include: The data acquisition module is used to acquire point cloud depth images and camera panoramic images; The feature extraction module is used to extract the first contour features of the target object in the point cloud depth image and the second contour features of the target object in the camera panoramic image; The mapping module is used to determine the positional mapping relationship between the point cloud depth image and the camera panoramic image based on the positional relationship between the first contour feature and the second contour feature; The point cloud and image registration module is used to perform position registration between the point cloud depth image and the camera panoramic image based on the position mapping relationship, and to reverse the pixel values ​​in the camera panoramic image based on the coordinate mapping relationship between the point cloud coordinates of the point cloud depth image and the pixel coordinates of the camera panoramic image, so that the pixel values ​​of the camera panoramic image are mapped to the point cloud depth image to generate a color point cloud, and to use the color point cloud as the target image after point cloud and image registration.

6. The apparatus as claimed in claim 5, characterized in that, The data acquisition module is specifically used for: Acquire a first image and a second image captured by an image acquisition device, wherein the first image and the second image are captured at the same camera moment from different shooting angles or at two adjacent camera moments; Select a common target interest point from the first interest point set corresponding to the first image and the second interest point set corresponding to the second image; Based on the first location information of the target interest point in the first image and the second location information of the target interest point in the second image, the positional mapping relationship between the first image and the second image is determined; Based on the location mapping relationship, the panoramic image of the camera is generated.

7. The apparatus as claimed in claim 5, characterized in that, The data acquisition module is also used for: Acquire a first point cloud set and a second point cloud set collected by a radar detection device, wherein the first point cloud set and the second point cloud set are obtained by scanning from different scanning angles at the same laser moment or at two adjacent laser moments; Select a candidate point cloud subset from the first point cloud set, and determine the target point cloud subset in the second point cloud set that corresponds to the candidate point cloud subset; Based on the positional relationship between the candidate point cloud subset and the target point cloud subset, a transformation matrix between the candidate point cloud subset and the target point cloud subset is constructed; Based on the transformation matrix, the candidate point cloud subset is transformed to the corresponding position of the target point cloud subset; Calculate the positional deviation between the candidate point cloud subset and the target point cloud subset; Determine whether the position deviation value is greater than a set threshold; If so, the transformation matrix is ​​iteratively updated until the position deviation value is less than or equal to the threshold, or the number of iterations is equal to the set maximum number of iterations; If not, then a complete set of scenic spots cloud collections is generated based on the transformation matrix.

8. The apparatus as claimed in claim 7, characterized in that, The data acquisition module is specifically used for: Select each candidate panoramic cloud located in the set of panoramic cloud spaces from the panoramic cloud set, and take each candidate panoramic cloud as the target panoramic cloud subset; The target panoramic point cloud subset is mapped to the pixel space corresponding to the point cloud space, and the coordinate mapping relationship between the point cloud space and the pixel space is recorded. Determine the vertical coordinates of each panoramic cloud in the target panoramic cloud subset; Calculate the average of the sums of all vertical coordinates to obtain the elevation value, and use the elevation value as the depth value of the point cloud depth image to generate the point cloud depth image.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-4.

Images