Image and point cloud registration method and device based on kmpe

Abstract

The application discloses a kind of KMPE-based image and point cloud registration method and equipment, comprising: obtaining target image to be registered and target point cloud to be registered, feature point extraction is carried out;The relationship model between corresponding coordinate system is established based on the feature point set extracted, and the rigid body registration optimization model of target image and target point cloud is established based on the function relationship;Based on the relationship model and the rigid body registration optimization model, the corresponding relationship is calculated, and the rigid body transformation relationship of target image and target point cloud is calculated;In the case where the deviation after rigid body transformation satisfies preset condition, the optimal rigid body transformation relationship of the target image and target point cloud registration is output.The embodiment of the application proposes the image and point cloud rigid body registration method of nuclear mean p power loss function, fully utilizes the robust characteristics of KMPE loss function to measurement noise and outlier, can effectively inhibit the influence of measurement noise and outlier on registration accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image and point cloud registration method and device based on KMPE. Background Technology

[0002] The purpose of image and 3D point cloud registration is to obtain the rigid transformation relationship between the image and the point cloud, so as to map the point cloud onto the image. It has wide applications in computer vision, robotics, virtual reality and augmented reality.

[0003] Currently, image and point cloud registration methods mainly fall into two categories: (1) Traditional methods. These methods use matching algorithms in the feature spaces of images and point clouds to estimate the rigid transformation relationship between the images and point clouds (Sattler T, Leibe B, Kobbelt L. Efficient & effective prioritized matching for large-scale image-based localization[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(9):1744-1756.). These methods are greatly affected by the initial values ​​of the rigid transformation relationship between the images and point clouds, making it difficult to obtain accurate rigid transformation relationships. (2) Deep learning-based methods. Most of these methods convert point clouds into depth images, then utilize convolutional neural networks (CNNs) to optimize the mesh-format information (Cattaneo D, Vaghi M, Ballardini AL, et al. CMRNet: camera to LiDAR-map registration [C] / / IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand: IEEE, 2019: 1283-1289.). CNNs can effectively handle cross-modal inputs and learn the correlation between depth and RGB, but if the initial rigid transformation relationship between the image and point cloud is not properly selected, the correct depth image cannot be obtained, thus reducing the accuracy of image-point cloud registration. Summary of the Invention

[0004] This application provides an image and point cloud registration method and device based on KMPE, which fully utilizes the robustness of the KMPE loss function to measurement noise and outliers to effectively suppress the influence of measurement noise and outliers on registration accuracy.

[0005] This application provides an image and point cloud registration method based on KMPE, including:

[0006] Acquire the target image and point cloud of the target to be registered;

[0007] Feature point extraction is performed on the target image and on the target point cloud.

[0008] A relationship model between the first feature point set of the extracted target point cloud and the second feature point set of the extracted target image is established based on the corresponding coordinate system, and a rigid body registration optimization model between the target image and the target point cloud is established based on the functional relationship.

[0009] Based on the relationship model and the rigid body registration optimization model, the correspondence between the first feature point set and the second feature point set is calculated, and the rigid body transformation relationship between the target image and the target point cloud is calculated.

[0010] If the deviation between the first feature point set and the second feature point set after rigid body transformation meets the preset conditions, the optimal rigid body transformation relationship for the registration of the target image and the target point cloud is output.

[0011] Optionally, establishing a relationship model between the corresponding coordinate systems based on the first feature point set of the extracted target point cloud and the second feature point set of the extracted target image, and establishing a rigid body registration optimization model between the target image and the target point cloud based on the functional relationship, includes:

[0012] Determine a feature point q in the first feature point set, whose three-dimensional coordinates in the camera coordinate system satisfy:

[0013]

[0014] The homogeneous coordinates of the points mapped by the three-dimensional coordinates in the image coordinate system of the target image satisfy the following:

[0015]

[0016] Based on the homogeneous coordinates, a relationship model is determined between feature points q in the first feature point set and mapped points in the image coordinate system, satisfying:

[0017]

[0018] A rigid body registration optimization model for the target image and target point cloud is established using the KMPE loss function, satisfying the following:

[0019]

[0020] stR T R = I3, det(R) = 1

[0021] Where σ represents the kernel parameter, used to control the size of the kernel window, p represents the power parameter, used to adjust the steepness of the KMPE loss curve, {q j ,s c(j)} represents the correspondence between point cloud feature points and image feature points.

[0022] Optionally, based on the relationship model and the rigid body registration optimization model, calculating the correspondence between the first feature point set and the second feature point set, and calculating the rigid body transformation relationship between the target image and the target point cloud, includes:

[0023] Given the intrinsic parameters of the camera used for image acquisition Minimum mean square error ε between the feature point set of the image and the feature point set of the point cloud after rigid body transformation min Maximum number of iterations k for rigid body registration max The initial values ​​for the rotation matrix R0 and translation vector t0 of the rigid body transformation are set to the current iteration number k = 1.

[0024] For k-1 iterations, based on the rigid body transformation relationship R between the target image and the target point cloud... k-1 and t k-1 The calculated value is used to calculate the feature point set of the point cloud under this rigid body transformation relationship. Midpoint q j The coordinates of the mapped point in the image coordinate system satisfy:

[0025]

[0026] Establish using the k-nearest neighbor search algorithm With image feature point set The correspondence is the correspondence between the first feature point set and the second feature point set.

[0027] Based on the correspondence between the image feature point set S and the point cloud feature point set Q updated after the k-th iteration Establish the rigid transformation relationship R between the target image and the target point cloud in the k-th iteration. k and t k The optimization model satisfies:

[0028]

[0029] Optionally, the IWO optimization algorithm is used to optimize the rigid body transformation relation R. k and t k The optimization model is used to solve the problem.

[0030] Optionally, regarding the rigid body transformation relation R k and t kDuring the iterative process of solving the optimization model, the rigid body transformation R is calculated. k and t k The mean square error between the first feature point set and the second feature point set:

[0031]

[0032] If ε k ≤ε min or k≥k max If the condition is met, the iteration terminates; otherwise, the iteration count is k = k + 1.

[0033] Optionally, before extracting feature points based on the target image, the acquired target image to be registered is further subjected to grayscale processing.

[0034] Feature point extraction based on the target image is achieved using the SIFT algorithm.

[0035] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned KMPE-based image and point cloud registration method.

[0036] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned KMPE-based image and point cloud registration method.

[0037] This application proposes an image and point cloud rigid body registration method using the kernel mean p-power error (KMPE) loss function. This method fully utilizes the robustness of the KMPE loss function against measurement noise and outliers, effectively suppressing the impact of measurement noise and outliers on registration accuracy.

[0038] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1This is a basic flowchart example of the KMPE-based image and point cloud registration method in this application embodiment;

[0041] Figure 2 This is a general flowchart example of the KMPE-based image and point cloud registration method in this application embodiment. Detailed Implementation

[0042] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0043] This application addresses the rigid body registration problem between 2D images and 3D point clouds containing noise and outliers. Embodiments of this application provide a KMPE-based image and point cloud registration method. It utilizes both Scale-invariant Feature Transform (SIFT) and 3D SIFT algorithms to extract feature point sets from the image and point cloud to be registered. Based on this, a KMPE loss function robust to noise and outliers is introduced to establish an optimized rigid body registration model for the image and point cloud, effectively suppressing the impact of noise and outliers on registration accuracy. Figure 1 As shown, the registration method of this application includes the following steps:

[0044] In step S101, the target image to be registered and the target point cloud to be registered are acquired. In a specific example, the target image to be registered can be a color image, and acquiring the target image to be registered may further include performing grayscale processing on the acquired target image to be registered. For example, in some examples, the target color image to be registered is... (W and H are the length and height of the image) and the point cloud of the target to be registered. The color image I is converted to grayscale to obtain its grayscale image.

[0045] In step S102, feature points are extracted from the target image and from the target point cloud. In some embodiments, feature point extraction from the target image is implemented using the SIFT algorithm.

[0046] Specifically, based on the aforementioned example, the SIFT algorithm can be used to extract feature points from the grayscale image I1 to obtain the feature point set of the color image I. The 3D SIFT algorithm is used to extract feature points from the 3D point cloud M to obtain the feature point set of the 3D point cloud M.

[0047] In step S103, a relationship model between the corresponding coordinate systems is established based on the first feature point set of the extracted target point cloud (feature point set S of target image I) and the second feature point set of the extracted target image (feature point set Q of 3D point cloud M), and a rigid body registration optimization model between the target image and the target point cloud is established based on the functional relationship.

[0048] In step S104, based on the relationship model and the rigid body registration optimization model, the correspondence between the first feature point set and the second feature point set is calculated, and the rigid body transformation relationship between the target image and the target point cloud is calculated.

[0049] In step S105, if the deviation between the first feature point set and the second feature point set after rigid body transformation meets the preset conditions, the optimal rigid body transformation relationship of the target image and the target point cloud registration is output.

[0050] This application proposes an image and point cloud rigid body registration method using the kernel mean p-th power loss function. It fully utilizes the robustness of the KMPE loss function against measurement noise and outliers, effectively suppressing the impact of measurement noise and outliers on registration accuracy.

[0051] In some embodiments, establishing a relationship model between corresponding coordinate systems based on the first feature point set of the extracted target point cloud and the second feature point set of the extracted target image, and establishing a rigid body registration optimization model between the target image and the target point cloud based on the functional relationship includes:

[0052] For a feature point q in a feature point set S of a 3D point cloud M, determine the 3D coordinates of feature point q in the first feature point set S, which satisfy the following:

[0053]

[0054] in,

[0055]

[0056]

[0057] Let represent the rotation matrix and translation vector of the rigid body transformation of the image and point cloud, respectively. The values ​​of the roll angle φ, pitch angle θ, and yaw angle ψ are respectively φ c ∈[-180°, 180°), θ c ∈[-90°, 90°] and ψ c ∈[-180°, 180°).

[0058] The homogeneous coordinates of the points mapped by the three-dimensional coordinates in the image coordinate system of the target image satisfy the following:

[0059]

[0060] Where λ is the unknown scale factor, and K is the camera intrinsic parameter matrix. Then... It can be represented as:

[0061]

[0062] in,

[0063] Based on the homogeneous coordinates, the relationship model between feature points q in the first feature point set and the mapping points in the image coordinate system is determined. That is, according to the aforementioned formulas (2) and (3), the mapping points of feature points q in the three-dimensional point cloud M in the image coordinate system can be obtained. The functional relation g satisfies:

[0064]

[0065] To effectively suppress the impact of measurement noise and outliers on the registration accuracy of images and point clouds, this application proposes to establish a rigid body registration optimization model for target images and target point clouds using the KMPE loss function, satisfying the following:

[0066]

[0067] stR T R = I3, det(R) = 1

[0068] Where σ represents the kernel parameter, used to control the size of the kernel window, p represents the power parameter, used to adjust the steepness of the KMPE loss curve, {q j ,s c(j)} represents the correspondence between point cloud feature points and image feature points. Therefore, an image and point cloud rigid body registration optimization model based on KMPE is established, transforming the image and point cloud rigid body registration problem into a problem of solving the optimization model (formula (5)).

[0069] In some embodiments, calculating the correspondence between the first feature point set and the second feature point set, and calculating the rigid transformation relationship between the target image and the target point cloud, based on the relationship model and the rigid body registration optimization model, includes:

[0070] Given the intrinsic parameters of the camera used for image acquisition Minimum mean square error ε between the feature point set of the image and the feature point set of the point cloud after rigid body transformation min Maximum number of iterations k for rigid body registration max The initial values ​​for the rotation matrix R0 and translation vector t0 of the rigid body transformation are set, and the current iteration number k = 1.

[0071] For k-1 iterations, based on the rigid body transformation relationship R between the target image and the target point cloud... k-1 and t k-1 The calculated value is used to calculate the feature point set of the point cloud under this rigid body transformation relationship. Midpoint q j Mapping point coordinates in the image coordinate system satisfy:

[0072]

[0073] Establish using the k-nearest neighbor search algorithm and The correspondence is the correspondence between the first feature point set and the second feature point set. That is, the correspondence between the image feature point set S and the point cloud feature point set Q.

[0074] Based on the correspondence between the image feature point set S and the point cloud feature point set Q updated after the k-th iteration Establish the rigid transformation relationship R between the target image and the target point cloud in the k-th iteration. k and t k The optimization model satisfies:

[0075]

[0076] In some embodiments, the IWO optimization algorithm is used to optimize the rigid body transformation relation R. k and t k The optimization model formula (7) is used to solve the problem, thereby updating the rigid body transformation relationship R between the image and the point cloud in the k-th iteration. k and t k .

[0077] In some embodiments, such as Figure 2 As shown, in the rigid body transformation relation R k and t k During the iterative process of solving the optimization model, the rigid body transformation R is calculated. k and t k The mean square error between the first feature point set and the second feature point set:

[0078]

[0079] If ε k ≤ε min or k≥k max If the condition is met, the iteration terminates; otherwise, the iteration count is k = k + 1, and the iteration is repeated based on the rigid body transformation relationship R between the image and the point cloud in step k-1. k-1 and t k-1The calculated value is used to calculate the feature point set of the point cloud under this rigid body transformation relationship. each point q j The coordinates of the mapped point in the image coordinate system.

[0080] Finally, the optimal rigid body transformation relationship between the output image and the point cloud registration is determined.

[0081] Output the rotation matrix R obtained in the k-th iteration. k Translation vector t k Its value is the optimal rigid body transformation relationship for image and point cloud registration.

[0082] The KMPE-based image and point cloud registration method proposed in this application has a good suppression effect on noise and outliers in images and point clouds, and can effectively improve the accuracy and robustness of rigid body registration of images and point clouds.

[0083] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned KMPE-based image and point cloud registration method.

[0084] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned KMPE-based image and point cloud registration method.

[0085] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0086] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. An image and point cloud registration method based on KMPE, characterized in that, include: Acquire the target image and point cloud of the target to be registered; Feature point extraction is performed on the target image and on the target point cloud. A relationship model between the corresponding coordinate systems is established based on the first feature point set of the extracted target point cloud and the second feature point set of the extracted target image, and a rigid body registration optimization model between the target image and the target point cloud is established based on the functional relationship. Based on the relationship model and the rigid body registration optimization model, the correspondence between the first feature point set and the second feature point set is calculated, and the rigid body transformation relationship between the target image and the target point cloud is calculated. If the deviation between the first feature point set and the second feature point set after rigid body transformation meets the preset conditions, the optimal rigid body transformation relationship for the registration of the target image and the target point cloud is output. A relationship model between corresponding coordinate systems is established based on the first feature point set of the extracted target point cloud and the second feature point set of the extracted target image. A rigid body registration optimization model between the target image and the target point cloud is then established based on this functional relationship, including: Determine the feature points in the first feature point set. q Its three-dimensional coordinates in the camera coordinate system satisfy: The homogeneous coordinates of the points mapped by the three-dimensional coordinates in the image coordinate system of the target image satisfy the following: Based on the homogeneous coordinates, the feature points in the first feature point set are determined. q The relationship model between the points mapped in the image coordinate system and the coordinate system satisfies: A rigid body registration optimization model for the target image and target point cloud is established using the KMPE loss function, satisfying the following: in, This represents kernel parameters used to control the size of the kernel window. p This represents the power parameter, used to adjust the steepness of the KMPE loss curve. This represents the correspondence between point cloud feature points and image feature points.

2. The image and point cloud registration method based on KMPE as described in claim 1, characterized in that, Based on the aforementioned relationship model and the rigid body registration optimization model, calculating the correspondence between the first feature point set and the second feature point set, and calculating the rigid body transformation relationship between the target image and the target point cloud, includes: Given the intrinsic parameters of the camera used for image acquisition Minimum mean square error between the feature point set of the image and the feature point set of the point cloud after rigid body transformation Maximum number of iterations for rigid body registration k max Initial values ​​of the rotation matrix for rigid body transformation R 0 and initial values ​​of translation vector t 0 sets the current iteration number to 0. k =1; right k -1 iteration, based on the rigid body transformation relationship between the target image and the target point cloud. R k-1 and t k-1 The calculated value is used to calculate the feature point set of the point cloud under this rigid body transformation relationship. midpoint The coordinates of the mapped point in the image coordinate system satisfy: use k Nearest neighbor search algorithm established With image feature point set The correspondence is the correspondence between the first feature point set and the second feature point set. ; According to the k Image feature point set updated after each iteration S and point cloud feature point set Q Correspondence between , establish the first k Rigid transformation relationship between target image and target point cloud in step iteration R k and t k The optimization model satisfies: 。 3. The KMPE-based image and point cloud registration method as described in claim 2, characterized in that, The IWO optimization algorithm is used to optimize rigid body transformation relationships. R k and t k The optimization model is used to solve the problem.

4. The KMPE-based image and point cloud registration method as described in claim 3, characterized in that, In the relationship of rigid body transformation R k and t k During the iterative process of solving the optimization model, the rigid body transformation is calculated. R k and t k The mean square error between the first feature point set and the second feature point set: like or If the condition is met, the iteration terminates; otherwise, the iteration count continues. k = k +1.

5. The KMPE-based image and point cloud registration method as described in claim 1, characterized in that, Before extracting feature points based on the target image, the process also includes grayscale processing of the acquired target image to be registered. Feature point extraction based on the target image is achieved using the SIFT algorithm.

6. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the KMPE-based image and point cloud registration method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the KMPE-based image and point cloud registration method as described in any one of claims 1 to 5.

Images