Point cloud image registration method, device, electronic device and storage medium

Abstract

The present invention discloses a registration method, device, electronic device and storage medium for point cloud images, the registration method for point cloud images includes: obtaining laser point cloud, image and initial image pose; extracting and matching feature points of the image; obtaining 3D points corresponding to each matching feature point based on the initial image pose; obtaining laser points corresponding to each 3D point in the laser point cloud by laser point backprojection, and determining the depth values ​​of each 3D point and the corresponding laser point respectively; performing BA optimization with the depth value of the laser point as an aid to obtain the optimized image pose; until the depth value of each 3D point and the corresponding laser point meets the preset conditions. The technical solution of the present invention improves the registration accuracy of point cloud images.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud image processing, and in particular to a point cloud image registration method, device, electronic device and storage medium. Background Art

[0002] Handheld laser scanning equipment, as an advanced surveying and mapping tool, is widely used in surveying and mapping science and technology, civil engineering, water conservancy engineering and other fields, greatly speeding up the speed of engineering operations. The high-precision point cloud results obtained by handheld laser scanning equipment enable research and applications in related fields to obtain more accurate and rich data support, but the laser point cloud only contains point cloud coordinate information and intensity information, and the color information of the laser point cloud needs to be obtained from other sensors. Among them, the camera can provide point cloud color and texture information, which can be used for point cloud visualization, classification and modeling.

[0003] However, before the camera provides accurate color information to the laser point cloud, the camera pose (i.e., image pose) needs to be registered with the laser point cloud. Currently, the color information corresponding to the point cloud is usually obtained by simultaneous measurement of laser radar and camera. However, this method is affected by the accuracy of external parameter calibration and time synchronization. Among them, the pose accuracy of some images is low, resulting in low registration accuracy of the point cloud image. Summary of the invention

[0004] The present invention provides a point cloud image registration method, aiming to improve the registration accuracy of the point cloud image.

[0005] To achieve the above object, the present invention proposes a point cloud image registration method, comprising:

[0006] Obtain laser point cloud, image and initial image pose;

[0007] Extracting and matching feature points of the images to determine matching feature points between the images;

[0008] Acquire the 3D points corresponding to the respective matching feature points based on the initial pose of the image;

[0009] Obtaining laser points corresponding to each 3D point in the laser point cloud by laser point backprojection, and determining depth values ​​of each 3D point and the corresponding laser point respectively;

[0010] BA optimization is performed with the depth value of the laser point as an aid to obtain an optimized image pose;

[0011] Based on the optimized image pose, the 3D points corresponding to each matching feature point are updated, the laser points corresponding to each 3D point are updated through laser point backprojection, the depth values ​​of each 3D point and the corresponding laser point are calculated, and it is confirmed whether the depth values ​​of each 3D point and the corresponding laser point meet the preset conditions;

[0012] If so, the latest optimized image pose is used as the final image pose result;

[0013] If not, jump to the step of performing BA optimization using the depth value of the laser point as an aid to obtain an optimized image pose.

[0014] In some embodiments, the step of extracting and matching feature points of the image includes:

[0015] Extract the feature points of each image, obtain the proximity relationship between images based on the prior position information of the images, and match each image with other images within a preset range.

[0016] In some embodiments, the step of acquiring 3D points corresponding to respective matching feature points based on the initial pose of the image includes:

[0017] For each set of matching feature points, triangulation is used to obtain the corresponding 3D points based on the initial image pose of the image to which it belongs.

[0018] In some embodiments, the step of acquiring laser points corresponding to each 3D point in the laser point cloud by laser point backprojection includes:

[0019] Search and obtain a set of neighboring laser points for each of the 3D points;

[0020] The laser points in the adjacent laser point set are sequentially projected onto the image to which the matching feature points corresponding to the corresponding 3D points belong, and the laser points corresponding to the respective matching feature points are determined.

[0021] In some embodiments, before the step of using the depth value of the laser point as an auxiliary for BA optimization, the point cloud image registration method further includes:

[0022] Confirm whether the depth value of each 3D point and the corresponding laser point meets the preset conditions;

[0023] If yes, the initial image pose is used as the final image pose result;

[0024] If not, a step of performing BA optimization with the depth value of the laser point as an aid is performed to obtain an optimized image pose.

[0025] In some embodiments, the step of using the depth value of the laser point as an aid to perform BA optimization includes:

[0026] Convert the 3D point from the world coordinate system to the camera coordinate system;

[0027] Normalize the coordinates and remove the distortion;

[0028] Convert the camera coordinate system into a pixel coordinate system to obtain a camera observation model;

[0029] Construct a reprojection error model with added depth information e=q-b+w×f(d i -d l ), where q is the pixel coordinate of the matching feature point, function b is the pixel coordinate of the 3D point obtained by back projection, and d i and d l They are the depth values ​​of the 3D point and the corresponding laser point respectively, f indicates the absolute value of the depth value difference, and w is the weight.

[0030] In some embodiments, the preset condition is that the average depth value difference between the depth value of each 3D point and the depth value of the corresponding laser point is less than a preset threshold.

[0031] The present invention also provides a point cloud image registration device, comprising:

[0032] The acquisition module is used to obtain the laser point cloud, image and initial image pose;

[0033] An extraction and matching module, used for extracting and matching feature points of the images to determine matching feature points between the images;

[0034] An estimation module, configured to obtain 3D points corresponding to respective matching feature points based on the initial image pose, and to update the 3D points corresponding to respective matching feature points based on the optimized image pose;

[0035] A back-projection module, used for obtaining laser points corresponding to each 3D point in the laser point cloud by back-projection of laser points, and updating laser points corresponding to each 3D point by back-projection of laser points;

[0036] A calculation module, used to determine the depth value of each 3D point and the corresponding laser point respectively;

[0037] An optimization module, used to perform BA optimization with the depth value of the laser point as an aid to obtain an optimized image pose;

[0038] A judgment module is used to confirm whether the depth value of each 3D point and the corresponding laser point meets the preset conditions;

[0039] The iteration module is used to use the latest optimized image pose as the final image pose result when it is confirmed that the depth value of each 3D point and the corresponding laser point meets the preset conditions, and to call the optimization module when it is confirmed that the depth value of each 3D point and the corresponding laser point does not meet the preset conditions.

[0040] The present invention also proposes an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned point cloud image registration method when executed by the processor.

[0041] The present invention further proposes a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned point cloud image registration method are implemented.

[0042] The technical solution of the registration method of the point cloud image of the present invention first obtains the laser point cloud and the initial pose of the image, extracts and matches the feature points of the image, and obtains the 3D points corresponding to the matching feature points based on the initial pose of the image and the matching feature points of the image, and then uses the back projection of the laser point cloud to obtain the laser points corresponding to each 3D point, and uses the depth value of the laser point as an auxiliary to perform BA iterative optimization, so that the depth value of the 3D point and the depth value of the corresponding laser point meet the end condition of the iterative optimization, thereby outputting an accurate image pose. By using the pre-established laser point cloud and the initial pose of the image as constraints, the image pose is optimized by BA, so that a more accurate image pose is obtained, which effectively improves the registration accuracy of the point cloud image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the flow chart of the first embodiment of the point cloud image registration method of the present invention;

[0044] Figure 2 It is a flow chart of the second embodiment of the point cloud image registration method of the present invention;

[0045] Figure 3 It is a schematic diagram of the flow chart of the third embodiment of the point cloud image registration method of the present invention;

[0046] Figure 4 It is a schematic diagram of the flow chart of the fourth embodiment of the point cloud image registration method of the present invention;

[0047] Figure 5 A schematic diagram of modules of an embodiment of a point cloud image registration device of the present invention;

[0048] Figure 6 It is a schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention provides a point cloud image registration method, which can be applied to a three-dimensional reconstruction device, such as a handheld laser scanning device. The implementation terminal of the point cloud image registration method of this embodiment can be an electronic device, which can be a laser scanning device, a desktop computer, a laptop computer, a tablet computer, a server, and other computing devices.

[0051] like Figure 1 As shown, Figure 1 It is a flow chart of the first embodiment of the point cloud image registration method of the present invention.

[0052] In this embodiment, the point cloud image registration method includes:

[0053] Step S10, obtaining laser point cloud, image and initial image pose;

[0054] Among them, the laser point cloud can be obtained by scanning the laser radar of the device, the image is taken by the camera of the device, the number of images is multiple (for example, 500, 1000), and the initial pose of the image is the initial pose of the camera (that is, the position and orientation of the camera in the world coordinate system).

[0055] Step S20, extracting and matching feature points of the images to determine matching feature points between the images;

[0056] In some embodiments, extracting and matching feature points of images may include: extracting feature points of each image, obtaining the proximity relationship between images according to the prior position information of the image, and matching each image with other images within a preset range. The extraction of feature points of the image may adopt the sift-gpu algorithm. The SIFT feature is a local feature of the image, which remains invariant to rotation, scale scaling, and brightness changes, and also maintains a certain degree of stability to changes in viewing angles, reflection changes, and noise; and the GPU version of the sift algorithm can further accelerate the extraction of feature points, thereby improving the processing speed of the feature point extraction step. The feature point matching may adopt a spatial matching method to obtain the proximity relationship between images according to the prior position information of the image, that is, according to the initial image posture of each image, determine the image adjacent to each image (for example, according to the initial image posture of the image, determine other images within the preset radius of the image coordinates), and sequentially match each image with other adjacent images. In this way, most invalid matches can be filtered out, the number of matches can be reduced, and the matching speed can be greatly improved. Among them, the matching feature points are feature points corresponding to the same object in different images. A feature point on one image may be matched from another image, or may be matched from multiple other images, that is, the number of feature points in a set of matching feature points is two or more.

[0057] Step S30, obtaining 3D points corresponding to each matching feature point based on the initial pose of the image;

[0058] After obtaining the matching feature points, the coordinates of the 3D points corresponding to each group of matching feature points can be estimated in combination with the initial image pose of the image to which the matching feature points belong; for example, for each group of matching feature points, triangulation is used to obtain the corresponding 3D points based on the initial image pose of the image to which it belongs. Among them, triangulation is the process of recovering the position of three-dimensional space points from two-dimensional feature points of two or more views in computer vision. When the initial image poses of two images are known, and the matching feature points between them are known, the positions of these matching feature points in three-dimensional space can be estimated by triangulation. In this embodiment, the matching feature points between each image have been determined through the above step S20, and the initial image pose of each image has been obtained in step S10. Therefore, the 3D points corresponding to each group of matching feature points can be restored separately by triangulation.

[0059] For example, in this embodiment, direct linear transformation can be used for triangulation to restore the 3D point coordinates corresponding to the feature points; specifically, as follows:

[0060] The camera internal parameter K, the external parameters of the initial image pose of the two matching feature points are T1 and T2, the two feature points corresponding to the 3D point are divided into P1 and P2, and the pixel coordinates , the camera matrix ,in , ;

[0061] Pixel coordinates , where X is the 3D point coordinate. By cross multiplication to eliminate the scalar factor, three equations are obtained for each matching feature point, two of which are linearly independent. For the first image, ,in ,

[0062] Expand to get in yes The first row of these components about X are linear. Similarly, three equations can be obtained for the second image; take two equations from each to get four equations with four unknown quantities, forming The equation of

[0063] pass untie ,get , that is, get the 3D point coordinates.

[0064] Step S40, obtaining laser points corresponding to each 3D point in the laser point cloud by laser point back projection, and determining depth values ​​of each 3D point and the corresponding laser point respectively;

[0065] In the case where the image and the laser point cloud are completely aligned, the depth value of the 3D point obtained in step S20 should be equal to the depth value of the corresponding laser point; therefore, after obtaining the 3D points corresponding to each matching feature point, the laser points corresponding to the 3D points are found from the laser point cloud by laser point backprojection, that is, the laser points in the laser point cloud are projected to the image to which the matching feature point belongs, and a laser point that overlaps or is closest to the projection and the matching feature point is found, and the laser point is the laser point corresponding to the 3D point corresponding to the matching feature point. Since each 3D point corresponds to a set of matching feature points, and a set of matching feature points includes at least two matching feature points, the number of laser points corresponding to a 3D point is also at least two. After obtaining the laser points corresponding to each 3D point, the depth value of each 3D point and the depth value of the corresponding laser point are calculated and determined. Among them, the depth value of the 3D point is the Z value of the 3D point in the image coordinate system, that is, the vertical distance from the 3D point to the image plane, and the depth value of the laser point is the Z value of the laser point in the image coordinate system, that is, the vertical distance from the laser point to the image plane.

[0066] Step S50, performing BA optimization with the depth value of the laser point as an aid to obtain an optimized image pose;

[0067] Among them, BA optimization, namely Bundle Adjustment optimization, is based on the principle that the light reflected from each feature point is finally converged to the optical center of the camera after optimal adjustment by adjusting the camera pose and the spatial position of the 3D point. This process involves the optimization of the poses of multiple cameras and the spatial coordinates of the landmark points under the poses; its core is to construct a cost function that describes the difference between the observed value and the value predicted by the model; the goal of the cost function is to minimize these differences (i.e., errors). In this embodiment, a pre-established laser point cloud (i.e., the laser point cloud obtained in step S10) is used as a reference, and BA optimization is performed by obtaining the real depth value (i.e., the depth value of the laser point in the laser point cloud) as an aid; that is, the error term of the depth information is added to the reprojection error model constructed by BA optimization, so as to optimize the more accurate image pose.

[0068] Step S60, based on the optimized image pose, the 3D points corresponding to the matching feature points are updated, the laser points corresponding to the 3D points are updated by laser point backprojection, the depth values ​​of the 3D points and the corresponding laser points are calculated, and it is confirmed whether the depth values ​​of the 3D points and the corresponding laser points meet the preset conditions;

[0069] After the optimized image pose is obtained through optimization, since the image pose of each image has changed, the 3D points corresponding to each matching feature point will also change. Therefore, it is necessary to redetermine the 3D points corresponding to each matching feature point based on the optimized image pose, that is, based on the optimized image pose, directly use triangulation to redetermine the 3D points corresponding to each matching feature point. Then determine the laser point corresponding to the updated 3D point (the specific method can refer to step 40, which will not be repeated here), and calculate the depth value of each 3D point and the corresponding laser point. After obtaining the depth value of each 3D point and the depth value of the laser point corresponding to each 3D point, determine whether the depth value of each 3D point and the depth value of the corresponding laser point meet the preset conditions, that is, determine whether the end condition of the iterative optimization is reached, so as to determine whether to end the iteration.

[0070] Among them, the preset condition can be: the average value of the depth difference between the depth value of all 3D points and the corresponding laser point is less than a preset threshold (for example, 5cm); the calculation method of the average value of the depth difference between the depth value of all 3D points and the corresponding laser point can be: for each 3D point, the depth value is subtracted from the corresponding laser point, and the absolute value is taken to obtain the corresponding multiple depth differences, the depth differences corresponding to all 3D points are summed, and then divided by the number of laser points corresponding to all 3D points, and the average value of the depth difference between the depth value of all 3D points and the corresponding laser point is obtained. Of course, the preset condition can also be other conditions, for example, the total number of 3D points whose depth difference with the corresponding laser point reaches the preset threshold is less than the preset number (for example, 10), or the proportion of 3D points whose depth difference with the corresponding laser point reaches the preset threshold is less than the preset percentage (for example, 2%).

[0071] Step S70: If yes, the image pose obtained by the latest optimization is used as the final image pose result;

[0072] When it is determined that the depth value of each 3D point and the depth value of the corresponding laser point meet the preset conditions, it means that the optimization result has met the requirements. At this time, the iterative optimization is terminated, and the latest optimized image pose is used as the final image pose result (i.e., camera pose), that is, the accurate image pose of each image is obtained, that is, the accurate registration of the laser point cloud and the camera image is achieved, so that a point cloud image with high registration accuracy can be obtained.

[0073] If not, jump to the step of performing BA optimization with the depth value of the laser point as an aid to obtain the optimized image pose.

[0074] If it is determined that the depth value of each 3D point and the depth value of the corresponding laser point do not meet the preset condition, jump to execute the above step S50 and continue iterative optimization until the depth value of each 3D point and the depth value of the corresponding laser point meet the preset condition.

[0075] The technical solution of the registration method of the point cloud image of this embodiment first obtains the laser point cloud and the initial pose of the image, extracts and matches the feature points of the image, and obtains the 3D points corresponding to the matching feature points based on the initial pose of the image and the matching feature points of the image, and then uses the back projection of the laser point cloud to obtain the laser points corresponding to each 3D point, and uses the depth value of the laser point as an auxiliary to perform BA iterative optimization, so that the depth value of the 3D point and the depth value of the corresponding laser point meet the end condition of the iterative optimization, thereby outputting an accurate image pose. By using the pre-established laser point cloud and the initial pose of the image as constraints, the image pose is optimized by BA, so that a more accurate image pose is obtained, which effectively improves the registration accuracy of the point cloud image.

[0076] like Figure 2 As shown, Figure 2It is a flow chart of the second embodiment of the point cloud image registration method of the present invention.

[0077] In this embodiment, in the above step S40, the step of obtaining the laser points corresponding to each 3D point in the laser point cloud by laser point back projection includes:

[0078] Step S41, searching and acquiring a set of neighboring laser points of each 3D point;

[0079] The search for a set of neighboring laser points of a 3D point may be performed by searching and filtering from a laser point cloud based on preset conditions. For example, a set of neighboring laser points of a 3D point may be a set of all laser points within a preset radius centered on the 3D point, or a set of a preset number of laser points closest to the 3D point. In other embodiments, it may also be a set of laser points that meet other preset conditions.

[0080] Step S42, projecting the laser points in the set of adjacent laser points sequentially to the image to which the matching feature points corresponding to the corresponding 3D points belong, and determining the laser points corresponding to the respective matching feature points.

[0081] Assume that a 3D point P corresponds to feature points of multiple images (i.e. a set of matching feature points) , search for laser points within a 20cm range centered on the 3D point to obtain a set of neighboring laser points: Taking image 1 as an example, the rotation matrix and translation components of image 1 are , the camera's intrinsic parameter matrix is ​​K. The process of reprojecting the laser point to the image is as follows:

[0082] First, transform the laser point from the world coordinate system to the camera coordinate system. This can be done by applying a rotation matrix and a translation vector: Next, Projection from the camera coordinate system to the image plane is done by normalizing the coordinates: Finally, the internal parameter matrix K is used to convert the normalized coordinates into pixel coordinates .

[0083] Project the laser points in the adjacent laser point set onto image 1 in sequence to obtain feature points The corresponding laser points can also get the feature points Corresponding depth value. Follow the above process for other images to get all the laser points and their depth values ​​corresponding to a 3D point; similarly, get all the laser points corresponding to all 3D points.

[0084] The technical solution of this embodiment is to filter out the adjacent laser point set of each 3D point by setting conditional search, that is, to filter out the laser points in the laser power source that are not related to the 3D point, and then back-project the laser points in the adjacent laser point set to the image, so as to find out the laser points corresponding to each matching point, that is, the laser points corresponding to the 3D point. Since only the laser points in the adjacent laser point set obtained by the search need to be projected to the image, the number of laser points processed by the back-projection is greatly reduced, and the processing speed of obtaining the laser points corresponding to the 3D point is effectively improved, that is, the efficiency is improved.

[0085] like Figure 3 As shown, Figure 3 It is a flow chart of the third embodiment of the point cloud image registration method of the present invention.

[0086] In this embodiment, before the step of using the depth value of the laser point as an auxiliary for BA optimization, the point cloud image registration method further includes:

[0087] Step S80, confirming whether the depth value of each 3D point and the corresponding laser point meets the preset conditions;

[0088] In special cases, the accuracy of the initial image pose may be very high, and optimization may not be necessary. If optimization is not required, the depth value of each 3D point and the depth value of the corresponding laser point can meet the preset conditions. For example, the preset condition is: the average depth value difference between the depth value of each 3D point and the depth value of the corresponding laser point is less than a preset threshold (for example, 5cm), or the preset condition is: the difference between the depth value of each 3D point and the depth value of any corresponding laser point is less than a preset threshold. Therefore, before performing the BA optimization processing step, it is first determined whether optimization is required to avoid performing unnecessary optimization processing steps.

[0089] Step S90: If yes, the initial image pose is used as the final image pose result;

[0090] If the depth value of each 3D point and the depth value of the corresponding laser point meet the preset conditions, it means that the accuracy of the initial image pose is high and no optimization is required. At this time, the initial image pose is output as the final image pose result.

[0091] If not, execute the above step S50.

[0092] If it is determined that the depth value of each 3D point and the depth value of the corresponding laser point do not meet the preset condition, then the subsequent step S50 is continued to be executed to optimize the image pose.

[0093] The technical solution of this embodiment performs a judgment before optimization to confirm whether optimization is necessary, so that in special cases where the initial pose accuracy of the image is high enough, subsequent optimization steps are not executed, and the registration of the point cloud image is completed more quickly.

[0094] like Figure 4 As shown, Figure 4 It is a flow chart of the fourth embodiment of the point cloud image registration method of the present invention.

[0095] In this embodiment, in the above step S50, the step of using the depth value of the laser point as an auxiliary for BA optimization includes:

[0096] Step S51, converting the 3D point from the world coordinate system to the camera coordinate system;

[0097] Set the world coordinates (X, Y, Z) of the 3D point P to the camera coordinate system: .

[0098] Step S52, normalizing the coordinates to remove distortion;

[0099] Normalize the coordinates ,

[0100] Remove distortion .

[0101] Step S53, converting the camera coordinate system into a pixel coordinate system to obtain a camera observation model;

[0102] Then convert the camera coordinate system to the pixel coordinate system , combining the above, the camera observation model is .

[0103] Step S54: construct a reprojection error model e = q - b + w × f (d i -d l );

[0104] We can first construct a reprojection error model e = qb, and then add depth information to the reprojection error model to obtain a reprojection error model with added depth information e = q-b + w × f (d i -d l ), where q is the pixel coordinate of the matching feature point, function b is the pixel coordinate h (R, t, X, Y, Z) obtained by back projection of the 3D point, d i and d l They are the depth values ​​of the 3D point and the corresponding laser point respectively. f indicates the absolute value of the depth value difference in meters. w is the weight (for example, 100).

[0105] It should be noted that, in the case where there are no contradictions or conflicts between the solutions of all the above embodiments of the present application, the above embodiments can be arbitrarily combined or combined to form new embodiments.

[0106] The present invention also proposes a point cloud image registration device, which is a virtual device corresponding to the above-mentioned point cloud image registration method.

[0107] like Figure 5 As shown, in this embodiment, the point cloud image registration device includes:

[0108] An acquisition module 10 is used to acquire laser point clouds, images, and initial image poses;

[0109] The extraction and matching module 20 is used to extract and match feature points of the images to determine matching feature points between the images;

[0110] An estimation module 30, configured to obtain 3D points corresponding to respective matching feature points based on an initial image posture, and to update 3D points corresponding to respective matching feature points based on an optimized image posture;

[0111] A back-projection module 40 is used to obtain laser points corresponding to each 3D point in the laser point cloud by back-projection of the laser points, and to update the laser points corresponding to each 3D point by back-projection of the laser points;

[0112] A calculation module 50, for determining the depth value of each 3D point and the corresponding laser point respectively;

[0113] An optimization module 60, used to perform BA optimization with the depth value of the laser point as an aid to obtain an optimized image pose;

[0114] A judgment module 70 is used to determine whether the depth value of each 3D point and the corresponding laser point meets a preset condition;

[0115] The iteration module 80 is used to use the latest optimized image pose as the final image pose result when it is confirmed that the depth value of each 3D point and the corresponding laser point meets the preset conditions, and to call the optimization module when it is confirmed that the depth value of each 3D point and the corresponding laser point does not meet the preset conditions.

[0116] The present invention also provides an electronic device, referring to Figure 6 , Figure 6 It is a schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention.

[0117] The electronic device of the embodiment of the present invention can be a laser scanning device, a desktop computer, a notebook, a palm computer, a server or other computing device. Figure 6As shown, the electronic device may include: a processor 1001 (such as a CPU), a network interface 1004, a user interface 1003, a memory 1005 and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit, such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0118] Those skilled in the art will understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] like Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a computer program.

[0120] exist Figure 6 In the electronic device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the computer program stored in the memory 1005. When the computer program is called and executed by the processor 1001, the steps of the above-mentioned point cloud image registration method are implemented.

[0121] Based on the computer program proposed in the aforementioned embodiment, the present invention further proposes a storage medium, which stores the computer program. When the computer program is executed by a controller, the point cloud image registration method recorded in the aforementioned embodiment is implemented.

[0122] The point cloud image registration device, electronic device and storage medium of the present invention can implement the steps of the above-mentioned point cloud image registration method, and therefore have at least all the beneficial effects brought about by the technical solution of the above-mentioned point cloud image registration method embodiment, which will not be described one by one here.

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0124] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0126] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0127] The above description is only a partial or preferred embodiment of the present invention. Neither the text nor the drawings can limit the scope of protection of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the overall concept of the present invention, or direct / indirect application in other related technical fields are included in the scope of protection of the present invention.

Claims

1. A point cloud image registration method, characterized in that: include: Obtain laser point cloud, image and initial image pose; Extracting and matching feature points of the images to determine matching feature points between the images; Acquire the 3D points corresponding to the respective matching feature points based on the initial pose of the image; Obtaining laser points corresponding to each 3D point in the laser point cloud by laser point backprojection, and determining depth values ​​of each 3D point and the corresponding laser point respectively; BA optimization is performed with the depth value of the laser point as an aid to obtain an optimized image pose; Based on the optimized image pose, the 3D points corresponding to each matching feature point are updated, the laser points corresponding to each 3D point are updated through laser point backprojection, the depth values ​​of each 3D point and the corresponding laser point are calculated, and it is confirmed whether the depth values ​​of each 3D point and the corresponding laser point meet the preset conditions; If so, the latest optimized image pose is used as the final image pose result; If not, jump to the step of performing BA optimization with the depth value of the laser point as an aid to obtain an optimized image pose; The step of acquiring laser points corresponding to each 3D point in the laser point cloud by laser point back projection comprises: Search and obtain a set of neighboring laser points for each of the 3D points; Projecting the laser points in the set of adjacent laser points sequentially to the image to which the matching feature points corresponding to the corresponding 3D points belong, and determining the laser points corresponding to the respective matching feature points; The step of using the depth value of the laser point as an auxiliary for BA optimization includes: Convert the 3D point from the world coordinate system to the camera coordinate system; Normalize the coordinates and remove the distortion; Convert the camera coordinate system into a pixel coordinate system to obtain a camera observation model; Construct a reprojection error model with added depth information e=q-b+w×f(d i -d l ), where q is the pixel coordinate of the matching feature point, function b is the pixel coordinate of the 3D point obtained by back projection, and d i and d l They are the depth values ​​of the 3D point and the corresponding laser point respectively, f indicates the absolute value of the depth value difference, and w is the weight.

2. The point cloud image registration method according to claim 1, characterized in that: The step of extracting and matching feature points of the image comprises: Extract the feature points of each image, obtain the proximity relationship between images based on the prior position information of the images, and match each image with other images within a preset range.

3. The point cloud image registration method according to claim 1, characterized in that: The step of acquiring 3D points corresponding to each matching feature point based on the initial pose of the image comprises: For each set of matching feature points, triangulation is used to obtain the corresponding 3D points based on the initial image pose of the image to which it belongs.

4. The point cloud image registration method according to claim 1, characterized in that: Before the step of using the depth value of the laser point as an auxiliary for BA optimization, the point cloud image registration method further includes: Confirm whether the depth value of each 3D point and the corresponding laser point meets the preset conditions; If yes, the initial image pose is used as the final image pose result; If not, a step of performing BA optimization with the depth value of the laser point as an aid is performed to obtain an optimized image pose.

5. The point cloud image registration method according to any one of claims 1 to 4, characterized in that: The preset condition is that the average depth value difference between the depth value of each 3D point and the depth value of the corresponding laser point is less than a preset threshold.

6. A point cloud image registration device, characterized in that: include: The acquisition module is used to obtain the laser point cloud, image and initial image pose; An extraction and matching module, used for extracting and matching feature points of the images to determine matching feature points between the images; An estimation module, configured to obtain 3D points corresponding to respective matching feature points based on the initial image pose, and to update the 3D points corresponding to respective matching feature points based on the optimized image pose; A back-projection module, used for obtaining laser points corresponding to each 3D point in the laser point cloud by back-projection of laser points, and updating laser points corresponding to each 3D point by back-projection of laser points; A calculation module, used to determine the depth value of each 3D point and the corresponding laser point respectively; An optimization module, used to perform BA optimization with the depth value of the laser point as an aid to obtain an optimized image pose; A judgment module is used to confirm whether the depth value of each 3D point and the corresponding laser point meets the preset conditions; An iteration module, used to use the latest optimized image pose as the final image pose result when it is confirmed that the depth value of each 3D point and the corresponding laser point meets the preset condition, and to call the optimization module when it is confirmed that the depth value of each 3D point and the corresponding laser point does not meet the preset condition; The back projection module is specifically used for: Search and obtain a set of neighboring laser points for each of the 3D points; Projecting the laser points in the set of adjacent laser points sequentially to the image to which the matching feature points corresponding to the corresponding 3D points belong, and determining the laser points corresponding to the respective matching feature points; The optimization module is specifically used for: Convert the 3D point from the world coordinate system to the camera coordinate system; Normalize the coordinates and remove the distortion; Convert the camera coordinate system into a pixel coordinate system to obtain a camera observation model; Construct a reprojection error model with added depth information e=q-b+w×f(d i -d l ), where q is the pixel coordinate of the matching feature point, function b is the pixel coordinate of the 3D point obtained by back projection, and d i and d l They are the depth values ​​of the 3D point and the corresponding laser point respectively, f indicates the absolute value of the depth value difference, and w is the weight.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the point cloud image registration method as described in any one of claims 1 to 5 are implemented.

8. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the point cloud image registration method according to any one of claims 1 to 5.

Images