Registration method and system of point cloud and image

Abstract

The invention relates to the technical field of image registration, and discloses a point cloud and image registration method and system. The method comprises the steps that a calibration plate and a calibration plate coordinate system are constructed, the position of the calibration plate is changed, three-dimensional point clouds of a target object when the calibration plate is located at different positions are obtained through a laser radar, and two-dimensional images of the target object when the calibration plate is located at different positions are obtained through a camera; internal parameters and external parameters of the camera are determined through the calibration tool and the two-dimensional image; according to the three-dimensional point cloud, the two-dimensional image and the calibration plate, calculating a rotation matrix and a translation vector between a laser radar coordinate system and a camera coordinate system; projecting the three-dimensional point cloud to a camera coordinate system according to the rotation matrix and the translation vector to obtain a two-dimensional coordinate corresponding to the three-dimensional point cloud; and determining two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal reference and the external reference, and assigning the pixels of the two-dimensional image to the three-dimensional point cloud to obtain registration data. According to the invention, the accuracy of registration can be improved.

Description

Technical Field

[0001] This application relates to the field of image registration technology, and particularly to a method and system for registering point cloud and image. Background Art

[0002] Registration algorithms are mainly used to align two or more images or point clouds in spatial positions so that the same features are preferably in the same positions. In the existing registration methods, the parameters between the radar and the camera are mainly calibrated by a simple hand-eye calibration method, resulting in certain errors in the registration results. Summary of the Invention

[0003] To solve the problem that the registration method in the existing technology easily leads to errors in the registration results, this application provides a method and system for registering point cloud and image.

[0004] In a first aspect, this application provides a method for registering point cloud and image, including:

[0005] Construct a calibration board and a calibration board coordinate system, change the position of the calibration board, obtain the three-dimensional point cloud of the target object when the calibration board is at different positions through a lidar, and obtain the two-dimensional image of the target object when the calibration board is at different positions through a camera;

[0006] Determine the internal parameters and external parameters of the camera through a calibration tool and the two-dimensional image;

[0007] Calculate the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system according to the three-dimensional point cloud, the two-dimensional image and the calibration board;

[0008] Project the three-dimensional point cloud into the camera coordinate system according to the rotation matrix and the translation vector to obtain the two-dimensional coordinates corresponding to the three-dimensional point cloud;

[0009] Determine the two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal parameters and the external parameters, and assign the pixels of the two-dimensional image to the three-dimensional point cloud to obtain registration data.

[0010] In an implementation manner, before changing the position of the calibration board, it includes:

[0011] Fix the relative positions of the lidar and the camera.

[0012] In an implementation manner, the determining of the internal parameters of the camera includes:

[0013] Calibrate the internal parameters of the camera through a calibration tool:

[0014]

[0015] Among them, A is the internal parameter of the camera, and f x represents the focal length of the camera in the x direction, and f y represents the focal length of the camera in the y direction. u0 represents the offset of the origin of the pixel plane to the camera imaging plane in the u direction, and v0 represents the offset of the origin of the pixel plane to the camera imaging plane in the v direction.

[0016] In one implementation, determining the external parameters of the camera includes:

[0017] Obtaining the external parameters between the calibration board and the camera coordinate system at different positions:

[0018]

[0019] Among them, T i represents the external parameter, represents the rotation matrix between the calibration board coordinate system and the camera coordinate system at the i-th position, represents the translation vector between the calibration board coordinate system and the camera coordinate system at the i-th position.

[0020] In one implementation, the method further includes:

[0021] Performing linear fitting on the three-dimensional point cloud to obtain a fitted line;

[0022] Selecting any point on the fitted line as the first target point, and converting the fitted line and the first target point from the lidar coordinate system to the camera coordinate system:

[0023]

[0024] R represents the rotation matrix from the lidar coordinate system to the camera coordinate system, and t represents the translation vector from the lidar coordinate system to the camera coordinate system. represents the coordinates of the fitted line in the camera coordinate system corresponding to represents the coordinates of the first target point on the fitted line in the camera coordinate system corresponding to the coordinate point.

[0025] In one implementation, the method further includes:

[0026] Constructing a line-plane constraint equation according to the fitted line and the first target point:

[0027]

[0028] Among them, represents the perpendicular distance from the origin of the camera coordinate system to the calibration board plane, The calculation formula is as follows:

[0029]

[0030] Wherein, represents the Euclidean distance from the origin of the camera coordinate system to the origin of the calibration board, [0 0 1] T represents the normal vector of the calibration board.

[0031] In one implementation, the method further includes:

[0032] Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane, and obtaining the normal vector of the fitting plane;

[0033] Selecting any point on the fitting plane as the second target point, and transforming the normal vector of the fitting plane and the second target point from the lidar coordinate system to the camera coordinate system:

[0034]

[0035]

[0036] Wherein, represents the coordinates of the normal vector s of the fitting plane in the camera coordinate system i corresponding to, represents the coordinates of the second target point e in the camera coordinate system i corresponding to.

[0037] In one implementation, the method further includes:

[0038] Constructing a plane-plane constraint equation according to the fitting plane and the second target point:

[0039]

[0040] Wherein, represents parallel.

[0041] In one implementation, calculating the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system includes:

[0042] Substituting the three-dimensional point cloud into the overdetermined equation composed of the line-plane constraint equation and the plane-plane constraint equation respectively;

[0043] Solving the overdetermined equation by the least squares method to calculate the rotation matrix and the translation vector.

[0044] In a second aspect, the present application provides a point cloud and image registration system, including:

[0045] A construction module for constructing a calibration board and establishing a calibration board coordinate system, changing the position of the calibration board, obtaining the three-dimensional point cloud of the target object when the calibration board is at different positions through a lidar, and obtaining the two-dimensional image of the target object when the calibration board is at different positions through a camera;

[0046] A determination module for determining the internal parameters and external parameters of the camera through a calibration tool and the two-dimensional image;

[0047] A calculation module for calculating the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system according to the three-dimensional point cloud, the two-dimensional image and the calibration board;

[0048] A projection module for projecting the three-dimensional point cloud into the camera coordinate system according to the rotation matrix and the translation vector to obtain the two-dimensional coordinates corresponding to the three-dimensional point cloud;

[0049] A registration module for determining the two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal parameters and the external parameters, and assigning the pixels of the two-dimensional image to the three-dimensional point cloud to obtain registration data.

[0050] The embodiments of the present application have the following beneficial effects:

[0051] The point cloud and image registration method provided by the present application obtains the three-dimensional point cloud and two-dimensional image of the target object when the calibration board is at different positions, then calculates the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system with the calibration board as a reference, and then registers the three-dimensional point cloud according to the rotation matrix and translation vector, improving the registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application, the drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the protection scope of the present application. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0053] Figure 1 Shows a schematic flowchart of a point cloud and image registration method provided by an embodiment of the present application;

[0054] Figure 2 Shows a schematic flowchart of a fitting line conversion method provided by an embodiment of the present application;

[0055] Figure 3 Shows a schematic flowchart of a fitting plane conversion method provided by an embodiment of the present application;

[0056] Figure 4 The schematic diagram of the framework structure of a point cloud and image registration system provided by an embodiment of the present application is shown. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0058] Generally, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0059] In the following text, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are only intended to represent specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be construed as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0060] In addition, the terms "first", "second", "third", etc. are only used for differential description and cannot be understood as indicating or implying relative importance.

[0061] Unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as commonly understood by those of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in a general-use dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0062] Next, some implementation manners of the present application will be described in detail in conjunction with the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0063] Registration algorithms are mainly used to align two or more images or point clouds in spatial position so that the same features are as close as possible to the same position. Registration algorithms can be divided into two categories: image registration and point cloud registration. Image registration is to align multiple images containing the same scene in spatial position, while point cloud registration is mainly used for the precise alignment of point clouds in three-dimensional space. In order to improve the accuracy of data, in the prior art, image acquisition and point cloud acquisition are usually performed on certain objects simultaneously, and then the point cloud is registered according to the image data, that is, point cloud-image registration, so that the features of the object can be obtained from both the image and the point cloud aspects.

[0064] Referring to Figure 1 , Figure 1 FIG. is a schematic flowchart of a method for registering a point cloud and an image provided in this embodiment. This method can improve the registration accuracy of the point cloud and the image. The method includes:

[0065] S101. Construct a calibration board and construct a calibration board coordinate system, change the position of the calibration board, obtain the three-dimensional point cloud of the target object at different positions of the calibration board through a lidar, and obtain the two-dimensional image of the target object at different positions of the calibration board through a camera.

[0066] The target object can be an object to be recognized. For example, if certain features of the object to be recognized are to be obtained from both the point cloud and the image, the three-dimensional point cloud of the target object can be obtained through a lidar at this time, and the two-dimensional image of the target object can be obtained through a camera.

[0067] When the lidar and the camera collect information about the target object, the positions, angles, and coordinate system construction methods of the two may be different. Therefore, before collecting the three-dimensional point cloud and two-dimensional image of the target object, the relative positions of the lidar and the camera need to be fixed first.

[0068] The calibration board can be a black and white checkerboard calibration board. The black and white checkerboard calibration board is a calibration tool widely used in fields such as visual measurement, machine vision, and photogrammetry. It is usually composed of black and white square grids, forming a pattern similar to a checkerboard. The sizes and layouts of these grids are precisely known.

[0069] First, the size of the black and white checkerboard calibration board can be set. For example, it can be set to 120 cm × 72 cm. The black and white square grids with a side length of 12 cm are evenly distributed on the black and white checkerboard calibration board. This is convenient for constructing the calibration board coordinate system and positioning.

[0070] Then fix the relative positions of the lidar and the camera, and change the position of the checkerboard calibration board to collect the three-dimensional point cloud P i ={P i |1 ≤ i ≤ m}, and the two-dimensional image Ii = {I i | 1 ≤ i ≤ m}, where m corresponds to the number of poses of the checkerboard calibration board, and P i and I i respectively represent the three-dimensional point cloud and the two-dimensional image collected when the pose of the checkerboard calibration board is i; Classify the collected three-dimensional lidar data P i according to the beam of the lidar to obtain where k represents the beam of the radar.

[0071] S102. Determine the internal parameters and external parameters of the camera through a calibration tool and the two-dimensional image.

[0072] Among them, the calibration tool can adopt the calibration toolbox of Matlab.

[0073] S103. Calculate the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system according to the three-dimensional point cloud, the two-dimensional image and the calibration board.

[0074] Taking the calibration board as the reference coordinate system, determine the rotation matrix and translation vector between the camera coordinate system and the calibration board coordinate system, and the relationship between the lidar coordinate system and the calibration board coordinate system, so as to determine the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system.

[0075] S104. Project the three-dimensional point cloud into the camera coordinate system according to the rotation matrix and the translation vector to obtain the two-dimensional coordinates corresponding to the three-dimensional point cloud.

[0076] According to the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system, transform the three-dimensional point cloud, so as to assign the pixel value in the two-dimensional image to the corresponding point cloud, and realize the registration of the point cloud.

[0077] S105. Determine the two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal parameters and the external parameters, and assign the pixels of the two-dimensional image to the three-dimensional point cloud to obtain the registration data.

[0078] According to the two-dimensional coordinates, the internal parameters and the external parameters of the camera, the corresponding relationship between the two-dimensional coordinates and the two-dimensional image can be determined, so as to determine the pixels corresponding to the two-dimensional coordinates, and then assign the pixels to the three-dimensional point cloud to realize the registration.

[0079] In this embodiment, the three-dimensional point cloud and the two-dimensional image of the target object are obtained through the calibration board, and then, taking the calibration board as the reference, calculate the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system, and then register the three-dimensional point cloud according to the rotation matrix and translation vector, improving the accuracy of registration.

[0080] In one implementation, after obtaining the three-dimensional point cloud and the two-dimensional image, the three-dimensional point cloud and the two-dimensional image can also be filtered.

[0081] Specifically, for the three-dimensional point cloud, the mean value d and variance q of all points within a 50-cm circular area around each point cloud in the lidar coordinate system can be calculated first, and a distance threshold d is set. Points exceeding the threshold are marked as outliers and removed from the three-dimensional point cloud.

[0082] For the two-dimensional image, the Gaussian function is used to measure the similarity between pixels. The more similar, the greater the corresponding weight, filtering out the noise in the image while highlighting the edge details.

[0083] In this embodiment, by processing the three-dimensional point cloud and the two-dimensional image using different filtering methods, the accuracy of registration is improved.

[0084] In one implementation, determining the internal parameters of the camera includes:

[0085] Calibrating the internal parameters of the camera using a calibration tool.

[0086] Specifically, for the two-dimensional images I of different checkerboard calibration plate positions i ={I i |1≤i≤m}, the Matlab calibration toolbox can be used to calibrate the internal parameters of the camera, obtaining the internal parameter matrix A of the camera and the external parameters T between the checkerboard calibration plate and the camera coordinate system at different positions i i .

[0087]

[0088] Among them, A is the internal parameter of the camera, f x represents the focal length of the camera in the x direction, f y represents the focal length of the camera in the y direction, u0 represents the offset of the origin of the pixel plane to the camera imaging plane in the u direction, and v0 represents the offset of the origin of the pixel plane to the camera imaging plane in the v direction.

[0089]

[0090] Among them, T i represents the external parameter, represents the rotation matrix between the calibration plate coordinate system and the camera coordinate system at the i-th position, represents the translation vector between the calibration plate coordinate system and the camera coordinate system at the i-th position.

[0091] Referring to Figure 2 , in one implementation, the method further includes: steps S106 - S107.

[0092] S106. Perform a straight-line fitting on the three-dimensional point cloud to obtain a fitted straight line.

[0093] S107. Select any point on the fitted straight line as the first target point, and transform the fitted straight line and the first target point from the lidar coordinate system to the camera coordinate system.

[0094] Specifically, for the three-dimensional lidar data of different wire bundles under calibration plates at different positions Use the least squares method to perform straight-line fitting to obtain a fitted straight line Randomly select a point on the fitted straight line As the first target point, the fitted straight line And the first target point Are transformed from the radar coordinate system to the camera coordinate system:

[0095]

[0096] R represents the rotation matrix between the lidar coordinate system and the camera coordinate system, and t represents the translation vector between the lidar coordinate system and the camera coordinate system. Represents the coordinates of the fitted straight line in the camera coordinate system Corresponding coordinates, Represents the coordinates of the first target point on the fitted straight line in the camera coordinate system Corresponding coordinate point.

[0097] In one implementation, the method further includes:

[0098] Construct a line-plane constraint equation according to the fitted straight line and the first target point:

[0099]

[0100] Among them, Represents the perpendicular distance from the origin of the camera coordinate system to the calibration plate plane, The calculation formula of is:

[0101]

[0102] Among them, Represents the Euclidean distance from the origin of the camera coordinate system to the origin of the calibration plate, [0 0 1] T Represents the normal vector of the calibration plate.

[0103] Referring to Figure 3 , in one implementation, the method further includes: steps S108 - S109.

[0104] S108. Perform a plane fitting on the three-dimensional point cloud to obtain a fitted plane, and obtain the normal vector of the fitted plane.

[0105] S109. Select any point on the fitting plane as the second target point, and transform the normal vector of the fitting plane and the second target point from the lidar coordinate system to the camera coordinate system.

[0106] Specifically, for the three-dimensional lidar data P i ={P i |1≤i≤m}, perform plane fitting using the least squares method to obtain the fitting plane S i , then the normal vector of the fitting plane is s i . Randomly select a point e i on the fitting plane as the second target point, and transform the normal vector s i of the fitting plane and the second target point e i from the lidar coordinate system to the camera coordinate system:

[0107]

[0108] where, represents the coordinates corresponding to the normal vector s i of the fitting plane in the camera coordinate system, represents the coordinates corresponding to the second target point e i in the camera coordinate system.

[0109] In one implementation, the method further includes:

[0110] Construct a plane-plane constraint equation based on the fitting plane and the second target point:

[0111]

[0112] where, represents parallel.

[0113] In one implementation, calculating the rotation matrix and translation vector between the lidar coordinate system and the camera coordinate system includes:

[0114] Substitute the three-dimensional point cloud into the overdetermined equation composed of the line-plane constraint equation and the plane-plane constraint equation respectively;

[0115] Solve the overdetermined equation by the least squares method to calculate the rotation matrix and the translation vector.

[0116] Specifically, collect the three-dimensional lidar data P i ={P iSubstitute {1 ≤ i ≤ m} into Formulas 5 and 6 to form m line-plane constraint equations, and substitute them into Formulas 10 and 11 to form m plane-plane constraint equations. Then, combine the m line-plane constraint equations and the m plane-plane constraint equations into an overdetermined equation and solve it using the least squares method to obtain the rotation matrix R and the translation vector t.

[0117] In this embodiment, by constructing a large number of constraint equations between the lidar and the camera, the calibration accuracy can be improved, and more accurate registration can be achieved.

[0118] In one implementation, the method further includes:

[0119] Project the three-dimensional point cloud in the lidar coordinate system to the camera coordinate system through the rotation matrix R and the translation vector t, then project the point cloud in the camera coordinate system to the pixel coordinate system through the camera internal parameter matrix A, and then determine whether the two-dimensional point cloud coordinates in the camera coordinate system are within the range of the pixel coordinate system. If not, they are removed from the point cloud data; finally, construct an information storage module to store the point cloud coordinates of the corresponding camera coordinate system in the pixel coordinate system, where the constructed information storage module should have the same value as the size of the image captured by the camera.

[0120] Then construct a depth information storage module. The size used to store the point cloud in this module should be the same as that of the information storage module. Calculate the Euclidean distance from the two-dimensional point cloud coordinates in the camera coordinate system to the origin of the camera coordinate system and fill it into the depth information module, and store the corresponding point cloud data at the same time. Correlate the point cloud data in the depth information storage module with the image pixels captured by the camera, and assign the color information in the image pixels to the point cloud to obtain the registration result.

[0121] In this embodiment, by projecting the three-dimensional point cloud to the camera coordinate system, determining the corresponding two-dimensional coordinates of the three-dimensional point cloud, and then assigning the pixel value corresponding to the coordinate to the three-dimensional point cloud, the registration of the point cloud and the image is realized, and the registration accuracy is improved.

[0122] Refer to Figure 4 , this application provides a point cloud and image registration system 400, including:

[0123] A construction module 401 for constructing a calibration board and constructing a calibration board coordinate system, changing the position of the calibration board, obtaining the three-dimensional point cloud of the target object when the calibration board is at different positions through the lidar, and obtaining the two-dimensional image of the target object when the calibration board is at different positions through the camera.

[0124] A determination module 402 for determining the internal parameters and external parameters of the camera through a calibration tool and the two-dimensional image.

[0125] A calculation module 403, configured to calculate a rotation matrix and a translation vector between a lidar coordinate system and a camera coordinate system according to the three-dimensional point cloud, the two-dimensional image, and the calibration board.

[0126] A projection module 404, configured to project the three-dimensional point cloud into the camera coordinate system according to the rotation matrix and the translation vector to obtain two-dimensional coordinates corresponding to the three-dimensional point cloud.

[0127] A registration module 405, configured to determine two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal parameters, and the external parameters, and assign the pixels of the two-dimensional image to the three-dimensional point cloud to obtain registration data.

[0128] It can be understood that the system in this embodiment corresponds to the point cloud and image registration method in the above embodiment. The optional items in the above embodiment are equally applicable to this embodiment, so they will not be described again here.

[0129] The present application further provides a computer device. Exemplarily, the computer device includes a processor and a memory. The memory stores a computer program, and the processor runs the computer program to enable the computer device to execute the above point cloud and image registration method or the functions of each module in the above point cloud and image registration system.

[0130] Among them, the processor may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0131] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory is used to store a computer program, and after receiving an execution instruction, the processor can execute the computer program accordingly.

[0132] This application also provides a computer storage medium for storing the computer program used in the above computer device. Among them, the computer storage medium can be a readable storage medium, a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium can include, but is not limited to: USB flash drives, external hard drives, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disks or optical discs and other various media that can store program codes.

[0133] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and structure diagrams in the drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the block can occur in a different order from that marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the structure diagram and / or flowchart, as well as the combination of blocks in the structure diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0134] In addition, each functional module or unit in various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0135] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.

[0136] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. A method for registering a point cloud with an image, characterized in that: include: Constructing a calibration plate and a calibration plate coordinate system, changing the position of the calibration plate, obtaining a three-dimensional point cloud of the target object when the calibration plate is at different positions through a laser radar, and obtaining a two-dimensional image of the target object when the calibration plate is at different positions through a camera; Determining the intrinsic and extrinsic parameters of the camera by using a calibration tool and the two-dimensional image; Calculate the rotation matrix and translation vector between the laser radar coordinate system and the camera coordinate system according to the three-dimensional point cloud, the two-dimensional image and the calibration plate; Projecting the three-dimensional point cloud to the camera coordinate system according to the rotation matrix and the translation vector to obtain two-dimensional coordinates corresponding to the three-dimensional point cloud; According to the two-dimensional coordinates, the internal parameters and the external parameters, the two-dimensional image pixels corresponding to the two-dimensional coordinates are determined, and the pixels of the two-dimensional image are assigned to the three-dimensional point cloud to obtain registration data.

2. The point cloud and image registration method according to claim 1, characterized in that: Before changing the position of the calibration plate, the method includes: The relative positions of the laser radar and the camera are fixed.

3. The method for registering a point cloud with an image according to claim 1, characterized in that: The determining the internal parameters of the camera includes: Calibrate the intrinsic parameters of the camera using the calibration tool: Among them, A is the intrinsic parameter of the camera, f x Indicates the focal length of the camera in the x direction, f y It represents the focal length of the camera in the y direction, u0 represents the offset from the origin of the pixel plane to the camera imaging plane in the u direction, and v0 represents the offset from the origin of the pixel plane to the camera imaging plane in the v direction.

4. The point cloud and image registration method according to claim 3, characterized in that: The determining of the external parameters of the camera includes: Obtain the external parameters between the calibration plate and the camera coordinate system at different positions: Among them, T i Represents external reference, represents the rotation matrix between the calibration plate coordinate system and the camera coordinate system at the i-th position, Represents the translation vector between the calibration plate coordinate system and the camera coordinate system at the i-th position.

5. The point cloud and image registration method according to claim 4, characterized in that: The method further comprises: Performing straight line fitting on the three-dimensional point cloud to obtain a fitting straight line; Select any point on the fitted straight line as the first target point, and transform the fitted straight line and the first target point from the laser radar coordinate system to the camera coordinate system: R represents the rotation matrix between the laser radar coordinate system and the camera coordinate system, t represents the translation vector between the laser radar coordinate system and the camera coordinate system, Represents the fitted straight line in the camera coordinate system The corresponding coordinates, Indicates the first target point on the fitted line in the camera coordinate system The corresponding coordinate points.

6. The point cloud and image registration method according to claim 5, characterized in that: The method further comprises: Construct a line-surface constraint equation based on the fitting straight line and the first target point: in, Represents the vertical distance from the origin of the camera coordinate system to the plane of the calibration plate, The calculation formula is: in, Represents the Euclidean distance from the origin of the camera coordinate system to the origin of the calibration plate, [0 0 1] T Represents the normal vector of the calibration plate.

7. The point cloud and image registration method according to claim 6, characterized in that: The method further comprises: Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane, and acquiring a normal vector of the fitting plane; Select any point on the fitting plane as the second target point, and transform the fitting plane normal vector and the second target point from the laser radar coordinate system to the camera coordinate system: in, Represents the fitted plane normal vector s in the camera coordinate system i The corresponding coordinates, Represents the second target point e in the camera coordinate system i The corresponding coordinates.

8. The point cloud and image registration method according to claim 7, characterized in that: The method further comprises: Construct a surface constraint equation based on the fitting plane and the second target point: in, Indicates parallelism.

9. The point cloud and image registration method according to claim 8, characterized in that: The step of calculating the rotation matrix and the translation vector between the laser radar coordinate system and the camera coordinate system includes: Substituting the three-dimensional point cloud into the line-surface constraint equation and the surface-surface constraint equation to form an overdetermined equation; The overdetermined equation is solved by the least square method to calculate the rotation matrix and the translation vector.

10. A point cloud and image registration system, characterized in that: include: A construction module is used to construct a calibration plate and a calibration plate coordinate system, change the position of the calibration plate, obtain a three-dimensional point cloud of the target object when the calibration plate is in different positions through a laser radar, and obtain a two-dimensional image of the target object when the calibration plate is in different positions through a camera; A determination module, used to determine the intrinsic parameters and extrinsic parameters of the camera through a calibration tool and the two-dimensional image; A calculation module, used to calculate the rotation matrix and translation vector between the laser radar coordinate system and the camera coordinate system according to the three-dimensional point cloud, the two-dimensional image and the calibration plate; A projection module, used to project the three-dimensional point cloud to the camera coordinate system according to the rotation matrix and the translation vector, so as to obtain the two-dimensional coordinates corresponding to the three-dimensional point cloud; A registration module is used to determine the two-dimensional image pixels corresponding to the two-dimensional coordinates according to the two-dimensional coordinates, the internal parameters and the external parameters, and assign the pixels of the two-dimensional image to the three-dimensional point cloud to obtain registration data.

Images