A Multi-Sensor Based Point Cloud Mosaic Method and System

Abstract

This application belongs to the field of visual measurement technology, and specifically discloses a point cloud stitching method and system based on multi-sensors. The point cloud stitching method based on multi-sensors provided by this application extracts and fuses data from multiple sensors including a camera, a monocular line laser scanner, and an inertial measurement unit, constructs a high-precision target homogeneous transformation matrix, and combines this target homogeneous transformation matrix to convert the coordinates of the center line point set of the first image at each frame time in the first coordinate system to the coordinates in the second coordinate system, and performs point cloud stitching on the coordinates of the center line point set of the first image at each frame time in the second coordinate system, avoiding the problem of inaccurate stitching caused by the low coincidence degree of the monocular line laser scan point cloud, and improving the point cloud stitching accuracy and the accuracy of reconstruction.

Description

Technical Field

[0001] This application belongs to the technical field of visual measurement, and more specifically, relates to a point cloud stitching method and system based on multi-sensors. Background Art

[0002] The point cloud obtained by a monocular line laser scanner scanning an object has the characteristic of less overlap. When using the traditional Iterative Closest Point (ICP) method to solve the transformation matrix, there is a problem of low accuracy. In this method, errors will gradually accumulate during the process of processing the line laser point cloud, resulting in low stitching accuracy.

[0003] Therefore, how to avoid the problem of low point cloud stitching accuracy caused by low overlap of the monocular line laser scanned point cloud is an urgent problem to be solved at present. Summary of the Invention

[0004] Aiming at the defects of the prior art, the purpose of this application is to provide a point cloud stitching method and system based on multi-sensors, aiming to solve the problem of low point cloud stitching accuracy caused by low overlap of the monocular line laser scanned point cloud.

[0005] To achieve the above purpose, in the first aspect, this application provides a point cloud stitching method based on multi-sensors, including:

[0006] Obtain the measurement data at all frame times during the process of scanning the object. The measurement data includes the first image, the second image, the angular velocity measurement value and the acceleration measurement value of the object measured by the inertial measurement unit. The first image is the image of the object scanned by the monocular line laser scanner, and the second image is the image containing the preset mark captured by the imaging device;

[0007] Execute at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame times are all converted to the coordinates in the second coordinate system in the first coordinate system. The first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0008] Perform point cloud stitching according to the coordinates of the center line point set of the first image corresponding to all frame times in the second coordinate system;

[0009] Among them, the transformation process includes:

[0010] Based on the pixel coordinates of each point in the center line point set corresponding to the first frame time, obtain the first coordinate of each point in the center line point set corresponding to the first frame time in the first coordinate system. The first frame time is any frame time;

[0011] Based on the first coordinate and the target homogeneous transformation matrix, the first coordinate is transformed into a second coordinate in a second coordinate system at a second frame time. The target homogeneous transformation matrix is a homogeneous transformation matrix between the coordinates in a first coordinate system at a first frame time and the coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame time and the second frame time, and the second frame time is other frame times except the first frame time.

[0012] In some embodiments, the obtaining manner of each point in the center line point set of the first image corresponding to the first frame time includes:

[0013] Determine a candidate center line point set according to the centroid of the laser stripe in the target binary image, where the target binary image is a binary image containing the laser stripe corresponding to the first image corresponding to the first frame time;

[0014] Determine the center line point set according to the candidate center line point set.

[0015] In some embodiments, determining a candidate center line point set according to the centroid of the laser stripe in the target binary image includes:

[0016] Obtain the first centroid of each row of pixels in the horizontal direction and the set of vertical centroid points in the target binary image, where the set of vertical centroid points is a set composed of the second centroids of each column of pixels in the target binary image in the vertical direction;

[0017] Obtain the target second centroid in the set of vertical centroid points that is closest to each first centroid;

[0018] Obtain the candidate center line point set according to each first centroid and the target second centroid corresponding to each first centroid.

[0019] In some embodiments, the obtaining manner of the target homogeneous transformation matrix includes:

[0020] Obtain the angular velocity measurement value and the acceleration measurement value of the object corresponding to all frame times during the process of scanning the object;

[0021] Based on the angular velocity measurement value and the acceleration measurement value, obtain the pose increment information from the first frame time to the second frame time;

[0022] Based on the pose increment information, obtain the first transformation matrix from the first frame time to the second frame time;

[0023] Based on the second image corresponding to the first frame time, obtain the third coordinate of the preset marker in the first coordinate system at the first frame time;

[0024] Based on the second image corresponding to the second frame time, obtain the fourth coordinate of the preset marker in the first coordinate system at the second frame time;

[0025] Determine a second transformation matrix from the first frame moment to the second frame moment according to the third coordinate and the fourth coordinate;

[0026] Determine a target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix.

[0027] In some embodiments, determining a target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix includes:

[0028] Input the first transformation matrix and the second transformation matrix into a target model to obtain output data of the target model;

[0029] Determine a target homogeneous transformation matrix according to the output data;

[0030] Among them, the acquisition method of the target model includes:

[0031] Input training data into a preset model for training until the loss function of the preset model converges. The training data includes a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix;

[0032] Determine a target model according to the converged preset model.

[0033] In some embodiments, determining a center line point set according to a candidate center line point set includes:

[0034] Obtain the unit normal vector of each data point in the candidate center line point set;

[0035] According to the angle between adjacent unit normal vectors, obtain at least one target data point with an angle greater than a preset threshold from the candidate center line point set;

[0036] Interpolate the set composed of at least one target data point based on the cubic non-rational B-spline curve interpolation method to obtain a center line point set.

[0037] In some embodiments, preprocess the first image at the first frame moment to obtain a target binary image, including:

[0038] Perform filtering processing on the first image based on a preset filtering algorithm to obtain a third image;

[0039] Perform binarization processing on the third image to obtain a fourth image;

[0040] Perform processing on the fourth image based on morphological operations to obtain a target binary image.

[0041] In a second aspect, the present application provides a point cloud stitching system based on multi-sensors, including:

[0042] An acquisition module, configured to acquire measurement data at all frame moments during the process of scanning an object, where the measurement data includes a first image, a second image, an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset marker captured by an image acquisition device;

[0043] A processing module, configured to perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments are all transformed to the coordinates in a second coordinate system in the first coordinate system, the first coordinate system is the coordinate system where the image acquisition device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0044] A stitching module, configured to perform point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system;

[0045] Wherein, the transformation process includes:

[0046] Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame moment, obtain the first coordinates of each point in the center line point set corresponding to the first frame moment in the first coordinate system, and the first frame moment is any frame moment;

[0047] Based on the first coordinates and the target homogeneous transformation matrix, transform the first coordinates to the second coordinates in the second coordinate system of the second frame moment, the target homogeneous transformation matrix is the homogeneous transformation matrix between the coordinates in the first coordinate system from the first frame moment to the second frame moment to the coordinates in the second coordinate system, the target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame moment and the second frame moment, and the second frame moment is other frame moments except the first frame moment.

[0048] In a third aspect, the present application provides an / image signal generator / network device / transmitter / terminal / base station / industrial computer, including: at least one memory, configured to store a program; at least one processor, configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any of some embodiments of the first aspect.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor is caused to execute the method described in the first aspect or any of some embodiments of the first aspect.

[0050] In a fifth aspect, the present application provides a computer program product, and when the computer program product runs on a processor, the processor is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0051] Generally speaking, compared with the prior art, the above technical solutions conceived by this application have the following beneficial effects:

[0052] The point cloud stitching method and system based on multiple sensors provided by this application extract and fuse the data of multiple sensors including cameras, monocular line laser scanners, and inertial measurement units, construct a high-precision target homogeneous transformation matrix, and combine this target homogeneous transformation matrix to transform the coordinates of the center line point set of the first image at each frame time in the first coordinate system to the coordinates in the second coordinate system, and perform point cloud stitching on the coordinates of the center line point set of the first image at each frame time in the second coordinate system, avoiding the problem of inaccurate stitching caused by the low coincidence degree of the monocular line laser scan point cloud, and improving the point cloud stitching accuracy and the accuracy of reconstruction. Description of the Drawings

[0053] Figure 1 is a schematic flowchart of the point cloud stitching method based on multiple sensors provided by an embodiment of this application;

[0054] Figure 2 is a schematic structural diagram of the data measurement system provided by an embodiment of this application;

[0055] Figure 3 is a schematic diagram of the coordinate system of the data measurement system constructed by an embodiment of this application;

[0056] Figure 4 is a schematic structural diagram of the point cloud stitching system based on multiple sensors provided by an embodiment of this application;

[0057] Figure 5 is a schematic structural diagram of the electronic device provided by an embodiment of this application. Detailed Embodiments

[0058] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0059] The term "and / or" in this article is an association relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article represents an "or" relationship between associated objects. For example, A / B represents A or B.

[0060] The terms "first", "second", etc. in the description and claims of this document are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first image, the second image, etc. are used to distinguish different images, rather than to describe a specific order of the images.

[0061] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] To avoid the problem of inaccurate point cloud stitching caused by low overlap of monocular line laser scan point clouds, many scholars choose to use external sensors to obtain the transformation matrix of the laser point cloud. Among them, the laser tracker can provide a high-precision transformation matrix, but it is expensive and not easy to carry; the Inertial Measurement Unit (IMU) can obtain high-precision data through integration, but due to its inherent zero drift problem, it is difficult to ensure long-term use; the monocular vision sensor is a relatively economical choice, and it has no cumulative error in the process of solving the transformation matrix, but the single precision is relatively low.

[0063] Based on this, an embodiment of the present application provides a multi-sensor based point cloud stitching method and system. By using a monocular line laser scanner, a camera, and an IMU, measurement data at all frame times during the process of scanning an object is obtained. Among them, the monocular line laser scanner, the camera, and the IMU obtain data synchronously to obtain a first image, a second image containing a preset marker, an acceleration measurement value, and an angular velocity measurement value. The first image is preprocessed, and an improved gray center of gravity algorithm is used to roughly extract the center line of the scanned laser stripe to obtain a candidate center line point set. After processing the candidate center line point set, its corresponding center line point set is obtained. Based on the internal parameters of the monocular line laser scanner and the pixel coordinates of the points in the center line point set, its three-dimensional coordinates in the coordinate system where the camera is located are calculated. Euler integration is performed on the acceleration measurement value and the angular velocity measurement value measured by the IMU to construct a pose increment matrix of the monocular line laser scanner; the ChArUco system is used to enhance ChArUco marker detection and pose estimation, and the transformation matrix between key frame times is calculated; the trained Variational Auto-Encoder (VAE) model is used to fuse the above data obtained by multiple sensors to obtain the target homogeneous transformation matrix between key frame times; the three-dimensional coordinates of the points in the center line point set in the coordinate system where the camera is located are frame-by-frame transformed to the camera coordinate system where the monocular line laser scanner is located using the target homogeneous transformation matrix between all frame times during the process of scanning the object, and the point cloud is stitched. In the embodiment of the present application, the preset marker uses a ChArUco marker.

[0064] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

[0065] See Figure 1 , an embodiment of the present application provides a multi-sensor based point cloud stitching method, including: step 110, step 120, and step 130.

[0066] Step 110, obtain measurement data at all frame times during the process of scanning an object. The measurement data includes a first image, a second image, the angular velocity measurement value and the acceleration measurement value of the object measured by an inertial measurement unit. The first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset marker captured by a camera;

[0067] Step 120, perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame times are converted to the coordinates in the second coordinate system. The first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0068] Step 130, perform point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame times in the second coordinate system;

[0069] Among them, the transformation process includes:

[0070] Based on the pixel coordinates of each point in the center line point set corresponding to the first frame moment, obtain the first coordinates of each point in the center line point set corresponding to the first frame moment in the first coordinate system, and the first frame moment is any frame moment;

[0071] Based on the first coordinates and the target homogeneous transformation matrix, convert the first coordinates to the second coordinates in the second coordinate system at the second frame moment. The target homogeneous transformation matrix is the homogeneous transformation matrix between the coordinates in the first coordinate system from the first frame moment to the second frame moment and the coordinates in the second coordinate system. The target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame moment and the second frame moment, and the second frame moment is other frame moments except the first frame moment.

[0072] In specific implementation, adopt as Figure 2 shown in the data measurement system to collect the measurement data of all frame moments included in the process of scanning the object. This measurement system consists of a monocular line laser scanner, an imaging device (such as a camera), an IMU, and multiple preset marker boards. Among them, the monocular line laser scanner is composed of a camera and a laser generator. In the embodiments of the present application, the preset marker board contains a preset marker, and this preset marker board adopts a ChArUco marker board containing ChArUco markers.

[0073] Please further refer to Figure 3 , establish a coordinate system for the data measurement system, where {W} represents the world coordinate system, {C} represents the coordinate system where the camera is located, that is, the first coordinate system, {C′} represents the camera coordinate system where the monocular line laser scanner is located, that is, the second coordinate system, and {A} represents the coordinate system where the ChArUco marker is located.

[0074] When starting to scan the object, ensure that there is at least one ChArUco marker in the camera's field of view initially. When the monocular line laser scanner scans the object, the synchronization accuracy of the first image of the object obtained by the line monocular laser scanner, the second image containing the ChArUco marker obtained by the camera, and the acceleration measurement value and angular velocity measurement value of the object measured by the IMU can be improved through the following steps, and errors caused by picture misalignment can be avoided.

[0075] Specifically, use the python function to create multiple threads to improve the shooting synchronization accuracy of the monocular line laser scanner and the camera;

[0076] Obtain the time stamp, use the same time stamp to name the images to avoid image confusion, and record the corresponding time stamp when storing the data measured by the corresponding IMU. Among them, different time stamps represent different frame moments.

[0077] Different from the monocular camera which has no cumulative error but poor accuracy in the process of solving the transformation matrix, the embodiment of the present application uses an IMU, which can obtain a high-precision pose transformation matrix in a short period of time. By fusing multi-sensor data, the zero-drift problem existing in the IMU is solved, effectively making up for the shortcomings of both, and reducing the cost while ensuring the accuracy.

[0078] Perform the following transformation process at least once until the coordinates of each point in the centerline point set of the first image corresponding to all frame times are all converted to the coordinates in the second coordinate system in the first coordinate system.

[0079] According to the coordinates of each point in the centerline point set of the first image corresponding to all frame times obtained in the second coordinate system, perform point cloud stitching. Wherein, the transformation process may specifically include:

[0080] For any two frame times: the first frame time and the second frame time.

[0081] According to the pixel coordinates (u, v) of each point in the centerline point set corresponding to the first image obtained by the monocular line laser scanner at the first frame time, calculate the three-dimensional coordinates (x C , y C , z C ) of each point in the centerline set corresponding to the first frame time in the coordinate system where the camera is located, that is, the first coordinate.

[0082] Specifically, assume that the camera internal parameter matrix K of the monocular line laser scanner is:

[0083]

[0084] Among them, f x is the ratio factor of the focal length of the camera in the horizontal direction (x-axis direction) to the pixel, f y is the ratio factor of the focal length of the camera in the vertical direction (y-axis direction) to the pixel, c x is the position of the center point on the camera imaging plane in the horizontal direction, c y is the position of the center point on the camera imaging plane in the vertical direction.

[0085] Convert the pixel coordinates (u, v) of each point in the centerline point set to the coordinates (x, y) in the normalized coordinate system {C} where the camera is located:

[0086]

[0087] Use the known laser plane equation to calculate the depth information:

[0088] Solve the depth information r as: Calculate the three-dimensional coordinates (x C , y C , z C ) of each point in the centerline point set in the coordinate system of the camera using the obtained depth information, which are respectively: z C = r. Where a, b, c, and d are all preset coefficients.

[0089] According to the target homogeneous transformation matrix, transform the first coordinate at the first frame time to the coordinate in the coordinate system of the monocular line laser scanner at the second frame time, that is, the second coordinate.

[0090] It should be noted that the target homogeneous transformation matrix from the first frame time to the second frame time is obtained by fusing the measurement data acquired by each sensor including the camera, IMU, and monocular line laser scanner at the first frame time and the second frame time. The target homogeneous transformation matrix is the homogeneous transformation matrix from the coordinate in the coordinate system of the camera to the coordinate in the coordinate system of the monocular line laser scanner between the first frame time and the second frame time, and is used to represent the conversion relationship from the coordinate in the coordinate system of the camera to the coordinate in the coordinate system of the monocular line laser scanner between the first frame time and the second frame time.

[0091] It should be noted that the multi-sensor based point cloud stitching method provided in the embodiments of the present application solves the problems of small overlap of line laser point clouds and large ICP registration errors by fusing multi-sensor data, and can be widely applied to various 3D reconstruction and measurement tasks, improving the efficiency and accuracy of industrial inspection and measurement.

[0092] The multi-sensor based point cloud stitching method provided in the embodiments of the present application extracts and fuses the data of multiple sensors including a camera, a monocular line laser scanner, and an inertial measurement unit, constructs a high-precision target homogeneous transformation matrix, combines the target homogeneous transformation matrix, converts the coordinates of the centerline point set of the first image at each frame time in the first coordinate system to the coordinates in the second coordinate system, and performs point cloud stitching on the coordinates of the centerline point set of the first image at each frame time in the second coordinate system, avoiding the problem of inaccurate stitching caused by low overlap of monocular line laser scan point clouds, and improving the point cloud stitching accuracy and the accuracy of reconstruction.

[0093] Further, in some embodiments, the acquisition method of the centerline point set of the first image corresponding to the first frame time may include:

[0094] Determine the candidate centerline point set according to the centroid of the laser stripe in the target binary image, where the target binary image is the binary image containing the laser stripe corresponding to the first image corresponding to the first frame time;

[0095] Determine the center line point set according to the candidate center line point set.

[0096] In a specific implementation, according to the first image at the first frame obtained by the monocular line laser scanner, obtain a binary image containing laser stripes corresponding to the first image at the first frame.

[0097] Use this binary image as the target binary image, and obtain the candidate center line point set according to the centroid of the laser stripe in the target binary image.

[0098] Obtain the center line point set according to the candidate center line point set.

[0099] Furthermore, in some embodiments, determining the candidate center line point set according to the centroid of the laser stripe in the target binary image may include:

[0100] Obtain the first centroid and the longitudinal centroid point set of each row of pixels in the horizontal direction in the target binary image respectively. The longitudinal centroid point set is a set composed of the second centroids of each column of pixels in the target binary image in the vertical direction;

[0101] Obtain the target second centroid in the longitudinal centroid point set that is closest to each first centroid;

[0102] Obtain the candidate center line point set according to each first centroid and the target second centroid corresponding to each first centroid respectively.

[0103] In a specific implementation, preprocess the first image at the first frame obtained by the monocular line laser scanner to obtain the target binary image.

[0104] Use the improved gray centroid algorithm to roughly extract the center line of the scanned laser stripe, and there is:

[0105] Use the light intensity weighting of the image pixel points to calculate the centroid of the η-th row of pixels in the horizontal direction in the target binary image (η, y η )(i.e., the first centroid) and the centroid of the β-th column of pixels in the vertical direction (x β , β)(i.e., the second centroid), that is, the second centroid, where η is the number of pixel rows of the target binary image, and β is the number of pixel columns of the target binary image, as shown in the following formula:

[0106]

[0107] Among them, γ(α, β) is the pixel value of the α-th row and β-th column in the target binary image, n is the total number of pixel columns of the target binary image, and m is the total number of pixel rows of the target binary image.

[0108] The set composed of the second centroids of each column of pixels in the vertical direction obtained from the target binary image is the longitudinal centroid point set (Xβ , β).

[0109] For the centroid of the pixels in the η-th row of the target binary image in the horizontal direction (η, y η ), calculate the distance between it and each data point in the set of vertical centroid points (X β , β), and find the second centroid closest to (η, y β ) from the set of vertical centroid points (X η ). This second centroid is the target second centroid, assumed to be (x' β , y' β ), as shown in the following formula:

[0110]

[0111] Similarly, the minimum value of the distances between the centroids of the pixels in the remaining rows of the target binary image in the horizontal direction and each data point in the set of vertical centroid points (X β ) can be calculated.

[0112] Calculate the midpoint of (η, y η ) and (x' β , y' β ) to obtain the data point (x′, y′) in the candidate centerline point set Ω, specifically as shown in the following formula:

[0113]

[0114] Similarly, the remaining data points in the candidate centerline point set Ω can be calculated.

[0115] Furthermore, in some embodiments, determining the centerline point set according to the candidate centerline point set may include:

[0116] Obtain the unit normal vectors of each data point in the candidate centerline point set;

[0117] According to the angle between adjacent unit normal vectors, obtain at least one target data point with an angle greater than a preset threshold from the candidate centerline point set;

[0118] Based on the cubic non-rational B-spline curve interpolation method, interpolate the set composed of at least one target data point to obtain the centerline point set.

[0119] In a specific implementation, for global smoothing of the candidate centerline point set extracted using the improved gray centroid method, there is:

[0120] Calculate the unit normal vectors N1 = {n1, n2,..., n of each data point in the candidate centerline point set Ω m}, where n1 is the unit normal vector of the first data point in the candidate centerline point set Ω, and n1 = n2, n2 ~ n m are the unit normal vectors of the second data point to the m-th data point in the candidate centerline point set Ω, respectively, and n m = n m-1 . As shown in the following formula:

[0121] n ε = k ε / ||k ε ||, ε = 2, 3, 4, …, m - 1

[0122] where k ε is the curvature at the ε-th data point p ε in the candidate centerline point set Ω, and n ε is the unit normal vector at the data point p ε .

[0123] Calculate the angle θ m between adjacent unit normal vectors n ε and n ε-1 in the unit normal vector N1 = {n1, n2, …, n ε , compare the size relationship between this angle θ ε and the preset threshold. If θ ε is greater than the preset threshold, then the data point p ε corresponding to n ε is used as the target data point. As shown in the following formula:

[0124]

[0125] Similarly, the angles between other adjacent unit normal vectors can be calculated. By comparing their size relationships with the preset threshold, all target data points with angles greater than the preset threshold can be found.

[0126] Use the cubic non-rational B-spline curve interpolation method to interpolate the set composed of these target data points to obtain the centerline point set Ω L . Among them, the number of data points in the centerline point set Ω L can be flexibly adjusted according to needs.

[0127] Furthermore, in some embodiments, preprocessing the first image at the first frame moment to obtain the target binary image may include:

[0128] Based on the preset filtering algorithm, filter the first image to obtain the third image;

[0129] Perform binarization processing on the third image to obtain the fourth image;

[0130] Based on morphological operations, the fourth image is processed to obtain a target binary image.

[0131] In a specific implementation, a preset filtering algorithm is used to filter the first image to remove image noise in the first image, and the obtained image is the third image. In the embodiments of the present application, the preset filtering algorithm may be used alone, such as Gaussian filtering, median filtering, etc., or may be used in combination with each other, and the embodiments of the present application do not limit this.

[0132] The obtained third image is binarized to convert the third image into black and white, and the obtained image is the fourth image.

[0133] Using morphological operations, the obtained fourth image is processed by dilation, erosion, opening operation, and closing operation, and connected component screening is performed to obtain a target binary image containing laser stripes.

[0134] Further, in some embodiments, the method for obtaining the target homogeneous transformation matrix may include:

[0135] Obtaining the angular velocity measurement value and acceleration measurement value of the object corresponding to all frame times during the process of scanning the object;

[0136] Based on the angular velocity measurement value and acceleration measurement value, obtaining the pose increment information from the first frame time to the second frame time;

[0137] Based on the pose increment information, obtaining the first transformation matrix from the first frame time to the second frame time;

[0138] Based on the second image corresponding to the first frame time, obtaining the third coordinate of the preset marker in the first coordinate system at the first frame time;

[0139] Based on the second image corresponding to the second frame time, obtaining the fourth coordinate of the preset marker in the first coordinate system at the second frame time;

[0140] According to the third coordinate and the fourth coordinate, determining the second transformation matrix from the first frame time to the second frame time;

[0141] According to the first transformation matrix and the second transformation matrix, determining the target homogeneous transformation matrix.

[0142] In a specific implementation, the Euler integral is performed on the acceleration measurement value and angular velocity measurement value measured by the IMU to obtain the pose increment information between two key frame times of the camera, and a pose increment matrix of the monocular line laser scanner is constructed, and this pose increment matrix is the first transformation matrix. In the embodiments of the present application, any two frame times are used as key frames.

[0143] Specifically, calculate the rotation increment from the first frame time (e.g., the i-th frame time) to the second frame time (e.g., the j-th frame time) based on the angular velocity measurement values and acceleration measurement values of the object corresponding to all frame times during the process of scanning the object.

[0144]

[0145] where {W} is the world coordinate system and {I} is the coordinate system where the IMU is located. is the angular velocity measurement value at the k-th frame time, and Δt is the time step.

[0146] According to the rotation increment and the acceleration measurement value at the i-th frame time calculate the velocity increment of the IMU from the i-th frame time to the j-th frame time

[0147]

[0148] where is the acceleration measurement value at the k-th frame time.

[0149] According to the velocity increment and the rotation increment calculate the displacement increment from the i-th frame time to the j-th frame time

[0150]

[0151] Take the rotation increment and the displacement increment as the pose increment information from the i-th frame time to the j-th frame time, and construct the transformation matrix from the i-th frame time to the j-th frame time according to this pose increment information and for this transformation matrix take the inverse to obtain the first transformation matrix from the i-th frame time to the j-th frame time.

[0152]

[0153] where T1 is the first transformation matrix from the i-th frame time to the j-th frame time.

[0154] Use the ChArUco system to enhance the ChArUco marker detection and pose estimation, and calculate the pose transformation matrix T2 from the i-th frame time to the j-th frame time. Specifically:

[0155] Input the second image containing the ChArUco marker at the first frame moment (e.g., the i-th frame moment) collected into the pre-trained Deep ChArUco network for feature extraction and key point (i.e., each corner point of the ChArUco marker) detection, and obtain the pixel coordinates corresponding to the preliminary key points and the QR code information of the ChArUco marker. Use the RefineNet network to refine the detected corner coordinates to obtain the pixel coordinates of the key points at the sub-pixel level. And use the solvePnP function in the OpenCV library, input the corner coordinates detected by the RefineNet network, the arrangement order of the three-dimensional information of the ChArUco marker, and the size of the ChArUco marker, and calculate the three-dimensional coordinates T of the ChArUco marker in the camera coordinate system C at the i-th frame moment i C , and this three-dimensional coordinate T i C is the third coordinate.

[0156] Based on the same method, obtain the three-dimensional coordinates of the ChArUco marker in the camera coordinate system C at the j-th frame moment This three-dimensional coordinate is the fourth coordinate.

[0157] Use homogeneous transformation matrix operation to calculate the pose transformation matrix T2 between this third coordinate T i C and the fourth coordinate , and this pose transformation matrix is the second transformation matrix.

[0158]

[0159] Among them, is obtained by taking the inverse of the third coordinate T i C .

[0160] According to the obtained first transformation matrix T1 and second transformation matrix T2, obtain the target homogeneous transformation matrix T.

[0161] Furthermore, in some embodiments, determining the target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix may include:

[0162] Input the first transformation matrix and the second transformation matrix into the target model to obtain the output data of the target model;

[0163] Determine the target homogeneous transformation matrix according to the output data;

[0164] Among them, the acquisition method of the target model includes:

[0165] Input the training data into a preset model for training until the loss function of the preset model converges. The training data includes a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix;

[0166] Determine the target model according to the converged preset model.

[0167] In a specific implementation, in the embodiments of the present application, the trained target model is used to fuse the first transformation matrix T1 and the second transformation matrix T2 obtained by multiple sensors to obtain the target homogeneous transformation matrix T between any two frame moments. In the embodiments of the present application, the trained VAE model is used as the target model.

[0168] Preprocess the transformation matrices (T1, T2) into a form suitable for input to the VAE model, flatten them from 4x4 matrices into 16-dimensional vectors, and combine them. The encoder maps the input data (T1, T2) to the latent space, and uses a two-layer non-linear network to reconstruct the input data (T1, T2) to obtain the latent variable z, which follows the following distribution function:

[0169]

[0170] where μ represents the expectation and σ represents the variance.

[0171] The encoder network consists of two connected layers, uses the ReLU activation function, and outputs two 16-dimensional branches, namely the mean branch f μ (T1, T2) and the log variance branch Use the log variance σ 2 =exp(logσ 2 ) to calculate the variance σ 2 . Use the reparameterization trick z = μ + σ ⊙ ∈ to sample the latent variable z from the normal distribution, where, is a random variable sampled from the standard normal distribution.

[0172] The three-layer network decoder performs a non-linear transformation on the latent variable, maps the reconstructed latent variable z back to the original data space and outputs a 16-dimensional vector, and folds this vector into a 4x4 transformation matrix to obtain the target homogeneous transformation matrix.

[0173] It should be noted that the target model can be obtained in the following way:

[0174] Input the training data into a preset model for training until the loss function of the preset model converges. The training data includes a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix; Obtain the target model according to the converged preset model. In the embodiments of the present application, the preset model uses the VAE model.

[0175] Input the preset homogeneous transformation matrix obtained in the simulation (the true value T′ of the obtained target homogeneous transformation matrix T can be used as the preset homogeneous transformation matrix) and the data (T1, T2) with added noise data (such as Gaussian noise) into the VAE model, and calculate the loss function The formula is as follows:

[0176]

[0177] Among them, q(z|T1,T2) represents the posterior distribution of the latent variable, p(z) represents the prior distribution of the latent variable, and follows Represents the expected value of the error of the latent variable z reconstructed from T′, D KL (q(z|T1,T2)||p(z)) represents the divergence between the posterior distribution and the prior distribution.

[0178] Use the gradient descent method to optimize the loss function, calculate the loss function and perform backpropagation to update the VAE model parameters until the loss function converges.

[0179] Use the target homogeneous transformation matrix T to convert the three-dimensional coordinates of each point in the centerline point set frame by frame to the coordinate system of the monocular linear laser scanner, and splice the point cloud.

[0180] Assume that the set of three-dimensional coordinates of each point in the centerline point set corresponding to the j-th frame moment in the coordinate system of the camera is (X Cj , Y Cj , Z Cj ). Among them, the homogeneous representation of the coordinates of the o-th point is [x Cj , y Cj , z Cj , 1] T , and use the target homogeneous transformation matrix T to convert (X Cj , Y Cj , Z Cj ) to the coordinate system of the monocular line laser scanner at the i-th frame moment: [x Ci , y Ci , z Ci , 1] T = T[x Cj , y Cj , z Cj , 1] T . Sequentially convert the three-dimensional coordinates of each point in the centerline point set at all frame moments to the coordinate system of the monocular line laser scanner, and complete the point cloud splicing.

[0181] The point cloud stitching method based on multiple sensors provided by the embodiments of the present application can effectively process various sensor data, including IMU and monocular line laser scanners, by using a VAE model. By using the VAE model to fuse the multiple sensor data, a high-precision homogeneous transformation matrix can be generated, reducing the error of the point cloud with less overlapping parts during the stitching process and improving the accuracy of the final reconstruction.

[0182] The point cloud stitching system based on multiple sensors provided by the present invention will be described below. The point cloud stitching system based on multiple sensors described below can be referred to correspondingly with the point cloud stitching method described above.

[0183] Please further refer to Figure 4 , the point cloud stitching system based on multiple sensors provided by the embodiments of the present application includes: an acquisition module 410, a processing module 420, and a stitching module 430.

[0184] The acquisition module 410 is configured to acquire the measurement data at all frame times during the process of scanning an object. The measurement data includes a first image, a second image, the angular velocity measurement value and the acceleration measurement value of the object measured by an inertial measurement unit. The first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset marker captured by a camera.

[0185] The processing module 420 is configured to perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame times are all transformed to the coordinates in the second coordinate system. The first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located.

[0186] The stitching module 430 is configured to perform point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame times in the second coordinate system.

[0187] Wherein, the transformation process includes:

[0188] Based on the pixel coordinates of each point in the center line point set corresponding to the first frame time, obtain the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system. The first frame time is any frame time.

[0189] Based on the first coordinates and the target homogeneous transformation matrix, transform the first coordinates to the second coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is the homogeneous transformation matrix between the coordinates in the first coordinate system from the first frame time to the second frame time and the coordinates in the second coordinate system. The target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame time and the second frame time. The second frame time is other frame times except the first frame time.

[0190] The point cloud stitching system based on multiple sensors provided by the embodiments of the present application extracts and fuses the data of multiple sensors including a camera, a monocular line laser scanner, and an inertial measurement unit, constructs a high-precision target homogeneous transformation matrix, and combines the target homogeneous transformation matrix to convert the coordinates of the center line point set of the first image at each frame time in the first coordinate system to the coordinates in the second coordinate system, and performs point cloud stitching on the coordinates of the center line point set of the first image at each frame time in the second coordinate system, avoiding the problem of inaccurate stitching caused by low coincidence degree of the monocular line laser scan point cloud, and improving the point cloud stitching accuracy and the accuracy of reconstruction.

[0191] It can be understood that for the detailed function implementation of each of the above units / modules, reference can be made to the introduction in the foregoing method embodiments, which will not be elaborated here.

[0192] It should be understood that the above device is used to execute the method in the above embodiments. For the corresponding program modules in the device, the implementation principle and technical effects are similar to those described in the above method. The working process of the device can refer to the corresponding process in the above method, which will not be elaborated here.

[0193] Based on the method in the above embodiments, the embodiments of the present application provide an electronic device, as Figure 5 shown. The electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute the method in the above embodiments.

[0194] In addition, when the logical instructions in the above memory 530 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0195] Based on the method in the above embodiments, the embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on a processor, the processor is caused to execute the method in the above embodiments.

[0196] Based on the method in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, it causes the processor to execute the method in the above embodiments.

[0197] It can be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0198] The method steps in the embodiments of the present application may be implemented in a hardware manner or by a processor executing software instructions. The software instructions may consist of corresponding software modules. The software modules may be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0200] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0201] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-sensor-based point cloud stitching method, characterized in that Including: Obtaining measurement data at all frame moments during the process of scanning an object, where the measurement data includes a first image, a second image, an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit. The first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset marker captured by an imaging device; Performing at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments are all transformed to the coordinates in a second coordinate system under the first coordinate system. The first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located; Performing point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system; Wherein, the transformation process includes: Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame moment, obtaining the first coordinates of each point in the center line point set corresponding to the first frame moment in the first coordinate system, where the first frame moment is any frame moment; Based on the first coordinates and a target homogeneous transformation matrix, transforming the first coordinates to the second coordinates in the second coordinate system of the second frame moment. The target homogeneous transformation matrix is the homogeneous transformation matrix between the coordinates in the first coordinate system from the first frame moment to the second frame moment and the coordinates in the second coordinate system. The target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame moment and the second frame moment, and the second frame moment is other frame moments except the first frame moment; The obtaining method of the target homogeneous transformation matrix includes: Obtaining the angular velocity measurement value and the acceleration measurement value of the object corresponding to all frame moments during the process of scanning the object; Based on the angular velocity measurement value and the acceleration measurement value, obtaining the pose increment information from the first frame moment to the second frame moment; Based on the pose increment information, obtaining the first transformation matrix from the first frame moment to the second frame moment; Based on the second image corresponding to the first frame moment, obtaining the third coordinates of the preset marker at the first frame moment in the first coordinate system; Based on the second image corresponding to the second frame moment, obtaining the fourth coordinates of the preset marker at the second frame moment in the first coordinate system; Determining the second transformation matrix from the first frame moment to the second frame moment according to the third coordinates and the fourth coordinates; Determining the target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix.

2. The multi-sensor-based point cloud stitching method according to claim 1, wherein The obtaining method of the center line point set of the first image corresponding to the first frame moment includes: Determining a candidate center line point set according to the centroid of the laser stripe in a target binary image, where the target binary image is a binary image containing the laser stripe corresponding to the first image corresponding to the first frame moment; Determining the center line point set according to the candidate center line point set.

3. The multi-sensor based point cloud stitching method according to claim 2, characterized in that, The determining a candidate center line point set according to the centroid of the laser stripe in the target binary image includes: Obtain the first centroid and the longitudinal centroid point set of each row of pixels in the horizontal direction of the target binary image respectively, where the longitudinal centroid point set is a set composed of the second centroids of each column of pixels in the vertical direction of the target binary image; Obtain the target second centroid in the longitudinal centroid point set that is closest to each first centroid; Obtain a candidate center line point set according to each first centroid and the target second centroid corresponding to each first centroid respectively.

4. The multi-sensor-based point cloud stitching method according to claim 1, wherein, The determining the target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix includes: Input the first transformation matrix and the second transformation matrix into a target model to obtain the output data of the target model; Determine the target homogeneous transformation matrix according to the output data; Among them, the obtaining method of the target model includes: Input training data into a preset model for training until the loss function of the preset model converges. The training data includes a preset homogeneous transformation matrix, noise data, the first transformation matrix, and the second transformation matrix; Determine the target model according to the converged preset model.

5. The multi-sensor based point cloud stitching method according to claim 2, wherein The determining the center line point set according to the candidate center line point set includes: Obtain the unit normal vector of each data point in the candidate center line point set; According to the angle between adjacent unit normal vectors, obtain at least one target data point in the candidate center line point set whose angle is greater than a preset threshold; Interpolate the set composed of the at least one target data point based on the cubic non-rational B-spline curve interpolation method to obtain the center line point set.

6. The multi-sensor-based point cloud stitching method according to claim 2, characterized in that, Preprocess the first image at the first frame time to obtain the target binary image, including: Perform filtering processing on the first image based on a preset filtering algorithm to obtain a third image; Perform binarization processing on the third image to obtain a fourth image; Process the fourth image based on morphological operations to obtain the target binary image.

7. A multi-sensor-based point cloud stitching system, characterized in that, It includes: An acquisition module, configured to acquire measurement data at all frame times during the process of scanning an object. The measurement data includes a first image, a second image, the angular velocity measurement value and the acceleration measurement value of the object measured by an inertial measurement unit. The first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset mark captured by an imaging device; A processing module, configured to perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame times are all converted to the coordinates in the second coordinate system. The first coordinate system is the coordinate system where the camera is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located; A stitching module, configured to perform point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame times in the second coordinate system; Among them, the transformation process includes: Based on the pixel coordinates of each point in the center line point set corresponding to the first frame moment, obtain the first coordinates of each point in the center line point set corresponding to the first frame moment in the first coordinate system, where the first frame moment is any frame moment; Based on the first coordinates and the target homogeneous transformation matrix, convert the first coordinates to the second coordinates in the second coordinate system at the second frame moment. The target homogeneous transformation matrix is the homogeneous transformation matrix between the coordinates in the first coordinate system from the first frame moment to the second frame moment to the coordinates in the second coordinate system. The target homogeneous transformation matrix is determined according to the measurement data corresponding to the first frame moment and the second frame moment, and the second frame moment is other frame moments except the first frame moment; The obtaining method of the target homogeneous transformation matrix includes: Obtain the angular velocity measurement values and acceleration measurement values of the object corresponding to all frame moments during the scanning of the object; Based on the angular velocity measurement values and the acceleration measurement values, obtain the pose increment information from the first frame moment to the second frame moment; Based on the pose increment information, obtain the first transformation matrix from the first frame moment to the second frame moment; Based on the second image corresponding to the first frame moment, obtain the third coordinates of the preset mark at the first frame moment in the first coordinate system; Based on the second image corresponding to the second frame moment, obtain the fourth coordinates of the preset mark at the second frame moment in the first coordinate system; According to the third coordinates and the fourth coordinates, determine the second transformation matrix from the first frame moment to the second frame moment; According to the first transformation matrix and the second transformation matrix, determine the target homogeneous transformation matrix.

8. An electronic device, characterized in that, It includes: At least one memory for storing a computer program; At least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program runs on the processor, the processor is caused to execute the method according to any one of claims 1-6.

Images