Point cloud splicing method and system based on multiple sensors

Through multi-sensor data fusion, a high-precision target homogeneous transformation matrix is ​​constructed, which solves the problem of low splicing accuracy caused by low overlap of point clouds in monocular laser scanning, and achieves more efficient point cloud splicing and reconstruction accuracy.

CN119067843AActive Publication Date: 2024-12-03HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202411094065.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The problem of low overlap of point clouds caused by the monocular laser scanning point clouds is difficult to effectively solve the problem of low point cloud splicing accuracy.

Method used

The multi-sensor data fusion method is adopted, including a monocular laser scanner, a camera and an inertial measurement unit. By acquiring and processing the measurement data at each frame time, a high-precision target homogeneous transformation matrix is ​​constructed to achieve accurate splicing of point clouds.

Benefits of technology

The point cloud splicing accuracy and reconstruction accuracy are improved, and the splicing inaccuracy caused by low overlap of point clouds with monocular laser scanning is avoided, which reduces costs and improves the efficiency of industrial detection and measurement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067843A_ABST
    Figure CN119067843A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of vision measurement, and particularly discloses a point cloud splicing method and system based on multiple sensors. According to the point cloud splicing method based on the multiple sensors, the high-precision target homogeneous transformation matrix is constructed by extracting and fusing the data of the multiple sensors including the camera, the monocular laser scanner and the inertial measurement unit, and the point cloud splicing method based on the multiple sensors is obtained by combining the target homogeneous transformation matrix. Converting coordinates of the center line point set of the first image in the first coordinate system at each frame moment into coordinates in a second coordinate system, and performing point cloud splicing on the coordinates of the center line point set of the first image in the second coordinate system at each frame moment, the problem of inaccurate splicing caused by low contact ratio of monocular laser scanning point clouds is avoided, and the point cloud splicing precision and the reconstruction accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of visual measurement technology, and more specifically, to a point cloud stitching method and system based on multiple sensors. Background Art

[0002] The point cloud obtained by scanning an object with a monocular line laser scanner has the characteristic of less overlap. When using the traditional closest point search method (Iterative Closest Point, ICP) to solve the transformation matrix, there is a problem of low accuracy. This method will gradually accumulate errors in the process of processing line laser point clouds, resulting in low splicing accuracy.

[0003] Therefore, how to avoid the problem of low point cloud stitching accuracy caused by low overlap of monocular line laser scanning point clouds is an urgent problem that needs to be solved. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a point cloud stitching method and system based on multiple sensors, aiming to solve the problem of low point cloud stitching accuracy caused by low overlap of monocular line laser scanning point clouds.

[0005] To achieve the above objectives, in a first aspect, the present application provides a point cloud stitching method based on multiple sensors, comprising:

[0006] Acquire measurement data of all frame moments in the process of scanning the object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image being an image of the object scanned by a monocular line laser scanner, and the second image being an image containing a preset mark taken by an imaging device;

[0007] Perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, where the first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0008] Perform point cloud stitching according to the coordinates of the center line point set of the first image corresponding to all frame moments in the second coordinate system;

[0009] The transformation process includes:

[0010] Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, where the first frame time is any frame time;

[0011] Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is ​​a homogeneous transformation matrix from the coordinates in the first coordinate system from the first frame time to the second frame time to the coordinates in the second coordinate system. The target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time. The second frame time is the frame time other than the first frame time.

[0012] In some embodiments, a method for acquiring each point in the center line point set of the first image corresponding to the first frame time includes:

[0013] Determine a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image, where the target binary image is a binary image containing the laser stripes corresponding to the first image corresponding to the first frame moment;

[0014] A centerline point set is determined based on the candidate centerline point set.

[0015] In some embodiments, determining a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image includes:

[0016] Obtain the first centroid of each row of pixels in the target binary image in the horizontal direction and the longitudinal centroid point set, the longitudinal centroid point set is a set of the second centroid of each column of pixels in the target binary image in the vertical direction;

[0017] Obtain the second center of gravity of the target that is closest to each first center of gravity in the longitudinal center of gravity point concentration;

[0018] A candidate centerline point set is obtained according to each first center of gravity and the target second center of gravity corresponding to each first center of gravity.

[0019] In some embodiments, the target homogeneous transformation matrix is ​​obtained by:

[0020] Obtaining the angular velocity measurement value and acceleration measurement value of the object corresponding to all frame moments in the process of scanning the object;

[0021] Based on the angular velocity measurement value and the acceleration measurement value, the incremental posture information from the first frame time to the second frame time is obtained;

[0022] Based on the position and posture increment information, a first transformation matrix from the first frame time to the second frame time is obtained;

[0023] Based on the second image corresponding to the first frame time, obtaining the third coordinate of the preset mark at the first frame time in the first coordinate system;

[0024] Based on the second image corresponding to the second frame time, obtaining the fourth coordinate of the preset mark at the second frame time in the first coordinate system;

[0025] Determine a second transformation matrix from the first frame time to the second frame time according to the third coordinate and the fourth coordinate;

[0026] A target homogeneous transformation matrix is ​​determined according to the first transformation matrix and the second transformation matrix.

[0027] In some embodiments, determining a target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix includes:

[0028] Inputting the first transformation matrix and the second transformation matrix into the target model to obtain output data of the target model;

[0029] According to the output data, determine the target homogeneous transformation matrix;

[0030] The target model is obtained by:

[0031] Inputting training data into a preset model for training until a loss function of the preset model converges, the training data including a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix;

[0032] Determine the target model based on the preset model after convergence.

[0033] In some embodiments, determining a centerline point set according to a candidate centerline point set includes:

[0034] Obtain the unit normal vector of each data point in the candidate centerline point set;

[0035] According to the angle between adjacent unit normal vectors, at least one target data point having an angle greater than a preset threshold is obtained from the candidate centerline point set;

[0036] Based on the cubic non-rational B-spline curve interpolation method, a set consisting of at least one target data point is interpolated to obtain a centerline point set.

[0037] In some embodiments, preprocessing the first image at the first frame moment to obtain a target binary image includes:

[0038] Based on a preset filtering algorithm, filtering the first image to obtain a third image;

[0039] Binarizing the third image to obtain a fourth image;

[0040] Based on the morphological operation, the fourth image is processed to obtain a target binary image.

[0041] In a second aspect, the present application provides a point cloud stitching system based on multiple sensors, comprising:

[0042] An acquisition module, used to acquire measurement data at all frame moments in the process of scanning an object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image being an image of the object scanned by a monocular line laser scanner, and the second image being an image containing a preset mark taken by an imaging device;

[0043] A processing module, used for performing at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, the first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0044] A stitching module, used for performing point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system;

[0045] The transformation process includes:

[0046] Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, where the first frame time is any frame time;

[0047] Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is ​​a homogeneous transformation matrix from the coordinates in the first coordinate system from the first frame time to the second frame time to the coordinates in the second coordinate system. The target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time. The second frame time is the frame time other than the first frame time.

[0048] In a third aspect, the present application provides an / image signal generator / network device / transmitter / terminal / base station / industrial computer, comprising: at least one memory for storing programs; at least one processor for executing programs stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any embodiments of the first aspect.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any embodiments of the first aspect.

[0050] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0051] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0052] The multi-sensor based point cloud stitching method and system provided in the present application constructs a high-precision target homogeneous transformation matrix by extracting and fusing data from multiple sensors including a camera, a monocular line laser scanner and an inertial measurement unit. Combined with the target homogeneous transformation matrix, the coordinates of the center line point set of the first image at each frame moment in the first coordinate system are converted to coordinates in the second coordinate system, and point cloud stitching is performed on the coordinates of the center line point set of the first image at each frame moment in the second coordinate system, thereby avoiding the problem of inaccurate stitching due to low overlap of the monocular line laser scanning point cloud and improving the point cloud stitching accuracy and reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a multi-sensor based point cloud stitching method provided in an embodiment of the present application;

[0054] Figure 2 is a structural diagram of a data measurement system provided in an embodiment of the present application;

[0055] Figure 3 is a schematic diagram of a coordinate system of a data measurement system constructed in an embodiment of the present application;

[0056] Figure 4 is a structural schematic diagram of a multi-sensor based point cloud stitching system provided in an embodiment of the present application;

[0057] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0060] The terms "first", "second", etc. in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. For example, a first image and a second image are used to distinguish different images rather than to describe a specific order of images.

[0061] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] In order to avoid the problem of inaccurate point cloud stitching caused by low overlap of monocular line laser scanning point cloud, many scholars choose to use external sensors to obtain the transformation matrix of laser point cloud. Among them, laser tracker can provide high-precision transformation matrix, but it is expensive and not easy to carry; Inertial Measurement Unit (IMU) can obtain high-precision data through integration, but its inherent zero drift problem makes it difficult to guarantee long-term use; Monocular vision sensor is a more economical choice, which has no cumulative error in the process of solving the transformation matrix, and the single precision is relatively low.

[0063] Based on this, an embodiment of the present application provides a point cloud stitching method and system based on multiple sensors, which obtains measurement data of all frame moments included in the process of scanning an object through a monocular line laser scanner, a camera, and an IMU. Among them, the monocular line laser scanner, the camera, and the IMU synchronously acquire data to obtain a first image, a second image containing a preset mark, acceleration measurement values, and angular velocity measurement values. The first image is preprocessed, and the center line of the scanning laser stripe is roughly extracted using an improved grayscale centroid algorithm to obtain a candidate center line point set, and the candidate center line point set is processed to obtain its corresponding center line point set. Based on the internal parameters of the monocular line laser scanner and the pixel coordinates of the points in the center line point set, its three-dimensional coordinates in the coordinate system where the camera is located are calculated. The acceleration and angular velocity measurements measured by the IMU are Euler-integrated to construct the position increment matrix of the monocular line laser scanner; the ChArUco system is used to enhance the ChArUco marker detection and attitude estimation, and the transformation matrix between key frame moments is calculated; the trained variational auto-encoder (VAE) model is used to fuse multiple sensors to obtain the above data and obtain the target homogeneous transformation matrix between key frame moments; the target homogeneous transformation matrix between all frame moments in the scanned object is used to convert the three-dimensional coordinates of the points in the center line point concentration in the camera coordinate system frame by frame to the camera coordinate system where the monocular line laser scanner is located, and the point cloud is spliced. In the embodiment of the present application, the preset marker uses a ChArUco marker.

[0064] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0065] See also Figure 1 , an embodiment of the present application provides a point cloud stitching method based on multiple sensors, including: step 110, step 120 and step 130.

[0066] Step 110, obtaining measurement data of all frame moments in the process of scanning the object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image taken by a camera and containing a preset mark;

[0067] Step 120, performing at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, the first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0068] Step 130, performing point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system;

[0069] The transformation process includes:

[0070] Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, where the first frame time is any frame time;

[0071] Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is ​​a homogeneous transformation matrix from the coordinates in the first coordinate system from the first frame time to the second frame time to the coordinates in the second coordinate system. The target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time. The second frame time is the frame time other than the first frame time.

[0072] In the specific implementation, the following Figure 2 The data measurement system shown collects measurement data of all frame moments included in the process of scanning an object. The measurement system consists of a monocular line laser scanner, an imaging device (e.g., a camera), an IMU, and a plurality of preset marker plates. Among them, the monocular line laser scanner is composed of a camera and a laser generator. In an embodiment of the present application, the preset marker plate contains a preset marker, and the preset marker plate adopts a ChArUco marker plate containing a ChArUco marker.

[0073] Please see further Figure 3 , establish a coordinate system for the data measurement system, where {W} represents the world coordinate system, {C} represents the coordinate system where the camera is located, that is, the first coordinate system, {C′} represents the camera coordinate system where the monocular line laser scanner is located, that is, the second coordinate system, and {A} represents the coordinate system where the ChArUco marker is located.

[0074] When starting to scan an object, make sure that there is at least one ChArUco marker in the camera's field of view in the initial state. When a monocular line laser scanner is scanning an object, the following steps can be used to improve the synchronization accuracy of the first image of the object obtained by the monocular line laser scanner, the second image containing the ChArUco marker obtained by the camera, and the acceleration and angular velocity measurements of the object measured by the IMU to avoid errors caused by image misalignment.

[0075] Specifically, a python function is used to create multithreading to improve the shooting synchronization accuracy of the monocular line laser scanner and the camera;

[0076] Get the timestamp, use the same timestamp to name the image to avoid image confusion, and record the corresponding timestamp when storing the data measured by the corresponding IMU. Different timestamps represent different frame moments.

[0077] Unlike monocular cameras, which have no cumulative errors but poor accuracy in the process of solving the transformation matrix, the embodiment of the present application uses IMU, which can obtain high-precision posture transformation matrix in a short period of time. By fusing multi-sensor data, the zero drift problem of IMU is solved, effectively compensating for the shortcomings of the two, and reducing costs while ensuring accuracy.

[0078] The following transformation process is performed at least once until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system.

[0079] Point cloud stitching is performed according to the coordinates of each point in the center line point set of the first image corresponding to all the acquired frame moments in the second coordinate system. The transformation process may specifically include:

[0080] For any two frame times: the first frame time and the second frame time.

[0081] According to the pixel coordinates (u, v) of each point in the center line point set corresponding to the first image acquired by the monocular line laser scanner at the first frame time, the three-dimensional coordinates (x, y) of each point in the center line point set corresponding to the first frame time in the coordinate system of the camera are calculated. C ,y C ,z C ), which is the first coordinate.

[0082] Specifically, it is assumed that the camera intrinsic parameter matrix K of the monocular line laser scanner is:

[0083]

[0084] Among them, f x is the ratio factor between the focal length of the camera in the horizontal direction (x-axis direction) and the pixel, f y is the ratio factor between the focal length of the camera in the vertical direction (y-axis direction) and the pixel, c x is the horizontal position of the center point on the camera imaging plane, c y It is the vertical position of the center point on the camera imaging plane.

[0085] Convert the pixel coordinates (u, v) of each point in the center line point set to the coordinates (x, y) in the normalized camera coordinate system {C}:

[0086]

[0087] Calculate the depth information using the known laser plane equation:

[0088] Solve the depth information r as: Use the depth information obtained to calculate the three-dimensional coordinates (x C ,y C ,z C ), respectively: z C =r. Among them, a, b, c, d are all preset coefficients.

[0089] According to the target homogeneous transformation matrix, the first coordinate at the first frame time is transformed, and the first coordinate is converted to the coordinate in the coordinate system where the monocular line laser scanner is located at the second frame time, that is, the second coordinate.

[0090] It should be noted that the target homogeneous transformation matrix from the first frame time to the second frame time is obtained by fusing the measurement data obtained by various sensors including the camera, IMU and monocular line laser scanner at the first frame time and the second frame time. The target homogeneous transformation matrix is ​​a homogeneous transformation matrix between the coordinates in the coordinate system of the camera from the first frame time to the second frame time to the coordinates in the coordinate system of the monocular line laser scanner, and is used to characterize the conversion relationship between the coordinates in the coordinate system of the camera from the first frame time to the second frame time to the coordinates in the coordinate system of the monocular line laser scanner.

[0091] It should be noted that the multi-sensor based point cloud stitching method provided in the embodiment of the present application solves the problem of small overlap of line laser point clouds and large error in ICP registration by fusing multi-sensor data. It can be widely used in various three-dimensional reconstruction and measurement tasks to improve the efficiency and accuracy of industrial inspection and measurement.

[0092] The multi-sensor based point cloud stitching method provided in the embodiment of the present application extracts and fuses data from multiple sensors including a camera, a monocular line laser scanner and an inertial measurement unit to construct a high-precision target homogeneous transformation matrix. Combined with the target homogeneous transformation matrix, the coordinates of the center line point set of the first image at each frame moment in the first coordinate system are converted to coordinates in the second coordinate system, and point cloud stitching is performed on the coordinates of the center line point set of the first image at each frame moment in the second coordinate system, thereby avoiding the problem of inaccurate stitching due to low overlap of the monocular line laser scanning point cloud and improving the point cloud stitching accuracy and reconstruction accuracy.

[0093] Furthermore, in some embodiments, a method for acquiring a center line point set of a first image corresponding to a first frame time may include:

[0094] Determine a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image, where the target binary image is a binary image containing the laser stripes corresponding to the first image corresponding to the first frame moment;

[0095] A centerline point set is determined based on the candidate centerline point set.

[0096] In a specific implementation, based on the first image at the first frame moment acquired by the monocular line laser scanner, a binary image containing laser stripes corresponding to the first image at the first frame moment is obtained.

[0097] The binary image is used as a target binary image, and a candidate centerline point set is obtained according to the center of gravity of the laser stripes in the target binary image.

[0098] According to the candidate centerline point set, a centerline point set is obtained.

[0099] Furthermore, in some embodiments, determining a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image may include:

[0100] Obtain the first centroid of each row of pixels in the target binary image in the horizontal direction and the longitudinal centroid point set, the longitudinal centroid point set is a set of the second centroid of each column of pixels in the target binary image in the vertical direction;

[0101] Obtain the second center of gravity of the target that is closest to each first center of gravity in the longitudinal center of gravity point concentration;

[0102] A candidate centerline point set is obtained according to each first center of gravity and the target second center of gravity corresponding to each first center of gravity.

[0103] In a specific implementation, the first image at the first frame moment acquired by the monocular line laser scanner is preprocessed to obtain a target binary image.

[0104] The improved grayscale centroid algorithm is used to roughly extract the center line of the scanning laser stripe:

[0105] The horizontal center of gravity (η,y) of the nth row of pixels in the target binary image is calculated using the image pixel intensity weights. η ) (i.e., the first centroid) and the centroid of the β-th column of pixels in the vertical direction (x β ,β) (i.e., the second centroid), i.e., the second centroid, where η is the number of pixel rows of the target binary image, and β is the number of pixel columns of the target binary image, as shown in the following formula:

[0106]

[0107] Among them, γ(α,β) is the pixel value of the αth row and βth column in the target binary image, n is the total number of pixel columns in the target binary image, and m is the total number of pixel rows in the target binary image.

[0108] The set of the second centroids of each column of pixels in the acquired target binary image in the vertical direction is the longitudinal centroid point set (X β,β).

[0109] For the horizontal centroid (η,y η ), calculate its vertical center of gravity point set (X β ,β) and from the longitudinal centroid point set (X β ,β) to find the value that is consistent with (η,y η ) is the second centroid closest to the target, assuming that (x' β ,y' β ), as shown below:

[0110]

[0111] Similarly, the horizontal centroid and the vertical centroid point set (X β ,β) is the minimum distance between each data point.

[0112] Calculate (η,y η ) and (x' β ,y' β ), and obtain the data point (x′, y′) in the candidate centerline point set Ω, as shown in the following formula:

[0113]

[0114] Similarly, the remaining data points in the candidate centerline point set Ω can be calculated.

[0115] Further, in some embodiments, determining the centerline point set according to the candidate centerline point set may include:

[0116] Obtain the unit normal vector of each data point in the candidate centerline point set;

[0117] According to the angle between adjacent unit normal vectors, at least one target data point having an angle greater than a preset threshold is obtained from the candidate centerline point set;

[0118] Based on the cubic non-rational B-spline curve interpolation method, a set consisting of at least one target data point is interpolated to obtain a centerline point set.

[0119] In the specific implementation, the candidate centerline point set extracted by the improved grayscale centroid method is globally smoothed, as follows:

[0120] Calculate the unit normal vector N1 = {n1, n2, ..., n m}, where n1 is the unit normal vector of the first data point in the candidate centerline point set Ω, and n1 = n2, n2 ~ ​​nm are the unit normal vectors from the second data point to the mth data point in the candidate centerline point set Ω, and n m =n m-1 . As shown in the following formula:

[0121] n ε =k ε / ||k ε ||,ε=2,3,4,…,m-1

[0122] Among them, k ε is the εth data point p in the candidate centerline point set Ω ε The curvature at n ε is the data point p ε The unit normal vector at .

[0123] Calculate the unit normal vector N1 = {n1, n2, ..., n m} adjacent unit normal vector n ε With n ε-1 The angle θ between ε , compare the angle θ ε The relationship between the size of the preset threshold, if θ ε If it is greater than the preset threshold, n ε The corresponding data point p ε As the target data point, as shown in the following formula:

[0124]

[0125] Similarly, the angles between other adjacent unit normal vectors can be calculated, and by comparing the angles with a preset threshold, all target data points whose angles are greater than the preset threshold can be found.

[0126] The set of target data points is interpolated using the cubic non-rational B-spline curve interpolation method to obtain the center line point set Ω L Among them, the center line point set Ω L The number of data points in can be flexibly adjusted as needed.

[0127] Furthermore, in some embodiments, preprocessing the first image at the first frame moment to obtain a target binary image may include:

[0128] Based on a preset filtering algorithm, filtering the first image to obtain a third image;

[0129] Binarizing the third image to obtain a fourth image;

[0130] Based on the morphological operation, the fourth image is processed to obtain a target binary image.

[0131] In a specific implementation, a preset filtering algorithm is used to filter the first image to remove image noise in the first image, and the obtained image is the third image. In the embodiment of the present application, the preset filtering algorithm can be a Gaussian filter, a median filter, or other algorithms used alone, or in combination with each other, and the embodiment of the present application does not limit this.

[0132] The obtained third image is binarized and converted into black and white, and the obtained image is the fourth image.

[0133] The obtained fourth image is processed by dilation, erosion, opening and closing operations using morphological operations, and the connected domain is screened out to obtain a target binary image containing laser stripes.

[0134] Furthermore, in some embodiments, the method of obtaining the target homogeneous transformation matrix may include:

[0135] Obtaining the angular velocity measurement value and acceleration measurement value of the object corresponding to all frame moments in the process of scanning the object;

[0136] Based on the angular velocity measurement value and the acceleration measurement value, the incremental posture information from the first frame time to the second frame time is obtained;

[0137] Based on the position and posture increment information, a first transformation matrix from the first frame time to the second frame time is obtained;

[0138] Based on the second image corresponding to the first frame time, obtaining the third coordinate of the preset mark at the first frame time in the first coordinate system;

[0139] Based on the second image corresponding to the second frame time, obtaining the fourth coordinate of the preset mark at the second frame time in the first coordinate system;

[0140] Determine a second transformation matrix from the first frame time to the second frame time according to the third coordinate and the fourth coordinate;

[0141] A target homogeneous transformation matrix is ​​determined according to the first transformation matrix and the second transformation matrix.

[0142] In a specific implementation, the acceleration measurement value and the angular velocity measurement value measured by the IMU are Euler-integrated to obtain the incremental pose information of the camera between two key frame moments, and the pose increment matrix of the monocular line laser scanner is constructed. The pose increment matrix is ​​the first transformation matrix. In the embodiment of the present application, any two frame moments are taken as key frames.

[0143] Specifically, the rotation increment from the first frame moment (for example, the i-th frame moment) to the second frame moment (for example, the j-th frame moment) is calculated according to the angular velocity measurement values ​​and acceleration measurement values ​​of the object corresponding to all frame moments in the scanning process of the object.

[0144]

[0145] Where, {W} is the world coordinate system, {I} is the coordinate system where the IMU is located, is the angular velocity measurement at the kth frame, and Δt is the time step.

[0146] According to the rotation increment and the acceleration measurement value at the i-th frame Calculate the velocity increment of the IMU from the i-th frame to the j-th frame

[0147]

[0148] in, is the acceleration measurement value at the kth frame.

[0149] According to the speed increment and rotation increment Calculate the displacement increment from the i-th frame to the j-th frame

[0150]

[0151] Rotate Increment and displacement increment and as the pose increment information from the i-th frame to the j-th frame, and construct the transformation matrix from the i-th frame to the j-th frame based on the pose increment information And the transformation matrix Inverse it to get the first transformation matrix between the i-th frame time and the j-th frame time.

[0152]

[0153] Among them, T1 is the first transformation matrix between the i-th frame time and the j-th frame time.

[0154] Use the ChArUco system to enhance ChArUco marker detection and pose estimation, and calculate the pose transformation matrix T2 between the i-th frame time and the j-th frame time. Specifically:

[0155] Input the second image containing the ChArUco mark at the first frame moment (for example, the i-th frame moment) collected into the pre-trained Deep ChArUco network for feature extraction and key point (i.e., each corner point of the ChArUco mark) detection to obtain the pixel coordinates corresponding to the preliminary key points and the QR code information of the ChArUco mark. Use the RefineNet network to refine the detected corner point coordinates to obtain the sub-pixel pixel coordinates of the key points. And use the solvePnP function in the OpenCV library, input the corner point coordinates detected by the RefineNet network, the arrangement order of the three-dimensional information of the ChArUco mark, and the size of the ChArUco mark, and calculate the three-dimensional coordinates T of the ChArUco mark in the camera coordinate system C at the i-th frame moment i C , the three-dimensional coordinate T i C This is the third coordinate.

[0156] Based on the same method, the three-dimensional coordinates of the ChArUco marker in the camera coordinate system C at the jth frame moment are obtained The three-dimensional coordinates This is the fourth coordinate.

[0157] Use homogeneous transformation matrix operation to calculate the third coordinate T i C and the fourth coordinate The posture transformation matrix T2 between them is the second transformation matrix.

[0158]

[0159] in, By the third coordinate T i C After inversion, we get .

[0160] According to the obtained first transformation matrix T1 and second transformation matrix T2, the target homogeneous transformation matrix T is obtained.

[0161] Further, in some embodiments, determining the target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix may include:

[0162] Inputting the first transformation matrix and the second transformation matrix into the target model to obtain output data of the target model;

[0163] According to the output data, determine the target homogeneous transformation matrix;

[0164] The target model is obtained by:

[0165] Inputting training data into a preset model for training until a loss function of the preset model converges, the training data including a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix;

[0166] Determine the target model based on the preset model after convergence.

[0167] In a specific implementation, the trained target model is used in the embodiment of the present application to fuse the first transformation matrix T1 and the second transformation matrix T2 acquired by multiple sensors to obtain the target homogeneous transformation matrix T between any two frame times. The trained VAE model is used as the target model in the embodiment of the present application.

[0168] The transformation matrix (T1, T2) is preprocessed into a form suitable for input to the VAE model, flattened from a 4x4 matrix to a 16-dimensional vector, and combined. The encoder maps the input data (T1, T2) to the latent space, and reconstructs the input data (T1, T2) using a two-layer nonlinear network to obtain the latent variable z, which obeys the following distribution function:

[0169]

[0170] Among them, μ represents the expectation and σ represents the variance.

[0171] The encoder network consists of two connected layers, using the ReLU activation function, and outputs two 16-dimensional branches, namely the mean branch f μ (T1,T2) and log-variance branch Using the logarithmic variance σ 2 =exp(logσ 2 ) Calculate the variance σ 2 The latent variable z is sampled from a normal distribution using the reparameterization trick z = μ + σ⊙∈, where is a random variable sampled from a standard normal distribution.

[0172] The three-layer network decoder performs a nonlinear transformation on the latent variables, maps the reconstructed latent variable z back to the original data space to output a 16-dimensional vector, folds the vector into a 4x4 transformation matrix, and obtains the target homogeneous transformation matrix.

[0173] It should be noted that the target model can be obtained in the following ways:

[0174] Input the training data into the preset model for training until the loss function of the preset model converges, the training data includes a preset homogeneous transformation matrix, noise data, a first transformation matrix, and a second transformation matrix; and obtain the target model according to the preset model after convergence. In the embodiment of the present application, the preset model adopts a VAE model.

[0175] Input the preset homogeneous transformation matrix obtained in the simulation (the true value T′ of the target homogeneous transformation matrix T can be used as the preset homogeneous transformation matrix) and the data (T1, T2) with added noise data (such as Gaussian noise) into the VAE model, and calculate the loss function The formula is as follows:

[0176]

[0177] Among them, q(z|T1,T2) represents the posterior distribution of the latent variable, p(z) represents the prior distribution of the latent variable, and obeys represents the expected value of the error of the latent variable z reconstructed by T′, D KL (q(z|T1,T2)||p(z)) represents the divergence between the posterior distribution and the prior distribution.

[0178] Use the gradient descent method to optimize the loss function, calculate the loss function and perform backpropagation to update the VAE model parameters until the loss function converges.

[0179] The target homogeneous transformation matrix T is used to transform the three-dimensional coordinates of each point in the center line point set frame by frame into the coordinate system of the monocular linear laser scanner, and the point cloud is stitched.

[0180] Assume that the three-dimensional coordinate set of each point in the center line point set corresponding to the jth frame in the camera coordinate system is (X Cj ,Y Cj ,Z Cj ), where the homogeneous representation of the coordinates of the oth point is [x Cj ,y Cj ,z Cj ,1] T , use the target homogeneous transformation matrix T to transform (X Cj ,Y Cj ,Z Cj ) is converted to the coordinate system of the monocular line laser scanner at the i-th frame: [x Ci ,y Ci ,z Ci ,1] T =T[x Cj ,y Cj ,z Cj ,1] T The center line points of all frame moments are concentrated in turn, and the three-dimensional coordinates of each point in the camera coordinate system are converted to the coordinate system of the monocular line laser scanner to complete the point cloud stitching.

[0181] The multi-sensor based point cloud stitching method provided in the embodiment of the present application can effectively process multiple sensor data, including IMU and monocular line laser scanner, using the VAE model. The multi-sensor data can be fused using the VAE model to generate a high-precision homogeneous transformation matrix, thereby reducing the error of point clouds with fewer overlapping parts during the stitching process and improving the accuracy of the final reconstruction.

[0182] The multi-sensor based point cloud stitching system provided by the present invention is described below. The multi-sensor based point cloud stitching system described below and the multi-sensor based point cloud stitching method described above can refer to each other.

[0183] Please see further Figure 4 The multi-sensor based point cloud stitching system provided in the embodiment of the present application includes: an acquisition module 410, a processing module 420 and a stitching module 430.

[0184] An acquisition module 410 is used to acquire measurement data at all frame moments in the process of scanning the object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image taken by a camera and containing a preset mark;

[0185] A processing module 420 is used to perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, where the first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located;

[0186] A stitching module 430, configured to stitch point clouds according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system;

[0187] The transformation process includes:

[0188] Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, where the first frame time is any frame time;

[0189] Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in the second coordinate system at the second frame time. The target homogeneous transformation matrix is ​​a homogeneous transformation matrix from the coordinates in the first coordinate system from the first frame time to the second frame time to the coordinates in the second coordinate system. The target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time. The second frame time is the frame time other than the first frame time.

[0190] The multi-sensor based point cloud stitching system provided in the embodiment of the present application extracts and fuses data from multiple sensors including a camera, a monocular line laser scanner and an inertial measurement unit to construct a high-precision target homogeneous transformation matrix. Combined with the target homogeneous transformation matrix, the coordinates of the centerline point set of the first image at each frame moment in the first coordinate system are converted to coordinates in the second coordinate system, and point cloud stitching is performed on the coordinates of the centerline point set of the first image at each frame moment in the second coordinate system, thereby avoiding the problem of inaccurate stitching due to low overlap of the monocular line laser scanning point cloud and improving the point cloud stitching accuracy and reconstruction accuracy.

[0191] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0192] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0193] Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, such as Figure 5 As shown, the electronic device may include: a processor (Processor) 510, a communication interface (Communications Interface) 520, a memory (Memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the method in the above embodiment.

[0194] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0195] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0196] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0197] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0198] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0200] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0201] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A point cloud stitching method based on multiple sensors, characterized in that: include: Acquire measurement data of all frame moments in the process of scanning the object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image is an image of the object scanned by a monocular line laser scanner, and the second image is an image containing a preset mark taken by an imaging device; Perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, where the first coordinate system is the coordinate system where the imaging device is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located; Performing point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system; The transformation process includes: Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, wherein the first frame time is any frame time; Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in a second coordinate system at a second frame time, the target homogeneous transformation matrix is ​​a homogeneous transformation matrix between coordinates in the first coordinate system from the first frame time to the second frame time to coordinates in the second coordinate system, the target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time, and the second frame time is a frame time other than the first frame time.

2. The multi-sensor point cloud stitching method according to claim 1, characterized in that: The method for acquiring the center line point set of the first image corresponding to the first frame time includes: Determine a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image, wherein the target binary image is a binary image containing the laser stripes corresponding to the first image corresponding to the first frame time; The centerline point set is determined according to the candidate centerline point set.

3. The multi-sensor point cloud stitching method according to claim 2, characterized in that: The step of determining a candidate centerline point set according to the center of gravity of the laser stripes in the target binary image includes: Obtaining a first centroid of each row of pixels in the target binary image in the horizontal direction and a longitudinal centroid point set, wherein the longitudinal centroid point set is a set of second centroids of each column of pixels in the target binary image in the vertical direction; Obtaining the target second center of gravity that is closest to each first center of gravity in the longitudinal center of gravity point concentration; A candidate centerline point set is obtained according to each first center of gravity and the target second center of gravity corresponding to each first center of gravity.

4. The multi-sensor point cloud stitching method according to claim 1, characterized in that: The method for obtaining the target homogeneous transformation matrix includes: Obtaining the angular velocity measurement value and acceleration measurement value of the object corresponding to all frame moments in the process of scanning the object; Based on the angular velocity measurement value and the acceleration measurement value, obtaining the posture increment information from the first frame time to the second frame time; Based on the pose increment information, obtaining a first transformation matrix from the first frame time to the second frame time; Based on the second image corresponding to the first frame time, obtaining the third coordinate of the preset mark at the first frame time in the first coordinate system; Based on the second image corresponding to the second frame time, acquiring the fourth coordinate of the preset mark at the second frame time in the first coordinate system; Determine a second transformation matrix from the first frame time to the second frame time according to the third coordinate and the fourth coordinate; The target homogeneous transformation matrix is ​​determined according to the first transformation matrix and the second transformation matrix.

5. The multi-sensor point cloud stitching method according to claim 4, characterized in that: The step of determining the target homogeneous transformation matrix according to the first transformation matrix and the second transformation matrix comprises: Inputting the first transformation matrix and the second transformation matrix into a target model to obtain output data of the target model; Determining the target homogeneous transformation matrix according to the output data; The target model is obtained by: Inputting training data into a preset model for training until a loss function of the preset model converges, the training data comprising a preset homogeneous transformation matrix, noise data, the first transformation matrix, and the second transformation matrix; The target model is determined according to the preset model after convergence.

6. The multi-sensor point cloud stitching method according to claim 2, characterized in that: The step of determining the centerline point set according to the candidate centerline point set includes: Obtaining a unit normal vector of each data point in the candidate centerline point set; According to the angle between adjacent unit normal vectors, obtaining at least one target data point whose angle is greater than a preset threshold from the candidate centerline point set; Based on the cubic non-rational B-spline curve interpolation method, the set consisting of the at least one target data point is interpolated to obtain the centerline point set.

7. The multi-sensor point cloud stitching method according to any one of claims 1 to 6, characterized in that: The preprocessing of the first image at the first frame moment to obtain the target binary image includes: Based on a preset filtering algorithm, filtering the first image to obtain a third image; Performing binarization processing on the third image to obtain a fourth image; Based on morphological operations, the fourth image is processed to obtain the target binary image.

8. A point cloud stitching system based on multiple sensors, characterized in that: include: an acquisition module, used to acquire measurement data at all frame moments in the process of scanning the object, the measurement data including a first image, a second image, and an angular velocity measurement value and an acceleration measurement value of the object measured by an inertial measurement unit, the first image being an image of the object scanned by a monocular line laser scanner, and the second image being an image containing a preset mark taken by an imaging device; A processing module, configured to perform at least one transformation process until the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the first coordinate system are converted to coordinates in the second coordinate system, wherein the first coordinate system is the coordinate system where the camera is located, and the second coordinate system is the coordinate system where the monocular line laser scanner is located; A stitching module, used for performing point cloud stitching according to the coordinates of each point in the center line point set of the first image corresponding to all frame moments in the second coordinate system; The transformation process includes: Based on the pixel coordinates of each point in the center line point set of the first image corresponding to the first frame time, obtaining the first coordinates of each point in the center line point set corresponding to the first frame time in the first coordinate system, wherein the first frame time is any frame time; Based on the first coordinates and the target homogeneous transformation matrix, the first coordinates are converted to second coordinates in a second coordinate system at a second frame time, the target homogeneous transformation matrix is ​​a homogeneous transformation matrix between coordinates in the first coordinate system from the first frame time to the second frame time to coordinates in the second coordinate system, the target homogeneous transformation matrix is ​​determined according to the measurement data corresponding to the first frame time and the second frame time, and the second frame time is a frame time other than the first frame time.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Standard cylinder-based linear laser scanning robot hand-eye calibration method

    CN113681559A

  • Real-time pose estimation method and device, electronic equipment and storage medium

    CN115661252A

  • Novel point cloud splicing method based on coding plane target

    CN118212124A

Cited By

  • RTK-based multi-legged robot collaborative mapping method and system

    CN120101769A

  • A multi-legged robot cooperative mapping method and system based on RTK

    CN120101769B