Multi-sensor fusion positioning method and system based on lidar point cloud data

Abstract

The present invention relates to the field of laser ranging, and discloses a multi-sensor fusion positioning method and system based on laser radar point cloud data, the method comprising: calculating the correction conversion matrix of the laser radar point cloud in the laser radar point cloud frame, using the correction conversion matrix to perform coordinate system correction on the laser radar point cloud to obtain a correction point cloud; synchronizing the time record of the target image with the time record of the correction point cloud frame to obtain a time-synchronized image and a time-synchronized frame; extracting image feature points in the time-synchronized image, extracting frame feature points in the time-synchronized frame; establishing a feature conversion matrix between image feature points and frame feature points, aligning image feature points with frame feature points using the feature conversion matrix, obtaining aligned image feature points and aligned frame feature points; fusing the aligned image feature points with the aligned frame feature points to obtain fused feature data, and using the fused feature data to locate the target position of the target to be located. The present invention can improve the accuracy of multi-sensor fusion positioning.

Description

Technical Field

[0001] The present invention relates to the field of laser ranging, and in particular to a multi-sensor fusion positioning method and system based on laser radar point cloud data. Background Art

[0002] LiDAR point cloud data is three-dimensional point cloud data measured by a LiDAR sensor. Since the amount of three-dimensional point cloud data is large, it is difficult to identify targets from a large amount of point cloud data. Therefore, the existing technology fuses the data collected by multiple other sensors with the LiDAR point cloud data, which makes it easier to identify target objects from the LiDAR point cloud data.

[0003] At present, the steps of multi-sensor fusion in the prior art are the fusion of data collected by sensors. In this process, time synchronization, feature extraction, feature fusion, and target point detection are required. The existing fusion technology converts 3D point clouds into 2D images to facilitate the fusion between two 2D images. However, due to the loss of longitudinal axis information in the process of converting 3D point clouds into 2D images, the fusion accuracy is insufficient. Secondly, since 3D point clouds are irregular in shape, irregular point clouds are difficult to fuse with images. Therefore, the prior art will also convert irregular point clouds into regular point clouds with similar shapes. However, the result is obvious. Similar point clouds are used instead of the original irregular point clouds during fusion, resulting in insufficient accuracy of the fusion results. Therefore, the accuracy of multi-sensor fusion positioning based on lidar point cloud data is insufficient. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a multi-sensor fusion positioning method and system based on lidar point cloud data, which can improve the accuracy of multi-sensor fusion positioning based on lidar point cloud data.

[0005] In a first aspect, the present invention provides a multi-sensor fusion positioning method based on laser radar point cloud data, comprising:

[0006] After scanning the target to be located by using a laser radar sensor, a laser radar point cloud frame of the target to be located is obtained, a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame is calculated, a coordinate system correction is performed on the laser radar point cloud by using the correction transformation matrix to obtain a corrected point cloud, and a corrected point cloud frame of the corrected point cloud is obtained;

[0007] Using an image sensor to collect a target image of the target to be located, synchronizing the time record of the target image with the time record of the correction point cloud frame to obtain a time-synchronized image and a time-synchronized frame;

[0008] Extracting image feature points in the time-synchronized image, and extracting frame feature points in the time-synchronized frame;

[0009] Establishing a feature conversion matrix between the image feature points and the frame feature points, and aligning the image feature points and the frame feature points using the feature conversion matrix to obtain aligned image feature points and aligned frame feature points;

[0010] The aligned image feature points and the aligned frame feature points are fused to obtain fused feature data, and the target position of the target to be located is located using the fused feature data.

[0011] In a possible implementation manner of the first aspect, calculating a correction transformation matrix of the lidar point cloud in the lidar point cloud frame includes:

[0012] The correction transformation matrix of the LiDAR point cloud in the LiDAR point cloud frame is calculated using the following formula:

[0013] ;

[0014] in, represents the correction transformation matrix, express The laser radar point cloud at time is relative to The relative position of the LiDAR point cloud at time, Indicates the time when the scanning of the lidar point cloud frame starts. Indicates the end time of a scan of the laser radar point cloud frame. represents the fixed scanning time required to scan the first point cloud, j represents the sequence number of the fixed scanning time, i represents the sequence number of the lidar point cloud, represents the starting time of scanning the i-th lidar point cloud, Indicates that when scanning the i-th point cloud, To the end of the scan The time required.

[0015] In a possible implementation manner of the first aspect, using the correction transformation matrix to perform coordinate system correction on the laser radar point cloud to obtain a corrected point cloud includes:

[0016] Based on the correction transformation matrix, the laser radar point cloud is corrected in the coordinate system using the following formula to obtain a corrected point cloud:

[0017] ;

[0018] in, represents the corrected point cloud, represents the three-dimensional point cloud coordinates of the correction point cloud, represents the three-dimensional point cloud coordinates of the laser radar point cloud, represents the correction transformation matrix, Represents the 3D point cloud coordinates of the calibrated point cloud Get the corrected point cloud .

[0019] In a possible implementation manner of the first aspect, synchronizing the time record of the target image with the time record of the corrected point cloud frame to obtain a time-synchronized image and a time-synchronized frame includes:

[0020] The synchronization moment between the time record of the target image and the time record of the correction point cloud frame is calculated using the following formula:

[0021] ;

[0022] in, Indicates the synchronization time, represents the total number of synchronization moments, represents the u-th moment in the time record of the target image a, represents the vth moment in the time record of the corrected point cloud frame b, Indicates from indivual Select the smallest absolute value of the difference, Indicated in When is the minimum absolute value of the difference, The value of The values ​​are similar to , Choose one as ;

[0023] Selecting a target time record of the target image and a target time record of the correction point cloud frame that are consistent with the synchronization moment from the time record of the target image and the time record of the correction point cloud frame respectively;

[0024] The target time record of the target image and the target image and the correction point cloud frame corresponding to the target time record of the correction point cloud frame are used as the time synchronization image and the time synchronization frame.

[0025] In a possible implementation manner of the first aspect, extracting image feature points from the time-synchronized image includes:

[0026] The image feature points in the time-synchronized image are extracted using the following formula:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, represents the image feature points, represents the point-wise convolutional layer, represents BatchNormalization, represents the local convolutional layer, represents the convolutional layer, represents the time-synchronized image, , , , represents the extracted intermediate features, represents the spatial pyramid pooling layer, represents upsampling, It stands for CSPDarknet53 to 2-Stage FPN, which realizes the feature fusion between low-level feature maps and high-level feature maps based on the CSPDarknet53 backbone network and 2-Stage FPN (Feature Pyramid Network). Represents an efficient multi-scale attention module.

[0033] In a possible implementation manner of the first aspect, extracting the frame feature points in the time synchronization frame includes:

[0034] The point cloud coordinate matrix of the point cloud in the time synchronization frame is constructed using the following format:

[0035] ;

[0036] in, express The point cloud coordinate matrix, represents the three-dimensional coordinates of the point cloud in the time synchronization frame, represents the point cloud in the time synchronization frame, m represents the total number of point clouds in the time synchronization frame, Represents the sequence number of the point cloud in the time synchronization frame;

[0037] Performing zero-mean conversion on each row of values ​​in the point cloud coordinate matrix to obtain a zero-mean matrix;

[0038] The covariance matrix of the zero-mean matrix is ​​constructed using the following formula:

[0039] ;

[0040] in, represents the covariance matrix, represents the three-dimensional coordinates of the point cloud in the time synchronization frame, m represents the total number of point clouds in the time synchronization frame, represents the sequence number of the point cloud in the time synchronization frame, ;

[0041] The eigenvalues ​​and eigenvectors of the covariance matrix are calculated using the following formula:

[0042] ;

[0043] in, Indicated by The derived eigenvalues ​​are Indicated by , The derived eigenvectors are represents a diagonal matrix, The diagonal elements of The characteristic value of express The three groups of eigenvectors are composed of three-order matrix eigenvectors. represents the covariance matrix, express The third-order matrix eigenvector composed of;

[0044] Using the target eigenvector corresponding to the minimum eigenvalue among the eigenvalues ​​in the eigenvector as the normal vector of the point cloud in the time synchronization frame;

[0045] Based on the normal vector, the curvature of the point cloud in the time synchronization frame is calculated using the following formula:

[0046] ;

[0047] in, represents the curvature, represents the normal vector of the s-th point cloud in the time synchronization frame, represents the normal vectors of the point clouds other than the s-th point cloud in the time synchronization frame;

[0048] Based on the curvature, the frame feature points in the time synchronization frame are calculated using the following formula:

[0049] ;

[0050] in, represents the frame feature point, represents the fully connected layer, represents a multi-layer perceptron, represents the curvature.

[0051] In a possible implementation manner of the first aspect, establishing a feature conversion matrix between the image feature points and the frame feature points includes:

[0052] The feature conversion matrix between the image feature points and the frame feature points is established using the following formula:

[0053] ;

[0054] in, represents the feature transformation matrix, represents the matrix composed of the internal and external parameters of the image sensor corresponding to the image feature points, f represents the total number of internal and external parameters of the image sensor, represents a matrix composed of the internal and external parameters of the lidar sensor corresponding to the frame feature points, and q represents the total number of the internal and external parameters of the lidar sensor.

[0055] In a possible implementation manner of the first aspect, the fusing the aligned image feature points with the aligned frame feature points to obtain fused feature data includes:

[0056] Detecting a target area of ​​the target to be located in the aligned image feature points;

[0057] Identifying a first target feature point in the aligned frame feature points that is within the target area;

[0058] The second target feature point corresponding to the first target feature point in the frame feature point is used as the fused feature data.

[0059] In a possible implementation manner of the first aspect, locating a target position of the target to be located by using the fused feature data includes:

[0060] The central coordinates of the fused feature data are calculated using the following formula:

[0061] ;

[0062] in, represents the center coordinates of the fused feature data, Represents the coordinates of the fused feature data;

[0063] The center coordinates are used as the target position of the object to be located.

[0064] In a second aspect, the present invention provides a multi-sensor fusion positioning system based on laser radar point cloud data, the system comprising:

[0065] A point cloud acquisition module is used to obtain a laser radar point cloud frame of the target to be located after scanning the target to be located with a laser radar sensor, calculate a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame, perform coordinate system correction on the laser radar point cloud using the correction transformation matrix to obtain a corrected point cloud, and obtain a corrected point cloud frame of the corrected point cloud;

[0066] A time synchronization module, used to collect a target image of the target to be located by using an image sensor, synchronize the time record of the target image with the time record of the correction point cloud frame, and obtain a time-synchronized image and a time-synchronized frame;

[0067] A feature extraction module, used to extract image feature points in the time-synchronized image and frame feature points in the time-synchronized frame;

[0068] A feature alignment module, used to establish a feature conversion matrix between the image feature points and the frame feature points, and to align the image feature points and the frame feature points using the feature conversion matrix to obtain aligned image feature points and aligned frame feature points;

[0069] The position positioning module is used to fuse the aligned image feature points with the aligned frame feature points to obtain fused feature data, and use the fused feature data to locate the target position of the target to be located.

[0070] Compared with the prior art, the technical principle and beneficial effects of this solution are:

[0071] The embodiment of the present invention calculates the correction conversion matrix of the laser radar point cloud in the laser radar point cloud frame to unify the power sources belonging to different coordinate systems collected at different times into point clouds in the same coordinate system, which can improve the accuracy of subsequent point cloud feature recognition. The embodiment of the present invention synchronizes the time record of the target image with the time record of the correction point cloud frame for subsequent fusion of images and point clouds belonging to the same time, thereby reducing the error of the disordered time series on the fusion of subsequent images and point clouds. The embodiment of the present invention extracts image feature points in the time-synchronized image to detect the image feature points using the improved YOLO detection algorithm, thereby improving the accuracy of subsequent point cloud feature recognition. The accuracy of feature point detection is improved on the basis of the X detection algorithm. Furthermore, the embodiment of the present invention extracts the frame feature points in the time synchronization frame to characterize the motion changes of the point cloud in the time synchronization frame using the normal vector, thereby obtaining implicit motion change features. The embodiment of the present invention establishes a feature conversion matrix between the image feature points and the frame feature points to map the frame feature points to the image feature points using the feature conversion matrix. The image feature points are used to facilitate the identification of targets. The point clouds mapped to the image area of ​​the target to be detected can be identified, and these point clouds are used as the point clouds of the target to be detected, thereby reducing the problem of low recognition efficiency caused by the large amount of data for point cloud recognition. Therefore, the multi-sensor fusion positioning method and system based on laser radar point cloud data proposed in the embodiment of the present invention can improve the accuracy of multi-sensor fusion positioning based on laser radar point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0073] Figure 1 A schematic diagram of a flow chart of a multi-sensor fusion positioning method based on laser radar point cloud data provided by an embodiment of the present invention;

[0074] Figure 2 In one embodiment of the present invention Figure 1 A schematic diagram of one of the steps of a multi-sensor fusion positioning method based on lidar point cloud data is provided;

[0075] Figure 3 In one embodiment of the present invention Figure 1 A schematic diagram of another step of a multi-sensor fusion positioning method based on lidar point cloud data is provided;

[0076] Figure 4 A schematic diagram of a module of a multi-sensor fusion positioning system based on lidar point cloud data provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0077] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0078] The embodiment of the present invention provides a multi-sensor fusion positioning method based on laser radar point cloud data, and the execution subject of the multi-sensor fusion positioning method based on laser radar point cloud data includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present invention. In other words, the multi-sensor fusion positioning method based on laser radar point cloud data can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0079] See also Figure 1 FIG. 1 is a flow chart of a multi-sensor fusion positioning method based on laser radar point cloud data provided by an embodiment of the present invention. Figure 1 The multi-sensor fusion positioning method based on lidar point cloud data described in includes:

[0080] S1. After scanning the target to be located with a laser radar sensor, obtain a laser radar point cloud frame of the target to be located, calculate a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame, use the correction transformation matrix to perform coordinate system correction on the laser radar point cloud to obtain a corrected point cloud, and obtain a corrected point cloud frame of the corrected point cloud.

[0081] In the embodiment of the present invention, the laser radar point cloud frame refers to a set of point clouds collected within one scanning cycle of the laser radar sensor.

[0082] Furthermore, an embodiment of the present invention calculates a correction transformation matrix of the lidar point cloud in the lidar point cloud frame to unify power sources belonging to different coordinate systems collected at different times into point clouds under the same coordinate system, thereby improving the accuracy of subsequent point cloud feature recognition.

[0083] In one embodiment of the present invention, the calculating of the correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame includes: calculating the correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame using the following formula:

[0084] ;

[0085] in, represents the correction transformation matrix, express The laser radar point cloud at time is relative to The relative position of the LiDAR point cloud at time, Indicates the time when the scanning of the lidar point cloud frame starts. Indicates the end time of a scan of the laser radar point cloud frame. represents the fixed scanning time required to scan the first point cloud, j represents the sequence number of the fixed scanning time, i represents the sequence number of the lidar point cloud, represents the starting time of scanning the i-th lidar point cloud, Indicates that when scanning the i-th point cloud, To the end of the scan The time required.

[0086] In one embodiment of the present invention, the use of the correction transformation matrix to perform coordinate system correction on the laser radar point cloud to obtain the corrected point cloud includes: based on the correction transformation matrix, using the following formula to perform coordinate system correction on the laser radar point cloud to obtain the corrected point cloud:

[0087] ;

[0088] in, represents the corrected point cloud, represents the three-dimensional point cloud coordinates of the correction point cloud, represents the three-dimensional point cloud coordinates of the laser radar point cloud, represents the correction transformation matrix, Represents the 3D point cloud coordinates of the calibrated point cloud Get the corrected point cloud .

[0089] S2. Using an image sensor to collect a target image of the target to be located, synchronizing the time record of the target image with the time record of the correction point cloud frame to obtain a time-synchronized image and a time-synchronized frame.

[0090] The embodiment of the present invention synchronizes the time record of the target image with the time record of the correction point cloud frame for subsequent fusion of images and point clouds belonging to the same time, thereby reducing the error of the subsequent fusion between images and point clouds caused by the disordered time sequence.

[0091] In one embodiment of the present invention, referring to Figure 2As shown, the step of synchronizing the time record of the target image with the time record of the correction point cloud frame to obtain a time-synchronized image and a time-synchronized frame includes:

[0092] S201, using the following formula to calculate the synchronization time between the time record of the target image and the time record of the correction point cloud frame:

[0093] ;

[0094] in, Indicates the synchronization time, represents the total number of synchronization moments, represents the u-th moment in the time record of the target image a, represents the vth moment in the time record of the corrected point cloud frame b, Indicates from indivual Select the smallest absolute value of the difference, Indicated in When is the minimum absolute value of the difference, The value of The values ​​are similar to , Choose one as ;

[0095] S202, selecting a target time record of the target image and a target time record of the correction point cloud frame that are consistent with the synchronization time from the time record of the target image and the time record of the correction point cloud frame respectively;

[0096] S203 , taking the target time record of the target image and the target image and the corrected point cloud frame corresponding to the target time record of the corrected point cloud frame as the time-synchronized image and the time-synchronized frame.

[0097] S3. Extract image feature points from the time-synchronized image, and extract frame feature points from the time-synchronized frame.

[0098] The embodiment of the present invention extracts image feature points from the time-synchronized image to detect the image feature points using an improved YOLOX detection algorithm, thereby improving the accuracy of feature point detection based on the improved YOLOX detection algorithm.

[0099] In one embodiment of the present invention, the extracting the image feature points in the time-synchronized image includes: extracting the image feature points in the time-synchronized image using the following formula:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, represents the image feature points, represents the point-wise convolutional layer, represents BatchNormalization, represents the local convolutional layer, represents the convolutional layer, represents the time-synchronized image, , , , represents the extracted intermediate features, represents the spatial pyramid pooling layer, represents upsampling, It stands for CSPDarknet53 to 2-Stage FPN, which realizes the feature fusion between low-level feature maps and high-level feature maps based on the CSPDarknet53 backbone network and 2-Stage FPN (Feature Pyramid Network). Represents an efficient multi-scale attention module.

[0106] Furthermore, the embodiment of the present invention extracts frame feature points in the time-synchronized frame to characterize the motion change of the point cloud in the time-synchronized frame using normal vectors, thereby obtaining implicit motion change features.

[0107] In one embodiment of the present invention, the extracting the frame feature points in the time synchronization frame includes: constructing a point cloud coordinate matrix of the point cloud in the time synchronization frame using the following format:

[0108] ;

[0109] in, express The point cloud coordinate matrix, represents the three-dimensional coordinates of the point cloud in the time synchronization frame, represents the point cloud in the time synchronization frame, m represents the total number of point clouds in the time synchronization frame, Represents the sequence number of the point cloud in the time synchronization frame;

[0110] Each row of values ​​in the point cloud coordinate matrix is ​​zero-meaned to obtain a zero-mean matrix; the covariance matrix of the zero-mean matrix is ​​constructed using the following formula:

[0111] ;

[0112] in, represents the covariance matrix, represents the three-dimensional coordinates of the point cloud in the time synchronization frame, m represents the total number of point clouds in the time synchronization frame, represents the sequence number of the point cloud in the time synchronization frame, ;

[0113] The eigenvalues ​​and eigenvectors of the covariance matrix are calculated using the following formula:

[0114] ;

[0115] in, Indicated by The derived eigenvalues ​​are Indicated by , The derived eigenvectors are represents a diagonal matrix, The diagonal elements of The characteristic value of express The three groups of eigenvectors are composed of three-order matrix eigenvectors. represents the covariance matrix, express The third-order matrix eigenvector composed of;

[0116] The target eigenvector corresponding to the minimum eigenvalue in the eigenvalues ​​in the eigenvector is used as the normal vector of the point cloud in the time synchronization frame; based on the normal vector, the curvature of the point cloud in the time synchronization frame is calculated using the following formula:

[0117] ;

[0118] in, represents the curvature, represents the normal vector of the s-th point cloud in the time synchronization frame, represents the normal vectors of the point clouds other than the s-th point cloud in the time synchronization frame;

[0119] Based on the curvature, the frame feature points in the time synchronization frame are calculated using the following formula:

[0120] ;

[0121] in, represents the frame feature point, represents the fully connected layer, represents a multi-layer perceptron, represents the curvature.

[0122] S4. Establish a feature conversion matrix between the image feature points and the frame feature points, and use the feature conversion matrix to align the image feature points and the frame feature points to obtain aligned image feature points and aligned frame feature points.

[0123] The embodiment of the present invention establishes a feature conversion matrix between the image feature points and the frame feature points, so as to map the frame feature points to the image feature points using the feature conversion matrix. The image feature points facilitate target identification and can identify point clouds mapped to the image area of ​​the target to be detected. These point clouds are used as point clouds of the target to be detected, thereby reducing the problem of low recognition efficiency caused by the large amount of data in point cloud recognition.

[0124] In one embodiment of the present invention, the establishing of the feature conversion matrix between the image feature points and the frame feature points includes: establishing the feature conversion matrix between the image feature points and the frame feature points using the following formula:

[0125] ;

[0126] in, represents the feature transformation matrix, represents the matrix composed of the internal and external parameters of the image sensor corresponding to the image feature points, f represents the total number of internal and external parameters of the image sensor, represents a matrix composed of the internal and external parameters of the lidar sensor corresponding to the frame feature points, and q represents the total number of the internal and external parameters of the lidar sensor.

[0127] Optionally, the process of aligning the image feature points with the frame feature points using the feature conversion matrix is: mapping the frame feature points to the image feature points using the feature conversion matrix, that is, achieving feature point overlap between the image feature points and the frame feature points.

[0128] S5. Fusing the aligned image feature points with the aligned frame feature points to obtain fused feature data, and using the fused feature data to locate the target position of the target to be located.

[0129] In one embodiment of the present invention, referring to Figure 3 As shown, the fusion of the aligned image feature points and the aligned frame feature points to obtain fused feature data includes:

[0130] S301, detecting a target area of ​​the target to be located in the aligned image feature points;

[0131] S302, identifying a first target feature point in the alignment frame feature points that is within the target area;

[0132] S303: Use the second target feature point corresponding to the first target feature point in the frame feature point as the fused feature data.

[0133] Exemplarily, the process of detecting the target area of ​​the target to be located in the aligned image feature points is implemented by image recognition technology, and the purpose is to use image recognition technology to identify feature points and feature point areas belonging to the target to be located in the image feature points.

[0134] In one embodiment of the present invention, the method of locating the target position of the target to be located by using the fused feature data includes: calculating the center coordinates of the fused feature data by using the following formula:

[0135] ;

[0136] in, represents the center coordinates of the fused feature data, Represents the coordinates of the fused feature data;

[0137] The center coordinates are used as the target position of the object to be located.

[0138] It can be seen that the embodiment of the present invention calculates the correction conversion matrix of the laser radar point cloud in the laser radar point cloud frame to unify the power sources belonging to different coordinate systems collected at different times into point clouds in the same coordinate system, which can improve the accuracy of subsequent point cloud feature recognition. The embodiment of the present invention synchronizes the time record of the target image with the time record of the correction point cloud frame for subsequent fusion of images and point clouds belonging to the same time, thereby reducing the error of the disordered time series on the fusion of subsequent images and point clouds. The embodiment of the present invention extracts image feature points in the time-synchronized image to detect the image feature points using the improved YOLOX detection algorithm, thereby improving the accuracy of subsequent point cloud feature recognition. The accuracy of feature point detection is improved on the basis of the LOX detection algorithm. Furthermore, the embodiment of the present invention extracts frame feature points in the time-synchronized frame to characterize the motion changes of the point cloud in the time-synchronized frame using normal vectors, thereby obtaining implicit motion change features. The embodiment of the present invention establishes a feature conversion matrix between the image feature points and the frame feature points to map the frame feature points to the image feature points using the feature conversion matrix. The image feature points facilitate target identification and can identify point clouds mapped to the image area of ​​the target to be detected. These point clouds are used as point clouds of the target to be detected, thereby reducing the problem of low recognition efficiency caused by the large amount of data in point cloud recognition. Therefore, a multi-sensor fusion positioning method based on lidar point cloud data proposed in an embodiment of the present invention can improve the accuracy of multi-sensor fusion positioning based on lidar point cloud data.

[0139] like Figure 4 The figure shows a functional module diagram of a multi-sensor fusion positioning system based on laser radar point cloud data according to the present invention.

[0140] The multi-sensor fusion positioning system 400 based on laser radar point cloud data of the present invention can be installed in an electronic device. According to the functions implemented, the multi-sensor fusion positioning system based on laser radar point cloud data can include a point cloud acquisition module 401, a time synchronization module 402, a feature extraction module 403, a feature alignment module 404 and a position positioning module 405. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0141] In the embodiment of the present invention, the functions of each module / unit are as follows:

[0142] The point cloud acquisition module 401 is used to obtain a laser radar point cloud frame of the target to be located after scanning the target to be located with a laser radar sensor, calculate a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame, perform coordinate system correction on the laser radar point cloud using the correction transformation matrix to obtain a corrected point cloud, and obtain a corrected point cloud frame of the corrected point cloud;

[0143] The time synchronization module 402 is used to collect the target image of the target to be located by using an image sensor, synchronize the time record of the target image with the time record of the correction point cloud frame, and obtain a time-synchronized image and a time-synchronized frame;

[0144] The feature extraction module 403 is used to extract image feature points in the time-synchronized image and extract frame feature points in the time-synchronized frame;

[0145] The feature alignment module 404 is used to establish a feature conversion matrix between the image feature points and the frame feature points, and use the feature conversion matrix to align the image feature points and the frame feature points to obtain aligned image feature points and aligned frame feature points;

[0146] The position positioning module 405 is used to fuse the aligned image feature points with the aligned frame feature points to obtain fused feature data, and use the fused feature data to locate the target position of the target to be located.

[0147] In detail, each module in the multi-sensor fusion positioning system 400 based on laser radar point cloud data in the embodiment of the present invention is used in the same manner as described above. Figure 1 The multi-sensor fusion positioning method based on lidar point cloud data has the same technical means and can produce the same technical effects, so I will not go into details here.

[0148] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

Claims

1. A multi-sensor fusion positioning method based on laser radar point cloud data, characterized in that: The method comprises: After scanning the target to be located by using a laser radar sensor, a laser radar point cloud frame of the target to be located is obtained, a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame is calculated, a coordinate system correction is performed on the laser radar point cloud by using the correction transformation matrix to obtain a corrected point cloud, and a corrected point cloud frame of the corrected point cloud is obtained; Using an image sensor to collect a target image of the target to be located, synchronizing the time record of the target image with the time record of the correction point cloud frame to obtain a time-synchronized image and a time-synchronized frame; Extracting image feature points in the time-synchronized image, and extracting frame feature points in the time-synchronized frame; Establishing a feature conversion matrix between the image feature points and the frame feature points, and aligning the image feature points and the frame feature points using the feature conversion matrix to obtain aligned image feature points and aligned frame feature points; Fusing the aligned image feature points with the aligned frame feature points to obtain fused feature data, and using the fused feature data to locate the target position of the target to be located; The step of calculating a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame comprises: The correction transformation matrix of the LiDAR point cloud in the LiDAR point cloud frame is calculated using the following formula: ； in, represents the correction transformation matrix, express The laser radar point cloud at time is relative to The relative position of the LiDAR point cloud at time, Indicates the time when the scanning of the lidar point cloud frame starts. Indicates the end time of a scan of the laser radar point cloud frame. represents the fixed scanning time required to scan the first point cloud, j represents the sequence number of the fixed scanning time, i represents the sequence number of the lidar point cloud, represents the starting time of scanning the i-th lidar point cloud, Indicates that when scanning the i-th point cloud, To the end of the scan the time required; The method of using the correction transformation matrix to perform coordinate system correction on the laser radar point cloud to obtain a corrected point cloud includes: Based on the correction transformation matrix, the coordinate system of the laser radar point cloud is corrected using the following formula to obtain a corrected point cloud: ； in, represents the corrected point cloud, represents the three-dimensional point cloud coordinates of the correction point cloud, represents the three-dimensional point cloud coordinates of the laser radar point cloud, represents the correction transformation matrix, Represents the 3D point cloud coordinates of the calibrated point cloud Get the corrected point cloud ; The step of synchronizing the time record of the target image with the time record of the corrected point cloud frame to obtain a time-synchronized image and a time-synchronized frame includes: The synchronization moment between the time record of the target image and the time record of the correction point cloud frame is calculated using the following formula: ； in, Indicates the synchronization time, represents the total number of synchronization moments, represents the u-th moment in the time record of the target image a, represents the vth moment in the time record of the corrected point cloud frame b, Indicates from indivual Select the smallest absolute value of the difference, Indicated in When the absolute value of the difference is the smallest, , Choose one as ; Selecting a target time record of the target image and a target time record of the correction point cloud frame that are consistent with the synchronization moment from the time record of the target image and the time record of the correction point cloud frame respectively; The target time record of the target image and the target image and the correction point cloud frame corresponding to the target time record of the correction point cloud frame are used as the time synchronization image and the time synchronization frame.

2. The method according to claim 1, characterized in that: The extracting of image feature points from the time-synchronized image comprises: The image feature points in the time-synchronized image are extracted using the following formula: ； ； ； ； ； in, represents the image feature points, represents the point-wise convolutional layer, represents batch normalization, represents the local convolutional layer, represents the convolutional layer, represents the time-synchronized image, , , , represents the extracted intermediate features, represents the spatial pyramid pooling layer, represents upsampling, Indicates the feature fusion between low-level feature maps and high-level feature maps. Represents an efficient multi-scale attention module.

3. The method according to claim 1, characterized in that The extracting the frame feature points in the time synchronization frame comprises: The point cloud coordinate matrix of the point cloud in the time synchronization frame is constructed using the following format: ； in, express The point cloud coordinate matrix, represents the three-dimensional coordinates of the point cloud in the time synchronization frame, represents the point cloud in the time synchronization frame, m represents the total number of point clouds in the time synchronization frame, Represents the sequence number of the point cloud in the time synchronization frame; Performing zero-mean conversion on each row of values ​​in the point cloud coordinate matrix to obtain a zero-mean matrix; The covariance matrix of the zero-mean matrix is ​​constructed using the following formula: ； in, represents the covariance matrix, m represents the total number of point clouds in the time synchronization frame, represents the sequence number of the point cloud in the time synchronization frame, ; The eigenvalues ​​and eigenvectors of the covariance matrix are calculated using the following formula: ； in, Indicated by The derived eigenvalues ​​are Indicated by , The derived eigenvectors are represents a diagonal matrix, The diagonal elements of The characteristic value of express The three groups of eigenvectors are composed of three-order matrix eigenvectors. represents the covariance matrix, express The third-order matrix eigenvectors are composed of express The three sets of eigenvectors of the third-order matrix eigenvector U and the diagonal matrix and The product of the eigenvectors of the third-order matrix is ​​equal to the covariance matrix The corresponding eigenvalues ​​and eigenvectors when ; Using the target eigenvector corresponding to the minimum eigenvalue among the eigenvalues ​​in the eigenvector as the normal vector of the point cloud in the time synchronization frame; Based on the normal vector, the curvature of the point cloud in the time synchronization frame is calculated using the following formula: ； in, represents the curvature, represents the normal vector of the s-th point cloud in the time synchronization frame, represents the normal vectors of the point clouds other than the s-th point cloud in the time synchronization frame; Based on the curvature, the frame feature points in the time synchronization frame are calculated using the following formula: ； in, represents the frame feature point, represents the fully connected layer, represents a multi-layer perceptron, represents the curvature.

4. The method according to claim 1, characterized in that: The establishing of a feature conversion matrix between the image feature points and the frame feature points comprises: The feature conversion matrix between the image feature points and the frame feature points is established using the following formula: ； in, represents the feature transformation matrix, represents the matrix composed of the internal and external parameters of the image sensor corresponding to the image feature points, f represents the total number of internal and external parameters of the image sensor, represents a matrix composed of the internal and external parameters of the lidar sensor corresponding to the frame feature points, and q represents the total number of the internal and external parameters of the lidar sensor.

5. The method according to claim 1, characterized in that The fusing the aligned image feature points with the aligned frame feature points to obtain fused feature data includes: Detecting a target area of ​​the target to be located in the aligned image feature points; Identifying a first target feature point in the aligned frame feature points that is within the target area; The second target feature point corresponding to the first target feature point in the frame feature point is used as the fused feature data.

6. The method according to claim 1, characterized in that The method of locating the target position of the target to be located by using the fused feature data includes: The center coordinates of the fused feature data are calculated using the following formula: ； in, represents the center coordinates of the fused feature data, Represents the coordinates of the fused feature data; The center coordinates are used as the target position of the object to be located.

7. A multi-sensor fusion positioning system based on laser radar point cloud data, the system implementing the method according to any one of claims 1 to 6, characterized in that: The system comprises: A point cloud acquisition module is used to obtain a laser radar point cloud frame of the target to be located after scanning the target to be located with a laser radar sensor, calculate a correction transformation matrix of the laser radar point cloud in the laser radar point cloud frame, perform coordinate system correction on the laser radar point cloud using the correction transformation matrix to obtain a corrected point cloud, and obtain a corrected point cloud frame of the corrected point cloud; A time synchronization module, used to collect a target image of the target to be located by using an image sensor, synchronize the time record of the target image with the time record of the correction point cloud frame, and obtain a time-synchronized image and a time-synchronized frame; A feature extraction module, used to extract image feature points in the time-synchronized image and frame feature points in the time-synchronized frame; A feature alignment module, used to establish a feature conversion matrix between the image feature points and the frame feature points, and to align the image feature points and the frame feature points using the feature conversion matrix to obtain aligned image feature points and aligned frame feature points; The position positioning module is used to fuse the aligned image feature points with the aligned frame feature points to obtain fused feature data, and use the fused feature data to locate the target position of the target to be located.

Images