Line feature extraction model training method and device, point cloud matching method and device

Abstract

The embodiments of this specification provide a training method and device for a line feature extraction model, and a point cloud matching method and device, wherein the training method for the line feature extraction model includes: constructing a simulated point cloud training set; then training the point cloud detector according to the simulated point cloud training set to obtain an initial extraction model; then obtaining at least two frames of real point cloud data, and determining reference line features corresponding to at least two frames of real point cloud data according to the initial extraction model; then determining a real point cloud training set according to the reference line features; then training the initial extraction model according to the real point cloud training set to obtain a line feature extraction model. In this way, the knowledge learned from the simulated point cloud data can be transferred to the real point cloud data, thereby improving the training accuracy of the line feature extraction model.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of intelligent driving technology, and in particular to a training method for a line feature extraction model. One or more embodiments of this specification also relate to a point cloud matching method, a training device for a line feature extraction model, a point cloud matching device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer and Internet technology, intelligent driving technology has also flourished. Intelligent driving technology combines a variety of sensors to perceive the surrounding environment, such as laser sensors, sonar, global positioning systems, odometers and inertial measurement units. The control system can interpret the sensor information to identify appropriate navigation paths, obstacles and related signs.

[0003] Intelligent driving relies on high-precision positioning technology. Laser sensors can sense the position of external obstacles and store and compare them in the form of three-dimensional point cloud data to calculate the position change of the sensor itself relative to external objects. In the existing technology, feature points can be extracted and described based on the three-dimensional point cloud, and the three-dimensional point cloud data of the previous and next frames can be matched to determine the position change of the sensor itself relative to external objects. The accuracy of matching the three-dimensional point cloud data of the previous and next frames based on the point features in the three-dimensional point cloud is low. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a training method for a line feature extraction model. One or more embodiments of this specification also relate to a point cloud matching method, a training device for a line feature extraction model, a point cloud matching device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a training method for a line feature extraction model is provided, comprising:

[0006] Constructing a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label;

[0007] According to the simulation point cloud training set, the point cloud detector is trained to obtain an initial extraction model;

[0008] Acquire at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to the initial extraction model;

[0009] Determine a real point cloud training set according to the reference line feature, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label;

[0010] The initial extraction model is trained according to the real point cloud training set to obtain the line feature extraction model.

[0011] Optionally, constructing a simulation point cloud training set includes:

[0012] Constructing at least two mesh models according to at least two types of preset vertices and edges;

[0013] Sampling at least two grid models to obtain simulation point cloud data corresponding to the at least two grid models;

[0014] Setting a first line feature label for a simulation sample point in the simulation point cloud data, wherein the first line feature label is used to indicate whether the simulation sample point is a straight line point or a non-straight line point;

[0015] According to the simulation point cloud data, a simulation point cloud training set is constructed.

[0016] Optionally, setting a first-line feature label for a simulation sample point in the simulation point cloud data includes:

[0017] Performing linear interpolation on preset vertices and edges of at least two mesh models to obtain straight line sample points included in at least two mesh models;

[0018] Searching for extended sample points whose distances from straight line sample points are less than a distance threshold in the non-straight line sample points of the simulation point cloud data set;

[0019] The first line feature labels of the straight line sample points and the extended sample points in at least two grid models are set as straight line points, and the first line feature labels of other sample points are set as non-straight line points.

[0020] Optionally, constructing a simulation point cloud training set based on the simulation point cloud data includes:

[0021] Obtaining a third set number of non-linear sample points from the real point cloud data as noise data;

[0022] According to the preset ratio, the simulated point cloud data and the noise data are mixed to obtain the simulated point cloud training set.

[0023] Optionally, the point cloud detector is trained according to the simulated point cloud training set to obtain an initial extraction model, including:

[0024] The simulation point cloud training set is input into the point cloud detector to obtain the first prediction line feature of each simulation sample point in the simulation point cloud training set;

[0025] According to the first predicted line features and the first line feature labels of each simulated sample point, the first loss value of the point cloud detector is calculated, the parameters of the point cloud detector are adjusted based on the first loss value, and the operation steps of constructing a simulated point cloud training set are returned to execute until the first training stop condition is reached, thereby obtaining a trained initial extraction model.

[0026] Optionally, calculating a first loss value of the point cloud detector according to the first predicted line feature and the first line feature label of each simulation sample point includes:

[0027] According to the ratio of positive samples to negative samples in each simulation sample point, the weight coefficients of positive samples and negative samples are determined, wherein the positive samples are points in the simulation sample points whose first line feature labels are straight line points, and the negative samples are points in the simulation sample points whose first line feature labels are non-straight line points;

[0028] Calculate a first sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the positive sample;

[0029] Calculate a second sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the negative sample;

[0030] According to the weight coefficients of the positive samples and the negative samples, the first sub-loss value and the second sub-loss value are weightedly fused to obtain the first loss value of the point cloud detector.

[0031] Optionally, according to the reference line features, a real point cloud training set is determined, including:

[0032] According to the reference line features, determine the target straight line sample points whose detection rate is greater than the detection rate threshold among the real straight line sample points of at least two frames of real point cloud data;

[0033] Performing expansion processing on the target straight line sample points to obtain expanded straight line sample points, wherein the expansion processing includes region growing, nearest neighbor interpolation and / or straight line fitting processing;

[0034] The second line feature labels of the target straight line sample points and the expanded straight line sample points are set as straight line points, and the second line feature labels of other sample points are set as non-straight line points to obtain a real point cloud training set.

[0035] Optionally, the initial extraction model is trained according to a real point cloud training set to obtain a line feature extraction model, including:

[0036] Input the real point cloud training set into the initial extraction model to obtain the second prediction line features of each real sample point in the real point cloud training set;

[0037] According to the second predicted line features and the second line feature labels of each real sample point, the second loss value of the initial extraction model is calculated, the model parameters of the initial extraction model are adjusted based on the second loss value, and the operation step of acquiring at least two frames of real point cloud data is returned to execute until the second training stop condition is reached, thereby obtaining a trained line feature extraction model.

[0038] According to a second aspect of an embodiment of this specification, a point cloud matching method is provided, including:

[0039] Get the two frames of point cloud data collected;

[0040] Inputting two frames of point cloud data into a line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the line feature extraction model of any one of the first aspects above;

[0041] According to the straight line points in the two frames of point cloud data, point cloud matching is performed on the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0042] Optionally, the two frames of point cloud data include a first frame of point cloud data and a second frame of point cloud data;

[0043] According to the straight line points in the two frames of point cloud data, point cloud matching is performed on the two frames of point cloud data to obtain the pose transformation parameters between the two frames of point cloud data, including:

[0044] According to the straight line points in the two frames of point cloud data, the straight lines included in the two frames of point cloud data are determined respectively;

[0045] Each straight line in the first frame of point cloud data is matched with the corresponding target straight line in the second frame of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0046] According to a third aspect of an embodiment of this specification, a training device for a line feature extraction model is provided, comprising:

[0047] A construction module is configured to construct a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label;

[0048] A first training module is configured to train a point cloud detector to obtain an initial extraction model based on a simulated point cloud training set;

[0049] A first determination module is configured to obtain at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to an initial extraction model;

[0050] A second determination module is configured to determine a real point cloud training set according to the reference line feature, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label;

[0051] The second training module is configured to train the initial extraction model according to the real point cloud training set to obtain the line feature extraction model.

[0052] According to a fourth aspect of the embodiments of this specification, a point cloud matching device is provided, including:

[0053] An acquisition module is configured to acquire two frames of collected point cloud data;

[0054] An input module is configured to input two frames of point cloud data into a line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the line feature extraction model of any one of the first aspects above;

[0055] The matching module is configured to perform point cloud matching on two frames of point cloud data according to straight line points in the two frames of point cloud data to obtain posture transformation parameters between the two frames of point cloud data.

[0056] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0057] Memory and processor;

[0058] The memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the training method of the line feature extraction model of any one of the first aspects above or the steps of the point cloud matching method of any one of the second aspects above.

[0059] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the training method of the line feature extraction model of any one of the first aspects or the steps of the point cloud matching method of any one of the second aspects.

[0060] An embodiment of the present specification provides a training method for a line feature extraction model, which first constructs a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label; then, based on the simulation point cloud training set, the point cloud detector is trained to obtain an initial extraction model; then, at least two frames of real point cloud data are acquired, and based on the initial extraction model, reference line features corresponding to at least two frames of real point cloud data are determined; then, based on the reference line features, a real point cloud training set is determined, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label; then, the initial extraction model is trained based on the real point cloud training set to obtain a line feature extraction model. In this case, the point cloud detector can be preliminarily trained based on the constructed simulation training samples, and then the initial extraction model obtained from the preliminary training can be used to label the real point cloud data to obtain a real point cloud training set. The initial extraction model is then further trained based on the real point cloud training set. The training accuracy of the line feature extraction model is relatively high. Subsequently, the straight line features in the point cloud data in the real scene can be accurately identified based on the line feature extraction model, and then the two frames of point cloud data can be matched based on the straight line features to improve the matching accuracy of the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of a method for training a line feature extraction model provided by an embodiment of this specification;

[0062] Figure 2a It is a schematic diagram of a process of constructing a simulation point cloud training set provided by an embodiment of this specification;

[0063] Figure 2b is a schematic diagram of a network structure of a point cloud detector provided by an embodiment of this specification;

[0064] Figure 2c is a schematic diagram of a training process of a line feature extraction model provided by an embodiment of this specification;

[0065] Figure 3 is a flow chart of a point cloud matching method provided by an embodiment of this specification;

[0066] Figure 4 It is a structural schematic diagram of a training device for a line feature extraction model provided by an embodiment of this specification;

[0067] Figure 5 is a schematic diagram of the structure of a point cloud matching device provided by an embodiment of this specification;

[0068] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0069] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0070] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0071] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0072] First, the terms involved in one or more embodiments of this specification are explained.

[0073] Autonomous positioning: The software automatically receives sensor data, runs positioning algorithms, and estimates the 3D position and 3D posture of the device.

[0074] Point cloud: It is a collection of massive points that express the spatial distribution and surface characteristics of the target in the same spatial reference system. The point cloud obtained based on the laser measurement principle includes three-dimensional coordinates (XYZ) and laser reflection intensity (Intensity).

[0075] LiDAR Point Cloud: The surrounding obstacles sensed and recorded by laser sensors are recorded in the computer in the form of three-dimensional scattered point data.

[0076] LiDAR Point Cloud Registration: Given two frames of laser point clouds, the pairing relationship between the points in the two frames of laser point clouds is solved, and the calculation process of restoring the three-dimensional transformation between the laser point clouds is restored.

[0077] Iterative Closest Point (ICP): A laser point cloud matching method that directly pairs the closest points in two point clouds to restore the 3D transformation between the point clouds.

[0078] LiDAR Point Cloud Features: Feature information extracted from the LiDAR point cloud by algorithms to facilitate matching and tracking between LiDAR point clouds.

[0079] KITTI Dataset: It is a computer vision algorithm evaluation dataset in the scenario of autonomous driving. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection and 3D tracking in the vehicle environment. The KITTI dataset contains real image data collected in scenes such as urban areas, rural areas and highways. Each image contains up to 15 vehicles and 30 pedestrians, as well as various degrees of occlusion and truncation. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2km of visual odometry sequences and more than 200k images of 3D annotated objects, sampled and synchronized at a frequency of 10Hz.

[0080] Dynamic Graph Convolutional Neural Network (DGCNN): A convolutional neural network architecture that can directly use point clouds as input, and can obtain enough local information for tasks such as point cloud classification and segmentation.

[0081] Position and Orientation: Indicates the position and attitude, including pitch angle, yaw angle and roll angle.

[0082] Rigid transformation: A transformation that only changes the position of an object without changing its shape and size, that is, a linear transformation in which the distance between points remains unchanged, the angle between lines remains unchanged, and the dot product remains unchanged is a rigid transformation.

[0083] It should be noted that in the context of tackling new logistics and promoting the development of smart cities, unmanned driving and its related technologies are one of the important sectors. On open roads, the use of low-cost equipment to achieve high-precision and robust autonomous positioning can provide accurate location information and path planning basis for the vehicle's automatic driving. The positioning system based on the laser sensor needs to complete the registration between the two frames of laser point clouds before and after, so as to calculate the relative position of the sensor at two times and in the map. Due to the actual scanning characteristics of the laser sensor itself that the resolution decreases with the distance from the object, the matching method for extracting feature points from the laser point cloud often has problems such as low repeatability of feature points and accuracy affected by resolution. Therefore, this specification proposes a method for automatically obtaining straight line features (corner dividing lines, rod-shaped objects, etc.) representing the actual scene in the point cloud, so as to more clearly capture the features in the point cloud and complete the relative position calculation. The method provided in the embodiment of this specification can be deployed and used on conventional embedded platforms, such as NVIDIA Jetson.

[0084] At present, the original point cloud can be projected into two-dimensional space along different directions, and the two-dimensional straight line support area can be extracted from the two-dimensional image. The two-dimensional straight line support area can be back-projected into three-dimensional space to obtain the three-dimensional straight line support area, thereby extracting the straight line features in the point cloud and performing point cloud matching. However, the above scheme is based on the point cloud projection method, which projects the point cloud into two-dimensional space along different directions and extracts line segments on the two-dimensional image. It is only applicable to uniformly distributed structured point clouds. The single-frame LiDAR point cloud is not evenly distributed after being projected into two-dimensional space. The two-dimensional straight line extraction method is difficult to extract complete straight lines, and it is easy to have mislabeling and missing labels. Therefore, this scheme cannot be applied to single-frame LiDAR point clouds.

[0085] In addition, the original point cloud can be segmented into a set of small planes, the edge points of the small planes with direction vectors can be extracted, and the edge points can be clustered into straight lines according to the direction vectors to perform point cloud matching. However, the above scheme extracts straight lines based on the method of local plane segmentation of point clouds and plane edge point clustering, which has high requirements on the point cloud plane and is also only applicable to uniformly distributed structured point clouds. Single-frame lidar point clouds have missing and empty areas, which will generate a large number of false detection points when extracting plane edge points, resulting in low accuracy of straight line extraction.

[0086] Furthermore, if you want to train an extraction model that can detect straight line points in a point cloud, you need a large amount of point cloud data with straight line feature annotations. Currently, you can use manual annotation methods to annotate the straight line points in each frame of the point cloud. However, this solution has a low degree of automation and requires a lot of manpower costs. It does not have the ability to be directly deployed on unmanned vehicles. In addition, this method is difficult to achieve a high annotation accuracy and is prone to mislabeling and missing labels.

[0087] Therefore, a training method for a line feature extraction model is provided in an embodiment of the present specification. The point cloud detector can be preliminarily trained based on the constructed simulation training samples, and then the initial extraction model obtained by the preliminary training is used to label the real point cloud data to obtain a real point cloud training set, and then the initial extraction model is further trained based on the real point cloud training set. The model training method of migrating from simulated three-dimensional point cloud to real three-dimensional point cloud is adopted, and manual labeling is not required, which can improve the labeling accuracy and ensure the training accuracy of the line feature extraction model.

[0088] In this specification, a training method for a line feature extraction model is provided. This specification also involves a point cloud matching method, a training device for a line feature extraction model, a point cloud matching device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0089] Figure 1 A flowchart of a method for training a line feature extraction model provided according to an embodiment of the present specification is shown, including steps 102 to 110.

[0090] Step 102: construct a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label.

[0091] It should be noted that the simulated point cloud training set refers to a set of forged three-dimensional point clouds that imitate real scenes, which is used for preliminary training of the point cloud detector. The first set quantity is the number of training sample points included in the simulated point cloud training set. The first line feature label refers to the sample label of the simulated sample point, which is used to indicate whether the simulated sample point is a straight line point or a non-straight line point.

[0092] In practical applications, we can first imitate the real scene to forge a simulated point cloud training set. Each simulated sample point in the forged simulated point cloud training set can carry a corresponding sample label, which can be used for subsequent preliminary training of the point cloud detector based on the simulated point cloud training set to provide initial training data for the point cloud detector.

[0093] In an optional implementation of this embodiment, a simulation point cloud training set is constructed, and the specific implementation process may be as follows:

[0094] Constructing at least two mesh models according to at least two types of preset vertices and edges;

[0095] Sampling at least two grid models to obtain simulation point cloud data corresponding to the at least two grid models;

[0096] Setting a first line feature label for a simulation sample point in the simulation point cloud data, wherein the first line feature label is used to indicate whether the simulation sample point is a straight line point or a non-straight line point;

[0097] According to the simulation point cloud data, a simulation point cloud training set is constructed.

[0098] It should be noted that there may be many different types of straight lines in real scenes, such as plane intersections (such as wall corner dividing lines) and rod-shaped objects (such as indicator poles, flagpoles, etc.). Therefore, for different types of straight lines, corresponding mesh models can be constructed based on preset vertices and edges, which serve as the skeleton structure for subsequent sampling to obtain point cloud data. In specific implementation, since different types of straight lines have different distribution characteristics, corresponding preset vertices and edges can be pre-set for any type of straight line. By connecting the preset vertices and edges, the mesh model corresponding to the straight line type can be pulled up, that is, the mesh model is connected by the corresponding preset vertices and edges.

[0099] In addition, after constructing the grid models corresponding to different types of straight lines, each grid model can be sampled to obtain the simulation point cloud data corresponding to each grid model, that is, a plurality of simulation sample points are sampled from the grid model to complete the preparation of training data, and a first-line feature label is added to each simulation sample point to indicate whether the simulation sample point is a straight line point or a non-straight line point. When sampling each grid model, a preset library function (such as an open3d library function) can be called to uniformly sample the grid model. The initial number of sampling points can be set to be relatively large (such as 10,000), and then the number of point clouds can be reduced to the required number of sampling points (such as 1,000) by using grid downsampling, or the required number of sampling points (such as 1,000) can be obtained by uniformly sampling directly from the grid model.

[0100] For example, Figure 2a 1 is a schematic diagram of a process of constructing a simulation point cloud training set provided by an embodiment of the present specification. Taking the straight line type as a plane-plane intersection line and a rod-shaped object as an example, for the plane-plane intersection line type, the corresponding plane-plane intersection line mesh model can be constructed based on the corresponding preset vertices and edges, such as Figure 2a As shown; for the rod-shaped object type, the corresponding rod-shaped object mesh model can be constructed based on the corresponding preset vertices and edges, as shown Figure 2a Then, the surface intersection line grid model and the rod-shaped object grid model can be sampled respectively to obtain the simulation point cloud data corresponding to the surface intersection line grid model and the simulation point cloud data corresponding to the rod-shaped object grid model, and the corresponding first line feature label can be set for each simulation sample point in the above simulation point cloud data.

[0101] In addition, when sampling at least two grid models to obtain simulation point cloud data corresponding to at least two grid models, each sampling point obtained by sampling can also be rigidly transformed to change the posture of the sampling point, but the positional relationship between the sampling points is not changed, such as moving the sampling points up by a preset amplitude (such as 1%), or rotating the sampling points by a preset angle, etc., so as to obtain the final simulation point cloud data, thereby improving the richness of the simulation point cloud data.

[0102] In the embodiments of the present specification, a corresponding grid model can be constructed and generated based on the general pattern of line features in the three-dimensional point cloud in the real scene, and then sampled into point cloud data to complete the preparation of training data, thereby improving the similarity between the simulated point cloud training set and the point cloud distribution in the real scene, ensuring the effectiveness of the constructed simulated point cloud training set, and further ensuring the accuracy of the subsequent preliminary training of the point cloud detector.

[0103] In an optional implementation of this embodiment, the sampled straight line points may be expanded, that is, the first line feature labels are set for the simulated sample points in the simulated point cloud data. The specific implementation process may be as follows:

[0104] Performing linear interpolation on preset vertices and edges of at least two mesh models to obtain straight line sample points included in at least two mesh models;

[0105] Searching for extended sample points whose distances from straight line sample points are less than a distance threshold in the non-straight line sample points of the simulation point cloud data set;

[0106] The first line feature labels of the straight line sample points and the extended sample points in at least two grid models are set as straight line points, and the first line feature labels of other sample points are set as non-straight line points.

[0107] It should be noted that in real scenes, the straight line points in the point cloud data may not be completely located on a straight line, and may float slightly up and down along the straight line. Therefore, when obtaining simulated point cloud data and setting the first-line feature label for each simulated sample point in the simulated point cloud data, the non-straight line point closest to the straight line point in the simulated point cloud data can also be set as a straight line point.

[0108] In practical applications, the edge between two preset vertices is a straight line, that is, linear interpolation is performed on the edge between the two preset vertices to obtain straight line sample points in the mesh model, and then an extended sample point whose distance from the straight line sample point is less than a distance threshold can be searched among the non-straight line sample points in the simulation point cloud data set. The extended sample point is close to the edge between the two preset vertices, that is, it floats slightly around the edge between the two preset vertices. Therefore, the extended sample point can also be set as a straight line point, and then the simulation sample points other than the straight line sample points and the extended sample points in the sampled simulation point cloud data are set as non-straight line points.

[0109] In the embodiments of the present specification, based on the distribution of point cloud data in real scenes, non-straight points floating around straight lines can also be expanded to straight line points, thereby being closer to the point cloud distribution in real scenes, ensuring the effectiveness of the constructed simulated point cloud training set, and further ensuring the accuracy of the subsequent preliminary training of the point cloud detector.

[0110] In an optional implementation of this embodiment, considering the presence of a large amount of noise and non-linear feature data in the point cloud data in the real scene, the constructed simulation point cloud training set can be strengthened by superimposing noise and adding non-linear feature point clouds, that is, constructing a simulation point cloud training set based on the simulation point cloud data. The specific implementation process can be as follows:

[0111] Obtaining a third set number of non-linear sample points from the real point cloud data as noise data;

[0112] According to the preset ratio, the simulated point cloud data and the noise data are mixed to obtain the simulated point cloud training set.

[0113] It should be noted that the third set number is the number of added noise data. The third set number can be set based on the number of simulated point cloud data obtained from the grid model. The number of simulated point cloud data and noise data of non-linear sample points obtained from the grid model satisfies the set ratio. For example, the ratio of simulated point cloud data to noise data can be 4:1. If the number of simulated sample points in the simulated point cloud data is 4000, then the number of non-linear sample points can be 1000. The mixed simulated point cloud training set includes 5000 simulated sample points, that is, the first set number is 5000.

[0114] In practical applications, the laser point cloud of the KITTI dataset can be grid downsampled, and some non-straight point clouds can be intercepted as the noise data, which are added to the simulated point cloud data obtained from the grid model. Finally, a set of training data is generated, namely the simulated point cloud training set. The simulated point cloud training set can include 4 sets of simulated point cloud data and 1 set of noise data. The 4 sets of simulated point cloud data can be composed of simulated point cloud data obtained by sampling different types of grid models, such as 2 sets of simulated point cloud data obtained by sampling face-to-face intersection grid models and 2 sets of simulated point cloud data obtained by sampling rod-shaped grid models, or 3 sets of simulated point cloud data obtained by sampling face-to-face intersection grid models and 1 set of simulated point cloud data obtained by sampling rod-shaped grid models, etc.

[0115] Using the above example, Figure 2a As shown, the simulated point cloud data obtained by sampling two groups of surface intersection line grid models, the simulated point cloud data obtained by sampling two groups of rod-shaped object grid models, and the noise data can be mixed to obtain a simulated point cloud training set.

[0116] In the embodiments of the present specification, there is a large amount of noise and non-straight line feature data in the point cloud data in the real scene. Therefore, noise data of non-straight line sample points can be added to the simulated point cloud data to enhance the diversity of the simulated point cloud training set. This makes the simulated point cloud training set closer to the point cloud distribution in the real scene, thereby ensuring the effectiveness of the constructed simulated point cloud training set and further ensuring the accuracy of the subsequent preliminary training of the point cloud detector.

[0117] Step 104: Based on the simulation point cloud training set, the point cloud detector is trained to obtain an initial extraction model.

[0118] It should be noted that, through the above-mentioned process of forging labeled data, a simulated point cloud training set that simulates the straight line features in the laser point cloud in the real scene can be generated in large quantities at low cost. The simulated point cloud training set includes a first set number of simulated sample points, and each simulated sample point carries a first line feature label. Therefore, the simulated point cloud training set can be input into the point cloud detector, and the point cloud detector is initially trained based on the line features and the first line feature labels predicted by the point cloud detector. Based on the training of the simulated point cloud training set, the general pattern of straight line features in the laser point cloud can be established and learned, providing a reference basis for subsequent training on real point cloud data.

[0119] In practical applications, the point cloud detector can be a DGCNN network, and the training platform can use an i77820x CPU, a Titan XP GPU, and an Adam optimizer. The initial learning rate is 0.001, and the learning rate decreases by 50% every 40,000 training steps.

[0120] In an optional implementation of this embodiment, the point cloud detector may be initially trained based on the line features and the first line feature labels predicted by the point cloud detector, that is, the point cloud detector may be trained to obtain an initial extraction model based on the simulated point cloud training set. The specific implementation process may be as follows:

[0121] The simulation point cloud training set is input into the point cloud detector to obtain the first prediction line feature of each simulation sample point in the simulation point cloud training set;

[0122] According to the first predicted line features and the first line feature labels of each simulated sample point, the first loss value of the point cloud detector is calculated, the parameters of the point cloud detector are adjusted based on the first loss value, and the operation steps of constructing a simulated point cloud training set are returned to execute until the first training stop condition is reached, thereby obtaining a trained initial extraction model.

[0123] In practical applications, the network structure of the point cloud detector is as follows: Figure 2b As shown, Figure 2bis a schematic diagram of a network structure of a point cloud detector provided by an embodiment of this specification, such as Figure 2b As shown in the figure, the point cloud detector is composed of three cascaded edge convolution modules, and the output of each edge convolution module is an n*64-dimensional feature vector. The simulated point cloud training set (n*3 dimensions) is input into the first edge convolution module of the point cloud detector, and the output of the previous edge convolution module is used as the input of the next edge convolution module. Finally, the output features of the three edge convolution modules are connected together and pooled through a fully connected layer to obtain a 1024-dimensional global feature. Then, the process is repeated to obtain an n*1024-dimensional global feature, which is then connected to the output features of the three edge convolution modules and the first prediction line features of each simulated sample point are obtained through a fully connected layer.

[0124] In addition, the first training stop condition is a condition that needs to be met for the point cloud detector to stop training. For example, the first training stop condition can be that the first loss value is less than the loss value threshold, or the number of iterations exceeds the number threshold. Among them, the loss value threshold is used to determine whether the difference between the predicted result and the actual result of the point cloud detector is small enough, and is generally set to be small; the number threshold is used to determine whether the number of iterations of the point cloud detector is large enough. Generally, convergence can be achieved through 50 iterations, and the accuracy rate can reach 98%, so the number threshold can be set to 50.

[0125] It should be noted that the first line feature label refers to the actual result of the simulation sample point, and the first predicted line feature is the line feature predicted by the point cloud detector for the simulation sample point. Therefore, based on the first predicted line feature and the first line feature label of the simulation sample point, the first loss value of the point cloud detector can be calculated, and then the model parameters of the point cloud detector can be adjusted according to the first loss value, and the operation steps of constructing a simulation point cloud training set can be returned to continue until the first training stop condition is reached, and the initial extraction model after the initial training is completed is obtained.

[0126] In specific implementation, the cross entropy loss function can be calculated based on the first predicted line features and first line feature labels of the simulated sample points to generate a loss value. Among them, the first line feature label refers to the processing result that the point cloud detector actually wants to output, that is, the first line feature label is the real result, and the simulated point cloud training set is input into the point cloud detector, and the first predicted line feature of the output simulated sample point is the predicted result. When the difference between the predicted result and the real result is small enough, it means that the predicted result is close enough to the real result, or when the number of iterations exceeds the number threshold, it means that enough rounds of training have been completed. At this time, the initial training of the point cloud detector is completed and the initial extraction model can be obtained.

[0127] In the embodiments of this specification, the difference between the predicted result and the actual result of the point cloud detector can be intuitively shown by calculating the loss value. Then, the point cloud detector is trained in a targeted manner and the parameters are adjusted, which can effectively improve the training rate and training effect of the point cloud detector.

[0128] Furthermore, after constructing the simulated point cloud training set, a preset proportion of simulated sample points can be selected from the simulated point cloud training set as the first training set, and the remaining simulated sample points can be used as the first test set. The first training set is input into the point cloud detector to obtain the first predicted line features of each simulated sample point in the first training set, thereby calculating the first loss value of the point cloud detector based on the first predicted line features and the first line feature labels of the simulated sample points in the first training set, and training the point cloud detector.

[0129] The ratio of the number of simulation sample points in the first training set to the number of simulation sample points in the first test set may be a preset ratio, such as the ratio of the first training set to the first test set is 4:1.

[0130] After the training is completed, the trained initial extraction model can be tested based on the first test set to determine whether the performance of the trained initial extraction model meets the requirements. Specifically, the first test set can be input into the initial extraction model to obtain the first prediction line features of each simulation sample point in the first test set. Based on the first prediction line features and the first line feature labels of the simulation sample points in the first test set, the recognition accuracy of the initial extraction model is calculated. If the recognition accuracy is greater than or equal to the accuracy threshold, it is determined that the performance of the trained initial extraction model meets the requirements, and subsequent labeling of the real point cloud data can be performed. If the recognition accuracy is lower than the accuracy threshold, it means that the performance of the trained initial extraction model does not meet the requirements. At this time, you can return to the operation steps of constructing a simulation point cloud training set, and continue to construct a simulation point cloud training set to further train the initial extraction model.

[0131] In an optional implementation of this embodiment, since the positive and negative samples in the simulation point cloud training set are unevenly distributed, and the negative samples are far more than the positive samples, the weight of the positive samples can be increased when calculating the loss value to solve the problem of imbalanced classification labels, that is, according to the first predicted line features and the first line feature labels of each simulation sample point, the first loss value of the point cloud detector is calculated. The specific implementation process can be as follows:

[0132] According to the ratio of positive samples to negative samples in each simulation sample point, the weight coefficients of positive samples and negative samples are determined, wherein the positive sample is a point in the simulation sample point whose first line feature label is a straight line point, and the negative sample is a point in the simulation sample point whose first line feature label is a non-straight line point, and the weight coefficient of the positive sample is greater than the weight coefficient of the negative sample;

[0133] Calculate a first sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the positive sample;

[0134] Calculate a second sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the negative sample;

[0135] According to the weight coefficients of the positive samples and the negative samples, the first sub-loss value and the second sub-loss value are weightedly fused to obtain the first loss value of the point cloud detector.

[0136] It should be noted that the positive sample is the point whose first line feature label is a straight line point in the simulated sample point, and the negative sample is the point whose first line feature label is a non-straight line point in the simulated sample point. Since the simulated sample points in the simulated point cloud training set include the simulated point cloud data obtained by sampling from the grid model and the non-straight line sample points obtained from the real point cloud data, and most of the points in the simulated point cloud data obtained by sampling from the grid model are non-straight line points, only a few are straight line points, such as Figure 2a As shown in the figure, the positive and negative samples in the simulation point cloud training set are unevenly distributed, and the negative samples are far more than the positive samples.

[0137] In practical applications, when the weight coefficients of positive and negative samples are determined based on the ratio of positive samples to negative samples, the weight coefficient of positive samples can be set much larger than that of negative samples due to the uneven distribution of positive and negative samples in the simulation point cloud training set, where negative samples far outnumber positive samples. For example, if the ratio of positive and negative samples in the simulation point cloud training set is 1:999, the weight coefficient of positive samples can be set to 0.9 and the weight coefficient of negative samples to 0.1, thereby increasing the influence of positive samples on the point cloud detector during training.

[0138] In an embodiment of the present specification, the ratio of positive samples to negative samples in the simulation sample points can be determined, and then the weight coefficients of the positive samples and the negative samples are respectively determined based on the ratio, and the weight coefficient of the positive sample is set to be greater than the weight coefficient of the negative sample. Then, the first sub-loss value of the point cloud detector is calculated based on the positive sample, and the second loss value of the point cloud detector is calculated based on the negative sample. Then, based on the weight coefficients of the positive sample and the negative sample, the first sub-loss value and the second sub-loss value are weightedly fused to obtain the first loss value of the point cloud detector, so that the weight of the positive sample can be increased when calculating the loss value to solve the problem of imbalance in classification labels.

[0139] Step 106: Acquire at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to the initial extraction model.

[0140] It should be noted that at least two frames of real point cloud data can be obtained from a real point cloud database, which can be a KITTI dataset. Specifically, at least two frames of laser point clouds in the KITTI dataset can be grid downsampled, and the grid size can be 0.25 meters. Each frame of laser point cloud can select a set number (such as 10,000) points closest to the zero point as the real sample points of the laser point cloud of this frame, and then perform a random rigid transformation on the real sample points of the laser point cloud of this frame. In addition, considering the planar motion characteristics of the data acquisition platform of the KITTI dataset (such as the vehicle can only be on the plane and cannot be flipped along the pitch angle), the rigid transformation here can be a two-dimensional rigid transformation. The number of transformations of the real sample points of each frame of laser point cloud can be pre-set, such as 200 times, and so on. At least two frames of real point cloud data can be obtained, and each frame of real point cloud data includes the set number of real sample points.

[0141] In practical applications, after obtaining at least two frames of real point cloud data, the at least two frames of real point cloud data can be input into a preliminarily trained initial extraction model. The initial extraction model can output predicted reference line features for each real sample point in the at least two frames of real point cloud data, that is, the initial extraction model can predict whether the type of each real sample point in the at least two frames of real point cloud data is a straight line point or a non-straight line point, that is, the reference line feature is used to indicate the type of each real sample point in the at least two frames of real point cloud data.

[0142] In an embodiment of the present specification, after obtaining at least two frames of real point cloud data, the at least two frames of real point cloud data can be input into a preliminarily trained initial extraction model, and the at least two frames of real point cloud data obtained can be labeled based on the initial extraction model to obtain training data in a real scene. Subsequently, the initial extraction model can be further trained to improve the accuracy of the line feature extraction model.

[0143] Step 108: Determine a real point cloud training set according to the reference line features, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label.

[0144] It should be noted that the real point cloud training set refers to a set of three-dimensional point clouds in real scenes, which is used to further adjust and train the initial extraction model. The second set number is the number of training sample points included in the real point cloud training set. The second line feature label refers to the sample label of the real sample point, which is used to indicate whether the real sample point is a straight line point or a non-straight line point. The real sample point carrying the second line feature label can be determined based on the reference line feature predicted by the initial extraction model.

[0145] In an optional implementation of this embodiment, the second line feature label of each real sample point in the real point cloud training set can be determined according to the detection rate of each real sample point in at least two frames of real point cloud data by the initial extraction model, that is, the real point cloud training set is determined according to the reference line feature. The specific implementation process can be as follows:

[0146] According to the reference line features, determine the target straight line sample points whose detection rate is greater than the detection rate threshold among the real straight line sample points of at least two frames of real point cloud data;

[0147] Performing expansion processing on the target straight line sample points to obtain expanded straight line sample points, wherein the expansion processing includes region growing, nearest neighbor interpolation and / or straight line fitting processing;

[0148] The second line feature labels of the target straight line sample points and the expanded straight line sample points are set as straight line points, and the second line feature labels of other sample points are set as non-straight line points to obtain a real point cloud training set.

[0149] It should be noted that each frame of real point cloud data may include multiple real sample points, including real straight line sample points and real non-straight line sample points. For the real straight line sample points in each frame of real point cloud data, if the reference line feature output by the initial detection model is also a straight line point, it can be determined that the real straight line sample point is correctly detected; if the reference line feature output by the initial detection model is a non-straight line point, it can be determined that the real straight line sample point is not correctly detected.

[0150] In practical applications, among the real straight line sample points of at least two frames of real point cloud data, the real straight line sample points whose detection rate is greater than the detection rate threshold can basically be correctly detected by the initial extraction model in most cases, and the real straight line sample points can be used as target straight line sample points, that is, the target straight line sample points that can be correctly detected in most cases are straight line points that can be correctly identified by the initial extraction model, and thus the second line feature label of the target straight line sample points should be straight line points. Among them, the detection rate threshold can be pre-set, such as the detection rate threshold can be set to 80%, and the detection rate of the real straight line sample points of 200 frames of real point cloud data is counted, and the real straight line sample points whose detection rate is greater than the detection rate threshold are used as target straight line sample points.

[0151] In addition, the straight line points in the point cloud data in the real scene may not be completely located on a straight line, and may float slightly up and down along the straight line. Therefore, in addition to the target straight line sample points correctly detected by the initial extraction model, the target straight line sample points can also be expanded to obtain expanded straight line sample points to complete the missing straight line points. Specifically, the target straight line sample points can be subjected to operations such as regional growing, nearest neighbor interpolation and / or straight line fitting processing to obtain expanded straight line sample points, and then the second line feature labels of the expanded straight line sample points are also set as straight line points, that is, the second line feature labels of the target straight line sample points and the expanded straight line sample points are both straight line points, and the second line feature labels of the real sample points other than the target straight line sample points and the expanded straight line sample points are set as non-straight line points, so as to obtain a labeled real point cloud training set.

[0152] In the embodiments of the present specification, each real sample point in the real point cloud training set can be labeled according to the detection rate of each real sample point by the initial extraction model, thereby improving the accuracy of the label of each real sample point, and the target straight line sample points can be processed by regional growing, nearest neighbor interpolation and / or straight line fitting to fill in the missing straight line points, thereby increasing the data richness of the real point cloud training set and ensuring the accuracy of subsequent further training of the initial extraction model.

[0153] Step 110: Train the initial extraction model according to the real point cloud training set to obtain a line feature extraction model.

[0154] In an optional implementation of this embodiment, the initial extraction model may be further trained based on the second predicted line features and second line feature labels of each real sample point in the real point cloud training set, that is, the initial extraction model is trained according to the real point cloud training set to obtain a line feature extraction model. The specific implementation process may be as follows:

[0155] Input the real point cloud training set into the initial extraction model to obtain the second prediction line features of each real sample point in the real point cloud training set;

[0156] According to the second predicted line features and the second line feature labels of each real sample point, the second loss value of the initial extraction model is calculated, the model parameters of the initial extraction model are adjusted based on the second loss value, and the operation step of acquiring at least two frames of real point cloud data is returned to execute until the second training stop condition is reached, thereby obtaining a trained line feature extraction model.

[0157] It should be noted that the second training stop condition here may be a condition that needs to be met for the initial extraction model to stop training. For example, the second training stop condition may be that the second loss value is less than the loss value threshold, or the number of iterations exceeds the number threshold. Generally, convergence can be achieved through 100 iterations, and the accuracy rate can reach 96%. Therefore, the number threshold can be set to 100. The second line feature label refers to the real result of the real sample point, and the second predicted line feature is the line feature predicted by the initial extraction model for the real sample point. Therefore, the second loss value of the initial extraction model can be calculated based on the second predicted line feature and the second line feature label of the real sample point, and then the model parameters of the initial extraction model are adjusted according to the second loss value, and the operation step of obtaining at least two frames of real point cloud data is returned to continue until the second training stop condition is met to obtain a trained line feature extraction model.

[0158] In specific implementation, the cross entropy loss function can be calculated based on the second predicted line features and second line feature labels of the real sample points to generate a loss value. Among them, the second line feature label refers to the processing result that the initial extraction model actually wants to output, that is, the second line feature label is the real result, and the real point cloud training set is input into the initial extraction model, and the second predicted line feature of the output real sample point is the predicted result. When the difference between the predicted result and the real result is small enough, it means that the predicted result is close enough to the real result, or when the number of iterations exceeds the number threshold, it means that enough rounds have been trained. At this time, the initial extraction model training is completed, and the final line feature extraction model can be obtained.

[0159] In the embodiments of this specification, the difference between the predicted results and the actual results of the initial extraction model can be intuitively shown by calculating the loss value. Further targeted training of the initial extraction model and adjustment of parameters can effectively improve the training rate and training effect of the initial extraction model and improve the accuracy.

[0160] Furthermore, after constructing the real point cloud training set, a preset proportion of real sample points can be selected from the real point cloud training set as the second training set, and the remaining real sample points can be used as the second test set. The second training set is input into the initial extraction model to obtain the second predicted line features of each real sample point in the second training set, so as to calculate the second loss value of the initial extraction model based on the second predicted line features and the second line feature labels of the real sample points in the second training set, and train the initial extraction model.

[0161] Among them, the ratio of the number of real sample points in the second training set and the second test set can be a preset ratio, which can be the same as or different from the ratio of the first training set and the first test set, such as the second training set: second test set can also be 4:1.

[0162] After the training is completed, the trained line feature extraction model can be tested based on the second test set to determine whether the performance of the trained line feature extraction model meets the requirements. Specifically, the second test set can be input into the line feature extraction model to obtain the second predicted line features of each real sample point in the second test set. Based on the second predicted line features and the second line feature labels of the real sample points in the second test set, the recognition accuracy of the line feature extraction model is calculated. If the recognition accuracy is greater than or equal to the accuracy threshold, it is determined that the performance of the trained line feature extraction model meets the requirements, and subsequent point cloud matching can be performed. If the recognition accuracy is lower than the accuracy threshold, it means that the performance of the trained line feature extraction model does not meet the requirements. At this time, you can return to execute the operation steps of obtaining at least two frames of real point cloud data, continue to label the real point cloud data, obtain the real point cloud training set, and further train the line feature extraction model.

[0163] As an example, Figure 2c is a schematic diagram of a training process of a line feature extraction model provided by an embodiment of this specification, such as Figure 2c As shown, the training process of the line feature extraction model includes four steps. The first step is to construct simulated training samples, the second step is to train the initial extraction model, the third step is to determine the real training samples (after rigid transformation, prediction, nearest neighbor interpolation and straight line fitting, etc.), and the fourth step is to train the line feature extraction model.

[0164] An embodiment of the present specification provides a training method for a line feature extraction model, which can perform preliminary training on a point cloud detector based on constructed simulation training samples, and then use the initial extraction model obtained from the preliminary training to label real point cloud data to obtain a real point cloud training set, and then further train the initial extraction model based on the real point cloud training set. The knowledge learned from the simulation point cloud data can be transferred to the real point cloud data, thereby improving the training accuracy of the line feature extraction model. Subsequently, the straight line features in the point cloud data in the real scene can be accurately identified directly based on the line feature extraction model, and then the two frames of point cloud data can be matched based on the straight line features, thereby improving the matching accuracy of the point cloud data.

[0165] Figure 3 A flowchart of a point cloud matching method provided according to an embodiment of the present specification is shown, including steps 302 to 306.

[0166] Step 302: Acquire two frames of collected point cloud data.

[0167] It should be noted that point cloud data of the actual scene can be collected by a laser sensor. When collecting point cloud data by the laser sensor, multiple frames of point cloud are collected at intervals of a set walking distance d (such as 0.3m), or multiple frames of point cloud are collected at intervals of a set time. In the embodiments of this specification, two frames of point cloud data are used as an example for detailed introduction. The two frames of point cloud data can be the first frame of point cloud data and the second frame of point cloud data collected by the laser sensor before and after. In some embodiments, the first frame of point cloud data and the second frame of point cloud data can be in a front-to-back adjacent relationship or a non-adjacent relationship.

[0168] Step 304: input the two frames of point cloud data into the line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the above-mentioned line feature extraction model.

[0169] It should be noted that the two frames of point cloud data can be respectively input into the line feature extraction model, and the line feature extraction model can identify which points in the two frames of point cloud data are straight line points, thereby obtaining the straight line points in the two frames of point cloud data.

[0170] Step 306: performing point cloud matching on the two frames of point cloud data according to the straight line points in the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0171] In an optional implementation of this embodiment, the two frames of point cloud data include a first frame of point cloud data and a second frame of point cloud data; point cloud matching is performed on the two frames of point cloud data according to straight line points in the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data. The specific implementation process can be as follows:

[0172] According to the straight line points in the two frames of point cloud data, the straight lines included in the two frames of point cloud data are determined respectively;

[0173] Each straight line in the first frame of point cloud data is matched with the corresponding target straight line in the second frame of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0174] In practical applications, the line feature extraction model can output straight line points in two frames of point cloud data. Based on the straight line points in the two frames of point cloud data, the straight lines included in the two frames of point cloud data can be determined respectively, and then each straight line in the first frame of point cloud data is matched with the corresponding target straight line in the second frame of point cloud data to obtain the pose transformation parameters between the two frames of point cloud data. That is, two close straight lines in the two frames of point cloud data are matched to determine the pose transformation parameters between the two frames of point cloud data.

[0175] It should be noted that in the field of autonomous driving, laser sensors can continuously collect images of actual scenes to obtain three-dimensional point cloud data. In order to determine the movement of the vehicle at the moment of the two frames of point cloud, the two frames of point cloud data collected can be matched to determine the posture transformation. Since the stability and repeatability of point features are not high, in the embodiment of the present application, the point cloud detector is initially trained based on forged simulated point cloud data, and then the real point cloud data is labeled based on the initial extraction model of the initial training, and then the initial extraction model is further trained based on the real point cloud data.

[0176] During the application stage, after collecting two frames of point cloud data of the actual scene through the laser sensor, the two frames of point cloud data can be directly input into the trained line feature extraction model. The line feature extraction model can output which points in the two frames of point cloud data are straight line points, and then the two frames of point clouds can be matched based on the straight line features.

[0177] An embodiment of the present specification provides a point cloud matching method, which can accurately identify straight line points in point cloud data in real scenes based on a line feature extraction model, match two frames of point cloud data based on straight line features, and is applicable to various real three-dimensional point cloud scenes. It can realize line feature extraction of non-uniform point clouds. In point cloud data scanned by a single frame of laser sensor, straight line points such as rods and surface intersection lines can be accurately extracted based on the line feature extraction model, including the intersection lines of uniform planes and non-uniform planes, thereby greatly improving the matching accuracy of point cloud data.

[0178] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for a line feature extraction model. Figure 4 FIG. 1 is a schematic diagram showing a structure of a training device for a line feature extraction model provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0179] A construction module 402 is configured to construct a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label;

[0180] A first training module 404 is configured to train a point cloud detector according to a simulated point cloud training set to obtain an initial extraction model;

[0181] A first determination module 406 is configured to obtain at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to the initial extraction model;

[0182] The second determination module 408 is configured to determine a real point cloud training set according to the reference line feature, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label;

[0183] The second training module 410 is configured to train the initial extraction model according to the real point cloud training set to obtain a line feature extraction model.

[0184] Optionally, the construction module 402 is further configured to:

[0185] Constructing at least two mesh models according to at least two types of preset vertices and edges;

[0186] Sampling at least two grid models to obtain simulation point cloud data corresponding to the at least two grid models;

[0187] Setting a first line feature label for a simulation sample point in the simulation point cloud data, wherein the first line feature label is used to indicate whether the simulation sample point is a straight line point or a non-straight line point;

[0188] According to the simulation point cloud data, a simulation point cloud training set is constructed.

[0189] Optionally, the construction module 402 is further configured to:

[0190] Performing linear interpolation on preset vertices and edges of at least two mesh models to obtain straight line sample points included in at least two mesh models;

[0191] Searching for extended sample points whose distances from straight line sample points are less than a distance threshold in the non-straight line sample points of the simulation point cloud data set;

[0192] The first line feature labels of the straight line sample points and the extended sample points in at least two grid models are set as straight line points, and the first line feature labels of other sample points are set as non-straight line points.

[0193] Optionally, the construction module 402 is further configured to:

[0194] Obtaining a third set number of non-linear sample points from the real point cloud data as noise data;

[0195] According to the preset ratio, the simulated point cloud data and the noise data are mixed to obtain the simulated point cloud training set.

[0196] Optionally, the first training module 404 is further configured to:

[0197] The simulation point cloud training set is input into the point cloud detector to obtain the first prediction line feature of each simulation sample point in the simulation point cloud training set;

[0198] According to the first predicted line features and the first line feature labels of each simulated sample point, the first loss value of the point cloud detector is calculated, the parameters of the point cloud detector are adjusted based on the first loss value, and the operation steps of constructing a simulated point cloud training set are returned to execute until the first training stop condition is reached, thereby obtaining a trained initial extraction model.

[0199] Optionally, the first training module 404 is further configured to:

[0200] According to the ratio of positive samples to negative samples in each simulation sample point, the weight coefficients of positive samples and negative samples are determined, wherein the positive sample is a point in the simulation sample point whose first line feature label is a straight line point, and the negative sample is a point in the simulation sample point whose first line feature label is a non-straight line point, and the weight coefficient of the positive sample is greater than the weight coefficient of the negative sample;

[0201] Calculate a first sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the positive sample;

[0202] Calculate a second sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the negative sample;

[0203] According to the weight coefficients of the positive samples and the negative samples, the first sub-loss value and the second sub-loss value are weightedly fused to obtain the first loss value of the point cloud detector.

[0204] Optionally, the second determining module 408 is further configured to:

[0205] According to the reference line features, determine the target straight line sample points whose detection rate is greater than the detection rate threshold among the real straight line sample points of at least two frames of real point cloud data;

[0206] Performing expansion processing on the target straight line sample points to obtain expanded straight line sample points, wherein the expansion processing includes region growing, nearest neighbor interpolation and / or straight line fitting processing;

[0207] The second line feature labels of the target straight line sample points and the expanded straight line sample points are set as straight line points, and the second line feature labels of other sample points are set as non-straight line points to obtain a real point cloud training set.

[0208] Optionally, the second training module 410 is further configured to:

[0209] Input the real point cloud training set into the initial extraction model to obtain the second prediction line features of each real sample point in the real point cloud training set;

[0210] According to the second predicted line features and the second line feature labels of each real sample point, the second loss value of the initial extraction model is calculated, the model parameters of the initial extraction model are adjusted based on the second loss value, and the operation step of acquiring at least two frames of real point cloud data is returned to execute until the second training stop condition is reached, thereby obtaining a trained line feature extraction model.

[0211] An embodiment of the present specification provides a training device for a line feature extraction model, which can perform preliminary training on a point cloud detector based on constructed simulation training samples, and then use the initial extraction model obtained from the preliminary training to label real point cloud data to obtain a real point cloud training set, and then further train the initial extraction model based on the real point cloud training set. The knowledge learned from the simulation point cloud data can be transferred to the real point cloud data, thereby improving the training accuracy of the line feature extraction model. Subsequently, the straight line features in the point cloud data in the real scene can be accurately identified directly based on the line feature extraction model, and then the two frames of point cloud data can be matched based on the straight line features, thereby improving the matching accuracy of the point cloud data.

[0212] The above is a schematic scheme of a training device for a line feature extraction model of this embodiment. It should be noted that the technical scheme of the training device for the line feature extraction model and the technical scheme of the training method for the line feature extraction model described above belong to the same concept, and the details not described in detail in the technical scheme of the training device for the line feature extraction model can be found in the description of the technical scheme of the training method for the line feature extraction model described above.

[0213] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for a line feature extraction model. Figure 5 FIG. 1 is a schematic diagram showing the structure of a point cloud matching device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0214] An acquisition module 502 is configured to acquire two frames of collected point cloud data;

[0215] An input module 504 is configured to input two frames of point cloud data into a line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the line feature extraction model;

[0216] The matching module 504 is configured to perform point cloud matching on the two frames of point cloud data according to the straight line points in the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0217] Optionally, the two frames of point cloud data include a first frame of point cloud data and a second frame of point cloud data; the matching module 504 is further configured to:

[0218] According to the straight line points in the two frames of point cloud data, the straight lines included in the two frames of point cloud data are determined respectively;

[0219] Each straight line in the first frame of point cloud data is matched with the corresponding target straight line in the second frame of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

[0220] An embodiment of the present specification provides a point cloud matching device, which can accurately identify straight line points in point cloud data in real scenes based on a line feature extraction model, match two frames of point cloud data based on straight line features, and is applicable to various real three-dimensional point cloud scenes. It can realize line feature extraction of non-uniform point clouds. In point cloud data scanned by a single frame of laser sensor, straight line points such as rods and surface intersection lines can be accurately extracted based on the line feature extraction model, including the intersection lines of uniform planes and non-uniform planes, thereby greatly improving the matching accuracy of point cloud data.

[0221] The above is a schematic scheme of a point cloud matching device of this embodiment. It should be noted that the technical scheme of the point cloud matching device and the technical scheme of the above point cloud matching method belong to the same concept, and the details not described in detail in the technical scheme of the point cloud matching device can be referred to the description of the technical scheme of the above point cloud matching method.

[0222] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0223] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0224] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0225] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0226] Among them, the processor 620 is used to execute the following computer executable instructions to implement the training method of the line feature extraction model or the steps of the point cloud matching method.

[0227] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the training method of the line feature extraction model or the point cloud matching method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the training method of the line feature extraction model or the point cloud matching method mentioned above.

[0228] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the training method of a line feature extraction model or the steps of a point cloud matching method.

[0229] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the training method of the line feature extraction model or the point cloud matching method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the training method of the line feature extraction model or the point cloud matching method mentioned above.

[0230] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0231] Computer instructions include computer program codes, which may be in source code form, object code form, executable files, or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0232] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0233] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0234] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A training method for a line feature extraction model, comprising: Constructing a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label; According to the simulation point cloud training set, the point cloud detector is trained to obtain an initial extraction model; Acquire at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to the initial extraction model; Determine a real point cloud training set according to the reference line feature, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label; The initial extraction model is trained according to the real point cloud training set to obtain a line feature extraction model.

2. According to the training method of the line feature extraction model in claim 1, the constructing a simulation point cloud training set comprises: Constructing at least two mesh models according to at least two types of preset vertices and edges; Sampling the at least two grid models to obtain simulation point cloud data corresponding to the at least two grid models; Setting a first line feature label for a simulation sample point in the simulation point cloud data, wherein the first line feature label is used to indicate whether the simulation sample point is a straight line point or a non-straight line point; The simulation point cloud training set is constructed according to the simulation point cloud data.

3. The training method for a line feature extraction model according to claim 2, wherein the step of setting a first line feature label for a simulation sample point in the simulation point cloud data comprises: Performing linear interpolation on preset vertices and edges of the at least two mesh models to obtain straight line sample points included in the at least two mesh models; Searching for extended sample points whose distance from the straight line sample points is less than a distance threshold among the non-straight line sample points of the simulated point cloud data set; The first line feature labels of the straight line sample points and the extended sample points in the at least two grid models are set as straight line points, and the first line feature labels of other sample points are set as non-straight line points.

4. The training method for the line feature extraction model according to claim 2, wherein constructing the simulation point cloud training set according to the simulation point cloud data comprises: Obtaining a third set number of non-linear sample points from the real point cloud data as noise data; The simulated point cloud data and the noise data are mixed according to a preset ratio to obtain the simulated point cloud training set.

5. The method for training a line feature extraction model according to claim 1, wherein the step of training a point cloud detector to obtain an initial extraction model based on the simulated point cloud training set comprises: Inputting the simulated point cloud training set into the point cloud detector to obtain the first prediction line feature of each simulated sample point in the simulated point cloud training set; According to the first predicted line features and the first line feature labels of the respective simulated sample points, a first loss value of the point cloud detector is calculated, parameters of the point cloud detector are adjusted based on the first loss value, and the operation steps of constructing a simulated point cloud training set are returned to be executed until the first training stop condition is reached, thereby obtaining a trained initial extraction model.

6. The training method for a line feature extraction model according to claim 5, wherein the first loss value of the point cloud detector is calculated according to the first predicted line features and the first line feature labels of the respective simulation sample points, comprising: Determine the weight coefficients of the positive samples and the negative samples according to the ratio of the positive samples to the negative samples in the respective simulation sample points, wherein the positive samples are points in the simulation sample points whose first line feature labels are straight line points, and the negative samples are points in the simulation sample points whose first line feature labels are non-straight line points, and the weight coefficients of the positive samples are greater than the weight coefficients of the negative samples; Calculating a first sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the positive sample; Calculating a second sub-loss value of the point cloud detector according to the first predicted line feature and the first line feature label of the negative sample; According to the weight coefficients of the positive sample and the negative sample, the first sub-loss value and the second sub-loss value are weightedly fused to obtain a first loss value of the point cloud detector.

7. The training method for a line feature extraction model according to any one of claims 1 to 6, wherein determining a real point cloud training set based on the reference line features comprises: Determine, according to the reference line feature, a target straight line sample point whose detection rate is greater than a detection rate threshold among the real straight line sample points of the at least two frames of real point cloud data; Performing expansion processing on the target straight line sample points to obtain expanded straight line sample points, wherein the expansion processing includes region growing, nearest neighbor interpolation and / or straight line fitting processing; The second line feature labels of the target straight line sample points and the expanded straight line sample points are set as straight line points, and the second line feature labels of other sample points are set as non-straight line points to obtain the real point cloud training set.

8. The method for training a line feature extraction model according to claim 7, wherein the initial extraction model is trained according to the real point cloud training set to obtain the line feature extraction model, comprising: Inputting the real point cloud training set into the initial extraction model to obtain the second prediction line feature of each real sample point in the real point cloud training set; According to the second predicted line features and the second line feature labels of each real sample point, the second loss value of the initial extraction model is calculated, the model parameters of the initial extraction model are adjusted based on the second loss value, and the operation step of acquiring at least two frames of real point cloud data is returned to execute until the second training stop condition is reached, so as to obtain a trained line feature extraction model.

9. A point cloud matching method, comprising: Get the two frames of point cloud data collected; Inputting the two frames of point cloud data into a line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the line feature extraction model according to any one of claims 1 to 8; Point cloud matching is performed on the two frames of point cloud data according to the straight line points in the two frames of point cloud data to obtain posture transformation parameters between the two frames of point cloud data.

10. The point cloud matching method according to claim 9, wherein the two frames of point cloud data include a first frame of point cloud data and a second frame of point cloud data; The step of performing point cloud matching on the two frames of point cloud data according to the straight line points in the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data includes: According to the straight line points in the two frames of point cloud data, respectively determine the straight lines included in the two frames of point cloud data; Each straight line in the first frame of point cloud data is matched with a corresponding target straight line in the second frame of point cloud data to obtain a posture transformation parameter between the two frames of point cloud data.

11. A training device for a line feature extraction model, comprising: A construction module is configured to construct a simulation point cloud training set, wherein the simulation point cloud training set includes a first set number of simulation sample points, and the simulation sample points carry a first line feature label; A first training module is configured to train a point cloud detector according to the simulated point cloud training set to obtain an initial extraction model; A first determination module is configured to obtain at least two frames of real point cloud data, and determine reference line features corresponding to the at least two frames of real point cloud data according to the initial extraction model; A second determination module is configured to determine a real point cloud training set according to the reference line feature, wherein the real point cloud training set includes a second set number of real sample points, and the real sample points carry a second line feature label; The second training module is configured to train the initial extraction model according to the real point cloud training set to obtain a line feature extraction model.

12. A point cloud matching device, comprising: An acquisition module is configured to acquire two frames of collected point cloud data; An input module, configured to input the two frames of point cloud data into a line feature extraction model to obtain straight line points in the two frames of point cloud data, wherein the line feature extraction model is trained based on the training method of the line feature extraction model according to any one of claims 1 to 8; The matching module is configured to perform point cloud matching on the two frames of point cloud data according to the straight line points in the two frames of point cloud data to obtain the posture transformation parameters between the two frames of point cloud data.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the training method of the line feature extraction model described in any one of claims 1 to 8 or the steps of the point cloud matching method described in any one of claims 9-10.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the training method of the line feature extraction model described in any one of claims 1 to 8 or the steps of the point cloud matching method described in any one of claims 9-10.

Images