A method and device for fast instance segmentation of frame-by-frame point cloud based on nearest neighbor KNN algorithm

Abstract

The present invention discloses a method and device for fast instance segmentation of frame-by-frame point clouds based on the nearest neighbor KNN algorithm, which effectively improves the model's processing efficiency for frame-by-frame point cloud instance segmentation. It belongs to the field of three-dimensional computer vision. The method of the present invention comprises the following steps: first, registering the point cloud data of each two adjacent frames to obtain parameter transformation information between each two adjacent frames of point cloud data; standardizing the point cloud data and using the transformation information to achieve the conversion of the two frames of point clouds; building a fast instance segmentation algorithm for frame-by-frame point clouds based on the idea of the nearest neighbor KNN algorithm; optimizing the point cloud search speed; using the trained model and the true value to perform a negative feedback network optimization KNN model; and using the optimized model to obtain the prediction result of the next frame in advance. The fast instance segmentation algorithm for frame-by-frame point clouds proposed by the present invention greatly reduces the computing power in the model reasoning process and speeds up the model reasoning speed.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional computer vision and is a method and device for fast instance segmentation of frame-by-frame point clouds based on a nearest neighbor KNN algorithm. Background Art

[0002] With the rise of computer vision, deep learning, as a mainstream AI technology, has been successfully applied to various 2D vision problems. However, the lack of depth information has always been a drawback of 2D vision, leading to the widespread use of point cloud data. Point cloud data largely preserves the original geometric information without requiring any discretization. Therefore, it is the preferred representation for many scene understanding applications, such as autonomous driving and robotics. However, due to the large amount of point cloud data and the complexity of computation, practical applications are inefficient and have low accuracy. Currently, point cloud instance segmentation and object detection technologies face numerous challenges.

[0003] Currently, point cloud models generated after training test and verify each frame of point cloud data independently, but adjacent frames often exhibit significant similarity. However, no method for reasoning between adjacent frames of point cloud data has yet been proposed. A fast instance segmentation method for frame-by-frame point clouds can effectively address this repeated computation during the model inference process, saving significant computing power.

[0004] The K-Nearest Neighbors algorithm (KNN) is a non-parametric supervised learning classifier that uses proximity to classify or predict groups of individual data points. It is well-suited for point clouds with similar frame-by-frame data. However, this algorithm suffers from computational overhead when the point cloud data is large. Furthermore, when the sample size is imbalanced, the classification of new samples tends to favor the category that predominated in the training samples, which can easily lead to prediction errors. The Facebook AI Similarity Search (Faiss) algorithm is an open-source library developed by the Facebook AI team. Essentially, Faiss is a vector database. The search is based on a raw vector database, with a single vector as the basic unit. By default, a vector x is input and the k most similar vectors to x are returned. Before using Faiss to perform a similarity search on a query vector, the original vector set must be encapsulated into an index file and cached in memory to facilitate real-time query computation. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention proposes a frame-by-frame point cloud fast instance segmentation method and device based on the nearest neighbor KNN algorithm. The algorithm is optimized by taking advantage of the characteristics that the large amount of point cloud data causes excessive computational complexity and poor results when the samples are unbalanced, so that the algorithm greatly improves speed and efficiency.

[0006] The present invention adopts the following technical solutions to achieve the above-mentioned purpose: In a first aspect, the present invention provides a method for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm, the method comprising the following steps:

[0007] Step 1: For any two adjacent frames of point cloud data of the target object, use the iterative nearest neighbor method (ICP) to perform registration and obtain the first posture transformation parameters between the two adjacent frames of point cloud data of the target object;

[0008] Step 2: Based on the first posture transformation parameter obtained in step 1, the first frame of point cloud data of two adjacent frames of point cloud data of the target object is spliced into the second frame of point cloud data to obtain first registered point cloud data;

[0009] Step 3: Obtain basic training verification data, including the number m of the first frame point cloud of the target object and the coordinate z of the first frame point cloud i , the target object instance segmentation result g of the first frame point cloud i , the number of the first registration point cloud is n, and the coordinate of the point cloud data is z k , the target object instance segmentation result of the first registered point cloud The point cloud coordinates are used as input, the Euclidean distance is calculated to obtain the predicted classification probability, the target object instance segmentation result of the point cloud is used as output to train the KNN model and optimize it based on the prediction error; the first registered point cloud is input to the optimized KNN model, and the predicted point cloud target object instance segmentation result is output. Get the target object category.

[0010] Furthermore, the registration process of two adjacent frames of point cloud data in step 1 is as follows:

[0011] a) Preprocessing: filtering and cleaning the target object point cloud data;

[0012] b) Matching: Using the estimated matrix of coarse registration, find the nearest point of the target object point cloud;

[0013] c) Weighting: adjust the weights of some corresponding point pairs;

[0014] d) Eliminate unreasonable point clouds corresponding to target objects;

[0015] e) Optimize the objective function and calculate the loss;

[0016]

[0017] in and is the i-th point corresponding to the first frame point cloud and the registered target object point cloud, N p Represents the total number of point clouds; R represents the rotation matrix from the first frame point cloud to the second frame point cloud, and t represents the offset matrix;

[0018] f) Minimize Loss and find the current optimal transformation;

[0019]

[0020] g) Repeat bf until convergence.

[0021] Furthermore, in step 3, the initial parameters of the algorithm are set, the model K neighbor value is set to k = 7, the initial value of the number of iterations is t = 10, and the Euclidean distance parameter d is set to i (z i , g i ), the parameter index index of the point cloud data, where:

[0022] z i =[z1,z2,z3,...,z m ],

[0023] g i =[g1, g2, g3, ..., g m ] T

[0024] z k =[z1,z2,z3,...,z n ]

[0025] where z i Indicates the coordinates of the first frame point cloud, z m represents the coordinates of the mth point, g i represents the target object instance segmentation result of the first frame point cloud, g m Represents the instance segmentation result of the mth point; where z k Indicates the coordinates of the first frame point cloud after registration, z n Represents the coordinates of the nth point after registration.

[0026] Furthermore, in step 3, the specific process of KNN model training is as follows:

[0027] a) Set the coordinate input z of the point cloud i , the result of the point cloud instance segmentation is the classified point g i ;

[0028] b) Calculate record g iThe Euclidean distance d from each record in the validation dataset i (z i , g i ):

[0029]

[0030] c) for distance d i (z i , g i ) Sort from small to large;

[0031] d) Determine the frequency of occurrence of the top k point categories, where k is set to 7;

[0032] e) The category with the highest frequency of occurrence is used as the predicted category to obtain the training model KNN;

[0033] f) Using the first registered point cloud obtained in step 2 as the model input, the prediction result is:

[0034]

[0035] Among them, y n Represents the output prediction result of the model KNN corresponding to the nth registered point cloud.

[0036] Furthermore, in step 3, the KNN model optimization process is as follows:

[0037] a) Add index to point cloud coordinates;

[0038] b) Use the Faiss algorithm to encapsulate the point cloud search into an index file Index iile and cache it in memory to provide real-time query calculation;

[0039] c) Original values based on the base training validation data And predict data through KNN model Subtract and calculate the error G(y);

[0040]

[0041] d) G(y) is passed to the KNN model in the form of negative feedback, so that the model is optimized twice, and finally the training model Model_KNN is obtained;

[0042] e) The first registered point cloud obtained in step 2 is used as the input of the model Model_KNN, and the prediction result is:

[0043]

[0044] Among them, y′ nIndicates the prediction result output by the model Model_KNN corresponding to the nth registered point cloud.

[0045] Furthermore, the detailed steps of the Faiss algorithm are as follows:

[0046] a) Vectorize the original point cloud of the target object and divide it into M subspaces, and train the center point for each subspace;

[0047] b) Find the center point corresponding to the vector;

[0048] c) Subtract the corresponding center point from the vector to generate the residual vector;

[0049] d) Generate a secondary quantizer for the residual vector.

[0050] In a second aspect, the present invention also provides a device for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm.

[0051] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for fast instance segmentation of a frame-by-frame point cloud based on the nearest neighbor KNN algorithm is implemented.

[0052] The beneficial results of this invention are primarily manifested in the following: While traditional point cloud models generated after training test and verify each frame of point cloud data independently, adjacent frames of data exhibit significant similarity. However, a method for reasoning about adjacent point clouds has not yet been proposed. To improve this method, this invention incorporates the Faiss algorithm and optimizes the KNN algorithm, increasing the model's efficiency in processing frame-by-frame point cloud instance segmentation during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a flow chart of the present invention;

[0055] Figure 2 This is a flowchart of the traditional model reasoning process;

[0056] Figure 3Flowchart of the reasoning process of the present invention;

[0057] Figure 4 This is a macroscopic representation of the reasoning of the present invention;

[0058] Figure 5 A macroscopic representation of traditional model reasoning;

[0059] Figure 6 This is a structural diagram of a frame-by-frame point cloud fast instance segmentation device based on the nearest neighbor KNN algorithm of the present invention. DETAILED DESCRIPTION

[0060] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0061] Embodiment 1:

[0062] The present invention can be applied to any occasion in the point cloud reasoning stage, and can be applied to any occasion in which the data is annotated and attached. Figure 1 、 Figure 2 and Figure 3 Give an example.

[0063] In existing data annotation scenarios, it is currently necessary to annotate the target object point cloud signals collected by the LiDAR. Target object instance segmentation and annotation for frame-by-frame point clouds often results in high data repetition and is time-consuming. The present invention proposes a method for fast instance segmentation of frame-by-frame point clouds based on the nearest neighbor KNN algorithm. The method comprises the following steps:

[0064] Step 1: A set of target object point cloud data streams collected by the existing lidar needs to be labeled. For the incoming target object point cloud data stream, it is frame-processed and each frame of point cloud is represented as an independent point cloud input.

[0065] Step 2: In order to obtain the spatial transformation relationship between adjacent point cloud data of the target object, for any set of two adjacent frames of point cloud data of the target object, the iterative nearest neighbor method (ICP) is used to align the two adjacent frames of point cloud data to obtain the first posture transformation parameters between the two adjacent frames of point cloud data;

[0066] a) Preprocessing: filtering and cleaning the first and second frame point cloud data of the target object;

[0067] b) Matching: Use the estimated matrix of coarse registration to find the nearest neighbor points;

[0068] c) Weighting: adjust the weights of some corresponding point pairs;

[0069] d) Eliminate unreasonable corresponding points;

[0070] e) Optimize the objective function and calculate the loss;

[0071]

[0072] in and is the i-th point corresponding to the first frame point cloud and the registration target point cloud, N p Represents the total number of point clouds; R represents the rotation matrix from the first frame point cloud to the second frame point cloud, and t represents the offset matrix;

[0073] f) Minimize Loss and find the current optimal transformation;

[0074]

[0075] Step 3: Based on the first posture transformation parameter obtained in step 2, the first frame of point cloud data of two adjacent frames of point cloud data of the target object is spliced to the second frame of point cloud data using the parameter to obtain first registered point cloud data;

[0076] Step 4: Obtain basic training verification data, including the number m of the first frame point cloud of the target object and the coordinate z of the corresponding point cloud i , the target object instance segmentation result g of the point cloud i , the number of the first registration point cloud is n, and the coordinate of the point cloud data is z k , target object instance segmentation results of point cloud

[0077] z k =R * ·z i +t *

[0078] Step 5: Use the KNN model algorithm and set the initial parameters of the algorithm. The model K neighbor value is set to k=7, the initial value of the number of iterations is t=10, and the Euclidean distance parameter d i (z i , g i ), the parameter index index of the point cloud data, where:

[0079] z i =[z1,z2,z3,...,z m ],

[0080] g i =[g1, g2, g3, ..., g m ] T

[0081] z k =[z1,z2,z3,...,z n ]

[0082] where z i Indicates the coordinates of the first frame point cloud, g i Represents the target object instance segmentation result of the first frame point cloud; where z k Indicates the coordinates of the first frame point cloud after registration;

[0083] Step 6: Start KNN model training. The process is as follows:

[0084] a) Set the coordinate input z of the point cloud i , the result of the point cloud instance segmentation is the classified point g i ;

[0085] b) Calculate record g i The Euclidean distance d from each record in the validation dataset i (z i , g i ):

[0086]

[0087] c) for distance d i (z i , g i ) Sort from small to large;

[0088] d) determine the frequencies of occurrence of the top k point categories, where k is set to 7;

[0089] e) The category with the highest frequency of occurrence is used as the predicted category to obtain the training model KNN;

[0090] f) Using the first registered point cloud obtained in step 2 as the model input, the predicted classification result of the target object is obtained as follows:

[0091]

[0092] Example 2:

[0093] On the basis of Example 1, in order to further reduce the time spent by the KNN model in comparing and calculating the spatial relationship of point clouds, the present invention uses the Faiss algorithm to optimize the training model obtained by KNN. The process is as follows:

[0094] a) Add index to the point cloud coordinates in each point cloud file in turn;

[0095] b) Using the Facebook AI Similarity Search (Faiss) algorithm, the point cloud search construct is encapsulated into an index file and cached in memory, providing real-time query calculations and significantly reducing the amount of computation required to process the point cloud file.

[0096] c) Original values based on the validation dataset and prediction dataset Subtract and calculate the error G(y):

[0097]

[0098] d) G(y) is passed to the model in the form of negative feedback, so that the model is optimized twice, and finally the training model Model_KNN is obtained;

[0099] e) Using the first registered point cloud obtained in step 2 as the model input, the predicted classification result of the target object is obtained as follows:

[0100]

[0101] Example 3:

[0102] Based on Example 1 and Example 2, the present invention can also be used in the model reasoning stage to assist the model in improving the reasoning speed, such as Figure 4 and Figure 5 As shown, in the traditional reasoning stage, there is extremely strong data correlation information in the video stream point cloud, but due to the independence of the reasoning stage, the correlation information is ignored. Therefore, the method of the present invention can also effectively reduce the reasoning time of the model when applied to the model reasoning stage, thereby improving the reasoning speed.

[0103] Step 1: Reasoning is required for a set of target object point cloud data streams collected by the LiDAR. The incoming point cloud data stream is processed by extracting frames, and each frame of point cloud is represented as an independent point cloud input.

[0104] Step 2: Obtain the spatial transformation relationship between adjacent point cloud data. For any set of two adjacent frames of point cloud data of the target object, use the iterative nearest neighbor method (ICP) to align the two adjacent frames of point cloud data to obtain the first posture transformation parameters between the two adjacent frames of point cloud data.

[0105] Step 3: Based on Example 1, according to the first posture transformation parameter in Step 2, the first frame of point cloud data in the two adjacent frames of point cloud data is spliced to the second frame of point cloud data using the parameter to obtain first registered point cloud data;

[0106] z k =R * ·z i +t *

[0107] Step 4: In order to further reduce the time spent by the KNN model in comparing and calculating the spatial relationship of point clouds, the present invention uses the Faiss algorithm to optimize the training model obtained by KNN.

[0108]

[0109] Eventually As the output result of the second frame point cloud, after testing, the present invention can improve the reasoning speed of the model in this example, reduce the resource waste caused by computer computing power and the similarity between adjacent frames, and at the same time ensure a similarity level of up to 98% with the model reasoning result.

[0110] The beneficial results of this invention are primarily manifested in the following: While traditional point cloud models generated after training test and verify each frame of point cloud data independently, adjacent frames of data exhibit significant similarity. However, a method for reasoning about adjacent point clouds has not yet been proposed. To improve this method, this invention incorporates the Faiss algorithm and optimizes the KNN algorithm, increasing the model's efficiency in processing frame-by-frame point cloud instance segmentation during training.

[0111] Corresponding to the aforementioned embodiment of a method for fast instance segmentation of a frame-by-frame point cloud based on a nearest neighbor KNN algorithm, the present invention also provides an embodiment of a device for fast instance segmentation of a frame-by-frame point cloud based on a nearest neighbor KNN algorithm.

[0112] See also Figure 6 An embodiment of the present invention provides a device for fast instance segmentation of point clouds frame by frame based on a nearest neighbor KNN algorithm, comprising a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement a method for fast instance segmentation of point clouds frame by frame based on a nearest neighbor KNN algorithm in the above embodiment.

[0113] The embodiment of the device for fast instance segmentation of frame-by-frame point cloud based on the nearest neighbor KNN algorithm provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown in the figure, a hardware structure diagram of a device with data processing capability for a frame-by-frame point cloud fast instance segmentation device based on the nearest neighbor KNN algorithm provided by the present invention is provided. Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0114] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0116] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for fast instance segmentation of a frame-by-frame point cloud based on a nearest neighbor KNN algorithm in the above embodiment is implemented.

[0117] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0118] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A frame-by-frame point cloud fast instance segmentation method based on the nearest neighbor KNN algorithm, characterized in that: The method comprises the following steps: Step 1: For any two adjacent frames of point cloud data of the target object, use the iterative nearest neighbor method (ICP) to perform registration and obtain the first posture transformation parameters between the two adjacent frames of point cloud data of the target object; Step 2: Based on the first posture transformation parameter obtained in step 1, the first frame of point cloud data of two adjacent frames of point cloud data of the target object is spliced into the second frame of point cloud data to obtain first registered point cloud data; Step 3: Obtain basic training and verification data, including the number m of point clouds in the first frame of the target object and the coordinates of the first frame point cloud , the target object instance segmentation result of the first frame point cloud , the number of the first registration point cloud is n, and the coordinates of the point cloud data are , the target object instance segmentation result of the first registered point cloud ; Take the point cloud coordinates as input, calculate the Euclidean distance to get the predicted classification probability, use the target object instance segmentation result of the point cloud as output to train the KNN model and optimize it based on the prediction error; input the first registered point cloud into the optimized KNN model, and output the predicted target object instance segmentation result of the point cloud , get the target object category; The KNN model optimization process is as follows: a) Add index to point cloud coordinates; b) Use the Faiss algorithm to encapsulate the point cloud search structure into an index file and cache it in memory, providing real-time query calculation; c) Original values based on the base training validation data And predict data through KNN model Subtract and calculate the error ; d) Yes It is passed to the KNN model in the form of negative feedback, so that the model is optimized twice, and finally the training model Model_KNN is obtained; e) The first registered point cloud obtained in step 2 is used as the input of the model Model_KNN, and the prediction result is: in, Represents the prediction result output by the model Model_KNN corresponding to the nth registered point cloud; The detailed steps of the Faiss algorithm are as follows: a) Vectorize the original point cloud of the target object and divide it into M subspaces, and train the center point for each subspace; b) Find the center point corresponding to the vector; c) Subtract the corresponding center point from the vector to generate the residual vector; d) Generate a secondary quantizer for the residual vector.

2. The method for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm according to claim 1, characterized in that: The registration process of two adjacent frames of point cloud data in step 1 is as follows: a) Preprocessing: filtering and cleaning the target object point cloud data; b) Matching: Using the estimated matrix from the coarse registration, find the nearest point in the target object’s point cloud; c) Weighting: adjust the weights of some corresponding point pairs; d) Eliminate unreasonable corresponding target object point clouds; e) Calculate Loss. The Loss function is represented as: in and is the i-th point corresponding to the first frame point cloud and the registered target object point cloud, Represents the total number of point clouds; R represents the rotation matrix from the first frame point cloud to the second frame point cloud, and t represents the offset matrix; f) Minimize Loss and find the current optimal transformation ; g) Repeat bf until convergence.

3. The method for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm according to claim 1, characterized in that: In step 3, the initial parameters of the algorithm are set. The model K neighbor value is set to k = 7, the initial value of the number of iterations is t = 10, and the Euclidean distance parameter is set to , the parameter index index of the point cloud data, where: in Represents the coordinates of the first frame point cloud, Represents the coordinates of the mth point, where m is the number of point clouds in the first frame of the target object. Represents the target object instance segmentation result of the first frame point cloud, represents the instance segmentation result of the mth point; Represents the coordinates of the first frame point cloud after registration, Represents the coordinates of the nth point after registration, where n is the number of the first registered point cloud.

4. The method for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm according to claim 1, characterized in that: In step 3, the specific process of KNN model training is as follows: a) Set the coordinates of the first frame point cloud , the target object instance segmentation result of the first frame point cloud is ; b) Calculation records Euclidean distance to each record in the validation dataset : c) Distance Sort from smallest to largest; d) Determine the frequency of occurrence of the first k point categories, where the model K neighbor value k is set to 7; e) The category with the highest frequency is used as the predicted category to obtain the training model KNN; f) Take the first registered point cloud obtained in step 2 as the model input, Point cloud for the first frame The point cloud coordinates after registration are input into the model to obtain the prediction result: in, Represents the output prediction result of the model KNN corresponding to the nth point cloud after registration.

5. A device for fast instance segmentation of point clouds frame by frame based on the nearest neighbor KNN algorithm, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a frame-by-frame point cloud fast instance segmentation method based on the nearest neighbor KNN algorithm is implemented as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a frame-by-frame point cloud fast instance segmentation method based on the nearest neighbor KNN algorithm according to any one of claims 1 to 4 is implemented.

Images