Point cloud data processing method and device, and electronic device
By employing a primitive dictionary for bag-of-words pooling in point cloud data processing, global features are generated, solving the problem of information loss in existing technologies and improving the accuracy and efficiency of point cloud data processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-12
- Publication Date
- 2026-03-17
AI Technical Summary
Existing technologies for point cloud data processing, when using average pooling or global pooling, will lose some information, leading to a decrease in the performance of classification and retrieval tasks.
We employ bag-of-words pooling based on a pre-built primitive dictionary to determine the global features of point cloud data. We then extract local feature sets through the backbone network and use pooling layers to perform bag-of-words pooling to generate global features.
It improves the accuracy and efficiency of point cloud data processing, avoids information loss, and enhances the interpretability of global features.
Smart Images
Figure CN115204245B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a point cloud data processing method and device, electronic equipment and computer readable storage medium. BACKGROUND
[0002] Laser radar can obtain relatively dense point cloud information, and positioning technology based on laser radar sensors is widely used in the positioning field due to its high precision and good robustness. For example, in an unmanned driving system, point cloud data collected by a laser radar of an unmanned driving device is used to locate obstacles. Semantic information is obtained by segmenting point cloud data generated by the laser radar, and the semantic information is integrated into laser radar positioning and mapping to improve the mapping and positioning accuracy of the laser radar.
[0003] Using the powerful modeling capability of neural networks to process unordered point cloud data requires excluding the influence of data arrangement on the results. In the prior art, when extracting semantic information from point cloud data, average pooling or global pooling is usually used to fuse unordered data features extracted from the point cloud data to obtain a global representation of the point cloud data as semantic information. However, in the above two pooling processes, part of the information carried in the point cloud data is lost, resulting in a decrease in the performance of subsequent classification, retrieval and other tasks.
[0004] It can be seen that the point cloud data processing method in the prior art still needs to be improved. SUMMARY
[0005] Embodiments of the present application provide a point cloud data processing method, which helps to improve the point cloud data processing accuracy and efficiency.
[0006] In a first aspect, embodiments of the present application provide a point cloud data processing method, comprising:
[0007] determining a local feature set of point cloud data of a target object;
[0008] performing bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed primitive dictionary to determine a global feature of the point cloud data;
[0009] using the global feature as a feature representation of the target object and outputting to a downstream feature processing task.
[0010] In a second aspect, embodiments of the present application provide a point cloud data processing device, comprising:
[0011] a local feature set determination module configured to determine a local feature set of point cloud data of a target object;
[0012] The bag-of-words pooling processing module is configured to perform bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed base element dictionary, and determine a global feature of the point cloud data.
[0013] The global feature output module is configured to output the global feature as a feature expression of the target object to a downstream feature processing task.
[0014] In a third aspect, an electronic device is disclosed, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the point cloud data processing method when executing the computer program.
[0015] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the program is executable on a processor to implement the point cloud data processing method.
[0016] The point cloud data processing method disclosed in the embodiments of the present application determines the local feature set of the point cloud data of the target object, then performs bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed base element dictionary to determine the global feature of the point cloud data, and finally outputs the global feature as the feature expression of the target object to a downstream feature processing task, which helps to improve the accuracy of point cloud data processing.
[0017] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the embodiments can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0019] Figure 1 is a point cloud data processing method flowchart of the first embodiment of the present application;
[0020] Figure 2 is a pre-set point cloud processing model structure schematic diagram of the first embodiment of the present application;
[0021] Figure 3is a structural schematic diagram of a point cloud data processing apparatus according to Embodiment Two of the present application;
[0022] Figure 4 a block diagram of an electronic device for performing a method according to the present application is schematically shown; and
[0023] Figure 5 a storage unit for holding or carrying program code implementing a method according to the present application is schematically shown. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.
[0025] Embodiment One
[0026] A point cloud data processing method disclosed by the embodiments of the present application is as shown in Figure 1 The method comprises steps 110 to 130.
[0027] Step 110: determining a local feature set of point cloud data of a target object.
[0028] The target object in the embodiments of the present application can be an object or a scene.
[0029] The point cloud data processing method in the embodiments of the present application is applied to a preset point cloud processing model for processing point cloud data through a neural network, as shown in Figure 2 The preset point cloud processing model comprises a backbone network 210, a pooling layer 220 and a fully connected layer 230, as shown in Figure 2 The network structure of each layer network and the data processing flow between each layer network will be described below.
[0030] The backbone network 210 is configured to extract features of input point cloud data and determine a local feature set. In some embodiments of the present application, the backbone network 210 can adopt a convolutional neural network structure commonly used in the prior art, for example, can be composed of one or more convolutional modules. In other embodiments of the present application, the backbone network 210 can also adopt other network mechanisms for feature extraction, which are not limited in the present application.
[0031] The backbone network 210 is further configured to generate a primitive dictionary based on the preset point cloud model set and the local feature set. In some embodiments of the present application, the backbone network 210 can adopt a network structure for extracting primitives based on the point cloud model set in the prior art, that is, adopt a feature extraction network structure in the prior art, to extract a specified number of primitives capable of expressing each point cloud model in the point cloud model set, so as to generate the primitive dictionary.
[0032] In some embodiments of the present application, during the training of the preset point cloud processing model, the preset model parameter optimization method and the preset dictionary parameter optimization method are alternately used to update the primitive dictionary for multiple rounds, which will be described below.
[0033] The pooling layer 220 is configured to perform bag-of-words pooling processing on each local feature in the local feature set based on the primitive dictionary, to determine the global feature of the input point cloud data. In the embodiments of the present application, the pooling layer 220 adopts a network structure and data processing method different from those in the prior art. The pooling layer 220 includes a primitive dictionary. During the pooling operation of the local feature set output by the backbone network 210 through the pooling layer 220, the bag-of-words pooling operation is adopted in the embodiments of the present application, and the local feature set is pooled based on the primitive dictionary, to obtain the global feature corresponding to the local feature set.
[0034] The bag-of-words pooling operation adopted in the embodiments of the present application can effectively avoid the loss of point cloud information caused by feature dimension reduction when the local feature set obtained after feature extraction of the point cloud is pooled in the prior art, and effectively improves the accuracy of the output global feature.
[0035] The fully connected layer 230 is configured to perform feature mapping on the global feature output by the pooling layer, to obtain the output result of the preset point cloud processing model. For different preset point cloud processing models applied to different scenarios, the output results are different, and accordingly, the network structure of the fully connected layer 230 is also different. For example, for a preset point cloud processing model applied to a classification scenario, the fully connected layer 230 can be constructed based on a classification network structure. For another example, for a preset point cloud processing model applied to a search scenario, the fully connected layer 230 is constructed based on a logistic regression network. The specific network structure and implementation of the fully connected layer 230 are not limited in the present application.
[0036] During online application, after the point cloud data of the target object is output to the preset point cloud processing model, the backbone network 210 performs feature extraction on the input point cloud data, and the backbone network 210 outputs the local feature set of the point cloud data.
[0037] Taking an unmanned driving application scenario as an example, after a radar arranged on an unmanned driving device collects point cloud data of a driving scene, the point cloud data is input into the preset point cloud processing model, and the backbone network 210 outputs a local feature set matched with the driving scene.
[0038] The specific implementation of the preset point cloud processing model determining the local feature set of the point cloud data of the target object can adopt a technical solution for extracting a local feature set of point cloud data in the prior art, which is not limited in the embodiments of the present application and will not be described again.
[0039] In step 120, a bag-of-words pooling processing is performed on each local feature in the local feature set based on a pre-constructed primitive dictionary, and a global feature of the point cloud data is determined.
[0040] In some embodiments of the present application, the step of performing the bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data is performed by a pooling layer of the preset point cloud processing model. As described above, the pooling layer 220 includes a primitive dictionary, which is determined by training the preset point cloud processing model.
[0041] A primitive is a basic element constituting a point cloud database, such as a sphere, a cylinder, a polyhedron, etc. Different primitives constitute different shapes by changing the size and placement. Since it is difficult to enumerate the primitives constituting an object manually, an EM algorithm (Expectation-Maximum, also known as Expectation-Maximization algorithm) based method is proposed in the embodiments of the present application to learn primitive features in a high-dimensional feature space to determine a primitive dictionary.
[0042] In some embodiments of the present application, first, the primitives in the primitive dictionary (i.e., bag-of-words) are initialized using random numbers. Then, in the process of training the preset point cloud processing model, the primitives in the primitive dictionary are updated gradually. For example, the network parameters of the preset point cloud processing model, including the network parameters of the pooling layer, can be optimized by a preset model parameter optimization method. That is, the network parameters include the elements in the primitive dictionary and the operation parameters of the pooling processing. The preset model parameter optimization method can be a gradient descent method. After the training of the preset point cloud processing model is completed, the optimal values of the primitive dictionary are obtained.
[0043] Since there are many elements in the primitive dictionary, in order to improve the training convergence speed of the preset point cloud processing model, a hybrid optimization algorithm is used to optimize the primitive parameters of the primitive dictionary in some embodiments of the present application.
[0044] In some embodiments of the present application, the base element dictionary is determined by the following method: randomly initializing the base element dictionary in the preset point cloud processing model; training the preset point cloud processing model based on the point cloud sample, and in the process of training the preset point cloud processing model, alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the base element dictionary for multiple rounds. For example, on the basis of optimizing the network parameters of each layer network of the preset point cloud processing model by using the preset model parameter optimization method, the base element parameters of the base element dictionary in the pooling layer of the preset point cloud processing model are optimized by using the preset dictionary parameter optimization method; and the above process is repeated for multiple rounds of network parameter optimization.
[0045] In some embodiments of the present application, the alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the base element dictionary for multiple rounds includes: using a preset model parameter optimization method to perform a preset first number of rounds of model parameter iterative updates on the preset point cloud processing model, wherein the model parameters include the base element dictionary; using a preset dictionary parameter optimization method to perform one round of base element dictionary update on the base element dictionary; and iteratively performing the above preset first number of rounds of model parameter iterative updates and one round of base element dictionary update in turn until a training termination condition of the preset point cloud processing model is met.
[0046] For example, in the process of training the preset point cloud processing model based on the point cloud sample, first, the gradient descent method is used to perform 50 rounds of network parameter optimization of the preset point cloud processing model (wherein the network parameters include the base element parameters in the base element dictionary), and then the preset dictionary parameter optimization method is used to optimize the base element parameters of the base element dictionary in the pooling layer of the preset point cloud processing model. Next, the gradient descent method is repeatedly used to perform 50 rounds of network parameter optimization of the preset point cloud processing model, and then the preset dictionary parameter optimization method is used to optimize the base element parameters of the base element dictionary in the pooling layer of the preset point cloud processing model. The above process is iteratively optimized until a training termination condition of the preset point cloud processing model is met. The training termination condition can be that the number of iterations reaches a preset number threshold, or the model error of the preset point cloud processing model converges to a preset range.
[0047] In some embodiments of this application, the network parameters of the pooling layer include a similarity matrix. The step of updating the primitive dictionary using a preset dictionary parameter optimization method includes: calculating the similarity weight between each local feature in the local feature set of the current point cloud sample and each dictionary primitive in the latest updated primitive dictionary based on the similarity matrix using the similarity calculation kernel function; wherein the similarity weight indicates the closeness of the relationship between the corresponding local feature and the corresponding dictionary primitive; for each dictionary primitive, weighted summation is performed on each local feature based on the normalized similarity weight between each local feature and the dictionary primitive, and the value of the dictionary primitive is updated using the result of the weighted summation; wherein the normalized similarity weight is obtained by dividing the normalized similarity weight between the corresponding local feature and the dictionary primitive by the sum of the similarity weights between all local features and the dictionary primitive.
[0048] The following describes the local feature set of point cloud data as X = {x1, x2, ..., x...} n Let the set of primitives in the primitive dictionary be C = {c1, c2, ..., c}. k Taking [example] as an example, this document illustrates the specific implementation of the preset dictionary parameter optimization method for updating the primitive dictionary in one round. Here, n is a natural number representing the number of local features, and K is a natural number representing the number of primitives in the primitive dictionary, i.e., the number of primitives in the bag-of-words. The value of K is determined based on the performance of the preset point cloud processing model on the validation set.
[0049] First, the local feature x in the local feature set X of the current point cloud sample can be calculated using the following formula. i In the latest round of updates, the dictionary primitive c k Similarity weight
[0050] In Formula 1 above, S(a,b) is the similarity calculation kernel function. In some embodiments of this application, the similarity calculation kernel function is an inner product function, S(a,b) = a T b, T represents the similarity matrix; K is the number of primitives in the primitive dictionary. From the above formula, we know that S(x i ,c k ) represents a local feature x i With the dictionary primitive c in the primitive dictionary k similarity, Representing local features x iThe sum of the similarities between each local feature in the local feature set X and each dictionary primitive in the dictionary primitive set. The similarity weight between each local feature in the local feature set X and each dictionary primitive in the dictionary primitive set can be calculated by the above formula 1. As can be seen from the above formula 1, the similarity weight between a certain local feature and a certain dictionary primitive in the dictionary primitive set represents the closeness of the relationship between the local feature and the dictionary primitive, that is: the greater the similarity weight, the closer the relationship between the local feature x The greater the value, the closer the relationship between the local feature x i and the dictionary primitive c k , that is, the greater the contribution of the local feature x i to the dictionary primitive.
[0051] Next, based on the similarity weight between each local feature and each dictionary primitive, the value of the corresponding dictionary primitive is updated. For example, in some embodiments of the present application, the value of the dictionary primitive c k may be updated by the following formula 2:
[0052]
[0053] In the above formula 2, represents the similarity weight between the local feature x j and the dictionary primitive c k , represents the normalized similarity weight between the local feature x i and the dictionary primitive c k , and K is the number of dictionary primitives in the dictionary primitive set. As can be seen from the above formula 2, the updated dictionary primitive is fused according to the contribution of each local feature to the dictionary primitive, so the updated dictionary primitive can better represent the contribution of the local feature of the point cloud data to the global feature.
[0054] After the training of the preset point cloud processing model is completed, in the online test process, the preset point cloud processing model can be directly used to extract features from the obtained point cloud data to determine the local feature set of the point cloud data. Then, the pooling layer of the preset point cloud processing model is used to perform bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed dictionary primitive set to determine the global feature of the point cloud data. Finally, the global feature is classified and mapped or other operations are performed through the fully connected layer, so as to obtain the model output result.
[0055] In some embodiments of the present application, the bag-of-words pooling processing is performed on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data, including: determining the similarity of each local feature in the local feature set to each dictionary primitive in the pre-constructed primitive dictionary respectively, as the contribution value of the corresponding local feature to each dictionary primitive; for each dictionary primitive, aggregating the contribution values of the local features to the dictionary primitive to determine the aggregated contribution value of the local feature set to the dictionary primitive; and taking the distribution histogram of the aggregated contribution values of each dictionary primitive in the primitive dictionary as the global feature of the point cloud data.
[0056] For example, the similarity V i between the local feature x j and the dictionary primitive c ij may be calculated by the following formula 3.
[0057]
[0058] wherein T represents a similarity matrix. As can be seen from the above formula 3, the similarity V ij reflects the closeness of the relationship between the local feature x i and the dictionary primitive c j , and therefore, the similarity V ij may be used to represent the contribution value of the local feature x i to the dictionary primitive c j .
[0059] Next, for each dictionary primitive, the contribution values of each local feature in the local feature set X to the dictionary primitive can be aggregated to determine the aggregated contribution value of the local feature set X to the dictionary primitive. For example, the contribution values of all local features in the local feature set X to the dictionary primitive c j are added up, and the result of the addition is taken as the aggregated contribution value of the local feature set X to the dictionary primitive c j .
[0060] In order to improve the representation ability of the global feature for the point cloud data and improve the extraction efficiency of the global feature, only a small number of important local features are considered to contribute to the global feature. In some embodiments of the present application, the aggregation of the contribution values of the local features to the dictionary primitives to determine the aggregated contribution value of the local feature set to the dictionary primitive includes: aggregating the largest preset number of contribution values to the dictionary primitive to determine the aggregated contribution value of the local feature set to the dictionary primitive. For example, the primitives with smaller local feature similarity are regarded as noise, and only the contributions of the L primitives with the largest similarity are calculated. Then, the local feature x i contributes to the primitive cj Contribution V ij The following optimization formula applies:
[0061]
[0062] In formula 4 above, I i To be related to local feature x i The set of L dictionary primitives with the highest similarity is obtained by comparing them with local features x. i Dictionary primitives with low similarity to the local feature x i Setting the similarity to 0 allows us to ignore the contribution of primitives with low similarity to local features.
[0063] Next, for each of the aforementioned dictionary primitives, the formula can be used... The contribution values of all the local features in the local feature set X to the dictionary primitive are aggregated to determine the aggregated contribution value of the local feature set X to the dictionary primitive, where n is the number of local features in the local feature set X.
[0064] Following the method described above, the aggregate contribution value of the local feature set X to each dictionary primitive can be determined, for example, denoted as h1, h2, ... h1 respectively. n .
[0065] Then, based on the aggregated contribution values h1, h2, ... h of the local feature set X for each dictionary primitive, n Further determine the distribution histogram of the aggregation contribution values of the dictionary primitives, for example, by obtaining a distribution histogram H = [h1, h2, ..., h K This distribution histogram is also a histogram showing the contribution of local features of the point cloud data to each primitive. As a result of merging local features of the point cloud, this distribution histogram will serve as a global feature describing the point cloud for downstream tasks.
[0066] Taking a primitive dictionary containing three primitives—triangle, rectangle, and circle—as an example, a car can be simplified to two rectangles and four circles. Therefore, the histogram obtained by projecting the car's shape onto the primitive dictionary can be represented as (0, 2, 4). A pointed-roof house can be simplified to one triangle and one rectangle, and its histogram can be represented as (1, 1, 0). By using the histogram of the aggregation contribution values of the local feature set of the point cloud data to the primitive dictionary as the global feature of the point cloud data, the relationship between the pooled result and the input features can be clearly explained.
[0067] Step 130: The global features are used as the feature representation of the target object and output to the downstream feature processing task.
[0068] As described above, the preset point cloud processing model described in the embodiments of the present application can be applied to different scenarios. For the preset point cloud processing model applied to different scenarios, the backbone network 210 and the fully connected layer 230 can be implemented based on different network structures to output different results.
[0069] In some embodiments of the present application, taking the application of the preset point cloud processing model to the classification scenario as an example, the fully connected layer is a classification network structure, and the global feature is output as the feature expression of the target object to a downstream feature processing task, including: outputting the global feature as the feature expression of the target object to the fully connected layer, executing a classification task through the fully connected layer, and determining the allocation result matched by the target object. For example, in the unmanned driving scenario, after the radar arranged on the unmanned driving device collects the point cloud data of the driving scene, the point cloud data is input to the preset point cloud processing model, and the backbone network 210 outputs the local feature set matched by the driving scene. Then, the pooling layer 220 performs bag-of-words pooling processing on the local feature set output by the backbone network 210 based on the pre-trained element dictionary to obtain the global feature of the input point cloud data; finally, the fully connected layer 230 performs classification mapping on the global feature to output the category of the target object matched by the point cloud data.
[0070] The bag-of-words pooling processing method of the pooling layer described in the embodiments of the present application can also be applied to other point cloud processing architectures to replace the pooling layer in the existing point cloud processing architecture. The structure of the preset point cloud processing model in the embodiments of the present application will not be described one by one.
[0071] The point cloud data processing method disclosed in the embodiments of the present application determines the local feature set of the point cloud data of the target object, then performs bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed element dictionary to determine the global feature of the point cloud data, and finally outputs the global feature as the feature expression of the target object to a downstream feature processing task, which helps to improve the point cloud data processing accuracy.
[0072] The point cloud data processing method disclosed in the embodiments of the present application improves the pooling layer to fuse the local features of the point cloud through bag-of-words pooling processing, without compressing and dimensionality reducing the local features, thereby avoiding the loss of point cloud information and improving the point cloud data processing accuracy. On the other hand, since the bag-of-words pooling statistically analyzes the unordered local features and generates a histogram to reflect how the elements in the element dictionary constitute the global feature, the global feature is more interpretable.
[0073] In the point cloud data processing method disclosed in the embodiments of the present application, the output global feature can be applied to point cloud data application tasks such as classification tasks, search tasks, and point cloud segmentation tasks. By setting a pooling layer in the existing point cloud processing architecture, that is, by using the method disclosed in the embodiments of the present application to perform bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed base dictionary, the accuracy of performing downstream tasks such as classification tasks, search tasks, and point cloud segmentation based on the global feature output by the pooling layer can be improved.
[0074] For example, in a classification task performed on the ModelNet40 point cloud dataset containing 12311 point cloud models, the classification accuracy of the deep learning model PointNet processing point cloud data is 89.2% on the same test set, and after adding the pooling layer described in the embodiments of the present application, the classification accuracy of the deep learning model PointNet is 90.0%. For example, in a point cloud segmentation task performed on the ShapeNet and S3DIS point cloud datasets, the ShapeNet point cloud dataset contains 16881 point cloud models, and the S3DIS point cloud dataset contains scanned point clouds in 6 indoor scenes. On the S3DIS point cloud dataset, the segmentation accuracy of the deep learning model PointNet processing point cloud data is 78.62%, and after adding the pooling layer described in the embodiments of the present application, the classification accuracy of the deep learning model PointNet is 79.78%. Experimental data show that the method of performing bag-of-words pooling processing on the local feature set to obtain the global feature disclosed in the embodiments of the present application has higher accuracy.
[0075] Embodiment Two
[0076] The point cloud data processing device disclosed in the embodiments of the present application, as shown in Figure 3 The device comprises:
[0077] The local feature set determination module 310 is configured to determine a local feature set of point cloud data of a target object.
[0078] The bag-of-words pooling processing module 320 is configured to perform bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed base dictionary to determine a global feature of the point cloud data.
[0079] The global feature output module 330 is configured to output the global feature as a feature expression of the target object to a downstream feature processing task.
[0080] In some embodiments of the present application, the bag-of-words pooling processing module 320 is further configured to:
[0081] determine a similarity between each local feature in the set of local features and each dictionary primitive in a pre-constructed dictionary of primitives, as a contribution value of the local feature to each of the dictionary primitives;
[0082] aggregate the contribution values of the local features to the dictionary primitives, for each of the dictionary primitives, to determine an aggregated contribution value of the set of local features to the dictionary primitive;
[0083] generate a histogram of the aggregated contribution values of the dictionary primitives in the dictionary of primitives as a global feature of the point cloud data.
[0084] In some embodiments of the present application, the aggregating the contribution values of the local features to the dictionary primitives, for each of the dictionary primitives, to determine an aggregated contribution value of the set of local features to the dictionary primitive comprises:
[0085] aggregating a predetermined number of the largest contribution values of the local features to the dictionary primitive to determine the aggregated contribution value of the set of local features to the dictionary primitive.
[0086] In some embodiments of the present application, the step of performing bag-of-words pooling processing on each local feature in the set of local features based on a pre-constructed dictionary of primitives to determine a global feature of the point cloud data is performed by a pooling layer of a pre-set point cloud processing model, and the pooling layer comprises a dictionary of primitives determined by the following method:
[0087] randomly initializing the dictionary primitives in the dictionary of primitives included in the pre-set point cloud processing model;
[0088] training the pre-set point cloud processing model based on point cloud samples, and in the process of training the pre-set point cloud processing model, alternately using a pre-set model parameter optimization method and a pre-set dictionary parameter optimization method to update the dictionary of primitives for multiple rounds.
[0089] In some embodiments of the present application, the network parameters of the pooling layer include a similarity matrix, and the alternately updating the dictionary of primitives for multiple rounds using the pre-set model parameter optimization method and the pre-set dictionary parameter optimization method comprises:
[0090] performing a pre-set first number of rounds of model parameter iterative updates on the pre-set point cloud processing model using a pre-set model parameter optimization method, wherein the model parameters include the dictionary of primitives;
[0091] performing one round of dictionary of primitives updates on the dictionary of primitives using a pre-set dictionary parameter optimization method;
[0092] The preset first number of iteration times of model parameter iteration update and one iteration time of primitive dictionary update are sequentially and iteratively performed until a training termination condition of the preset point cloud processing model is met.
[0093] The one iteration time of primitive dictionary update of the primitive dictionary by using the preset dictionary parameter optimization method comprises:
[0094] The similarity calculation kernel function calculates the similarity weight of each local feature in the local feature set of the current point cloud sample and each dictionary primitive in the updated primitive dictionary in the last iteration time based on the similarity matrix; wherein, the similarity weight is used to indicate the relationship closeness of the corresponding local feature and the corresponding dictionary primitive.
[0095] For each dictionary primitive, the local features are weighted and accumulated based on the normalized similarity weight of each local feature and the dictionary primitive, and the value of the dictionary primitive is updated through the result of weighted accumulation; wherein, the normalized similarity weight is obtained by dividing the normalized similarity weight of the corresponding local feature and the dictionary primitive by the sum of the similarity weights of all local features and the dictionary primitive.
[0096] In some embodiments of the present application, as shown in Figure 2 The preset point cloud processing model comprises: a backbone network 210, a pooling layer 220, and a fully connected layer 230; wherein,
[0097] The backbone network 210 is configured to extract features from the input point cloud data and determine a local feature set.
[0098] The backbone network 210 is further configured to generate a primitive dictionary based on a preset point cloud model set and the local feature set.
[0099] The pooling layer 220 is configured to perform bag-of-words pooling processing on each local feature in the local feature set based on the primitive dictionary, and determine a global feature of the input point cloud data.
[0100] The fully connected layer 230 is configured to perform feature mapping on the global feature output by the pooling layer, and obtain an output result of the preset point cloud processing model.
[0101] The point cloud data processing device disclosed in the embodiments of the present application is used to implement the point cloud data processing method described in Embodiment One of the present application. The specific implementation of each module of the device will not be repeated, and the specific implementation of the corresponding steps of the method embodiments can be referred to.
[0102] The point cloud data processing apparatus disclosed by the embodiments of the present application determines the local feature set of the point cloud data of the target object, then performs bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data, and finally outputs the global feature as the feature expression of the target object to the downstream feature processing task, which helps to improve the point cloud data processing accuracy.
[0103] The point cloud data processing apparatus disclosed by the embodiments of the present application improves the pooling layer to fuse the point cloud local features through bag-of-words pooling processing, without the need for local feature compression and dimension reduction, thereby avoiding point cloud information loss and improving the point cloud data processing accuracy. On the other hand, since the bag-of-words pooling statistically analyzes the unordered local features and generates a histogram to reflect how the elements in the primitive dictionary constitute the global feature, the global feature is more interpretable.
[0104] In the point cloud data processing apparatus disclosed by the embodiments of the present application, the output global feature can be applied to point cloud data application tasks such as classification tasks, search tasks, and point cloud segmentation tasks. By setting the pooling layer disclosed by the embodiments of the present application in the existing point cloud processing architecture, i.e., using the method disclosed by the embodiments of the present application to perform bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data, the local feature set is pooled, which can improve the accuracy of executing downstream tasks such as classification tasks, search tasks, and point cloud segmentation based on the global feature output by the pooling layer.
[0105] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.
[0106] The point cloud data processing method and apparatus provided by the present application are described in detail above, and the principles and implementation modes of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and a core idea thereof; meanwhile, for those skilled in the art, the specific implementation mode and application range can be changed according to the idea of the present application; and the above description should not be understood as a limitation of the present application.
[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.
[0108] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of the present application. The present application can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer readable medium or can have the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.
[0109] For example, Figure 4 An electronic device that can implement the methods according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally comprises a processor 410 and a memory 420 and program code 430 stored on the memory 420 and executable on the processor 410, which when executed by the processor 410 implements the methods described in the above embodiments. The memory 420 can be a computer program product or a computer readable medium. The memory 420 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 420 has storage space 4201 for program code 430 of computer programs for performing any of the method steps in the above methods. For example, the storage space 4201 for program code 430 can comprise individual computer programs for implementing the various steps in the above methods, respectively. The program code 430 is computer readable code. The computer programs can be read out from or written into one or more computer program products. The computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. The computer programs include computer readable code which, when executed on the electronic device, causes the electronic device to perform the methods according to the above embodiments.
[0110] The embodiment of the present application further discloses a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the point cloud data processing method in the embodiment one of the present application.
[0111] Such a computer program product can be a computer readable storage medium, which can have a similar arrangement of storage segments, storage spaces, etc. as the memory 420 in the electronic device shown. The program code can be stored in the computer readable storage medium, for example, in a compressed form. The computer readable storage medium is usually a portable or fixed storage unit as described with reference to Figure 4 The program code can be stored in the computer readable storage medium, for example, in a compressed form. The computer readable storage medium is usually a portable or fixed storage unit as described with reference to Figure 5 The program code can be stored in the computer readable storage medium, for example, in a compressed form. The computer readable storage medium is usually a portable or fixed storage unit as described with reference to
[0112] The term "one embodiment", "an embodiment” or "one or more embodiments” as used herein means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
[0113] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
[0114] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a comma. Such listing does not imply that the devices must be co-located. The word "first”, "second”, "third” etc. does not imply any order. The terms "first”, "second”, "third” etc. are to be interpreted according to their meaning known to those skilled in the art. The application is not limited to the embodiments described herein but can vary within the scope of the appended claims.
[0115] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for processing point cloud data, characterized in that, The method comprises: determining a local feature set of point cloud data of a target object; performing bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed primitive dictionary to determine a global feature of the point cloud data; outputting the global feature as a feature representation of the target object to a downstream feature processing task; The step of performing bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data is performed by a pooling layer of a preset point cloud processing model, and the pooling layer comprises: a primitive dictionary, which is determined by the following method: randomly initializing dictionary primitives in the primitive dictionary included in the preset point cloud processing model; training the preset point cloud processing model based on point cloud samples, and in the process of training the preset point cloud processing model, alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the primitive dictionary for multiple rounds; The network parameters of the pooling layer include a similarity matrix, and the alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the primitive dictionary for multiple rounds comprises: performing preset first number of rounds of model parameter iterative updates on the preset point cloud processing model using the preset model parameter optimization method, wherein the model parameters include the primitive dictionary; performing one round of primitive dictionary updates on the primitive dictionary using the preset dictionary parameter optimization method; iteratively performing the preset first number of rounds of model parameter iterative updates and one round of primitive dictionary updates in turn until a training termination condition of the preset point cloud processing model is met; wherein the performing one round of primitive dictionary updates on the primitive dictionary using the preset dictionary parameter optimization method comprises: calculating, based on the similarity matrix, similarity weights of each local feature in a local feature set of a current point cloud sample and each dictionary primitive in the primitive dictionary updated in the latest round respectively through the similarity calculation kernel function; wherein the similarity weight is used to indicate the closeness of the relationship between the corresponding local feature and the corresponding dictionary primitive; for each dictionary primitive, weighting and accumulating each local feature based on the normalized similarity weight of each local feature and the dictionary primitive, and updating the value of the dictionary primitive through the result obtained by the weighting and accumulation; wherein the normalized similarity weight is obtained by dividing the normalized similarity weight of the corresponding local feature and the dictionary primitive by the sum of the similarity weights of all the local features and the dictionary primitive.
2. The method of claim 1, wherein, The step of performing bag-of-words pooling processing on each local feature in the local feature set based on the pre-constructed primitive dictionary to determine the global feature of the point cloud data comprises: determining the similarity of each local feature in the local feature set and each dictionary primitive in the pre-constructed primitive dictionary as the contribution value of the corresponding local feature to each dictionary primitive; for each dictionary primitive, aggregating the contribution values of the local features to the dictionary primitive to determine the aggregated contribution value of the local feature set to the dictionary primitive; A distribution histogram of the aggregated contribution values of each dictionary primitive in the primitive dictionary is taken as a global feature of the point cloud data.
3. The method of claim 2, wherein, The step of aggregating the contribution values of the local features to the dictionary primitives to determine the aggregated contribution values of the local feature set to the dictionary primitives comprises: The maximum preset number of contribution values to the dictionary primitives are aggregated to determine the aggregated contribution values of the local feature set to the dictionary primitives.
4. The method according to any one of claims 1 to 3, characterized in that, The preset point cloud processing model comprises a backbone network, a pooling layer and a fully connected layer, wherein the backbone network is configured to extract features of input point cloud data to determine a local feature set; The backbone network is further configured to generate a primitive dictionary based on a preset point cloud model set and the local feature set; The pooling layer is configured to perform bag-of-words pooling processing on each local feature in the local feature set based on the primitive dictionary to determine a global feature of the input point cloud data; The fully connected layer is configured to perform feature mapping on the global feature output by the pooling layer to obtain an output result of the preset point cloud processing model.
5. A point cloud data processing apparatus, characterized by comprising: Comprise: A local feature set determination module configured to determine a local feature set of point cloud data of a target object; A bag-of-words pooling processing module configured to perform bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed primitive dictionary to determine a global feature of the point cloud data; A global feature output module configured to output the global feature as a feature representation of the target object to a downstream feature processing task; The step of performing bag-of-words pooling processing on each local feature in the local feature set based on a pre-constructed primitive dictionary to determine a global feature of the point cloud data is performed by a pooling layer of a preset point cloud processing model, and the pooling layer comprises a primitive dictionary determined by the following steps: Randomly initializing a dictionary primitive in the primitive dictionary included in the preset point cloud processing model; Training the preset point cloud processing model based on point cloud samples, and in the process of training the preset point cloud processing model, alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the primitive dictionary for multiple rounds; The network parameters of the pooling layer comprise a similarity matrix, and the alternately using a preset model parameter optimization method and a preset dictionary parameter optimization method to update the primitive dictionary for multiple rounds comprises: Using a preset model parameter optimization method to perform preset first number of rounds of model parameter iterative updates on the preset point cloud processing model, wherein the model parameters comprise the primitive dictionary; Using a preset dictionary parameter optimization method to perform one round of primitive dictionary update on the primitive dictionary; Iteratively performing the preset first number of rounds of model parameter iterative updates and one round of primitive dictionary update in sequence until a training termination condition of the preset point cloud processing model is met; wherein the using a preset dictionary parameter optimization method to perform one round of primitive dictionary update on the primitive dictionary comprises: The similarity calculation kernel function calculates a similarity weight of each local feature in the local feature set of the current point cloud sample and each dictionary primitive in the updated base primitive dictionary in the last round respectively based on the similarity matrix; wherein the similarity weight is used to indicate the closeness of the relationship between the corresponding local feature and the corresponding dictionary primitive; For each dictionary primitive, the local features are weighted and accumulated based on the normalized similarity weight of each local feature and the dictionary primitive, and the value of the dictionary primitive is updated by the result obtained by the weighted accumulation; wherein the normalized similarity weight is obtained by dividing the normalized similarity weight of the corresponding local feature and the dictionary primitive by the sum of the similarity weights of all local features and the dictionary primitive.
6. The apparatus of claim 5, wherein, The bag-of-words pooling processing module is further used for: determining the similarity of each local feature in the local feature set and each dictionary primitive in the pre-constructed base primitive dictionary as the contribution value of the corresponding local feature to each dictionary primitive; for each dictionary primitive, aggregating the contribution values of the local features to the dictionary primitive to determine the aggregated contribution value of the local feature set to the dictionary primitive; the distribution histogram of the aggregated contribution values of each dictionary primitive in the base primitive dictionary is used as the global feature of the point cloud data.
7. An electronic device comprising a memory, a processor, and program code stored on the memory and executable on the processor, wherein, The processor executes the program code to implement the point cloud data processing method of any one of claims 1 to 4.
8. A computer-readable storage medium having stored thereon a program code, characterized in that, The program code is executed by the processor to implement the steps of the point cloud data processing method of any one of claims 1 to 4.