Three-dimensional point cloud analysis method, device, equipment, medium and program product

Abstract

The invention relates to a three-dimensional point cloud analysis method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring point cloud embedded data of an input point cloud; pooling the point cloud embedded data to obtain average pooling data and maximum pooling data; point token construction is carried out according to the position injector, the point cloud embedded data and the average pooling data, and candidate point tokens are obtained; performing cue word generation on the candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens; performing fine adjustment on the two-dimensional pre-training model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used for processing a three-dimensional point cloud analysis task. By adopting the method, the parameter quantity can be reduced, and the accuracy of three-dimensional point cloud analysis can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to three-dimensional point cloud analysis methods, devices, equipment, media and program products. Background Art

[0002] With the development of computer vision, parameter efficient fine-tuning (PEFT) technology has become increasingly mature. This technology effectively handles various downstream tasks by leveraging the rich semantic features and representation capabilities of large base models, while significantly reducing learning and storage costs. However, as an important research direction, 3D visual understanding faces greater challenges in data acquisition, which leads to the lack of large-scale pre-trained base models for 3D tasks. However, in existing technologies, acquiring and annotating high-quality 3D data requires a lot of resources and manpower, and synthetic data usually has problems with insufficient distribution diversity and poor practical applicability. Therefore, there is an urgent need for a point cloud analysis method that can improve the accuracy of point cloud analysis while reducing the amount of parameters. Summary of the invention

[0003] Based on this, it is necessary to provide a three-dimensional point cloud analysis method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of point cloud analysis while reducing the number of parameters in response to the above technical problems.

[0004] In a first aspect, the present application provides a three-dimensional point cloud analysis method, comprising:

[0005] Get the point cloud embedding data of the input point cloud;

[0006] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0007] constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0008] Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0009] The two-dimensional pre-trained model is fine-tuned through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0010] In one embodiment, the step of constructing a point token according to the position injector, the point cloud embedding data and the average pooling data to obtain a candidate point token includes:

[0011] Determining an origin point token based on the average pooled data;

[0012] determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector;

[0013] The relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token.

[0014] In one embodiment, the step of constructing a point token according to the position injector, the point cloud embedding data and the average pooling data to obtain a candidate point token includes:

[0015] Determining an origin point token based on the average pooled data;

[0016] determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector;

[0017] The relative position of the linear combination is injected into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0018] In one embodiment, generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token includes:

[0019] Prompt words are generated for the average pooled data and the maximum pooled data by a trainable prompt generator to obtain a target prompt word; parameters are shared between the prompt generator and the position injector;

[0020] A target point token is constructed according to the target prompt word and the candidate point token.

[0021] In one embodiment, the generating of prompt words for the average pooled data and the maximum pooled data by a trainable prompt generator to obtain a target prompt word includes:

[0022] Performing feature extraction on the average pooled data and the maximum pooled data through a trainable prompt generator to obtain overall feature data;

[0023] Prompt words are generated according to the overall feature data to obtain the target prompt words with the same dimension as the candidate point token.

[0024] In one embodiment, the method further comprises:

[0025] Get the data set to be predicted;

[0026] Performing a three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result;

[0027] The three-dimensional point cloud analysis results are visualized.

[0028] In a second aspect, the present application also provides a three-dimensional point cloud analysis device, comprising:

[0029] An acquisition module, used to acquire point cloud embedding data of an input point cloud;

[0030] A pooling processing module, used for performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0031] A token construction module, used for constructing point tokens according to a position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0032] A prompt word generation module, used to generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0033] The model fine-tuning module is used to fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Get the point cloud embedding data of the input point cloud;

[0036] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0037] constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0038] Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0039] The two-dimensional pre-trained model is fine-tuned through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Get the point cloud embedding data of the input point cloud;

[0042] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0043] constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0044] Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0045] The two-dimensional pre-trained model is fine-tuned through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Get the point cloud embedding data of the input point cloud;

[0048] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0049] constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0050] Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0051] The two-dimensional pre-trained model is fine-tuned through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0052] The above-mentioned three-dimensional point cloud analysis method, device, computer equipment, computer-readable storage medium and computer program product obtain point cloud embedding data of the input point cloud; perform pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; construct point tokens according to the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens; generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens; fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of the candidate point tokens, and prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of the point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point token generated by the embedding layer is aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decrease in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A diagram showing an application environment of a three-dimensional point cloud analysis method in an embodiment;

[0055] Figure 2 A schematic diagram of a process of a three-dimensional point cloud analysis method in one embodiment;

[0056] Figure 3 A schematic diagram of a process of a three-dimensional point cloud analysis method in one embodiment;

[0057] Figure 4 A color visualization image showing the effect of a three-dimensional point cloud analysis method in another embodiment;

[0058] Figure 5 A color visualization image showing the effect of a three-dimensional point cloud analysis method in another embodiment;

[0059] Figure 6 A schematic diagram of a flow chart of a three-dimensional point cloud analysis method in another embodiment;

[0060] Figure 7 A schematic diagram of a flow chart of a three-dimensional point cloud analysis method in another embodiment;

[0061] Figure 8 is a structural block diagram of a three-dimensional point cloud analysis device in one embodiment;

[0062] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The three-dimensional point cloud analysis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the point cloud embedding data of the input point cloud; performs pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; constructs point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; generates prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token; fine-tunes the two-dimensional pre-trained model through the target point token to obtain the target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In an exemplary embodiment, Figure 2As shown, a three-dimensional point cloud analysis method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S210. Among them:

[0066] Step S202, obtaining point cloud embedding data of the input point cloud.

[0067] Among them, the input point cloud can be the three-dimensional point cloud data of the data set to be trained obtained by the K nearest neighbor algorithm and the farthest point sampling method according to the data set to be trained, and the point cloud embedding data is obtained by converting the grouped input point cloud into structured and representative embedding data.

[0068] In some embodiments, the training set data to be predicted can be first obtained, and the K-nearest neighbor algorithm and farthest point sampling processing can be performed on the training set data to be predicted to obtain an input point cloud, and then the point cloud data after the input point cloud is grouped is converted into multiple groups of structured and representative point cloud embedding data.

[0069] Among them, the training set data to be predicted can use the ScanObjectNN data set and the ModelNet40 data set, the K nearest neighbor algorithm can use the KNN algorithm, the farthest point sampling processing can use the FPS algorithm, and the input point cloud can be grouped by random grouping or other grouping methods, which are not limited to this.

[0070] In some embodiments, specifically, a lightweight network (Point_Embed) is implemented to obtain point cloud embedding data of the input point cloud, which is in the form of the following formula (1):

[0071] (1)

[0072] Among them, Point_Embed can take many forms, such as PointNet, PointMLP, Point-PN, etc., input point cloud The point set after grouping from the input point cloud In order to simplify the subsequent embodiments, Indicate point of Neighborhood points A collection of .

[0073] It should be noted that the input point cloud is obtained by performing K-nearest neighbor algorithm and farthest point sampling processing on the training set data to be predicted, and then the point cloud data after grouping the input point cloud is converted into multiple groups of structured and representative point cloud embedding data. This not only enhances the utilization efficiency of the point cloud, but also aligns the point cloud embedding data with the input dimension of the subsequent two-dimensional pre-trained model that needs to be fine-tuned, thereby promoting the utilization of prior knowledge by the two-dimensional pre-trained model.

[0074] Step S204: performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data.

[0075] Among them, pooling processing includes average pooling processing and maximum pooling processing.

[0076] In some embodiments, the point cloud embedding data may be subjected to average pooling processing and maximum pooling processing through a pooling operation module, and the overall features of the point cloud embedding data may be captured through pooling processing, thereby obtaining average pooling data and maximum pooling data.

[0077] In some embodiments, the average pooling process can be performed , thereby obtaining the global token embedding of the point cloud embedding data of the input point cloud , and embed the global token into As the average pooled data, and the average pooled data can be used to represent the centroid of the input point cloud: .

[0078] In some embodiments, a maximum pooling process may be performed to obtain the maximum pooling data of the point cloud embedding data of the input point cloud, and the average pooling data may be expressed as .

[0079] Step S206, constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens.

[0080] Among them, the position injector is a Position Injector designed based on the position encoding in the two-dimensional pre-trained model Transformer. The position injector can inject sufficient position information from the source modality into the 3D token, thereby achieving more effective collaboration with the frozen two-dimensional pre-trained model Transformer.

[0081] In some embodiments, point tokens are constructed based on a position injector, point cloud embedding data and average pooling data to obtain candidate point tokens, including: determining the original point token based on the average pooling data; determining the relative position of the linear combination between the point cloud embedding data and the original point token based on the position injector; and injecting the relative position of the linear combination into the original point token through a preset learning model to obtain a candidate point token.

[0082] Among them, the original point token is the global token embedding data for each input point cloud in the initial state, the linear combination relative position represents the relative position difference between the current point cloud and the center of mass of the entire input point cloud, and the preset learning model refers to a pre-trained learning model containing learnable parameters, and is used to inject the linear combination relative position into the original point token.

[0083] In some embodiments, the average pooled data is used as the global token embedding data of the input point cloud and can be directly used to determine the original point token , and then determine the point cloud embedding data through the position injector and the origin token The relative position of the linear combination between .

[0084] Among them, the relative position of the linear combination can be expressed as .

[0085] In some embodiments, the relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token. The specific position injection process can be shown in the following formula (2):

[0086] (2)

[0087] in, is the candidate point token, and are learnable parameters.

[0088] In some embodiments, point tokens are constructed based on a position injector, point cloud embedding data and average pooling data to obtain candidate point tokens, including: determining the original point token based on the average pooling data; determining the relative position of the linear combination between the point cloud embedding data and the original point token based on the position injector; and injecting the relative position of the linear combination into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0089] Among them, the preset convolution kernel model may refer to a 1D convolution kernel.

[0090] In some embodiments, after determining the linear combination relative position between the point cloud embedding data and the original point token according to the position injector, the 1D convolution kernel injects the linear combination relative position into the original point token, so that this module can be seamlessly integrated into the existing model as an independent layer. Specifically, the calculation process of the candidate point token can be shown as formula (3):

[0091] (3)

[0092] in, refers to the preset convolution kernel model, refers to the relative position of the linear combination between the point cloud embedding data and the original point token, Represents the centroid of the point cloud embedding data.

[0093] In this embodiment, a Position Injector is proposed, which injects sufficient position information from the source modality into the 3D token, thereby achieving more effective collaboration with the frozen two-dimensional pre-trained model Transformer, and then making full use of the attention prior ability of the two-dimensional pre-trained model Transformer, thereby improving the analysis effect of three-dimensional point cloud data after subsequent model training is completed.

[0094] Step S208, generating prompt words for candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens.

[0095] Among them, the process of prompt word generation refers to attaching the generated prompt word to the candidate point token, and the target point token contains the target prompt word, which is used to implement the fine-tuning self-attention mechanism for different tasks for the two-dimensional pre-trained model.

[0096] In some embodiments, prompt words are generated for candidate point tokens based on average pooling data and maximum pooling data to obtain target point tokens, including: generating prompt words for average pooling data and maximum pooling data through a trainable prompt generator to obtain target prompt words; sharing parameters between the prompt generator and the position injector; and constructing the target point token based on the target prompt words and candidate point tokens.

[0097] In some embodiments, prompt words are generated for average pooling data and maximum pooling data through a trainable prompt generator to obtain target prompt words, including: extracting features from average pooling data and maximum pooling data through a trainable prompt generator to obtain overall feature data; generating prompt words based on the overall feature data to obtain target prompt words with the same dimension as the candidate point token.

[0098] Among them, a trainable prompt generator is used to generate prompt words, and the prompt words generated by the prompt generator have the following properties: 1) are closely related to the input point cloud; 2) are used to capture the overall information of the input point cloud; 3) have the same dimension as the token embedding.

[0099] In some embodiments, a trainable prompt generator is used to generate prompt words for the average pooled data and the maximum pooled data to obtain a target prompt word. Since the prompt generator adopts the same structure as token embedding and introduces the average pooled data and the maximum pooled data obtained by the pooling operation to capture the overall features of the input point cloud, the target prompt word is obtained, and then the target prompt word is attached to the candidate point token to obtain the target point token.

[0100] It should be noted that sharing parameters between the prompt generator and the position injector realizes parameter sharing of the token embedding module and the prompt generator module, thereby maintaining consistency between the point tokens that capture local features and the point prompts that encode overall features, while reducing training parameters.

[0101] In some embodiments, a trainable prompt generator is used to generate prompt words for average pooled data and maximum pooled data, and the generation process of the target prompt word is shown in the following formula (4):

[0102] (4)

[0103] in, Represents the maximum pooling data and represents average pooled data, Indicates the target cue word.

[0104] In this embodiment, the target prompt word constructed by average pooling data and maximum pooling data is permutation invariant to the group arrangement of the input point cloud, thereby ensuring that the model is less sensitive to the order of point groups, so that the prompt generator provides more stable global features and eliminates redundant information. At the same time, the target prompt word retains the geometric information of the input point cloud.

[0105] Step S210, fine-tuning the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task.

[0106] Among them, the fine-tuning process can adopt parameter efficient fine-tuning (PEFT) technology to alleviate overfitting and improve the generalization ability of the 3D model. The 3D point cloud analysis task can be ablation experiment tasks, object classification and other downstream tasks of 3D point cloud analysis. The 2D pre-trained model is a backbone network frozen on a large-scale data set.

[0107] In some embodiments, by inputting the target point token into a two-dimensional pre-trained model and passing through a multi-stage Transfarmer block in the two-dimensional pre-trained model, while continuously copying the target prompt word in the target point token, the two-dimensional pre-trained model is fine-tuned to obtain a target point cloud analysis model. The target point cloud analysis model can be used to process three-dimensional point cloud analysis tasks to improve the execution efficiency of three-dimensional point cloud analysis tasks. This fine-tuning method not only reduces the learning parameters required in model training, but also ensures the execution efficiency of the target point cloud analysis model in three-dimensional point cloud analysis tasks, so that it can be applied to related downstream tasks of three-dimensional point cloud analysis to improve the execution effect of each downstream task.

[0108] In the above three-dimensional point cloud analysis method, the point cloud embedding data of the input point cloud is obtained; the point cloud embedding data is pooled to obtain average pooling data and maximum pooling data; point tokens are constructed according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens; the two-dimensional pre-trained model is fine-tuned through the target point tokens to obtain the target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of the candidate point tokens, and prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of the point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point token generated by the embedding layer is aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decrease in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks.

[0109] In an exemplary embodiment, Figure 3 As shown, the three-dimensional point cloud analysis method further includes steps S302 to S306. Among them:

[0110] Step S302, obtaining a data set to be predicted.

[0111] The data set to be predicted may be a ScanObjectNN data set or a ModelNet40 data set, but is not limited thereto.

[0112] In some embodiments, the data set to be predicted may be obtained from a point cloud database to be predicted, or the data set to be predicted may be obtained in other ways, but is not limited thereto.

[0113] Step S304: performing a three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result.

[0114] Among them, the three-dimensional point cloud analysis is to analyze and calibrate the three-dimensional point cloud in the training set to be predicted, so as to determine the coordinate data of each three-dimensional point cloud.

[0115] In some embodiments, a three-dimensional point cloud analysis is performed on each object in the training set to be predicted according to the target point cloud analysis model to obtain coordinate data of each three-dimensional point cloud in each object as a three-dimensional point cloud analysis result.

[0116] In some embodiments, corresponding three-dimensional point cloud analysis results can be generated by combining specific downstream tasks of three-dimensional point cloud analysis. For example, for an object classification task, an object classification accuracy result can be obtained.

[0117] Step S306: Visually display the three-dimensional point cloud analysis results.

[0118] The visual display includes but is not limited to table data display and dynamic color map display.

[0119] In some embodiments, the three-dimensional point cloud analysis results may be combined with a specific downstream task of the three-dimensional point cloud analysis to generate specific downstream task result data, and the downstream task result data may be visualized.

[0120] In some embodiments, in order to further illustrate the contribution of each module, the present solution visualizes the feature distribution and corresponding response on the original input point cloud in the 3D point cloud analysis result, such as Figure 4 and Figure 5 As shown, Figure 4 The feature visualization results corresponding to the 3D point cloud analysis results on the ScanObjectNN dataset to be predicted are shown in color. Figure 5 A color visualization showing the effects of different modules. Specifically, when only Point-PN is used, there is overlap in the feature distribution between different categories, such as Figure 4 As shown in (a) in . Figure 4 (b) shows the feature distribution after the PosIn module is aligned with ViT-B (built on Point-PN). Figure 4 (c) shows the effect of using only PT, using the same point embedding module. Figure 5From the above, we can clearly see that PT and PosIn focus on different parts of the object; however, both highlight the object structure, thus enhancing the separability of the features. Figure 4 (d) in the figure shows the effect of adopting the APPT (APPTuning) framework in this solution. Figure 5 The third row of shows that APPT captures a relatively complete overall structure of the object. This helps explain why APPT achieves significant improvements in classification but is slightly inferior to Joint-MAE in segmentation, because the APPT encoder places more emphasis on the overall structure of the input. Intuitively, it can be seen that the feature distribution boundaries obtained by APPT are more obvious, and the feature separability has been significantly improved.

[0121] In some embodiments, as shown in Tables 1, 2, 3, 4 and 5 below, Table 1 shows the accuracy results (APPT(ours)) of classifying objects based on the three-dimensional point cloud analysis results on the ScanObjectNN dataset, and also adds the accuracy results of the existing three-dimensional point cloud analysis technologies (DNN-based model and Transformer-basedmodel) to achieve a more intuitive effect display.

[0122] Table 1 Object classification accuracy (%) results on the ScanObjectNN dataset

[0123]

[0124] Among them, Table 2 shows the accuracy results of classifying objects based on the three-dimensional point cloud analysis results on the ModelNet40 dataset, and also adds the accuracy results of the existing three-dimensional point cloud analysis technology to achieve a more intuitive effect display.

[0125] Table 2 Object classification accuracy results on the ModelNet40 dataset

[0126]

[0127]

[0128] It should be noted that the results are shown in Tables 1 and 2. On the ScanObjectNN dataset, this scheme experimentally demonstrates the effects of different point embedding methods, including Point-PN, PointNet, PointMLP, and Transformer, which are considered as baseline models. The APPT method designed by this scheme consistently outperforms the 3D pre-trained models by a large margin. For example, on the most challenging PB-T50-RS data classification, the pre-trained model Any2Point improves performance by 4.8% over Point-PN. In contrast, APPT shows a significant performance improvement of 5.7% after embedding with Point-PN, surpassing Point-PN. Compared with Any2Point, this scheme improves performance by 0.7%.

[0129] At the same time, this scheme also performs well on 2D pre-trained models and performs well compared with 3D pre-trained models. Specifically, in the OBJ-BG task, this scheme improves the performance by 1.5% over Joint-MAE and 2.4% over Point-MAE. In the OBJ-ONLY task, this scheme also surpasses Joint-MAE, achieving a 1.6% performance improvement and a 2.2% performance improvement over Point-MAE. In addition, in the specific scenario of PB-T50-RS, this scheme shows the most significant improvement. Compared with the best performing 3D pre-trained model, this scheme improves the performance by 6.5%.

[0130] On the ModelNet40 dataset, the accuracy of Point-PN and Transformer as baseline models are 93.8% and 91.4% respectively, while the fine-tuned model of this solution improves the accuracy by 0.4% over the baseline model. At the same time, compared with other 3D pre-trained models and 2D pre-trained models, this solution shows a very obvious performance lead. Compared with the 3D pre-trained model Joint-MAE, this solution improves the accuracy by 0.4%, and compared with the 2D pre-trained model Any2Point, it improves the accuracy by 0.1%. In addition, in the language mode, this solution performs better than the visual mode, reaching an accuracy of 95.1%, which is 0.8% higher than the language mode of Any2Point.

[0131] In some embodiments, a series of control experiments are set up to clearly demonstrate the specific impact of each component of APPT on performance improvement. The specific experimental results are shown in Tables 3 and 4 below. Among them, each module of APPT contributes to the performance improvement of the benchmark method (i.e., Point-PN), especially the combination of PT and PosIn brings the most significant performance improvement.

[0132] Table 3 Ablation experiments on the ScanObjectNN dataset

[0133]

[0134] Table 4 Ablation experiments on the ModelNet40 dataset

[0135]

[0136] In some embodiments, in order to further illustrate the effectiveness of this solution in different modes, this solution was additionally experimented in the PB-T50-RS part of ScanObjectNN, and the results are shown in Table 5 below. This solution used DivoV2 (2D), DeiT (2D) and RoBERTa (1D) as pre-trained models, and fine-tuned this solution, and also achieved good results. In particular, in the 2D mode, with Divo V2 as the pre-trained model, it achieved the same accuracy as ViT-B.

[0137] Table 5 Results on the ScanObjectNN dataset using different pre-training parameters

[0138]

[0139] In order to understand the solution of this application more clearly, Figure 6 and Figure 7 The details are as follows:

[0140] first, Figure 6 The core logic flow chart of the APPT framework proposed in this scheme is shown in the figure. Figure 6 The left side shows that the input point cloud is obtained from the object image through the FPS+k-NN algorithm and input into the APPT framework. In the APPT framework, the input point cloud is first grouped and then input into the token embedding module and the prompt word generation module respectively. In the token embedding module, a lightweight network (Point_Embed) is used to obtain the point cloud embedding data of the input point cloud, and then through the Position Injector, this module injects sufficient position information from the source modality into the 3D token, so that the generated candidate point token can achieve more effective collaboration with the solidified two-dimensional pre-trained model Transformer.

[0141] Among them, the token embedding module and the prompt word generation module share parameters during the training process, and the position information can refer to the relative position of the linear combination between the point cloud embedding data and the original point token.

[0142] Secondly, Figure 6The prompt word generation module in , through the pooling process of the point cloud embedding data, obtains the average pooling data and the maximum pooling data, and then generates prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data, and obtains the following Figure 7 The target point token shown on the left is composed of a general point token, a target point token, and a target prompt word. The two-dimensional pre-trained model is fine-tuned by the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0143] In this embodiment, the following technical effects are achieved by adopting the APPT architecture technology:

[0144] 1) We study the potential of 2D pre-trained models for 3D point cloud analysis and find that directly leveraging 2D prior knowledge for minimal fine-tuning can outperform 3D models trained from scratch.

[0145] 2) The proposed APPT framework is used to fine-tune the 2D pre-trained model for direct application in 3D point cloud analysis. The framework includes a point embedding module and a point sequencer for feature alignment. This scheme provides more structured features to the model through prompt adjustment.

[0146] 3) This scheme combines the permutation invariance theory to generate prompt words through the movement of points without adding additional parameters. At the same time, the permutation invariance model is used to obtain stronger robustness and generalization ability.

[0147] 4) Significantly improved the accuracy and efficiency of 3D point cloud data analysis by fine-tuning the pre-trained 2D visual transformer model so that it can be directly applied to 3D point cloud data.

[0148] 5) It solves the problem of scarce 3D data training samples, effectively utilizes the rich 2D image data resources, and reduces the dependence on large amounts of 3D training data.

[0149] 6) It provides a new solution for cross-dimensional visual tasks, broadens the application possibility of 2D pre-trained models in the 3D field, and promotes the development of cross-dimensional data processing technology.

[0150] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a 3D point cloud analysis device for implementing the 3D point cloud analysis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more 3D point cloud analysis device embodiments provided below can refer to the limitations of the 3D point cloud analysis method above, and will not be repeated here.

[0152] In an exemplary embodiment, Figure 8 As shown, a three-dimensional point cloud analysis device 800 is provided, including: an acquisition module 801, a pooling processing module 802, a token construction module 803, a prompt word generation module 804 and a model fine-tuning module 805, wherein:

[0153] An acquisition module 801 is used to acquire point cloud embedding data of an input point cloud;

[0154] A pooling processing module 802 is used to perform pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0155] A token construction module 803 is used to construct point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens;

[0156] A prompt word generation module 804 is used to generate prompt words for candidate point tokens according to average pooling data and maximum pooling data to obtain target point tokens;

[0157] The model fine-tuning module 805 is used to fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task.

[0158] In some embodiments, the token construction module 803 is also used to determine the original point token based on the average pooling data; determine the linear combination relative position between the point cloud embedded data and the original point token according to the position injector; and inject the linear combination relative position into the original point token through a preset learning model to obtain a candidate point token.

[0159] In some embodiments, the token construction module 803 is also used to determine the original point token based on the average pooling data; determine the relative position of the linear combination between the point cloud embedded data and the original point token based on the position injector; and inject the relative position of the linear combination into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0160] In some embodiments, the prompt word generation module 804 is also used to generate prompt words for average pooling data and maximum pooling data through a trainable prompt generator to obtain target prompt words; parameters are shared between the prompt generator and the position injector; and a target point token is constructed based on the target prompt word and the candidate point token.

[0161] In some embodiments, the prompt word generation module 804 is also used to extract features from the average pooled data and the maximum pooled data through a trainable prompt generator to obtain overall feature data; and generate prompt words based on the overall feature data to obtain target prompt words with the same dimension as the candidate point token.

[0162] In some embodiments, the device also includes: a visualization display module for acquiring a data set to be predicted; performing a three-dimensional point cloud analysis on the training set to be predicted according to a target point cloud analysis model to obtain a three-dimensional point cloud analysis result; and visually displaying the three-dimensional point cloud analysis result.

[0163] In the above-mentioned three-dimensional point cloud analysis device, the point cloud embedding data of the input point cloud is obtained; the point cloud embedding data is pooled to obtain average pooling data and maximum pooling data; point tokens are constructed according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens; the two-dimensional pre-trained model is fine-tuned by the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of the candidate point tokens, and prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of the point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point token generated by the embedding layer is aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decrease in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks.

[0164] Each module in the above three-dimensional point cloud analysis device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0165] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a two-dimensional pre-trained model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional point cloud analysis method is implemented.

[0166] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned three-dimensional point cloud analysis method when executing the computer program.

[0168] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional point cloud analysis method are implemented.

[0169] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned three-dimensional point cloud analysis method when executed by a processor.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0171] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional point cloud analysis method, characterized in that: The method comprises: Get the point cloud embedding data of the input point cloud; Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; constructing point tokens according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token; The two-dimensional pre-trained model is fine-tuned through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

2. The method according to claim 1, characterized in that The step of constructing a point token according to the position injector, the point cloud embedding data and the average pooling data to obtain a candidate point token includes: Determining an origin point token based on the average pooled data; determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector; The relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token.

3. The method according to claim 1, characterized in that The step of constructing a point token according to the position injector, the point cloud embedding data and the average pooling data to obtain a candidate point token includes: Determining an origin point token based on the average pooled data; determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector; The relative position of the linear combination is injected into the original point token through a preset convolution kernel model to obtain a candidate point token.

4. The method according to claim 1, characterized in that: The generating of prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point tokens includes: Prompt words are generated for the average pooled data and the maximum pooled data by a trainable prompt generator to obtain a target prompt word; parameters are shared between the prompt generator and the position injector; A target point token is constructed according to the target prompt word and the candidate point token.

5. The method according to claim 4, characterized in that The generating of prompt words for the average pooled data and the maximum pooled data by a trainable prompt generator to obtain a target prompt word includes: Performing feature extraction on the average pooled data and the maximum pooled data through a trainable prompt generator to obtain overall feature data; Prompt words are generated according to the overall feature data to obtain the target prompt words with the same dimension as the candidate point token.

6. The method according to claim 1, characterized in that The method further comprises: Get the data set to be predicted; Performing a three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result; The three-dimensional point cloud analysis results are visualized.

7. A three-dimensional point cloud analysis device, characterized in that: The device comprises: An acquisition module, used to acquire point cloud embedding data of an input point cloud; A pooling processing module, used for performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; A token construction module, used for constructing point tokens according to a position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; A prompt word generation module, used to generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token; The model fine-tuning module is used to fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images