3D Point Cloud Analysis Method, Apparatus, Device, Medium and Program Product

Abstract

This application relates to a three-dimensional point cloud analysis method, apparatus, device, medium, and program product. The method comprises: obtaining point cloud embedding data of an input point cloud; performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; constructing point tokens based on a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens; generating prompt words for the candidate point tokens based on the average pooling data and the maximum pooling data to obtain target point tokens; fine-tuning a two-dimensional pre-trained model using the target point tokens to obtain a target point cloud analysis model; and using the target point cloud analysis model to process three-dimensional point cloud analysis tasks. This method can improve the accuracy of three-dimensional point cloud analysis while reducing the number of parameters.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to three-dimensional point cloud analysis methods, devices, equipment, media and program products. Background Art

[0002] With the development of computer vision, parameter efficient fine-tuning (PEFT) technology has become increasingly mature. This technology effectively handles various downstream tasks by leveraging the rich semantic features and representation capabilities of large-scale base models, while significantly reducing learning and storage costs. However, as an important research direction, 3D visual understanding faces greater challenges in data acquisition, which has led to a lack of large-scale pre-trained base models for 3D tasks. However, in existing technologies, acquiring and annotating high-quality 3D data requires a lot of resources and manpower, and synthetic data usually has problems with insufficient distribution diversity and poor real-world applicability. Therefore, there is an urgent need for a point cloud analysis method that can improve the accuracy of point cloud analysis while reducing the number of parameters. Summary of the Invention

[0003] Based on this, it is necessary to provide a three-dimensional point cloud analysis method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of point cloud analysis while reducing the number of parameters to address the above technical problems.

[0004] In a first aspect, the present application provides a three-dimensional point cloud analysis method, comprising:

[0005] Get the point cloud embedding data of the input point cloud;

[0006] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0007] constructing point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens;

[0008] Generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0009] The two-dimensional pre-trained model is fine-tuned using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0010] In one embodiment, the step of constructing a point token based on the position injector, the point cloud embedding data, and the average pooling data to obtain a candidate point token includes:

[0011] determining an origin point token based on the average pooled data;

[0012] determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector;

[0013] The relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token.

[0014] In one embodiment, the step of constructing a point token based on the position injector, the point cloud embedding data, and the average pooling data to obtain a candidate point token includes:

[0015] determining an origin point token based on the average pooled data;

[0016] determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector;

[0017] The relative position of the linear combination is injected into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0018] In one embodiment, generating a prompt word for the candidate point token based on the average pooling data and the maximum pooling data to obtain a target point token includes:

[0019] Generate a prompt word for the average pooled data and the maximum pooled data using a trainable prompt generator to obtain a target prompt word; parameters are shared between the prompt generator and the position injector;

[0020] A target point token is constructed according to the target prompt word and the candidate point token.

[0021] In one embodiment, generating prompt words from the average pooled data and the maximum pooled data using a trainable prompt generator to obtain a target prompt word includes:

[0022] Performing feature extraction on the average pooled data and the maximum pooled data by a trainable prompt generator to obtain overall feature data;

[0023] Prompt words are generated according to the overall feature data to obtain the target prompt words with the same dimension as the candidate point token.

[0024] In one embodiment, the method further comprises:

[0025] Get the dataset to be predicted;

[0026] Performing a three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result;

[0027] The three-dimensional point cloud analysis results are visualized.

[0028] In a second aspect, the present application further provides a three-dimensional point cloud analysis device, comprising:

[0029] The acquisition module is used to obtain the point cloud embedding data of the input point cloud;

[0030] A pooling processing module, used to perform pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0031] a token construction module, configured to construct point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens;

[0032] a prompt word generation module, configured to generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0033] The model fine-tuning module is used to fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task.

[0034] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Get the point cloud embedding data of the input point cloud;

[0036] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0037] constructing point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens;

[0038] Generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0039] The two-dimensional pre-trained model is fine-tuned using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0041] Get the point cloud embedding data of the input point cloud;

[0042] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0043] constructing point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens;

[0044] Generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0045] The two-dimensional pre-trained model is fine-tuned using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0046] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0047] Get the point cloud embedding data of the input point cloud;

[0048] Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0049] constructing point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens;

[0050] Generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token;

[0051] The two-dimensional pre-trained model is fine-tuned using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks.

[0052] The above-mentioned three-dimensional point cloud analysis method, device, computer equipment, computer-readable storage medium and computer program product obtain point cloud embedding data of the input point cloud; pool the point cloud embedding data to obtain average pooling data and maximum pooling data; construct point tokens based on the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens; generate prompt words for the candidate point tokens based on the average pooling data and maximum pooling data to obtain target point tokens; fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through position injectors, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of candidate point tokens. In addition, prompt words are generated for candidate point tokens based on average pooling data and maximum pooling data to obtain target point tokens. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point tokens generated by the embedding layer are aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decline in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A diagram of an application environment of a three-dimensional point cloud analysis method in one embodiment;

[0055] Figure 2 1 is a flow chart of a three-dimensional point cloud analysis method according to an embodiment;

[0056] Figure 3 1 is a flow chart of a three-dimensional point cloud analysis method according to an embodiment;

[0057] Figure 4 A color visualization image is provided for showing the effect of a three-dimensional point cloud analysis method according to another embodiment;

[0058] Figure 5 A color visualization image is provided for showing the effect of a three-dimensional point cloud analysis method according to another embodiment;

[0059] Figure 6 is a schematic flow chart of a three-dimensional point cloud analysis method in another embodiment;

[0060] Figure 7 is a schematic flow chart of a three-dimensional point cloud analysis method in another embodiment;

[0061] Figure 8 is a structural block diagram of a three-dimensional point cloud analysis device in one embodiment;

[0062] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] The three-dimensional point cloud analysis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated on server 104, or placed on the cloud or other network servers. Server 104 obtains point cloud embedding data of the input point cloud; performs pooling on the point cloud embedding data to obtain average pooling data and maximum pooling data; constructs point tokens based on the position injector, point cloud embedding data, and average pooling data to obtain candidate point tokens; generates prompt words for the candidate point tokens based on the average pooling data and maximum pooling data to obtain target point tokens; fine-tunes the two-dimensional pre-trained model using the target point tokens to obtain a target point cloud analysis model; and the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In an exemplary embodiment, Figure 2As shown in the figure, a three-dimensional point cloud analysis method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps S202 to S210.

[0066] Step S202: Obtain point cloud embedding data of the input point cloud.

[0067] Among them, the input point cloud can be the three-dimensional point cloud data of the data set to be trained obtained by the K-nearest neighbor algorithm and the farthest point sampling method according to the data set to be trained, and the point cloud embedding data is obtained by converting the grouped input point cloud into structured and representative embedding data.

[0068] In some embodiments, the training set data to be predicted can be obtained first, and the K-nearest neighbor algorithm and farthest point sampling processing can be performed on the training set data to be predicted to obtain the input point cloud, and then the point cloud data after the input point cloud is grouped is converted into multiple groups of structured and representative point cloud embedding data.

[0069] Among them, the training set data to be predicted can use the ScanObjectNN dataset and the ModelNet40 dataset, the K nearest neighbor algorithm can use the KNN algorithm, the farthest point sampling processing can use the FPS algorithm, and the input point cloud can be grouped by random grouping or other grouping methods, which are not limited to these.

[0070] In some embodiments, specifically, a lightweight network (Point_Embed) is implemented to obtain point cloud embedding data of the input point cloud, which is in the form of the following formula (1):

[0071] (1)

[0072] Among them, Point_Embed can take many forms, such as PointNet, PointMLP, Point-PN, etc., input point cloud The point set after grouping from the input point cloud In order to simplify the subsequent embodiments, Indicates a point of Neighborhood points A collection of .

[0073] It should be noted that the input point cloud is obtained by performing K-nearest neighbor algorithm and farthest point sampling processing on the training set data to be predicted, and then the point cloud data after grouping the input point cloud is converted into multiple groups of structured and representative point cloud embedding data. This not only enhances the utilization efficiency of the point cloud, but also aligns the point cloud embedding data with the input dimension of the two-dimensional pre-training model that needs to be fine-tuned later, thereby promoting the utilization of prior knowledge by the two-dimensional pre-training model.

[0074] Step S204: performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data.

[0075] Among them, pooling processing includes average pooling processing and maximum pooling processing.

[0076] In some embodiments, the point cloud embedding data can be subjected to average pooling processing and maximum pooling processing through a pooling operation module, and the overall features of the point cloud embedding data can be captured through pooling processing, thereby obtaining average pooling data and maximum pooling data.

[0077] In some embodiments, the average pooling , thereby obtaining the global token embedding of the point cloud embedding data of the input point cloud and embed the global token into As the average pooled data, and the average pooled data can be used to represent the centroid of the input point cloud: .

[0078] In some embodiments, a maximum pooling process can be performed to obtain the maximum pooling data of the point cloud embedding data of the input point cloud, and the average pooling data can be expressed as .

[0079] Step S206 , constructing point tokens based on the position injector, point cloud embedding data, and average pooling data to obtain candidate point tokens.

[0080] Among them, the position injector is a Position Injector designed based on the position encoding in the two-dimensional pre-trained model Transformer. The position injector can inject sufficient position information from the source modality into the 3D token, thereby achieving more effective collaboration with the frozen two-dimensional pre-trained model Transformer.

[0081] In some embodiments, point tokens are constructed based on a position injector, point cloud embedding data, and average pooling data to obtain candidate point tokens, including: determining the original point token based on the average pooling data; determining the linear combination relative position between the point cloud embedding data and the original point token based on the position injector; and injecting the linear combination relative position into the original point token through a preset learning model to obtain a candidate point token.

[0082] Among them, the original point token is the global token embedding data for each input point cloud in the initial state, the linear combination relative position represents the relative position difference between the current point cloud and the center of mass of the entire input point cloud, and the preset learning model refers to a pre-trained learning model containing learnable parameters, and is used to inject the linear combination relative position into the original point token.

[0083] In some embodiments, the average pooled data is used as the global token embedding data of the input point cloud and can be directly used to determine the original point token , and then determine the point cloud embedding data through the position injector and origin token The relative position of the linear combination between .

[0084] Among them, the relative position of the linear combination can be expressed as .

[0085] In some embodiments, the relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token. The specific position injection process can be shown in the following formula (2):

[0086] (2)

[0087] in, is the candidate point token, and is a learnable parameter.

[0088] In some embodiments, point tokens are constructed based on a position injector, point cloud embedding data, and average pooling data to obtain candidate point tokens, including: determining the original point token based on the average pooling data; determining the relative position of the linear combination between the point cloud embedding data and the original point token based on the position injector; and injecting the relative position of the linear combination into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0089] The preset convolution kernel model may refer to a 1D convolution kernel.

[0090] In some embodiments, after determining the linear combination relative position between the point cloud embedding data and the original point token according to the position injector, the 1D convolution kernel injects the linear combination relative position into the original point token, so that this module can be seamlessly integrated into the existing model as an independent layer. Specifically, the calculation process of the candidate point token can be shown as formula (3):

[0091] (3)

[0092] in, Refers to the preset convolution kernel model, refers to the relative position of the linear combination between the point cloud embedding data and the original point token, Represents the centroid of the point cloud embedding data.

[0093] In this embodiment, the proposed Position Injector injects sufficient position information from the source modality into the 3D token, thereby achieving more effective collaboration with the frozen two-dimensional pre-trained model Transformer, and then making full use of the attention prior ability of the two-dimensional pre-trained model Transformer, thereby improving the analysis effect of three-dimensional point cloud data after subsequent model training is completed.

[0094] Step S208 : Generate prompt words for candidate point tokens based on the average pooling data and the maximum pooling data to obtain target point tokens.

[0095] Among them, the process of prompt word generation refers to attaching the generated prompt word to the candidate point token. The target point token contains the target prompt word, which is used to implement the fine-tuning self-attention mechanism for different tasks of the two-dimensional pre-trained model.

[0096] In some embodiments, prompt words are generated for candidate point tokens based on average pooling data and maximum pooling data to obtain target point tokens, including: generating prompt words for average pooling data and maximum pooling data through a trainable prompt generator to obtain target prompt words; sharing parameters between the prompt generator and the position injector; and constructing the target point token based on the target prompt words and the candidate point tokens.

[0097] In some embodiments, prompt words are generated for average pooled data and maximum pooled data by a trainable prompt generator to obtain target prompt words, including: extracting features from average pooled data and maximum pooled data by a trainable prompt generator to obtain overall feature data; generating prompt words based on the overall feature data to obtain target prompt words with the same dimension as the candidate point token.

[0098] Among them, a trainable prompt generator is used to generate prompt words, and the prompt words generated by the prompt generator have the following properties: 1) are closely related to the input point cloud; 2) are used to capture the overall information of the input point cloud; 3) have the same dimension as the token embedding.

[0099] In some embodiments, a trainable prompt generator is used to generate prompt words for the average pooled data and the maximum pooled data to obtain a target prompt word. Since the prompt generator adopts the same structure as the token embedding and introduces the average pooled data and the maximum pooled data obtained by the pooling operation to capture the overall features of the input point cloud, the target prompt word is obtained, and then the target prompt word is attached to the candidate point token to obtain the target point token.

[0100] It should be noted that sharing parameters between the hint generator and the position injector realizes parameter sharing of the token embedding module and the hint generator module, thereby maintaining consistency between the point tokens that capture local features and the point hints that encode global features, while reducing training parameters.

[0101] In some embodiments, a trainable prompt generator is used to generate prompt words for the average pooled data and the maximum pooled data, and the generation process of the target prompt word is shown in the following formula (4):

[0102] (4)

[0103] in, Represents the maximum pooling data and represents the average pooled data, Indicates the target cue word.

[0104] In this embodiment, the target prompt word constructed by average pooling data and maximum pooling data is permutation invariant to the group arrangement of the input point cloud, thereby ensuring that the model is less sensitive to the order of point groups, so that the prompt generator provides more stable global features and eliminates redundant information. At the same time, the target prompt word retains the geometric information of the input point cloud.

[0105] Step S210, fine-tuning the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task.

[0106] Among them, the fine-tuning process can use parameter-efficient fine-tuning (PEFT) technology to alleviate overfitting and improve the generalization ability of the 3D model. The 3D point cloud analysis task can be a downstream task of 3D point cloud analysis, such as ablation experiment tasks and object classification. The 2D pre-trained model is a backbone network frozen on a large-scale dataset.

[0107] In some embodiments, by inputting the target point token into the two-dimensional pre-training model and passing it through multiple stages of Transfarmer blocks in the two-dimensional pre-training model, while continuously copying the target prompt words in the target point token, the two-dimensional pre-training model is fine-tuned to obtain a target point cloud analysis model. The target point cloud analysis model can be used to process three-dimensional point cloud analysis tasks to improve the execution efficiency of three-dimensional point cloud analysis tasks. This fine-tuning method not only reduces the learning parameters required in model training, but also ensures the execution efficiency of the target point cloud analysis model in three-dimensional point cloud analysis tasks, thereby improving the execution effect of each downstream task when applied to related three-dimensional point cloud analysis downstream tasks.

[0108] In the above-mentioned three-dimensional point cloud analysis method, the point cloud embedding data of the input point cloud is obtained; the point cloud embedding data is pooled to obtain average pooling data and maximum pooling data; point tokens are constructed based on the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens; prompt words are generated for the candidate point tokens based on the average pooling data and maximum pooling data to obtain target point tokens; the two-dimensional pre-trained model is fine-tuned through the target point tokens to obtain the target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through position injectors, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of candidate point tokens. In addition, prompt words are generated for candidate point tokens based on average pooling data and maximum pooling data to obtain target point tokens. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point tokens generated by the embedding layer are aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decline in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks.

[0109] In an exemplary embodiment, Figure 3 As shown, the three-dimensional point cloud analysis method further includes steps S302 to S306. In which:

[0110] Step S302: Obtain the data set to be predicted.

[0111] The dataset to be predicted may be a ScanObjectNN dataset or a ModelNet40 dataset, but is not limited thereto.

[0112] In some embodiments, the dataset to be predicted may be obtained from a point cloud database to be predicted, or may be obtained in other ways, without limitation thereto.

[0113] Step S304 , performing a three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result.

[0114] Among them, the three-dimensional point cloud analysis is to analyze and calibrate the three-dimensional point cloud in the training set to be predicted, so as to determine the coordinate data of each three-dimensional point cloud.

[0115] In some embodiments, a three-dimensional point cloud analysis is performed on each object in the training set to be predicted according to the target point cloud analysis model to obtain coordinate data of each three-dimensional point cloud in each object as a three-dimensional point cloud analysis result.

[0116] In some embodiments, corresponding three-dimensional point cloud analysis results can be generated by combining specific downstream tasks of three-dimensional point cloud analysis. For example, for object classification tasks, object classification accuracy results can be obtained.

[0117] Step S306: Visually display the three-dimensional point cloud analysis results.

[0118] Among them, visual display includes but is not limited to tabular data display and dynamic color map display.

[0119] In some embodiments, the three-dimensional point cloud analysis results can be combined with specific downstream tasks of the three-dimensional point cloud analysis to generate specific downstream task result data, and the downstream task result data can be visualized.

[0120] In some embodiments, in order to further illustrate the contribution of each module, this solution visualizes the feature distribution and corresponding response on the original input point cloud in the 3D point cloud analysis results, such as Figure 4 and Figure 5 As shown, Figure 4 The following is a color map showing the feature visualization results corresponding to the 3D point cloud analysis results on the ScanObjectNN dataset to be predicted. Figure 5 The following color map shows the visual effects of different modules. Specifically, when only Point-PN is used, there is overlap in the feature distribution between different categories, such as Figure 4 As shown in (a) in . Figure 4 (b) shows the feature distribution after the PosIn module is aligned with ViT-B (built based on Point-PN). Figure 4 (c) in Figure 3 shows the effect of using only PT, using the same point embedding module. Figure 5From the above, we can clearly see that PT and PosIn focus on different parts of the object; however, both highlight the object structure, thereby enhancing the separability of features. Figure 4 (d) in the figure shows the effect of adopting the APPT (APPTuning) framework in this solution. Figure 5 The third row shows that APPT captures a relatively complete overall structure of the object. This helps explain why APPT achieves significant improvements in classification but slightly lags behind Joint-MAE in segmentation, as the APPT encoder places greater emphasis on the overall structure of the input. Intuitively, APPT achieves more distinct feature distribution boundaries and significantly improves feature separability.

[0121] In some embodiments, as shown in Tables 1, 2, 3, 4, and 5 below, Table 1 shows the accuracy results (APPT(ours)) of classifying objects based on three-dimensional point cloud analysis results on the ScanObjectNN dataset, and also adds the accuracy results of existing three-dimensional point cloud analysis technologies (DNN-based model and Transformer-basedmodel) to achieve a more intuitive effect display.

[0122] Table 1 Object classification accuracy (%) results on the ScanObjectNN dataset

[0123]

[0124] Among them, Table 2 shows the accuracy results of object classification based on 3D point cloud analysis results on the ModelNet40 dataset, and also adds the accuracy results of existing 3D point cloud analysis technologies to achieve a more intuitive effect display.

[0125] Table 2 Object classification accuracy results on the ModelNet40 dataset

[0126]

[0127]

[0128] It should be noted that the results are shown in Tables 1 and 2. On the ScanObjectNN dataset, this scheme experimentally demonstrated the effectiveness of different point embedding methods, including Point-PN, PointNet, PointMLP, and Transformer, which are considered as baseline models. The APPT method designed in this scheme consistently significantly outperforms the 3D pre-trained models in terms of performance. For example, in the most challenging PB-T50-RS data classification, the pre-trained model Any2Point improved the performance by 4.8% compared to Point-PN. In contrast, after using Point-PN embedding, APPT showed a significant performance improvement of 5.7%, surpassing Point-PN. Compared with Any2Point, this scheme improved the performance by 0.7%.

[0129] At the same time, this solution also performs well on 2D pre-trained models and performs well compared to 3D pre-trained models. Specifically, in the OBJ-BG task, this solution improves the performance of Joint-MAE by 1.5% and improves the performance by 2.4% compared to Point-MAE. In the OBJ-ONLY task, this solution also surpasses Joint-MAE, achieving a 1.6% performance improvement and a 2.2% performance improvement over Point-MAE. In addition, in the specific scenario of PB-T50-RS, this solution shows the most significant improvement. Compared with the best-performing 3D pre-trained model, this solution improves the performance by 6.5%.

[0130] On the ModelNet40 dataset, the accuracy of Point-PN and Transformer as baseline models was 93.8% and 91.4%, respectively. The fine-tuned model of this solution improved the accuracy by 0.4% over the baseline model. At the same time, compared with other 3D pre-trained models and 2D pre-trained models, this solution showed a very clear performance lead. Compared with the 3D pre-trained model Joint-MAE, this solution improved the accuracy by 0.4%, and compared with the 2D pre-trained model Any2Point, it improved the accuracy by 0.1%. In addition, in the language mode, this solution performed better than the visual mode, achieving an accuracy of 95.1%, which was 0.8% higher than the language mode of Any2Point.

[0131] In some embodiments, a series of control experiments are set up to clearly demonstrate the specific impact of each component of APPT on performance improvement. The specific experimental results are shown in Tables 3 and 4 below. Among them, each module of APPT contributes to the performance improvement of the baseline method (i.e., Point-PN), especially the combination of PT and PosIn brings the most significant performance improvement.

[0132] Table 3 Ablation experiments on the ScanObjectNN dataset

[0133]

[0134] Table 4 Ablation experiments on the ModelNet40 dataset

[0135]

[0136] To further illustrate the effectiveness of this solution in different modes, additional experiments were conducted on the PB-T50-RS portion of ScanObjectNN. The results are shown in Table 5 below. This solution also achieved good results by fine-tuning DivoV2 (2D), DeiT (2D), and RoBERTa (1D) as pre-trained models. In particular, in 2D mode, using Divo V2 as the pre-trained model, it achieved the same accuracy as ViT-B.

[0137] Table 5 Results on the ScanObjectNN dataset using different pre-training parameters

[0138]

[0139] In order to understand the solution of this application more clearly, Figure 6 and Figure 7 The details are as follows:

[0140] first, Figure 6 The core logic flow chart of the APPT framework proposed in this solution is shown in the figure. Figure 6 The left side shows that the input point cloud is obtained from the object image through the FPS+k-NN algorithm and input into the APPT framework. In the APPT framework, the input point cloud is first grouped and then input into the token embedding module and the prompt word generation module respectively. In the token embedding module, a lightweight network (Point_Embed) is used to obtain the point cloud embedding data of the input point cloud. Then, through the Position Injector, this module injects sufficient position information from the source modality into the 3D token, so that the generated candidate point token can achieve more effective collaboration with the solidified two-dimensional pre-trained model Transformer.

[0141] Among them, the token embedding module and the prompt word generation module share parameters during the training process, and the position information can refer to the relative position of the linear combination between the point cloud embedding data and the original point token.

[0142] Secondly, Figure 6The prompt word generation module in the point cloud embedding data is pooled to obtain average pooled data and maximum pooled data, and then the prompt word is generated for the candidate point token based on the average pooled data and the maximum pooled data. Figure 7 The target point token shown on the left is composed of a general point token, a target point token, and a target prompt word. The target point token is used to fine-tune the 2D pre-trained model to obtain the target point cloud analysis model; the target point cloud analysis model is used to process 3D point cloud analysis tasks.

[0143] In this embodiment, the following technical effects are achieved by adopting the APPT architecture technology:

[0144] 1) By studying the potential of 2D pre-trained models for 3D point cloud analysis, we find that directly leveraging 2D prior knowledge for minimal fine-tuning can outperform 3D models trained from scratch.

[0145] 2) This proposal proposes the APPT framework, which fine-tunes 2D pre-trained models for direct application to 3D point cloud analysis. The framework includes a point embedding module and a point sequencer for feature alignment. This proposal provides the model with more structured features through prompt adjustment.

[0146] 3) This solution incorporates permutation invariance theory to generate prompt words based on point motion without adding additional parameters. Furthermore, the permutation invariance model achieves enhanced robustness and generalization.

[0147] 4) Significantly improves the accuracy and efficiency of 3D point cloud data analysis by fine-tuning the pre-trained 2D visual transformer model so that it can be directly applied to 3D point cloud data.

[0148] 5) It solves the problem of scarce 3D data training samples, effectively utilizes the rich 2D image data resources, and reduces the dependence on large amounts of 3D training data.

[0149] 6) It provides a new solution for cross-dimensional visual tasks, broadens the application possibilities of 2D pre-trained models in the 3D field, and promotes the development of cross-dimensional data processing technology.

[0150] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0151] Based on the same inventive concept, embodiments of the present application also provide a 3D point cloud analysis device for implementing the aforementioned 3D point cloud analysis method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more 3D point cloud analysis device embodiments provided below can be found in the above-described limitations of the 3D point cloud analysis method and will not be further elaborated here.

[0152] In an exemplary embodiment, Figure 8 As shown, a three-dimensional point cloud analysis device 800 is provided, comprising: an acquisition module 801, a pooling processing module 802, a token construction module 803, a prompt word generation module 804 and a model fine-tuning module 805, wherein:

[0153] An acquisition module 801 is used to acquire point cloud embedding data of an input point cloud;

[0154] The pooling processing module 802 is used to perform pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data;

[0155] A token construction module 803 is used to construct point tokens based on the position injector, point cloud embedding data and average pooling data to obtain candidate point tokens;

[0156] A prompt word generation module 804 is used to generate prompt words for candidate point tokens based on the average pooling data and the maximum pooling data to obtain a target point token;

[0157] The model fine-tuning module 805 is used to fine-tune the two-dimensional pre-trained model through the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task.

[0158] In some embodiments, the token construction module 803 is also used to determine the original point token based on the average pooling data; determine the linear combination relative position between the point cloud embedding data and the original point token based on the position injector; and inject the linear combination relative position into the original point token through a preset learning model to obtain a candidate point token.

[0159] In some embodiments, the token construction module 803 is also used to determine the original point token based on the average pooling data; determine the relative position of the linear combination between the point cloud embedding data and the original point token based on the position injector; and inject the relative position of the linear combination into the original point token through a preset convolution kernel model to obtain a candidate point token.

[0160] In some embodiments, the prompt word generation module 804 is also used to generate prompt words for average pooled data and maximum pooled data through a trainable prompt generator to obtain target prompt words; parameters are shared between the prompt generator and the position injector; and a target point token is constructed based on the target prompt word and the candidate point token.

[0161] In some embodiments, the prompt word generation module 804 is also used to perform feature extraction on the average pooled data and the maximum pooled data through a trainable prompt generator to obtain overall feature data; generate prompt words based on the overall feature data to obtain target prompt words with the same dimension as the candidate point token.

[0162] In some embodiments, the device also includes: a visualization display module for obtaining a data set to be predicted; performing three-dimensional point cloud analysis on the training set to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result; and visually displaying the three-dimensional point cloud analysis result.

[0163] In the above-mentioned three-dimensional point cloud analysis device, the point cloud embedding data of the input point cloud is obtained; the point cloud embedding data is pooled to obtain average pooling data and maximum pooling data; point tokens are constructed according to the position injector, the point cloud embedding data and the average pooling data to obtain candidate point tokens; prompt words are generated for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain target point tokens; the two-dimensional pre-trained model is fine-tuned by the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks. Therefore, point tokens are constructed through position injectors, point cloud embedding data and average pooling data to obtain candidate point tokens, which can further improve the feature expression ability of candidate point tokens. In addition, prompt words are generated for candidate point tokens based on average pooling data and maximum pooling data to obtain target point tokens. The permutation invariance of the two types of pooling data can be used to help the model capture the overall pattern of point cloud data more deeply, thereby enhancing the robustness and generalization ability of the target point cloud analysis model. Moreover, the target point tokens generated by the embedding layer are aligned with the two-dimensional pre-trained model, so the two-dimensional pre-trained model can be directly fine-tuned to obtain the target point cloud analysis model, thereby significantly reducing the scale of training parameters, and ultimately achieving a significant reduction in the time and data scale required for point cloud training, while avoiding the decline in task accuracy due to overfitting, bringing significant performance improvements to 3D point cloud downstream tasks.

[0164] Each module in the above-mentioned 3D point cloud analysis device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0165] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store a two-dimensional pre-trained model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional point cloud analysis method is implemented.

[0166] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0167] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned three-dimensional point cloud analysis method when executing the computer program.

[0168] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional point cloud analysis method are implemented.

[0169] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned three-dimensional point cloud analysis method when executed by a processor.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0171] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0172] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A three-dimensional point cloud analysis method, characterized in that: The method comprises: Get the point cloud embedding data of the input point cloud; Performing pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; constructing point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens; Generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token; Fine-tuning the two-dimensional pre-trained model using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process the three-dimensional point cloud analysis task; Generating prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain the target point token includes: Generate a prompt word for the average pooled data and the maximum pooled data using a trainable prompt generator to obtain a target prompt word; parameters are shared between the prompt generator and the position injector; A target point token is constructed according to the target prompt word and the candidate point token.

2. The method according to claim 1, characterized in that The step of constructing a point token based on the position injector, the point cloud embedding data, and the average pooling data to obtain a candidate point token includes: determining an origin point token based on the average pooled data; determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector; The relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token.

3. The method according to claim 1, characterized in that The step of constructing a point token based on the position injector, the point cloud embedding data, and the average pooling data to obtain a candidate point token includes: determining an origin point token based on the average pooled data; determining a linear combination relative position between the point cloud embedding data and the original point token according to the position injector; The relative position of the linear combination is injected into the original point token through a preset convolution kernel model to obtain a candidate point token.

4. The method according to claim 1, wherein Generating prompt words for the average pooled data and the maximum pooled data by a trainable prompt generator to obtain a target prompt word includes: Performing feature extraction on the average pooled data and the maximum pooled data by a trainable prompt generator to obtain overall feature data; Prompt words are generated according to the overall feature data to obtain the target prompt words with the same dimension as the candidate point token.

5. The method according to claim 1, wherein The method further comprises: Get the dataset to be predicted; Performing a three-dimensional point cloud analysis on the dataset to be predicted according to the target point cloud analysis model to obtain a three-dimensional point cloud analysis result; The three-dimensional point cloud analysis results are visualized.

6. A three-dimensional point cloud analysis device, characterized in that: The device comprises: The acquisition module is used to obtain the point cloud embedding data of the input point cloud; A pooling processing module, used to perform pooling processing on the point cloud embedding data to obtain average pooling data and maximum pooling data; a token construction module, configured to construct point tokens according to a position injector, the point cloud embedding data, and the average pooling data to obtain candidate point tokens; a prompt word generation module, configured to generate prompt words for the candidate point tokens according to the average pooling data and the maximum pooling data to obtain a target point token; A model fine-tuning module, configured to fine-tune the two-dimensional pre-trained model using the target point token to obtain a target point cloud analysis model; the target point cloud analysis model is used to process three-dimensional point cloud analysis tasks; The prompt word generation module is also used to generate prompt words for the average pooled data and the maximum pooled data through a trainable prompt generator to obtain a target prompt word; parameters are shared between the prompt generator and the position injector; and a target point token is constructed based on the target prompt word and the candidate point token.

7. The device according to claim 6, characterized in that The token construction module is further configured to determine an original point token based on the average pooling data; determine a relative position of a linear combination between the point cloud embedding data and the original point token based on the position injector; The relative position of the linear combination is injected into the original point token through a preset learning model to obtain a candidate point token.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images