3D pre-training method and system based on diffusion model

Abstract

The embodiments of the present invention provide a 3D pre-training method and system based on a diffusion model. The method comprises: encoding visible point cloud word segmentation features by means of an encoder to obtain deep visible point cloud word segmentation features, wherein the visible point cloud word segmentation features are generated by obtained point cloud blocks; concatenating the deep visible point cloud word segmentation features and mask point cloud word segmentation features to obtain concatenated features, wherein the mask point cloud word segmentation features are generated by the obtained point cloud blocks; encoding the concatenated features by means of a condition embedding module to obtain a guidance condition of the diffusion model; and on the basis of the guidance condition of the diffusion model and a multi-stage uniform sampling strategy, determining a loss function, wherein the loss function is used for guiding a decoder to perform decoding, so as to complete training. The solution of the present invention comprehensively understands the point cloud while a network architecture is not changed and the calculation amount is not increased, thereby improving the model training effect and performance, and further saving the pre-training cost.

Description

A 3D pre-training method and system based on diffusion model Technical Field Embodiments of the present invention relate to the field of computer vision technology, and in particular to a 3D pre-training method and system based on a diffusion model. Background Art With the rapid development of 3D computer vision, point cloud has become an important data form for describing the shape and spatial position of 3D objects. Point cloud data has a wide range of applications, including 3D scene reconstruction, autonomous driving, intelligent robots and other fields. Research on 3D point cloud pre-training technology can provide important support for these tasks, thereby greatly reducing the time and resource costs of artificial intelligence model development and deployment. The current mainstream 3D pre-training methods can be divided into two categories: contrastive learning and mask modeling. Contrastive learning performs pre-training by comparing the similarities and differences between samples. For example, in the field of 3D point clouds, two point clouds are constructed from different perspectives, and the similarity of point features between the two point clouds is compared to achieve pre-training of the point cloud network; or only using a single-view depth map for feature contrast learning can also learn effective feature representations; or according to the cross-modal contrastive learning method, point clouds and rendered 2D images are contrasted to learn, which helps to learn generalizable point cloud representations. However, 3D pre-training methods based on contrastive learning often require the design of negative samples and large batch sizes. If the negative samples are not selected properly, the model may learn incorrect feature representations, resulting in performance degradation; large batch sizes may make the training process more unstable, the training effect is poor, and the model performance is affected, leading to the risk of model collapse. Mask modeling does not require the design of negative samples and large batch sizes, but current mask modeling mostly learns multi-level geometric knowledge by modifying the hierarchical network architecture, resulting in an increase in network computation. Summary of the invention In view of this, embodiments of the present invention provide a 3D pre-training method and system based on a diffusion model, an electronic device, and a computer storage medium to at least partially solve the above problems. According to a first aspect of an embodiment of the present invention, a 3D pre-training method based on a diffusion model is provided, comprising encoding a visible point cloud segmentation feature through an encoder to obtain a deep visible point cloud segmentation feature. word features, wherein the visible point cloud word segmentation features are generated by the acquired point cloud blocks; the deep visible point cloud word segmentation features are spliced ​​with the masked point cloud word segmentation features to obtain spliced ​​features, wherein the masked point cloud word segmentation features are generated by the acquired point cloud blocks; the spliced ​​features are encoded through a conditional embedding module to obtain the guiding conditions of the diffusion model; the loss function is determined according to the guiding conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to decode to complete the training. In one implementation, before encoding the visible point cloud segmentation features through an encoder to obtain the deep visible point cloud segmentation features, the process includes obtaining point cloud blocks through the farthest point sampling algorithm and the K nearest neighbor algorithm; centralizing the point cloud blocks to update the point cloud blocks; embedding the updated point cloud blocks into the segmentation features to obtain the point cloud segmentation features; and randomly masking the point cloud segmentation features to generate visible point cloud segmentation features and masked point cloud segmentation features, wherein the visible point cloud segmentation features are represented as The masked point cloud segmentation feature is expressed as To mask the point cloud segmentation features The number of g = sr is the visible point cloud word segmentation feature The number of random masking. In another implementation, a point cloud block is obtained by using a farthest point sampling algorithm and a K nearest neighbor algorithm, including sampling the preset point cloud data by using the farthest point sampling algorithm to obtain s center points, wherein the preset point cloud data includes n points, and the preset point cloud data is represented by X∈R n×3 , s center points are represented as Based on each center point C of s center points i , k points closest to each center point are selected as point cloud blocks through the K nearest neighbor algorithm, where each center point is represented by C i , the point cloud block is represented as Point cloud block The calculation formula is: In another implementation, the updated point cloud block is embedded into the word segmentation feature to obtain the point cloud word segmentation feature, including a simplified version of PointNet ξ with parameter φ φ (·) Update the point cloud block Embedded into word segmentation features In, it is expressed as: In another implementation, the visible point cloud segmentation feature is encoded by an encoder to obtain a deep visible point cloud segmentation feature, including inputting the position information of the visible point cloud segmentation feature into a position embedding formula ψ with a parameter τ τ In (·), the position features of the visible point cloud segmentation features are output, and the visible point cloud segmentation features are The position feature of the feature is expressed as The position features of the output visible point cloud segmentation features With visible point cloud segmentation features Connect to get the combined features; through the encoder Φ ρ The Transformer block (·) extracts the combined features and obtains the deep visible point cloud segmentation features. The deep visible point cloud segmentation features are expressed as Among them, the encoder Φ ρ (·) is composed of a standard Transformer block with a parameter of ρ. The visible point cloud segmentation feature is encoded by the encoder, and the formula for obtaining the deep visible point cloud segmentation feature is: In another implementation, the splicing features are encoded by a conditional embedding module to obtain the guiding conditions of the diffusion model, including a conditional embedding module f with a parameter ω. ω (·) Encode the splicing features to obtain the guiding condition c of the diffusion model, where the formula of the guiding condition c of the diffusion model is: In another implementation, the loss function is determined according to the guidance condition of the diffusion model and the multi-stage uniform sampling strategy, including defining the training target of the pre-trained model according to the guidance condition c of the diffusion model, and the formula is: The time step interval [1, T] for each preset point cloud data is divided into h stages: in Randomly sample a time step t from h stages, calculate h losses and average the h losses to get the loss function: According to a second aspect of an embodiment of the present invention, a 3D pre-training system based on a diffusion model is provided, comprising a first encoding module, configured to encode a visible point cloud segmentation feature through an encoder to obtain a deep visible point cloud segmentation feature, wherein the visible point cloud segmentation feature is generated by an acquired point cloud block; a splicing module, configured to splice the deep visible point cloud segmentation feature with the occluded point cloud segmentation feature to obtain a spliced ​​feature, wherein the occluded point cloud segmentation feature is generated by an acquired point cloud block; and a second encoding module, It is used to encode the splicing features through the conditional embedding module to obtain the guiding conditions of the diffusion model; the determination module is used to determine the loss function according to the guiding conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to decode to complete the training. According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to the first aspect. According to a fourth aspect of an embodiment of the present invention, there is provided a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method of the first aspect is implemented. In summary, the solution of the embodiment of the present invention achieves a comprehensive understanding of the point cloud and obtains richer geometric information by improving the pre-training method and utilizing the multi-level guidance characteristics of the diffusion model without changing the network architecture or increasing the amount of calculation, thereby improving the effect and performance of model training and further saving the cost of point cloud pre-training. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings. FIG1 is a flowchart of the steps of a 3D pre-training method based on a diffusion model according to an embodiment of the present invention. FIG2 is an overall framework diagram of a 3D pre-training method based on a diffusion model according to an embodiment of the present invention. FIG. 3A is a visualization result diagram of a 3D pre-training method based on a diffusion model according to an embodiment of the present invention. FIG. 3B is a comparison diagram of semantic segmentation results according to another embodiment of the present invention. FIG. 4 is a structural block diagram of a 3D pre-training system based on a diffusion model corresponding to the embodiment of FIG. 1 . FIG. 5 is a schematic structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION In order to have a clearer understanding of the technical features, purposes and effects of the embodiments of the present application, the specific implementation methods of the embodiments of the present application are now described with reference to the accompanying drawings. In this document, “exemplary” means “serving as an example, instance or illustration”, and any diagram or implementation described in this document as “exemplary” should not be interpreted as a more preferred or more advantageous technical solution. In order to simplify the drawings, only the parts related to the present application are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one or more of the parts with the same structure or function are schematically drawn or only one or more of them are marked. In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention. For ease of understanding, before describing the specific embodiments of the present invention in detail, an exemplary description is first given of the prior art of the present invention. Many studies in the existing technology, including VisualChatGPT[1]Wu C, Yin S, Qi W, et al.Visual chatgpt: Talking, drawing and editing with visual foundation models[J].arXiv preprint arXiv:2303.04671, 2023. and SAM[2]Kirillov A, Mintun E, Ravi N, et al.Segment anything[J].arXiv preprint arXiv:2304.02643, 2023., have demonstrated the excellent performance of pre-trained models in a wide range of downstream tasks. With the rapid development of 3D computer vision, point cloud has become an important data form for describing the shape and spatial position of 3D objects. The application scope of point cloud data is very wide, including 3D scene reconstruction, autonomous driving, intelligent robots and other fields. The research on 3D point cloud pre-training technology can provide important support for these tasks, thereby greatly reducing the time and resource cost of artificial intelligence model development and deployment. The current mainstream 3D pre-training methods can be divided into two categories: contrastive learning and mask modeling. Contrastive learning performs pre-training by comparing similarities and differences between samples, for example: In the field of 3D point cloud PointContrast[3]Xie S, Gu J, Guo D, et al.Pointcontrast: Unsupervised pre-training for 3d point cloud understanding[C] / / Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020: 574-591. By constructing two point clouds from different perspectives and comparing the similarity of point features between the two point clouds, the point cloud network can be pre-trained. DepthContrast[4]Zhang Z, Girdhar R, Joulin A, et al.Self-supervised pretraining of 3d features on any point-cloud[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2021:10252-10263.Effective feature representation can be learned by using only the depth map of a single view for feature contrast learning. CrossPoint[5]Afham M, Dissanayake I, Dissanayake D, et al.Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:9902-9912. A cross-modal contrastive learning method is proposed to compare point clouds and rendered 2D images, which helps to learn generalizable point cloud representations. However, the above contrastive learning-based methods often require the design of negative samples and large batch sizes, and cannot completely avoid model collapse, thus affecting performance. Different from contrastive learning, mask modeling pre-trains the encoder by recovering the mask information, without the need to design negative samples and large batch sizes, where: Point-BERT[6]Yu X, Tang L, Rao Y, et al.Point-bert: Pre-training 3d point cloud transformers with masked point modeling[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:19313-19322. Through dVAE[7]Rolfe J T.Discrete variational autoencoders[J].arXiv preprint arXiv:1609.02200, 2016. Convert point cloud patches into discrete tokens and predict the tokens of the masked part. Going a step further, Point-MAE[8]Pang Y, Wang W, Tay FEH, et al.Masked autoencoders for point cloud self-supervised learning[C] / / Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II.Cham: Springer Nature Switzerland, 2022: 604-621. directly masks and restores point cloud blocks, thereby pre-training the encoder. Point-M2AE[9]Zhang R, Guo Z, Gao P, et al.Point-M2AE: multi-scale masked autoencoders for hierarchical point cloud pre-training[J].arXiv preprint arXiv:2205.14401, 2022. A hierarchical network structure is constructed, which can gradually model geometric information and capture feature information, and achieves good results in downstream tasks. Joint-MAE

[0010] Guo Z, Li X, Heng P A. Joint-mae: 2d-3d joint masked autoencoders for 3d point cloud pre-training[J].arXiv preprint arXiv:2302.14007, 2023. Focusing on the correlation between 2D images and 3D point clouds, a hierarchical module is proposed for cross-modal interaction to reconstruct the masking information of the two modalities. However, Point-M2AE and Joint-MAE learn multi-level geometric knowledge by modifying the hierarchical network architecture, which greatly increases the amount of network computation. Therefore, the present invention analyzes the above problems and proposes solutions. Without changing the network architecture or increasing the amount of computation, by improving the training method and utilizing the gradual guidance characteristics of the diffusion model, the network can learn multi-level geometric knowledge by restoring noisy point clouds at different stages, thereby achieving better performance in various downstream tasks. The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention. See Figure 1 which is a flowchart of the steps of the 3D pre-training method based on the diffusion model according to an embodiment of the present invention, and Figure 2 which is an overall framework diagram of the 3D pre-training method based on the diffusion model according to an embodiment of the present invention. According to FIG. 1 and FIG. 2 , the embodiment of the present invention mainly includes the following steps: Step S110, encoding the visible point cloud segmentation features through an encoder to obtain deep visible point cloud segmentation features, wherein the visible point cloud segmentation features are generated by the acquired point cloud blocks. It should be understood that the encoder refers to a module or network structure that converts input data into a potential representation or feature vector, is responsible for extracting potential geometric features, and is composed of standard Transformer blocks. It should also be understood that the visible point cloud segmentation features are encoded by the encoder to obtain deep visible point cloud segmentation features so that subsequent processing steps can better understand and utilize these features. Step S120, concatenating the deep visible point cloud segmentation feature and the obscured point cloud segmentation feature to obtain a concatenated feature, wherein the obscured point cloud segmentation feature is generated by the acquired point cloud block. It should be understood that the deep visible point cloud segmentation features and the obscured point cloud segmentation features are concatenated to obtain the concatenated The purpose of the joint feature is to combine the two features to obtain a richer and more diverse feature representation and provide more comprehensive information for subsequent processing. Step S130, encoding the splicing features through a conditional embedding module to obtain the guiding conditions of the diffusion model. It should be understood that the conditional embedding module takes the splicing features as input and encodes them through a series of operations and network layers. The purpose of conditional embedding is to extract key information from the splicing features and generate guidance conditions for the diffusion model, which can help the decoder perform better decoding operations. Step S140, determining a loss function according to the guidance conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to perform decoding to complete the training. In summary, the solution of the embodiment of the present invention achieves a comprehensive understanding of the point cloud by improving the pre-training method and utilizing the multi-level guidance characteristics of the diffusion model without changing the network architecture or increasing the amount of calculation, obtains richer geometric information, improves the effect and performance of model training, and further saves the cost of point cloud pre-training. In addition, the 3D pre-training method of the embodiment of the present invention is highly flexible and can be extended to various network models to improve their performance. In one implementation, before encoding the visible point cloud segmentation features through an encoder to obtain the deep visible point cloud segmentation features, the process includes obtaining point cloud blocks through the farthest point sampling algorithm and the K nearest neighbor algorithm; centralizing the point cloud blocks to update the point cloud blocks; embedding the updated point cloud blocks into the segmentation features to obtain the point cloud segmentation features; and randomly masking the point cloud segmentation features to generate visible point cloud segmentation features and masked point cloud segmentation features, wherein the visible point cloud segmentation features are represented as The masked point cloud segmentation feature is expressed as To mask the point cloud segmentation features The number of g = sr is the visible point cloud word segmentation feature The number of random masking. It should be understood that generating visible point cloud segmentation features and occluded point cloud segmentation features can help the model learn features of different importance. For visible point cloud segmentation features, the model can focus on learning key and obvious features. And occluded point cloud segmentation features can help the model learn the ability to infer and fill in occlusions, and improve the ability to understand and model complete point cloud data. By randomly masking some point cloud segmentation features, the diversity of the data set can be increased, thereby improving the generalization ability of the model. Masking different point cloud segmentation features can simulate the occlusion situation in real scenes. This enables the model to better adapt to the processing of various occlusion situations. Through centralization processing, the position of the point cloud block can be normalized to the origin or specified position, eliminating the influence of the position on subsequent processing, and improving the consistency, stability and description ability of the data; by embedding the updated point cloud block into the word segmentation feature, the feature representation can be enriched and a better feature basis can be provided for subsequent processing tasks. In another implementation, a point cloud block is obtained by using a farthest point sampling algorithm and a K nearest neighbor algorithm, including sampling the preset point cloud data by using the farthest point sampling algorithm to obtain s center points, wherein the preset point cloud data includes n points, and the preset point cloud data is represented by X∈R n×3 , s center points are represented as Based on each center point C of s center points i , k points closest to each center point are selected as point cloud blocks through the K nearest neighbor algorithm, where each center point is represented by C i , the point cloud block is represented as Point cloud block The calculation formula is: In another implementation, the updated point cloud block is embedded into the word segmentation feature to obtain the point cloud word segmentation feature, including a simplified version of PointNet ξ with parameter φ φ (·) Update the point cloud block Embedded into word segmentation features In, it is expressed as: In another implementation, the visible point cloud segmentation feature is encoded by an encoder to obtain a deep visible point cloud segmentation feature, including inputting the position information of the visible point cloud segmentation feature into a position embedding formula ψ with a parameter τ τ In (·), the position feature of the visible point cloud segmentation feature is output. The position feature of the visible point cloud segmentation feature is expressed as The position features of the output visible point cloud segmentation features With visible point cloud segmentation features Connect to get the combined features; through the encoder Φ ρ The Transformer block (·) extracts the combined features and obtains the deep visible point cloud segmentation features. The deep visible point cloud segmentation features are expressed as Among them, the encoder Φ ρ(·) is composed of a standard Transformer block with a parameter of ρ. The visible point cloud segmentation feature is encoded by the encoder, and the formula for obtaining the deep visible point cloud segmentation feature is: In another implementation, the splicing features are encoded by a conditional embedding module to obtain the guiding conditions of the diffusion model, including a conditional embedding module f with a parameter ω. ω (·) Encode the splicing features to obtain the guiding condition c of the diffusion model, where the formula of the guiding condition c of the diffusion model is: In another implementation, the loss function is determined according to the guidance condition of the diffusion model and the multi-stage uniform sampling strategy, including defining the training target of the pre-trained model according to the guidance condition c of the diffusion model, and the formula is: The time step interval [1, T] for each preset point cloud data is divided into h stages: in Randomly sample a time step t from h stages, calculate h losses and average the h losses to get the loss function: It should be understood that in the fine-tuning stage, the encoder is separated from the diffusion model, and a specific decoder and task head are introduced for training to adapt to various 3D downstream tasks. It should also be understood that the present invention restores the original point cloud during the pre-training process, and can also perform pre-training by restoring the masked point cloud blocks. During the pre-training process, the diffusion model denoising network can be replaced with a transformer network structure, and the pre-training network Encoder can be replaced with any 2D / 3D feature network. The new pre-training model proposed in this invention does not require a hierarchical network structure. By improving the 3D pre-training method and utilizing the multi-level guidance characteristics of the diffusion model, it can achieve a comprehensive understanding of the original input, while enabling the network to obtain richer geometric information, thereby performing better in downstream tasks and effectively saving the cost of point cloud pre-training. In addition, the pre-training paradigm proposed in this invention can be flexibly extended to various network models and improve their performance. FIG3A is a visualization result diagram of the 3D pre-training method based on the diffusion model according to an embodiment of the present invention, wherein each row shows the original input point cloud, i.e., the preset point cloud data, the obscured point cloud, and the generated point cloud at different stages of the reverse diffusion process. According to FIG3A , even if 80% of the points are obscured, the present invention can still The scheme can still recover high-quality point cloud, which proves the effectiveness of the scheme of the present invention. FIG3B is a comparison diagram of semantic segmentation results of another embodiment of the present invention, showing a qualitative comparison of semantic segmentation results on the S3DIS dataset. The first column is the original input point cloud, the second to fourth columns are the detection results of PointNeXt, Point-MAE and the solution of the present invention, and the last column is the true label. The segmentation results of the solution of the present invention are closer to the true label, and the segmentation effect of the blackboard and the table is better. Compared with PointNeXt and Point-MAE trained from scratch, there are fewer mis-segments, more hierarchical geometric knowledge has been learned, and it can perform better in downstream tasks. 4 is a structural block diagram of a 3D pre-training system 400 based on a diffusion model corresponding to the embodiment of FIG. 1 . The 3D pre-training system 400 based on a diffusion model includes: The first encoding module 410 is used to encode the visible point cloud segmentation features through an encoder to obtain deep visible point cloud segmentation features, wherein the visible point cloud segmentation features are generated by the acquired point cloud blocks. The splicing module 420 is used to splice the deep visible point cloud segmentation feature and the occluded point cloud segmentation feature to obtain a spliced ​​feature, wherein the occluded point cloud segmentation feature is generated by the acquired point cloud block. The second encoding module 430 is used to encode the splicing features through the conditional embedding module to obtain the guiding conditions of the diffusion model. The determination module 440 is used to determine the loss function according to the guidance conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to perform decoding to complete the training. In summary, the solution of the embodiment of the present invention achieves a comprehensive understanding of the point cloud and obtains richer geometric information by improving the pre-training method and utilizing the multi-level guidance characteristics of the diffusion model without changing the network architecture or increasing the amount of calculation, thereby improving the effect and performance of model training and further saving the cost of point cloud pre-training. The system of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the system of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here. According to a third aspect of an embodiment of the present invention, an electronic device is provided. Referring to FIG. 5 , a structural block diagram of an electronic device 500 that can be used as a server or client of the present application is now described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of Digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein. The electronic device 500 may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 . The processor 502, the communication interface 504, and the memory 506 communicate with each other via the communication bus 508. The communication interface 404 is used to communicate with other electronic devices or servers. The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above method embodiment. Specifically, the program 510 may include program codes, which include computer operation instructions. The processor 502 may be a processor CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs. The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Program 510 can be specifically used to enable processor 502 to perform the following operations: encode the visible point cloud segmentation features through an encoder to obtain deep visible point cloud segmentation features; splice the deep visible point cloud segmentation features with the masked point cloud segmentation features to obtain spliced ​​features; encode the spliced ​​features through a conditional embedding module to obtain the guiding conditions of the diffusion model; determine the loss function based on the guiding conditions of the diffusion model and the multi-stage uniform sampling strategy. In addition, the specific implementation of each step in program 510 can refer to the corresponding The corresponding descriptions in the steps and units are not repeated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiments, which are not repeated here. It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention. The exemplary embodiments of the present invention further provide a computer storage medium storing a computer program, wherein the computer program implements the methods of the various embodiments of the present invention when executed by a processor. The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein. It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art. The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A 3D pre-training method based on a diffusion model, characterized in that: include: The visible point cloud segmentation feature is encoded by an encoder to obtain a deep visible point cloud segmentation feature, wherein the visible point cloud segmentation feature is generated by the acquired point cloud block; splicing the deep visible point cloud segmentation feature and the obscured point cloud segmentation feature to obtain a spliced ​​feature, wherein the obscured point cloud segmentation feature is generated by the acquired point cloud block; The splicing features are encoded by a conditional embedding module to obtain guiding conditions of a diffusion model; A loss function is determined according to the guiding conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to perform decoding to complete the training.

2. The method according to claim 1, characterized in that Before encoding the visible point cloud segmentation features through the encoder to obtain the deep visible point cloud segmentation features, the process includes: Obtain point cloud blocks through the farthest point sampling algorithm and K nearest neighbor algorithm; Centralizing the point cloud block to update the point cloud block; Embed the updated point cloud block into the word segmentation feature to obtain the point cloud word segmentation feature; The point cloud segmentation feature is randomly masked to generate the visible point cloud segmentation feature and the masked point cloud segmentation feature, wherein the visible point cloud segmentation feature is represented as The masked point cloud segmentation feature is expressed as is the masked point cloud segmentation feature The number of g = sr is the visible point cloud word segmentation feature The number of random masking.

3. The method according to claim 2, characterized in that The method of obtaining a point cloud block by using the farthest point sampling algorithm and the K nearest neighbor algorithm includes: The preset point cloud data is sampled by the farthest point sampling algorithm to obtain s center points, wherein the preset point cloud data includes n points, and the preset point cloud data is represented by X∈R n×3 , s center points are represented as Based on each center point C of s center points i , k points closest to each center point are selected as the point cloud block through the K nearest neighbor algorithm, where each center point is represented by C i , the point cloud The block is represented as Then the point cloud block The calculation formula is:

4. The method according to claim 3, characterized in that The updated point cloud block is embedded into the word segmentation feature to obtain the point cloud word segmentation feature, including: Through a simplified version of PointNetξ with parameter φ φ (·) Update the point cloud block Embedded into word segmentation features In, it is expressed as:

5. The method according to claim 4, characterized in that The encoding process of the visible point cloud segmentation features by the encoder to obtain the deep visible point cloud segmentation features includes: The position information of the visible point cloud segmentation feature is input into the position embedding formula ψ with parameter τ τ (·) outputs the position feature of the visible point cloud segmentation feature, and the position feature of the visible point cloud segmentation feature is expressed as The position feature of the output visible point cloud segmentation feature With the visible point cloud segmentation feature Connect to obtain the combined features; Through the encoder Φ ρ The Transformer block (·) extracts the combined features to obtain the deep visible point cloud segmentation features, which are expressed as Among them, the encoder Φ ρ (·) is composed of a standard Transformer block with a parameter of ρ. The visible point cloud segmentation feature is encoded by the encoder, and the formula for obtaining the deep visible point cloud segmentation feature is:

6. The method according to claim 5, characterized in that The splicing features are encoded by the conditional embedding module to obtain the guiding conditions of the diffusion model, including: Through the conditional embedding module f with parameter ω ω (·) Encode the splicing features to obtain the guiding condition c of the diffusion model, where the formula of the guiding condition c of the diffusion model is:

7. The method according to claim 6, characterized in that Determining the loss function according to the guidance conditions of the diffusion model and the multi-stage uniform sampling strategy includes: The training objective of the pre-training model is defined according to the guiding condition c of the diffusion model, and the formula is: The time step interval [1, T] for each preset point cloud data is divided into h stages: in Randomly sample a time step t from h stages, calculate h losses and average the h losses to get the loss function:

8. A 3D pre-training system based on a diffusion model, characterized in that: include: A first encoding module is used to encode the visible point cloud segmentation feature through an encoder to obtain a deep visible point cloud segmentation feature, wherein the visible point cloud segmentation feature is generated by the acquired point cloud block; A splicing module, used for splicing the deep visible point cloud segmentation feature with the obscured point cloud segmentation feature to obtain a spliced ​​feature, wherein the obscured point cloud segmentation feature is generated by the acquired point cloud block; A second encoding module, used for encoding the splicing features through a conditional embedding module to obtain a guiding condition of a diffusion model; The determination module is used to determine the loss function according to the guidance conditions of the diffusion model and the multi-stage uniform sampling strategy, and the loss function is used to guide the decoder to decode to complete the training.

9. An electronic device, characterized in that: include: processor; A memory for storing programs; The program comprises instructions, which, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images