Data augmentation and part segmentation method

Abstract

A computer implemented method for training a generative model is provided. The method includes steps of accessing a training dataset comprising labelled and unlabelled point cloud representations of objects. In a first training phase a variational autoencoder comprising a first and second encoder and a decoder is trained on the training dataset. In a second training phase first and second denoising diffusion models are trained based on data output from the first and second encoders.

Description

[0001] 202417825 DATA AUGMENTATION AND PART SEGMENTATION METHOD TECHNICAL FIELD The present disclosure relates to methods and systems for generative data augmentation and three-dimensional (3D) part segmentation. BACKGROUND In recent years 3D computer vision has made astonishing progress thanks to the advent of deep learning. One of the keys to this success lies in the abundance of annotated data. Compared with the annotation for two dimensional (2D) semantic segmentation, the annotation for 3D semantic segmentation is particularly time-consuming and laborious. A straightforward way to get more annotated data is to perform traditional data augmentation (TDA), which employs basic geometric transformations, like random rescaling, rotation, flipping & jittering on already labelled data. One shortcoming of this approach is that the variety of 3D shapes is small. Other approaches focused on extracting helpful information from samples with and without semantic segmentation annotations methods can be broadly classified into semi-supervised learning, weakly- supervised learning and unsupervised learning . The semi-supervised method refers to annotating only a subset of samples, such as selecting 5% to 10% of available 3D objects for annotation. Weakly-supervised methods denote those that annotate only a small number of points in each object or scene, with the most extreme case being the annotation of just one point per part, i.e., "one part, one click." The unsupervised method initially trains the segmentation model on unannotated point clouds, followed by fine- tuning on several annotated samples. In recent years, generative models such as variational autoencoders (VAEs), generative adversarial networks (GANs), and denoising diffusion models (DDMs) have achieved impressive results. This provides a new perspective and can be used to bridge the gap between data annotation and data generation. Diffusion models can generate high-quality samples for data augmentation in downstream 2D classification tasks, improving the prediction accuracy. However, these models focus on 2D global classification tasks, 202417825 which are more straightforward than 3D segmentation tasks. Existing 3D generative diffusion models have achieved outstanding results on point cloud-based 3D shape generation. However, these models focus on generating 3D shapes without any part segmentation information, which cannot facilitate the training of 3D segmentation tasks. There is therefore a need for a method for generating high-quality three-dimensional shapes with semantic part segmentation annotations, and, in particular, where a limited amount of labelled data is available. SUMMARY According to a first aspect, a computer implemented method for training a generative model is provided. The method comprises a) accessing a training dataset comprising point cloud representations of one or more objects, the training dataset comprising afirst subset of labelled point cloud representations, ^^^^ with labels, ^^^^, and a secondsubset comprising unlabelled point cloud representations, ^^^^, wherein the size, N, ofthe first subset is less than the size, M, of the second subset; b) assigning a set ofpredetermined labels, ^^^^, to the second subset, ^^^^, to obtain a modified trainingdataset, {{^^^^, ^^^^}, {^^^^, ^^^^}} c) initializing a generative model, the generative modelcomprising: i) a variational auto-encoder (VAE), wherein the VAE comprises: a firstencoder, ^^^^ , to encode an input comprising a labelled point cloud representation, as alatent representation in a first latent space; a second encoder, ^^ℎ , to encode an inputcomprising a labelled point cloud representation as a point-wise latent representation ina second latent space; and a decoder, ^^ , to decode an input comprising a sampledℎlatent representation from the first latent space and a sampled point-wise latent representation from the second latent space to obtain an output point cloudrepresentation of a three-dimensional object; ii) a first denoising-diffusion model, ^^^^ , todenoise a diffused latent representation from the first latent space; and iii) a seconddenoising-diffusion model, ^^ℎ , to denoise a diffused point-wise latent representationfrom the second latent space; and d) in a first training phase: training the VAE,{^^ ^^ ^^ ^^ ^^^^ , ^^ℎ , ^^ℎ }, based on the modified training dataset, {{^^ , ^^ }, {^^ ,^^ }}, wherein theVAE is trained to reconstruct {^^^^, ^^^^}, conditioned by {^^^^, ^^^^}; and e) in a secondtraining phase: training the first denoising diffusion model, ^^^^ , and the second denoising 202417825diffusion model, ^^ℎ , over a pre-determined number of discrete time steps, T, based onoutput obtained from the first trained encoder, ^^^^ , second trained encoder ^^ℎ whereinthe first denoising diffusion model, ^^ , and the second denoising diffusion model, ^^ , are^^ ℎconditioned based on the labels ^^^^,^^^^ .Preferably, for each labelled point cloud representation, (^^^^ ^^^^, ^^^^) in the first subset,^^^^,^^^^the method comprises obtaining latent representations of the point cloud{ }representation in the first and second latent spaces, based on output obtained from thefirst trained encoder, ^^ , and second trained encoder, ^^ ; applying the first trained^^ ℎdenoising diffusion models, ^^ , and second trained denoising diffusion model ^^ , for ^^-^^ ℎstep forward diffusion and, subsequently, ^^-step backward denoising to obtain a set of( ) variant latent representation, wherein ^^ ∈ 0..^^ ; applying the decoder, ^^ , to the setℎ^^ ^^ ^^×3 of variant latent representations to generate a set ^^ = {^^ ∈ ℝ : ^^ ∈ (0, .. ^^)}, of^^ ^^,^^^^ ^^ point cloud representations, ^^ , with labels, ^^ .^^,^^ ^^Preferably, the method comprises training a first predictive model, ^^, for inferring alabelling of a point cloud representation of a three-dimensional object, based on the first ^^ ^^^^ ^^ { } ^ ^ subset ^^ , ^^ and a further set of labelled point cloud representations ^^ = ^^ ,^^^^^^ ^^^^ with corresponding labels ^^ .Preferably, the method comprises applying the trained predictive model, ^^, to each point^^ ^^^ cloud representation in the second subset ^^ to obtain a set of inferred labels, ^^ for^^ the point cloud representations in the second subset, ^^ .^^ ^^ Preferably, for each point cloud representation, ^^ , in the second subset, ^^ and^^^^ ^^^ corresponding labels ^^^ in ^^ , the method comprises: obtaining latent representations^^ of the point cloud representation, ^^ , in the first and second latent spaces, based on^^output obtained from the first trained encoder, ^^ , second trained encoder ^^ ; applying^^ ℎthe first trained denoising diffusion models, ^^ , and second trained denoising diffusion^^model ^^ , for ^^-step forward diffusion and, subsequently, ^^-step backward denoising toℎ 202417825obtain a set of variant latent representation, wherein ^^ ∈ (0..^^); applying the decoder,to the set of variant latent representations to generate a set ^^^^ = {^^^^ : ∈ ℝ^^×3^^,^^ : ^^ ∈(0, .. ,^^)}, of point cloud representations, ^^^^ ^^^^,^^ , , with labels, ^^^ .Preferably, the method comprises training a second predictive model, ^^′, for inferringlabels in a point cloud representation of a three-dimensional object, based on the first ^^ ^^^^ ^^ { } ^ ^ subset ^^ ,^^ , the generated labelled point cloud representations ^^ = ^^ with^^^^^^ ^^^^^^ ^^ { } corresponding labels ^^ , and the generated point cloud representations ^^ = ^^ ,^^^^^^^^^ with corresponding labels ^^ .Preferably, the method comprises accessing a further point cloud representation, ^^^ , of athree-dimensional object; and applying the second trained predictive model, ^^′ to inferlabels, ^^^, of the further point cloud representation, ^^^ .Preferably, training the VAE, comprises maximizing a variational bound on a data log-likelihood function, and wherein ^^ , ^^ are posterior distributions for sampling a latent representation z^^ ^^ 0^^ ℎin the first latent space and a latent representation, h, in the second latent space; ^^ is0 ^^ℎthe prior distribution for reconstruction prediction; ^^ is a Kullback-Leibler^^^^regularization terms; λ and λ are hyperparameters for balancing the Kullback-Leiblerz h( ) regularization terms and reconstruction accuracy; and ^^ 0, ^^ is a normal distribution.Preferably, training the first denoising diffusion model, ^^ , comprises minimizing an^^( ) objective function, ℒ ^^ , wherein:^^ ^^ ^^ ^^ ^^( ) { } { } and wherein ^^ = ^^ ^^,^^ ; ^^ ∈ ^^ ,^^ ; ^^ ∈ ^^ ,^^ ; ^^ ~ ^^(0, ^^) and t is a time0 ^^ 202417825step uniformly sampled from {1 …^^}.Preferably, training the second denoising diffusion model, ^^ℎ , comprises minimizing anobjective function, wherein: and wherein time step uniformly sampled from {1 …^^}.Preferably N, is less than or equal to M / 10.Preferably, the VAE, and the first and second denoising diffusion models, ^^^^ , ^^ℎ areartificial neural networks. These and other aspects of the invention will be apparent from the embodiment(s) described below. BRIEF DESCRIPTION OF THE DRAWINGS For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: Figure 1 shows a block diagram of a generative model, according to an example; Figure 2 shows a flow diagram of a method for training a generative model, according to an example; Figures 3A – 3E show examples of point cloud representations of shapes; Figure 4 shows a neural network architecture, according to an example; Figure 5 shows a neural network architecture, according to an example; Figure 6 shows a neural network architecture, according to an example; 202417825 Figure 7 shows a neural network architecture, according to an example; Figure 8 shows a neural network architecture, according to an example; Figure 9 show examples of point cloud representations of shapes; Figure 10 illustrates a data processing system on which embodiments of the invention may be implemented. DETAILED DESCRIPTION Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein. Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate. The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof. 202417825 The methods and systems described herein may be used to generate novel three dimensional (3D) shapes with segmentation labels. The methods may further be used for generative data augmentation (GDA) in 3D part segmentation tasks. Starting with a small portion of labelled 3D shapes, these methods generate a large number of novel labelled 3D shapes, as well as providing an overall increase in the accuracy of object part segmentation models. Denoising Diffusion Model The methods described herein utilise denoising diffusion models (DDMs) conditioned by part segmentation labels. The DDMs described herein generate data by simulating astochastic process in a discrete-step manner. Given a sample ^^0~ ^^(^^0) from adistribution, a forward diffuse process gradually adds noise to the input to make itconverge to a Gaussian distribution after T steps, i.e., ^^(^^^^) ≈ ^^(0, ^^). During training,the diffusion model ^^^^ with parameters ^^ is trained to predict the noise ^^ to denoisethe diffused sample ^^^^. The training objective is: In equation (1), t is a time step uniformly sampled from {1, ... , ^^ }, ^^~^^(0, ^^) is thenoise for diffusing ^^0 to ^^^^, and a is the condition information, such as category labelsor description context.During inference, the DDM starts from a random sample ^^^^~^^(0, ^^) and denoises ititeratively until ^^ = 0. At each step, random noise ^^~^^(0, ^^) is added to theintermediate result ^^ to increase the randomness of the denoising process and,^^eventually, the diversity of the final generated results. The diffuse-denoise procedurefirstly diffuses the input feature ^^0 for ^^ steps (^^ < ^^) and then denoises the noisyfeature ^^ to ^^′^^ 0. During the forward diffusion process, more and more details of theinput data deteriorate, while in the backward process, besides denoising, random noise is also added to the sample at each step to make this process stochastic, thus resultingin the generation of an input sample variation. Therefore, ^^′ is similar to ^^ but not the00same after the diffuse-denoise procedure, i.e., some changes are introduced to ^^ while0 202417825the general characteristics of ^^0 are maintained, which enable the diffuse-denoiseprocedure to enhance the sampling variety. Generative Model The methods described herein provide a generative model for generating 3D point clouds with segmentation labels by introducing a segmentation conditioning module in variational autoencoder (VAE) and diffusion modules, enabling the model to learn the appropriate placement of points based on segmentation. Moreover, in order to utilize only a small portion of labelled data, a semi-supervised learning approach is used to train the generative model. Figure 1 is a diagram of a generative model 100, according to an example. The input tothe model 100 comprises point coordinates, ^^ ∈ ℝ^^×3, 110 and segmentation labels orzero padding, ^^ ∈ ℝ^^×^^ , 120 for labelled and unlabelled samples, respectively. Thegenerative model 100 comprises a hierarchical variational auto encoder (VAE)comprising a global encoder, ^^^^ , 130 point-level encoder, ^^ℎ , 140 and point-leveldecoder, ^^ℎ , 150. The hierarchical VAE maps inputs 110, 120 to a global latent ^^0 ∈ℝ^^^^ and point-level latent ℎ ∈ ℝ^^ × ^^0 ℎ . The generative model 100 comprises latentdiffusion models, ^^^^ , 160 and, ^^ℎ , 170 for these two latent spaces. All these modules areconditioned by ^^. The final output 180 of the generative model is the reconstructedpoint cloud during the training of VAE, or the novel generated point cloud with associated segmentation labels during inference. Figure 2 is a flow diagram of a method 200 for training the generative model 100, according to an example. At block 210, the method comprises accessing a training dataset comprising point cloud representations of one or more objects. The trainingdataset may comprise a first subset of labelled point cloud representations, ^^^^ = {^^^^^^ ∈ segmentation labels, ^^^^ = {^^^^ ∈ {0,1}^^ ×^^^^ : ^^ = 1..^^}, and asecond subset comprising unlabelled point cloud representations, ^^^^ = {^^^^^^ ∈ℝ^^×3| ^^ = 1, .. The number, N, of labelled point clouds is less than the number,M, of unlabelled point clouds. In examples described herein N is significantly less than 202417825M. For example, N, may be 10% of M. The segmentation labels ^^^^ may be one-hotsegmentation labels {^^^^ ^^×^^^^ ∈ {0,1} | ^^ = 1, .. , ^^} where c is the number of sub-parts.Figure 3A is an example of a la ^^belled point cloud-representation 300 in the subset ^^ ofthe training dataset. The point cloud 300 represents an aircraft. Different segmentation labels are allocated to the main body 310, wings 320, engines 330 and tailfin 340. Figure 3B is an example of an unlabelled point cloud representation 350, also representing an ^^ aircraft, in the subset, ^^ , of the training dataset. In Figure 3B there is no distinctionbetween the regions of the point cloud 350 that represent the different components of the aircraft. ^^ At block 220, a set of pseudo-labels, ^^ , is assigned to the point clouds of the second^^ ^^ ^^ ^^ ^^{ } { } subset, ^^ , to obtain a modified training dataset, { ^^ , ^^ , ^^ ,^^ }. This enables the^^ generative model to be trained in a unified manner. The set of pseudo-labels ^^ maybe the set At block 230, the method 200 comprises initializing a generative model. According to examples described herein, the generative model comprises a variational auto-encoder ^^×3 ^^×^^ ^^^^ { } (VAE), ^^ ,^^ , ^^ , comprising a first encoder, ^^ : ℝ × ℝ → ℝ , to encode a^^ ℎ ℎ ^^labelled point cloud representation as a latent representation, ^^ , in a first latent space,0^^ ^^×3 ^^×^^ ^^ ^^×^^^^ ^^ ^^ ℝ , a second encoder, ^^ : ℝ × ℝ × ℝ → ℝ , to encode a point cloudℎ^^×^^^^ representation as a point-wise latent representation, ℎ , in a second latent space, ℝ ,0and a decoder, The decoder, decodes sampled latent representation from the first latent space and a sampled point-wise latent representation from the second latent space to obtain an output point cloud representation of a three-dimensional object. The generative model further comprises ^^ ^^×^^ ^^^^ ^^ a first denoising-diffusion model, ^^ : ℝ × ℝ × ℝ → ℝ , to denoise a diffused^^latent representation from the first latent space; and a second denoising-diffusion model, denoise a diffused point-wise representation from the second latent space; and 202417825At block 240, in a first training phase, the VAE, {^^^^ ,^^ℎ , ^^ℎ} is trained based on themodified training dataset, {{^^^^,^^^^}, {^^^^,^^^^}}. The VAE is trained to reconstruct{^^^^, ^^^^} conditioned by {^^^^, ^^^^}. Training the VAE, may comprisesmaximizing a variational lower bound on a data log-likelihood function, where: In equation (2), ^^^^^^ ,^^^^ℎ are posterior distributions for sampling a latent representationz0 in the first latent space and a latent representation, h0, in the second latent space; ^^^^ℎis the prior distribution for reconstruction prediction; ^^^^^^ is a Kullback-Leiblerregularization terms; λz and λh are hyperparameters for balancing the Kullback-Leiblerregularization terms and reconstruction accuracy; and ^^(0, ^^) is a normal distribution.At block 250, in a second training phase, the first denoising diffusion model, ^^^^ , and thesecond denoising diffusion model, ^^ℎ , are trained over a pre-determined number ofdiscrete time steps, T, based on output obtained from the first trained encoder, ^^^^ andsecond trained encoder ^^ℎ . The first denoising diffusion model, ^^^^ , and the seconddenoising diffusion model, ^^ℎ , are conditioned based on the labels ^^^^, ^^^^. Thegenerative model is trained in this semi-supervised approach to learn the reconstruction of the point clouds with or without the segmentation labels. According to examples,training the first denoising diffusion model, ^^^^ , may comprise minimizing an objectivefunction, ℒ(^^^^), where: In equation step uniformly sampled from {1 …^^}. Training the second denoising diffusion model, ^^ℎ ,may comprise minimizing an objective function, ℒ(^^ℎ), wherein: 202417825In equation is atime step uniformly sampled from {1 …^^}.Once the generative model is trained, it can be used to run a ^^-step diffuse-denoiseprocedure on ^^^^. For each labelled sample ^^^^ ^^ ^^^^, a set of re-generated variants ^^^^ = {^^^^,^^ ∈ℝ^^×3 : ^^ ∈ (0, .. ,^^)} with segmentation labels ^^^^, is obtained. Compared with ^^^^, some^^ ^^^^ local deformations appear on the generated variants ^^ . These novel local features are^^,^^learned from both labelled and unlabelled data. Through this process, segmentation annotations are transferred to the newly generated samples. ^^ Figure 3C depicts a generated variant point cloud 360, ^^ , according to an example.^^The point cloud 360 depicts a representation of an aircraft as a point cloud, similar to the point cloud representations 300, 350, but differing from both the representations 300, 350 in terms of local features. For example, the aircraft represented by the point cloud 360 has different shaped wings from the point cloud 300, and a different shaped tailfin from the point cloud 350. ^^ With the original labelled data ^^ and the generative data augmentation based on label^^ ^^ ^^×3 ^^ ×^^ { } ^ ^ transfer ^^ = ^^ : ^^ = 1.. ^^ , a neural network, ^^: ℝ → 0, 1 may be trained^^^^^^ ^^for inferring a pseudo-labelling of a point cloud representation. This neural network, ^^ ^^, may be applied to the unlabelled point cloud representations ^^ to determine a set^^^^ ^^ ×^^^ { } of inferred pseudo-labels ^^ = {^^^ ∈ 0, 1 : ^^ = 1, .. ,^^}.^^Figure 3D depicts a pseudo-labelling 370 of the point cloud 350, obtained using thetrained neural network, ^^, according to an example. The pseudo labelling 370 of thepoint cloud 350 contains some artifacts, where the neural network incorrectly predicts the label for points in the point cloud data. For example, the region 380 contains some mislabelled points. 202417825 Since the generative model obtained from the method 200 learns to condition on the segmentation labels, points with wrong predictions may be moved to more appropriate positions, while most structures with correct labels will be preserved after a diffuse- denoise procedure with a few steps. The trained generative model maybe be used to ^^ ^^ run a ^^′ -step diffuse-denoise procedure on ^^ , conditioning on the pseudo labels ^^^ .For each unlabelled sample ^^^^ ^^ ^^ ^^×3^^ a set of refined samples ^^^^ = {^^^^,^^′ ∈ ℝ : ^^′ ∈(0, .. , ^^)} is obta ^^ined with labels ^^^ .^^Figure 3E depicts a refined labelling 390. In the example, points with wrong predictions are moved to more appropriate positions, while most structures with correct labels are preserved after a diffuse-denoise procedure. ^^ ^^ From the above steps, the generative data augmentation {^^ , ^^ } is obtained from^^^^^^ ^^^^^^′^^ ^^ label transfer to the generated variants ^^ = { ^^ ∶ ^^ = 1..^^} and the refined pseudo-^^^^^^ ^^^^ ^^ ^^ ^^^^ labelled samples ^^ = {^^ : ^^ = 1..^^}. Finally the dataset {^^ , ^^ , ^^ } may be^^^^^^′ ^^ ^^^^^^ ^^^^^^′^^×3 ^^ ×^^ { } used to train a final segmentation network ^^′: ℝ → 0, 1 .Neural Network ArchitectureFigure 4 depicts a neural network architecture 400 for the global encoder ^^ , according^^^^ × 3 to an example. In Figure 4 a point cloud ^^ ∈ ℝ is concatenated with a segmentation^^ × ^^ label comprising zero padding ^^ ∈ ℝ where n and c are the number of points andpart types respectively. The input is passed through a series of point-voxel convolution (PVConv) modules 410 and down sampled from 2048, to 1024 points, and eventually to 256 points through set abstraction (SA) layers 420. A max pooling layer 430 outputs a global intermediate feature and a multilayer perceptron (MLP) 440 transforms the ^^^^ output to a global latent ^^ ∈ ℝ .0Figure 5 depicts an example implementation 500 of the module PVConv 410 shown in Figure 4. The PVConv module 500 combines the advantages of point-based and voxel- based methods in extracting neighbouring features. The point-based pipeline comprises 202417825 a linear layer 510, a group normalization (GN) 520, and a swish activation function 530. The voxel-based pipeline additionally comprises voxelize / de-voxelize layers 535, three- dimensional convolution 540, dropout 550, and a squeeze-and-excitation (SE) layer 560. In the SE layer 560, the feature is multiplied by an adaptively calibrated channel-wise factor. The outputs from these two pipelines are merged by a summation 570.Figure 6 depicts a neural network architecture 600 for the denoising diffusion model, ^^^^ .The zero padding ^^ ∈ ℝ^^ × ^^ is firstly accumulated along the point dimension to obtain^^ ^^^ ∈ ℝ ^ . For the labelled samples, ^^^^ is a vector of the number of points for each part,while for the unlabelled samples, it is a zero vector. ^^^^ is concatenated with the noisyglobal latent ^^^^ at time step t, and fed to a linear layer 610. The core of the diffusionmodel, ^^^^ , is a stacked ResNet 620, in which the feature is first fused with temporalembedding 630 by summation and then fused with the output from MLP 640 and SE layer 650. The output of the stacked ResNet is eventually transformed into the predicted noise through the output linear layer 660.The point-level encoder, ^^ℎ , and decoder, ^^ℎ , and the point-level denoising-diffusionmodel, ^^ℎ , adopt a similar 4-layer Point-Voxel Convolution Neural Network (PVCNN)architecture. These PVCNNs also utilise the PVConv module 500, however the GN layer 520 is replaced by an adaptive group normalization layer for conditioning on theglobal latent value ^^0. Figure 7 depicts an example implementation 700 of an adaptivegroup normalization layer. The adaptive GN 700 comprises a linear layer 710 and GN layer 720 similar to the GN layer 520. Figure 8 depicts a neural network architecture 800 for the point level denoising-diffusionmodel, ^^ℎ , according to an example. Each layer of the network 800 comprises PVConvmodules 810 except the deepest layer, SA modules 820, and feature propagation (FP) modules 830. The SA modules 820 and FP modules 830 are utilized to down- and up-sample the points. In the point-level diffusion ^^ℎ , intermediate features are concatenatedwith the temporal embedding of diffusion step, t. Additionally, the model comprisessegmentation conditioning (SC) modules 840, in which the segmentation labels / zero 202417825 paddings are concatenated with the intermediate features. GP modules 850 are also used at all layers. As the number of points decreases with increasing layers from 2048 points to 16 points and some small sub-parts such as the engine part of the airplane example depicted in Figure 3, only have a small portion of points, the global attention modules ensure that the points of small sub-parts can obtain sufficient information from other points of the same category. The methods and systems described herein implement a three-dimensional generative diffusion model with a segmentation conditioning module. The model described herein is capable of generating point clouds with segmentation labels to create high-quality and automatically annotated 3D shapes. The methods described herein reduce the manual labour for labelling tasks and boost the performance of segmentation neural networks. In addition, the methods described herein may be used to generate labelled three- dimensional shapes in arbitrary orientations from a small portion of labelled data. The generative model described herein learns the relative position among various parts within an object, rather than merely memorizing the absolute spatial positions of each part. When an object is in arbitrary orientations, the generative model can still place points in appropriate positions conditioned by the segmentation. Figure 9 shows examples of generated variants of objects in arbitrary orientations. In Figure 9 the left column 910 shows origin point clouds and the right three columns 920 show associated generated variants. Since the input point cloud contains not only the position information of each point but, as a whole, the pose information of the object,thus the global latent, ^^0 , and point-level latent, ℎ0, compressed by the VAE both containthe pose information of the object. Thus, the diffuse-denoise procedure may not only result in local structural deformation but also change the object pose. Figure 10 illustrates an example of a data processing system in which an embodiment of the present disclosure may be implemented. The data processing system 1000 comprises a processor 1010 connected to a local system bus 1020. The local system 202417825 bus connects the processor to a main memory 1030 and graphics display adaptor 1040, which may be connected to a display 1050. The data processing system may communicate with other systems via a wireless user interface adapter connected to the local system bus 1020, or via a wired network, for example, to a local area network. Additional memory 1060 may also be connected via the local system bus 1020. A suitable adaptor, such as wireless user interface adapter 1070, for other peripheral devices, such as a keyboard 1080 and mouse 1090, or other pointing device, allows the user to provide input to the data processing system. The data processing system 1000 is adapted to carry out the methods in accordance with the embodiments described herein. For example, the keyboard 1080 and mouse 1090 may function as a user input device for receiving information from the user, the processor 1010 may be adapted to carry out the steps of the method and the display 1050 adapted to display a particular view to the user. A computer product comprising instructions which, when run on a computer, such as the data processing system 1000, may be provided to cause the computer to execute the steps of the methods of the embodiments of the present invention outlined above. Experimental Data Datasets Experimental data is provided for two publicly available 3D datasets: ShapeNet (Yi, L et al., A scalable active framework for region annotation in 3d shape collections. ACMTransactions on Graphics (TOG) 35, 1 – 12 (2016),https: / / api.semanticscholar.org / CorpusID:28807122, 6, 8, 16) & PartNet (Mo, K et. Al.; A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. pp.909–918 (2019) 2, 8, 10, 16). ShapeNet and PartNet contain 16 and 24 categories of objects, respectively. In the experimental data presented herein, six non-overlapping categories are selected from these two datasets. In ShapeNet, the six selected categories are: airplane, car, guitar, 202417825 lamp, motorbike, and pistol categories. In PartNet, the six selected categories are: bed, bottle, chair, display, faucet, and laptop in the coarse-level segmentation. Each object in these categories contains three to six subparts. In each category, 10% of the samples have segmentation labels, while the remaining samples are unlabelled. For each object, 2048 points are sampled at random. Segmentation Network In the data provided herein, the segmentation networks are PointNet (Qi, C. et. Al.: Deep learning on point sets for 3d classification and segmentation.2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 77–85 (2016), https: / / api.semanticscholar.org / CorpusID:5115938) and SPoTr (Self-positioning point- based transformer for point cloud understanding. 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 21814–21823 (2023), https: / / api.semanticscholar.org / CorpusID:257804915). Results In the experiments, the mean accuracy (mAcc) and part-category mean IoU (mIoU) are used as evaluation metrics. PointNet is trained using six different approaches: (1) full supervision (FS): training the segmentation model using all the labelled samples (100%); (2) without any data augmentation: training the segmentation network using the labelled samples (10%) without any data augmentation approaches; (3) with TDA: training the segmentation network using the labelled samples (10%) with TDA such as random transform, random flipping, random rescaling, and jittering; (4) contrastive learning (CL): training the segmentation network using the labelled samples (10%) and the unlabelled samples (90%) with the contrastive learning method proposed in Jiang, L., Shi, S., Tian, Z., Lai, X., Liu, S., Fu, C.W., Jia, J.: Guided point contrastive learning for semi-supervised point cloud semantic 202417825 segmentation. In: Proceedings of the IEEE / CVF international conference on computer vision. pp.6423–6432 (2021)) (5) using the generative data augmentation (GDA) model provided herein based on label transfer (LT): training the segmentation network using the labelled samples (10%) and the GDA based on label transfer to the newly generated variants ^^^^ ^^^^^^^ from ^^ ^;(6) with GDA based on label transfer (LT) and pseudo label refinement (PLR): training the segmentation network using the labelled samples (10%), GDA based on label transfer ^^^^ , and GDA based on pseudo label refinement ^^^^^^^^^^ ^^^^^^′ .The results on the ShapeNet dataset are listed in Table 1. The results show that TDA, CL-based semi-supervised learning, and GDA all boost the training of PointNet. Among these methods, GDA with LT and PLR performs the best, while the improvement brought by CL is not significantly larger than that of TDA. In the average of 6 categories, GDA outperforms TDA by 2.89% and CL by 2.64% on mIoU, and 1.00% and 0.91% on mAcc. Table I The experiment results on the PartNet dataset are listed in Table II. GDA outperforms TDA and CL-based semi-supervised learning. In the average of 6 categories, GDA outperforms TDA by 4.52% and CL by 3.01% on mIoU, and 1.32% and 0.95% on mAcc. In the experiment of bottle category, the PointNet trained with 10% labeled data and GDA surpassed the one trained with full supervision on the mIoU metric. 202417825 Table II In a further experiment the performance of the SPoTr model was compared using TDA and GDA. The experimental results are listed in Table III. In the average of 6 categories, GDA outperforms TDA by 5.45% on mIoU and 1.99% on mAcc. Complex neural networks often require more training data; otherwise, overfitting may occur. These results confirm the effectiveness of GDA for various 3D neural networks, including complex models containing a large number of transformer modules. Table III Hyperparameters Tables IV-VII below illustrate example hyperparameters for the generative model. TableIV illustrates example hyperparameters for the global encoder, ^^^^ .

[0002] 202417825 Table IVTable V illustrates example hyperparameters for the global diffusion-denoise model, ^^^^ .Table VTable VI illustrates example hyperparameters for the point-level encoder ^^ℎ . In Table VI,layer 1 refers to the shallowest layer and layer 4 refers to the deepest layer. Table VITable VII illustrates example hyperparameters for the point-level decoder ^^ℎ . In Table VII,layer 1 refers to the shallowest layer and layer 4 refers to the deepest layer. 202417825 Table VIITable VIII illustrates example hyperparameters for the point-level decoder ^^ℎ . In TableVIII, layer 1 refers to the shallowest layer and layer 4 refers to the deepest layer. Table VIII 202417825 The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. The present inventions can be embodied in other specific apparatus and / or methods. The described embodiments are to be considered in all respects as illustrative and not restrictive. In particular, the scope of the invention is indicated by the appended claims rather than by the description and figures herein. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

202417825 CLAIMS1. A computer implemented method for training a generative model, the methodcomprising:a) accessing a training dataset comprising point cloud representations of one ormore objects, the training dataset comprising a first subset of labelled point cloudrepresentations, ^^^^ with labels, ^^^^, and a second subset comprising unlabelled pointcloud representations, ^^^^, wherein the size, N, of the first subset is less than the size,M, of the second subset;b) assigning a set of predetermined labels, ^^^^, to the second subset, ^^^^, to obtaina modified training dataset, {{^^^^,^^^^}, {^^^^, ^^^^}};c) initializing a generative model, the generative model comprising:i) a variational auto-encoder (VAE), wherein the VAE comprises:a first encoder, ^^^^ , to encode an input comprising a labelled pointcloud representation, as a latent representation in a first latent space; asecond encoder, ^^ℎ , to encode an input comprising a labelledpoint cloud representation as a point-wise latent representation in a second latent space; and adecoder, ^^ℎ , to decode an input comprising a sampled latentrepresentation from the first latent space and a sampled point-wise latent representation from the second latent space to obtain an output point cloud representation of a three-dimensional object; ii) a first denoising-diffusion model, ^^^^ , to denoise a diffused latentrepresentation from the first latent space; and iii) a second denoising-diffusion model, ^^ℎ , to denoise a diffused point-wiselatent representation from the second latent space; andd) in a first training phase:training the VAE,based on the modified training dataset,{{^^^^,^^^^}, {^^^^, ^^^^}}, wherein the VAE is trained to reconstruct {^^^^, ^^^^},conditioned by {^^^^, ^^^^}; and202417825e) in a second training phase:training the first denoising diffusion model, ^^^^ , and the second denoisingdiffusion model, ^^ℎ , over a pre-determined number of discrete time steps, T,based on output obtained from the first trained encoder, ^^^^ , second trainedencoder ^^ℎ ,wherein the first denoising diffusion model, ^^^^ , and the second denoising diffusionmodel, ^^ℎ , are conditioned based on the labels ^^^^, ^^^^.

2. The method of claim 1, comprising, for each labelled point cloud representation,(^^^^,^^^^) in the first subse { ^^ ^^}^^ ^^ t, ^^ , ^^ :obtaining latent representations of the point cloud representation in the first andsecond latent spaces, based on output obtained from the first trained encoder, ^^^^ , andsecond trained encoder, ^^ℎ ;applying the first trained denoising diffusion models, ^^^^ , and second traineddenoising diffusion model ^^ , for ^^-step forward diffusion and, subsequently, ^^-stepℎbackward denoising to obtain a set of variant latent representation, wherein ^^ ∈ (0.. ^^);applying the decoder, ^^ , to the set of variant latent representations toℎgenerate a set ^^^^ = {^^^^ ∈ ℝ^^×3: ^^ ∈ (0, ..^^)}, of point cloud representations, ^^^^^^ ^^,^^ ^^,^^ ,with labels, ^^^^^^.

3. The method of claim 2, comprising:training a first predictive model, ^^, for inferring a labelling of a point cloud^^ ^^ { } representation of a three-dimensional object, based on the first subset ^^ ,^^ and a^^ ^^ ^ ^ further set of labelled point cloud representations ^^ = ^^ , with corresponding^^^^^^ ^^^^ labels ^^ .

4. The method of claim 3, comprising: applying the trained predictive model, ^^, to^^ each point cloud representation in the second subset ^^ to obtain a set of inferred^^ ^^^ labels, ^^ for the point cloud representations in the second subset, ^^ .2024178255. The method of claim 4, comprising, for each point cloud representation, ^^^^^^ , inthe second subset, ^^^^ and corresponding labels ^^^^^ in ^^^^^:obtaining latent representations of the point cloud representation, ^^^^^^ , in the firstand second latent spaces, based on output obtained from the first trained encoder, ^^^^ ,second trained encoder ^^ℎ ;applying the first trained denoising diffusion models, ^^^^ , and second traineddenoising diffusion model ^^ℎ , for ^^-step forward diffusion and, subsequently, ^^-stepbackward denoising to obtain a set of variant latent representation, wherein ^^ ∈(0, .. , ^^);applying the decoder,to the set of variant latent representations toa set ^^^^ = {^^^^ ^^×3 ^^^,^^ : ∈ ℝ : ^^ ∈ (0, .. ,^^)}, of point cloud representations, ^^ ^^^,^^ , , withlabels, ^^^^^.

6. The method of claim 5, comprising:training a second predictive model, ^^′, for inferring labels in a point cloudrepresentation of a three-dimensional object, based on the first subset {^^^^,^^^^}, thegenerated labelled point cloud representations ^^^^ ^^^^^^^^ = ^^^^^^ with corresponding labels^^^^, and the generated point cloud representations ^^^^^^^^^^ = {^^^^}, with correspondinglabels ^^^^^.

7. The method of claim 6 comprising,accessing a further point cloud representation, ^^^ , of a three-dimensional object;and applying the second trained predictive model, ^^′ to infer labels, ^^^, of the furtherpoint cloud representation, ^^^ .2024178258. The method of claim 1, wherein training the VAE, {^^^^ , ^^ℎ , ^^ℎ }, comprisesmaximizing a variational lower bound on a data log-likelihood function,wherein:and, wherein ^^^^^^ , ^^^^ℎ are posterior distributions for sampling a latent representation z0in the first latent space and a latent representation, h0, in the second latent space; ^^^^ℎis the prior distribution for reconstruction prediction; ^^^^^^ is a Kullback-Leiblerregularization terms; λz and λh are hyperparameters for balancing the Kullback-Leiblerregularization terms and reconstruction accuracy; and ^^(0, ^^) is a normal distribution.

9. The method of claim 1, wherein training the first denoising diffusion model, ^^^^ ,comprises minimizing an objective function, ℒ(^^^^), wherein:and wherein ^^0 = ^^^^(^^,^^); ^^ ∈ {^^^^,^^^^}; ^^ ∈ {^^^^, ^^^^}; ^^ ~ ^^(0, ^^) and t is a timestep uniformly sampled from {1 …^^}.

10. The method of claim 1, wherein training the second denoising diffusion model,^^ℎ , comprises minimizing an objective function, ℒ(^^ℎ), wherein:and whereintime step uniformly sampled from {1 …^^}.

11. The method of claim 1, wherein N, is less than or equal to M / 10.

12. The method of claim 1, wherein the VAE, and the first and second denoisingdiffusion models, ^^^^ , ^^ℎ are artificial neural networks.202417825 13. A data processing system comprising: a processor; and a memory storing instructions that, when executed by the processor cause the processor to execute the method according to any one of claims 1 to 12.

14. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 12.

Images