Multi-domain matching point cloud joint generation method and system based on hierarchical diffusion model, and related device

Abstract

The invention provides a multi-domain matching point cloud joint generation method and system based on a hierarchical diffusion model and a related device, and the method comprises the following steps: constructing a single-domain point cloud diffusion model which is used for mapping an input point cloud into a corresponding potential variable, and learning a process of recovering an original point cloud from a noise point cloud; constructing a joint potential spatial diffusion model, wherein the joint potential spatial diffusion model is used for modeling joint distribution and correlativity among a plurality of point cloud domains in a potential space; based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer, realizing joint generation or condition generation of matched point cloud pairs; according to the method, through hierarchical design, the matching rationality, training stability and reasoning efficiency among multi-domain objects are remarkably improved while the generation quality and diversity are guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of 3D point cloud generation and modeling technology, specifically relating to a method, system and related apparatus for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model. Background Technology

[0002] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), digital humans, intelligent manufacturing, and autonomous driving, the ability to automatically generate and edit 3D content is becoming increasingly important. 3D point clouds, as a crucial data format for describing the geometry and spatial structure of objects, have been widely applied in scenarios such as virtual try-on, virtual photography, game development, industrial assembly simulation, and robot grasping. In these applications, systems often need to generate not only the 3D shape of a single object, but also combinations of multiple objects with semantic or geometric relationships, such as pairwise point cloud data of the human body and clothing, the human body and a chair, a table and a cup placed on it, or a robot and its tools.

[0003] To improve the quality of 3D generation, various 3D generation models have been proposed in existing technologies, such as voxel-based generation models, mesh-based generation models, implicit field-based generation models, and point cloud-based generative adversarial networks (GANs), variational autoencoders (VAEs), normalizing flows, and diffusion models. These methods have achieved good results in generating the shape of single objects, producing 3D point clouds or meshes with reasonable local details and overall structure. However, most of these methods focus on single-domain or single-class objects, and they still fall short when it is necessary to simultaneously model the interaction relationships, geometric constraints, and semantic consistency between multiple objects.

[0004] In practical applications, multi-domain paired point clouds often exhibit close and complex relationships. For example, in virtual try-on and digital human applications, the shape, size, and posture of the human body and clothing need to be highly matched; otherwise, situations may arise where clothing and the human body intertwine, are severely suspended, or are overstretched. In indoor scenes, the sitting posture of the person and chair, and the relative positions of the table and tabletop objects, also need to meet basic physical and functional constraints; otherwise, the generated scene will lack realism and practical value. Simply generating each object independently and then combining them through simple geometric overlay often fails to guarantee the rationality of the overall configuration.

[0005] To address the lack of collaborative modeling among multiple objects, some existing methods attempt to introduce conditional generation or joint modeling approaches. One type of method uses the shape or features of the first object as conditional input when generating a second object, aiming to generate a matching second object under conditional constraints. Another type of method directly concatenates the representations of multiple objects and performs joint modeling within a unified network or diffusion model. While these methods improve the consistency of pairwise or multi-object generation to some extent, they still suffer from insufficient control granularity, high computational costs, and difficulty in simultaneously considering diversity and reasonable coordination. In particular, directly performing joint diffusion or adversarial training in high-dimensional point cloud space can easily lead to training instability, difficulty in guaranteeing generation quality, and difficulty in flexibly supporting the conditional generation requirement of "giving one object and generating multiple objects with different combinations".

[0006] In summary, in the task of joint generation of multi-domain pairwise point clouds, how to effectively model the joint distribution and correlation between different object domains while maintaining the quality and diversity of generation, and how to achieve flexible and fine-grained condition control in the inference stage, remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, and related apparatus for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model. This invention solves the problems in the prior art where multiple objects are usually modeled independently, making it difficult to characterize the joint distribution relationship between objects, and where joint generation in high-dimensional point cloud space leads to excessive computational overhead and unstable training.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model, comprising the following steps: A single-domain point cloud diffusion model is constructed, which is used to map the input point cloud to the corresponding latent variables and learn the process of recovering the original point cloud from the noisy point cloud. A joint latent space diffusion model is constructed, which is used to model the joint distribution and correlation among multiple point cloud domains in the latent space; Based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer, point cloud pairs that are jointly generated or conditionally generated and matched are realized.

[0009] Preferably, the single-domain point cloud diffusion model includes a point cloud encoder, a normalized flow module, and a point cloud spatial noise predictor, wherein: The point cloud encoder is used to map multiple point clouds to be matched into latent variables and construct the posterior distribution corresponding to the latent variables. The normalized flow module is used to learn the invertible prior distribution of the latent variables and align the obtained invertible prior distribution with the posterior distribution of the latent variables. The point cloud spatial noise predictor is used to predict noise in the point cloud space given latent variables and time steps.

[0010] Preferably, the joint latent spatial diffusion model includes a joint latent spatial noise predictor, which is used to model the joint distribution of latent variable pairs in multiple point cloud domains.

[0011] Preferably, the specific method for jointly generating matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent spatial diffusion layer is as follows: The noise variables corresponding to multiple point cloud domains to be matched are denoised and split using a joint latent spatial diffusion layer to generate latent variables corresponding to each point cloud domain. A single-domain point cloud diffusion layer is used to generate a matching point cloud from multiple latent variables.

[0012] Preferably, the specific method for conditionally generating matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent spatial diffusion layer is as follows: Based on multiple point cloud domains to be matched, set the point cloud of any one of the point cloud domains; The set point cloud is used to generate corresponding latent variables using a single-domain point cloud diffusion layer; By using a joint latent spatial diffusion layer, the noise variables corresponding to the remaining point cloud domains in the multiple point cloud domains to be matched are denoised and split to generate the latent variables corresponding to each point cloud domain. The obtained latent variables are used to generate matching point clouds using the corresponding single-domain point cloud diffusion layer.

[0013] Secondly, the present invention provides a multi-domain matching point cloud joint generation system based on a hierarchical diffusion model, comprising: A single-domain model building unit is used to build a single-domain point cloud diffusion model, which maps the input point cloud to the corresponding latent variables and learns the process of recovering the original point cloud from the noisy point cloud. A joint model building unit is used to construct a joint latent space diffusion model, which is used to model the joint distribution and correlation between multiple point cloud domains in the latent space. The point cloud pair generation unit is used to jointly generate or conditionally generate matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the method described thereon.

[0015] Fourthly, the present invention provides a computing device cluster, comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to the method.

[0016] Fifthly, the present invention provides a computer program product, the computer program product including computer-executable instructions, which, when executed, implement the method described.

[0017] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described herein.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. By learning geometric details within each object domain using a point cloud diffusion model, and then explicitly modeling the statistical correlations and geometric constraints between multiple domains in a low-dimensional joint latent space, this method effectively avoids unreasonable phenomena such as interleaving and dangling that occur in existing single-domain generation methods during combination. It significantly improves the coordination of multi-domain objects such as human-clothing and table-object in terms of pose, scale, and spatial relationships. By compressing high-dimensional point clouds into a low-dimensional latent space for joint diffusion, computational complexity and memory overhead are greatly reduced, training stability and inference speed are enhanced, and the gradient oscillation and convergence difficulties that easily occur when directly modeling in high-dimensional point cloud space are overcome. Furthermore, this invention supports a decoupled conditional generation mechanism. Given a point cloud in one domain, by fixing its latent variables and back-sampling the latent variables of another domain in the joint latent space, fine-grained multi-directional matching and diverse combinations can be achieved, flexibly adapting to different application scenarios without retraining the model. The framework adopts a modular design, with each domain encoding and decoding module and the shared joint diffusion module being independent of each other. This facilitates expansion to new object combinations and task scenarios, improving the versatility and engineering usability of the method. It provides an efficient and reliable solution for applications that require multi-object collaborative generation, such as virtual try-on, scene compositing, and robot grasping.

[0019] In summary, this invention introduces a hierarchical structure of "intra-domain point cloud diffusion + latent space joint diffusion", which significantly improves the rationality of multi-domain paired point cloud cooperation and generation efficiency while ensuring generation quality. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall framework of the multi-domain pairwise point cloud joint generation method involved in the embodiments of the present invention; Figure 2 This is a schematic diagram of the single-domain point cloud diffusion encoding and decoding process involved in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the joint latent space diffusion modeling module involved in an embodiment of the present invention; Figure 4 This is a schematic diagram of the condition generation process based on joint latent space involved in an embodiment of the present invention. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] Example 1 This embodiment provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. First, within each object domain, the reversible relationship of "point cloud - latent code - noise" is learned through the point cloud diffusion model. Then, a one-dimensional latent diffusion model is constructed in the joint latent space composed of multi-domain latent codes to explicitly model the joint distribution of paired latent codes. This achieves both the generation of high-quality single-domain point clouds and the generation of paired point clouds with reasonable cooperation between multiple domains.

[0028] like Figure 1 As shown in the figure, this embodiment provides a method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model, including: A single-domain point cloud diffusion layer is constructed, which is used to map the input point cloud to the corresponding latent variables and learn the process of recovering the original point cloud from the noisy point cloud. A joint latent space diffusion layer is constructed, which is used to model the joint distribution and correlation among multiple point cloud domains in the latent space; Based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer, point cloud pairs that are jointly generated or conditionally generated and matched are realized.

[0029] Example 2 Based on Example 1, this example provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. The pre-constructed joint latent spatial diffusion layer includes a joint latent spatial noise predictor, which is used to model the joint distribution of latent variable pairs in multiple point cloud domains.

[0030] Example 3 Based on Example 1, this example provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. The single-domain point cloud diffusion layer includes a point cloud encoder, a normalized flow module, and a point cloud spatial noise predictor, wherein: The point cloud encoder is used to map multiple point clouds to be matched into latent variables and construct the posterior distribution corresponding to the latent variables. The normalized flow module is used to learn the invertible prior distribution of the latent variables and align the obtained invertible prior distribution with the posterior distribution of the latent variables. The point cloud spatial noise predictor is used to predict noise in the point cloud space given latent variables and time steps.

[0031] Example 4 Based on Example 3, this example provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. The point cloud encoder includes a multi-layer point feature extraction unit, a global convergence unit, and several fully connected layers, wherein: The multi-layer point feature extraction unit is used to extract geometric features from the local neighborhood to obtain local features; The global aggregation unit is used to aggregate the extracted local features to obtain global features; The fully connected layer is used to map the obtained global features to latent space parameters.

[0032] Example 5 Based on Example 1, this example provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. The specific method for jointly generating matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent space diffusion layer is as follows: The noise variables corresponding to multiple point cloud domains to be matched are denoised and split using a joint latent spatial diffusion layer to generate latent variables corresponding to each point cloud domain. A single-domain point cloud diffusion layer is used to generate a matching point cloud from multiple latent variables.

[0033] Example 6 Based on Example 1, this example provides a multi-domain matching point cloud joint generation method based on a hierarchical diffusion model. The specific method for conditionally generating matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent space diffusion layer is as follows: Based on multiple point cloud domains to be matched, set the point cloud of any one of the point cloud domains; The set point cloud is used to generate corresponding latent variables using a single-domain point cloud diffusion layer; By using a joint latent spatial diffusion layer, the noise variables corresponding to the remaining point cloud domains in the multiple point cloud domains to be matched are denoised and split to generate the latent variables corresponding to each point cloud domain. The obtained latent variables are used to generate matching point clouds using the corresponding single-domain point cloud diffusion layer.

[0034] Example 7 This application can be extended to any number and type of object domains, such as human body-clothing, table-object, robot-tool, etc. For ease of explanation, this embodiment uses two object domains as an example for detailed description, denoting the first point cloud domain as a point cloud. (For example, human body point cloud), the second point cloud region is denoted as point cloud. (For example, dot clouds in clothing). Superscript This indicates that the point cloud has not yet undergone a diffusion process and is a "clean" original point cloud. In a typical embodiment, the training dataset can be represented as... , among which, each pair These are paired samples that are registered in physical space or logically corresponding. This represents the number of samples in the dataset.

[0035] This embodiment provides a method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model, including the following steps: Step 1: Construct a single-domain point cloud diffusion layer This application independently trains single-domain point cloud diffusion models in the first and second point cloud domains, using the same modeling method but different network parameters. The following discussion uses the point cloud in the first point cloud domain as an example. The process will be explained using an example, and the second point is cloud domain. The modeling method is completely symmetrical.

[0036] 2.1 Point Cloud Preprocessing and Representation Input 3D point cloud Composed of several three-dimensional points, denoted as: Indicating the first point cloud The coordinates of the points This indicates the number of points in the first point cloud region.

[0037] To reduce scale and location differences and improve the numerical stability of network training, this embodiment performs preprocessing steps such as centering, scale normalization, and point resampling on each point cloud to obtain a standardized first point cloud domain. Its input dimension and value range are uniform throughout the entire dataset, facilitating batch training and inference. Second point cloud domain point cloud. Preprocessing can be performed using the same procedure.

[0038] 2.2 Point Cloud Encoder and Latent Posterior Distribution like Figure 2 As shown, this embodiment configures a point cloud encoder for each point cloud domain to process high-dimensional point clouds. Mapping to low-dimensional latent variables And explicitly model the posterior distribution of the latent variables.

[0039] In one specific embodiment, a point cloud encoder may include: a multi-layer point feature extraction unit for extracting geometric features from local neighborhoods; a global pooling unit (e.g., max pooling or attention aggregation) for pooling local features into global features; and several fully connected layers for mapping the global features to latent space parameters, namely the mean vector and log-variance vector of the posterior distribution. Let the parameters of the point cloud encoder be... Then the posterior distribution is defined as: , in, Describes the latent posterior distribution. and These are the mean vector and the diagonal covariance matrix, respectively.

[0040] During training, this embodiment employs a reparameterization technique to sample latent variables from the Gaussian posterior. : .

[0041] in, This represents noise sampled from a standard Gaussian distribution. This indicates element-wise multiplication.

[0042] In this way, the end-to-end differentiability of the entire network is maintained on the one hand, and the latent space can automatically adapt to the shape distribution of different point cloud samples on the other hand.

[0043] Second point: cloud domain point clouds Use another set of parameters The point cloud encoder undergoes the same processing to obtain the latent variables of the second point cloud domain. .

[0044] 2.3 Normalized Flow Learning with Flexible Priors To enhance the expressive power of the latent space and avoid overly restrictive distribution of latent variables due to simply fixing the priors, this embodiment introduces a normalized flow in each point cloud domain to learn complex and reversible prior distributions.

[0045] Taking the first cloud region as an example, let the normalized flow transformation be: , in, It consists of several cascaded reversible affine coupling layers, reversible activation functions, and scale translation units. Let these be the learnable parameters of the normalized flow in the first cloud domain. In the transformed space, auxiliary variables are defined. Follows a standard Gaussian distribution: .

[0046] Based on the density transformation formula of the invertible transformation, the prior distribution of the latent variables can be obtained: , in, This represents the prior probability density of the latent variables in the first cloud domain. This indicates determinant operations.

[0047] To make the encoded posterior distribution Aligning with this prior distribution, this embodiment adds a KL divergence regularization term during the training phase: , This indicates the KL loss of the first cloud domain. This represents the KL divergence operator.

[0048] By minimizing this term, the latent variable is... exist It exhibits a complex multimodal structure while maintaining an invertible mapping relationship with the standard Gaussian distribution, which facilitates the splicing of multi-domain latent variables and noise injection in a unified Gaussian space.

[0049] Secondly, the cloud domain also uses normalized flow. The prior distribution of the second cloud region is obtained. and its KL regularization terms .

[0050] 2.4 Point Cloud Spatial Forward Diffusion and Noise Scheduling In the process of single-domain point cloud diffusion, this embodiment defines the diffusion process from clean point cloud in the point cloud space. To noisy point clouds The forward diffusion sequence. Given a preset noise schedule. , To control the first The noise intensity injected at each time step can adopt linear increasing, cosine increasing or other monotonic strategies. In different embodiments, the choice can be made according to data complexity and training stability. This represents the total number of time steps for diffusion.

[0051] definition: .

[0052] At time step Above, noisy clouds It can be constructed in one step using a closed-form formula: .

[0053] This formula allows direct input from... during training. Noisy cloud sampled at any time step in one step This eliminates the need to explicitly simulate the entire forward diffusion path, thus significantly reducing the data generation overhead during training.

[0054] Second point: cloud domain point clouds The forward diffusion process in the point cloud space is the same, using the same noise scheduling.

[0055] 2.5 Noise Predictor and Total Loss in a Single Domain To recover a clean point cloud from a noisy point cloud, this embodiment trains a noise predictor in each point cloud domain. Taking the first point cloud domain as an example, the noise predictor is denoted as... Its parameters are The input is a noisy cloud. Time step embedding and the corresponding latent variables The output is a noise response. The predicted value.

[0056] The noise predictor can employ a U-Net-based point cloud network: the encoding path progressively downsamples point cloud features to capture the global structure, while the decoding path progressively upsamples and fuses shallow details, thus balancing local geometry and global semantics; time step These are then incorporated into the features of each layer via embedding vectors to reflect the distribution differences of different noise levels; latent variables It can be used as a conditional vector concatenated with intermediate features or applied to feature channels through affine modulation.

[0057] The training objective is expressed in the form of mean squared error: , in, For the first domain noise regression loss, To generate The actual noise used at that time.

[0058] Taking into account both latent spatial prior alignment and point cloud diffusion denoising, the total loss in the first point cloud domain in this embodiment can be written as: , in, Let the total loss function be the first point cloud domain. This is a balancing factor used to weigh the latent spatial alignment quality against the point cloud denoising accuracy. A similar total loss can be defined in the second point cloud domain. .

[0059] In practice, stochastic gradient descent or adaptive optimization algorithms (such as Adam) can be used to minimize the above loss until the single-domain diffusion model converges on the training set.

[0060] III. Second Phase: Joint Potential Spatial Diffusion Modeling In the first stage, the first and second point cloud domains acquired their respective latent spaces and diffusion generation capabilities. However, if latent variables are sampled independently from their respective prior distributions during inference... If the point clouds corresponding to these two latent variables lack cross-domain coupling, it is difficult to guarantee mutual matching. Therefore, this embodiment performs cross-domain coupling on the latent variable pairs in the latent space. Explicit modeling is performed on the joint distribution.

[0061] 3.1 Construction of Paired Latent Variables For each pair of samples in the training dataset First, the "clean" latent variables of the first and second point cloud domains are obtained using the point cloud encoder trained in the first stage. and : .

[0062] Then, the two are concatenated in a fixed order to form a joint latent variable. : .

[0063] The set of joint latent variables corresponding to all training samples This refers to the true data distribution in the joint potential space, which is used to train the subsequent joint potential diffusion model.

[0064] 3.2 Joint potential space forward diffusion like Figure 3 As shown, this embodiment constructs a one-dimensional potential diffusion process in the joint potential space. Given a set of joint noise schedules... ,definition: .

[0065] At time step The forward noise addition process for the joint latent variables is as follows:

[0066] in, Maintain and The same dimension includes two parts: the first point cloud domain latent variables and the second point cloud domain latent variables. With time steps... The increase, The distribution gradually approaches a standard Gaussian distribution, eventually reaching... Approximately isotropic noise.

[0067] Compared to constructing joint diffusion directly in high-dimensional point cloud space, joint diffusion in latent space significantly reduces the state dimension, thereby reducing computational complexity and improving the stability of the training process.

[0068] 3.3 Joint Noise Predictor and Training Objective To recover the joint latent variables from noise, this embodiment constructs a one-dimensional joint noise predictor. The parameter is denoted as At time step At that time, the input to the joint noise predictor is the noisy joint latent variable. and time step embedding The output is a noise response. Predicted values: .

[0069] During training, time steps are randomly selected. and noise Generate according to the forward diffusion formula And minimize the following joint diffusion loss: .

[0070] The loss Consistent with the training objective of the standard diffusion model, this is equivalent to minimizing the variational lower bound of the joint latent diffusion model. After training convergence, the joint noise predictor can accurately predict noise levels at different noise levels. By recovering the implicit structure, the latent variables are indirectly learned. The joint distribution of .

[0071] IV. Joint Generative Reasoning Process In the joint generation mode, this embodiment does not rely on any input point cloud, but directly generates a pair of matching point clouds from the noise in the joint latent space. The overall process is as follows: Figure 1 As shown in the diagram, the key steps are as follows: 1. Perform inverse denoising of the joint latent space and decompose it into two-domain latent variables. In the reverse generation stage, from the initial noise Starting from this point, the update formula for joint potential diffusion is: .

[0072] in The standard deviation coefficient, determined based on the preset noise scheduling, is used to control the back diffusion process during the [missing information - likely a specific step or event]. The intensity of randomness injected at each time step, typically , It is standard Gaussian noise.

[0073] From time step Iterate step by step to time step The denoised joint latent variables are obtained. This process is equivalent to sampling a pair of correlated latent variables from the joint latent space according to the learned joint distribution. The latent variables are split into two segments according to the predetermined dimensions to obtain the first point of cloud domain latent variables. Second point cloud domain latent variables .

[0074] 2. Single-domain point cloud diffusion decoding to generate point clouds For the first point cloud region, initialize the point cloud with standard Gaussian noise. , with latent variables As a condition, the first point cloud domain noise predictor is executed in the reverse order of the training phase. Step-by-step reverse diffusion, gradually denoising to obtain the generated point cloud. Similarly, for the second point cloud region, from the noisy point cloud... Starting with latent variables As a condition, perform backdiffusion to generate point clouds. .

[0075] because The point clouds generated are from the same sampling from the joint potential diffusion model, and statistically correspond to the same joint distribution sample. They exhibit high consistency and compatibility in terms of geometry, orientation, and spatial position.

[0076] V. Condition Generation Process Based on Joint Potential Space In the context of conditional generation, the goal of this embodiment is to generate a matching second point cloud domain (e.g., clothing point cloud) given a first point cloud domain (e.g., human body point cloud). To this end, this embodiment designs a conditional latent diffusion reasoning process based on a joint latent space. Its core feature is: clean latent variables in the first point cloud domain. The state remains constant, with noise added only at each time step to obtain an intermediate state. Used as a condition; second point: cloud domain latent variable Then, by progressively denoising along the reverse of the joint potential diffusion, it eventually converges to the same level. Matching latent variables .

[0077] The entire condition generation process is coordinated Figure 4 The process can be summarized as follows.

[0078] 1. Conditional Domain Point Cloud Encoding First, the given first point cloud region point cloud Input the point cloud encoder and normalized flow module of the first point cloud domain to obtain its clean latent variables in the latent space. : .

[0079] Throughout the condition generation process, this latent variable It no longer changes and is continuously used as conditional information.

[0080] 2. Initialization of the second point: cloud domain latent variables Initial noise of the second-point cloud region latent variable sampled from a standard Gaussian distribution. As a diffusion time step The second point of the potential state of the cloud domain at time t. Here This represents the maximum time step for joint potential diffusion.

[0081] 3. Conditional joint potential diffusion reverse process like Figure 4 As shown, at time step from During the decreasing process, for each time step Perform the following two updates until the time step decreases to... : (a) Fixing latent variables Perform forward noise addition to obtain the conditional noise state. : Noise scheduling using joint potential diffusion ,definition: .

[0082] At time step Fixed clean latent variables A one-time mapping to the state corresponding to the noise level: .

[0083] Although It does not participate in back diffusion itself, but at each time step, it constructs a conditional representation consistent with the current noise level through closed-loop forward noise addition. For use by the joint noise predictor.

[0084] (b) under the conditions Next, perform one-step inverse denoising on the second point of the cloud domain latent variables. Let the joint latent diffusion noise predictor be... In its implementation, it can output a vector of the same length as the joint latent variables. In this embodiment, the output corresponding to the second point cloud domain latent variable part is denoted as... At time step Given the current second-point cloud domain latent variables With conditions According to the following formula Perform a reverse update: .

[0085] The above formula is consistent with the backsampling formula of the standard diffusion model, but differs from the unconditional case in that the noise prediction term... The input explicitly depends on the conditional noise state. Therefore, the update trajectory of the second point cloud domain latent variable is forcibly constrained to be consistent with that of the given first point cloud domain latent variable. Near the matching joint distribution area.

[0086] By time step Repeat the above two steps, and finally... When conditional latent variables are obtained This latent variable is statistically similar to the latent variable in the given first point cloud domain. It exhibits strong cohesion and consistency.

[0087] 4. Target Domain Point Cloud Generation After completing the reverse update of the potential diffusion, this embodiment will obtain the second cloud domain latent variable. In the single-domain point cloud diffusion decoder input to the second point cloud domain, following the same point cloud spatial reverse diffusion process as in the first stage, from the noisy point cloud... Gradual denoising to generate target point cloud .

[0088] If it is necessary to reconstruct the first cloud domain, the fixed one can also be used. Input the point cloud diffusion reconstruction of the first point cloud domain In most applications, the original input point cloud can also be directly used. With the generated point cloud Combining these elements forms the final conditional generation of paired point clouds. .

[0089] Through the aforementioned conditional generation process, this embodiment utilizes a joint latent diffusion model to explicitly introduce fixed first point cloud domain latent variables as conditions at each noise level. It then performs progressive reverse denoising updates on the second point cloud domain latent variables, ensuring that the final generated second point cloud domain maintains a high degree of matching with the given first point cloud domain in terms of geometric structure and semantic attributes. This achieves high-quality conditional pairwise point cloud generation. This conditional generation process can also be extended to generate a first point cloud domain from a given second point cloud domain, or, in multi-domain scenarios, to generate the remaining point cloud domains from given partial point cloud domains, simply by changing the fixed latent variable components accordingly. The overall design of this embodiment exhibits good symmetry and scalability.

[0090] In summary, this embodiment achieves high-quality single-domain generation in the point cloud space by constructing a point cloud encoder, a normalized flow, and a point cloud diffusion noise predictor within the first and second point cloud domains, respectively. Furthermore, it introduces a joint latent diffusion and joint noise predictor into the joint latent space composed of spliced ​​latent variables from multiple domains to explicitly model the joint distribution of latent variable pairs. Finally, in the inference stage, it can achieve the joint generation of point clouds by directly sampling noise, and can also generate a point cloud of another domain that matches a given point cloud in a certain domain.

[0091] The hierarchical diffusion structure proposed in this embodiment effectively decouples "geometric detail modeling of high-dimensional point clouds" from "joint relationship modeling between multiple domains." While ensuring generation quality and diversity, it significantly reduces the computational complexity of joint modeling, improves the stability of training and inference, and has good generalization and scalability. It can be widely applied to various point cloud generation and understanding tasks such as human-clothing collaborative modeling, robot and tool co-design, and collaborative generation of multiple objects in a scene.

[0092] Example 8 This embodiment provides a multi-domain matching point cloud joint generation system based on a hierarchical diffusion model, including: A single-domain model building unit is used to build a single-domain point cloud diffusion model, which maps the input point cloud to the corresponding latent variables and learns the process of recovering the original point cloud from the noisy point cloud. A joint model building unit is used to construct a joint latent space diffusion model, which is used to model the joint distribution and correlation between multiple point cloud domains in the latent space. The point cloud pair generation unit is used to jointly generate or conditionally generate matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer.

[0093] Example 9 This embodiment also provides a computing device. The computing device includes a bus, a processor, a memory, and a communication interface. The processor, memory, and communication interface communicate with each other via the bus. The computing device can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device.

[0094] A bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, a bus can include a path for transmitting information between various components of a computing device (e.g., memory, processor, communication interfaces).

[0095] The processor may include any one or more of the following: central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), microprocessor (MP), or digital signal processor (DSP).

[0096] Memory can include volatile memory, such as random access memory (RAM). Processors can also include non-volatile memory. volatile memory, such as read-only memory (ROM). ROM (memory only), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0097] The memory stores executable program code, which the processor executes to implement the functions of the aforementioned units, thereby achieving, for example, the method described in Embodiment 1. That is, the memory may store instructions for the methods and functions relating to the computing device in any of the above embodiments.

[0098] The communication interface uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between computing devices and other devices or communication networks.

[0099] Example 10 This embodiment also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0100] The computing device cluster includes at least one computing device. The memory of one or more computing devices in the computing device cluster may store the same instructions for performing the methods and functions related to the computing devices in any of the above embodiments.

[0101] In some possible implementations, the memory of one or more computing devices in the computing device cluster may also store partial instructions for performing the methods and functions of the computing devices involved in any of the above embodiments. In other words, a combination of one or more computing devices can jointly execute the instructions for performing the methods and functions of the computing devices.

[0102] It should be noted that the memory in different computing devices within a computing device cluster can store different instructions, which are used to execute parts of the device's functions.

[0103] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Two computing devices are connected to each other via the network. Specifically, they connect to the network through communication interfaces in each computing device.

[0104] Embodiments of this disclosure also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions related to a computing device in any of the above embodiments.

[0105] Example 11 This embodiment also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, cause the processor to perform the methods and functions of the computing device involved in any of the above embodiments.

[0106] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0107] Example 12 This embodiment provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods as described above with reference to the accompanying drawings. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0108] Computer program code used to implement the methods of this disclosure may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the computer or other programmable data processing apparatus, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be performed. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0109] In the context of this disclosure, computer program code or related data may be carried on any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and so on. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0110] Computer-readable media can be any tangible medium that contains or stores programs for or relating to an instruction execution system, apparatus, or device, or a data storage device such as a data center containing one or more available media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0111] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model, characterized in that, Includes the following steps: A single-domain point cloud diffusion model is constructed, which is used to map the input point cloud to the corresponding latent variables and learn the process of recovering the original point cloud from the noisy point cloud. A joint latent space diffusion model is constructed, which is used to model the joint distribution and correlation among multiple point cloud domains in the latent space; Based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer, point cloud pairs that are jointly generated or conditionally generated and matched are realized.

2. The method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model according to claim 1, characterized in that, The single-domain point cloud diffusion model includes a point cloud encoder, a normalized flow module, and a point cloud spatial noise predictor, wherein: The point cloud encoder is used to map multiple point clouds to be matched into latent variables and construct the posterior distribution corresponding to the latent variables. The normalized flow module is used to learn the invertible prior distribution of the latent variables and align the obtained invertible prior distribution with the posterior distribution of the latent variables. The point cloud spatial noise predictor is used to predict noise in the point cloud space given latent variables and time steps.

3. The method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model according to claim 1, characterized in that, The joint latent spatial diffusion model includes a joint latent spatial noise predictor, which is used to model the joint distribution of latent variable pairs in multiple point cloud domains.

4. The method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model according to claim 1, characterized in that, The specific method for jointly generating matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent spatial diffusion layer is as follows: The noise variables corresponding to multiple point cloud domains to be matched are denoised and split using a joint latent spatial diffusion layer to generate latent variables corresponding to each point cloud domain. A single-domain point cloud diffusion layer is used to generate a matching point cloud from multiple latent variables.

5. The method for joint generation of multi-domain matching point clouds based on a hierarchical diffusion model according to claim 1, characterized in that, The specific method for generating conditional matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint latent spatial diffusion layer is as follows: Based on multiple point cloud domains to be matched, set the point cloud of any one of the point cloud domains; The set point cloud is used to generate corresponding latent variables using a single-domain point cloud diffusion layer; By using a joint latent spatial diffusion layer, the noise variables corresponding to the remaining point cloud domains in the multiple point cloud domains to be matched are denoised and split to generate the latent variables corresponding to each point cloud domain. The obtained latent variables are used to generate matching point clouds using the corresponding single-domain point cloud diffusion layer.

6. A multi-domain matching point cloud joint generation system based on a hierarchical diffusion model, characterized in that, include: A single-domain model building unit is used to build a single-domain point cloud diffusion model, which maps the input point cloud to the corresponding latent variables and learns the process of recovering the original point cloud from the noisy point cloud. A joint model building unit is used to construct a joint latent space diffusion model, which is used to model the joint distribution and correlation between multiple point cloud domains in the latent space. The point cloud pair generation unit is used to jointly generate or conditionally generate matching point cloud pairs based on the single-domain point cloud diffusion layer and the joint potential spatial diffusion layer.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 5.

8. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes computer-executable instructions that, when executed, implement the method of any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method of any one of claims 1 to 5.

Images