Conditional three-dimensional layout prediction

Abstract

The present disclosure relates, inter alia, to a computer-implemented method using a machine learning function that is pre-configured to employ an input 3D layout and a given noise level and predict an output 3D layout. The function is pre-configured with conditional discard regarding at least one layout parameter. The method further includes obtaining a set of condition inputs; and for each condition input, determining one or more condition candidate 3D layouts and determining a plurality of perturbed condition candidate 3D layouts. The method further includes applying a pre-configuration function to each perturbed condition candidate, where one layout parameter is discarded; reconstruction errors are calculated and averaged to obtain a score. This forms an improved solution for predicting a 3D layout.

Description

Technical Field

[0001] This disclosure relates to the field of computer programs and systems, and more specifically to methods, data structures and systems related to 3D layout prediction. Background Technology

[0002] Some available solutions for generating 3D scenes involve machine learning techniques, such as Generative Adversarial Networks (GANs).

[0003] Current technologies face significant limitations in achieving realistic and diverse scene arrangements. A realistic scene arrangement means that the scene configuration is semantically consistent and physically meaningful between objects and between objects and the floor plan. In other words, a realistic scene arrangement tends to foster strong semantic relationships between groups of objects and with the floor plan, and tends to place objects in a consistent physical manner. Therefore, a realistic scene arrangement often prevents: for example, placing a bed in the kitchen (object semantically inconsistent with the floor plan), placing an oven next to the bed (objects semantically inconsistent with each other), and placing objects without the necessary physical support (e.g., a teacup suspending itself instead of being placed on a table).

[0004] In this context, there is still a need for improved solutions for predicting 3D layouts. Summary of the Invention

[0005] Therefore, a method is provided using a computer implementation pre-configured to take an input 3D layout and a given noise level as a machine learning function, and is referred to hereinafter as the "usage method". The usage method includes obtaining the machine learning function. The 3D layout has a set of layout parameters including a planar graph, a 3D arrangement of one or more 3D bounding boxes, and a semantic category for each 3D bounding box. Each bounding box is defined in the 3D arrangement by a predetermined set of values ​​for one or more bounding box parameters. The input 3D layout includes a given planar graph, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the first 3D arrangement by a first value of the predetermined set of one or more bounding box parameters. The function is also pre-configured to predict an output 3D layout. The output 3D layout includes a given planar graph, a second 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of the predetermined set of one or more bounding box parameters. The function is configured to predict a second value of the predetermined set of one or more bounding box parameters that differs from the first value in the predetermined set of one or more bounding box parameters. The function is also pre-configured with conditional drop-out for at least one layout parameter, which includes a planar graph and / or a semantic category for each 3D bounding box. The method of use also includes obtaining a set of conditional inputs. Each conditional input includes a distinct value for one of the at least one layout parameter and the same value for each of the other layout parameters in the planar graph and the semantic category for each 3D bounding box. The method of use also includes determining one or more conditional candidate 3D layouts for each conditional input. Each conditional candidate 3D layout is the result of iterative sampling using the pre-configured function. The method of use also includes determining multiple perturbed conditional candidate 3D layouts for each conditional input. Each perturbed conditional candidate 3D layout is determined by adding appropriate noise to the corresponding conditional candidate 3D layout. The method of use also includes applying the pre-configured function to the perturbed conditional candidate 3D layout for each conditional input and each perturbed conditional candidate 3D layout, wherein one layout parameter is dropped, thereby obtaining a corresponding unconditional output. The method also includes: for each conditional input and for each perturbed conditional candidate 3D layout, calculating the reconstruction error between the corresponding conditional candidate 3D layout and the corresponding unconditional output. The method further includes: for each conditional input, averaging the reconstruction errors over multiple perturbed conditional candidate 3D layouts to obtain a score.

[0006] This usage method may include one or more of the following features:

[0007] - Iterative sampling using pre-configured functions includes iteration:

[0008] • Inject noise into the input 3D layout to obtain a perturbed input 3D layout;

[0009] • Apply the pre-configured function to the perturbed input 3D layout at least once to obtain the output 3D layout;

[0010] as well as

[0011] • Use the output 3D layout as input for the next iteration;

[0012] Optionally, the noise has a level that decreases with the depth of the iteration;

[0013] - The pre-configured functions are applied at least once in each iteration, including:

[0014] • Apply the pre-configured function to the perturbed input 3D layout to obtain the first output 3D layout;

[0015] • The first intermediate 3D layout is obtained by calculating the gradient step between the perturbed input 3D layout and the first output 3D layout;

[0016] • Apply the pre-configured function to the first intermediate 3D layout to obtain the second output 3D layout; and

[0017] • The gradient step size between the perturbed input 3D layout and the second output 3D layout is obtained by calculating the gradient step size.

[0018] The two intermediate 3D layouts are used to obtain the final 3D layout.

[0019] - One or more conditional candidate 3D layouts include the final result of iterative sampling;

[0020] Adding appropriate noise to the final result of iterative sampling includes: sampling the noise level and sampling corresponding noise based on the sampled noise level.

[0021] - One or more conditional candidate 3D layouts include one or more intermediate results from iterative sampling;

[0022] - Adding appropriate noise for each intermediate result of iterative sampling includes: sampling the corresponding noise based on the noise level of the intermediate iterations of the iterative sampling corresponding to the intermediate result;

[0023] - The method also includes sorting the conditional candidate 3D layouts, starting with the lowest score, according to the corresponding scores of the conditional candidate layouts; and / or

[0024] - The pre-configured function is parameterized as follows:

[0025]

[0026] in:

[0027] ·x σ This represents the first 3D arrangement of one or more given 3D bounding boxes.

[0028] · Represents a given plan view.

[0029] ·c i It is a list of given semantic categories.

[0030] σ is a given noise level.

[0031] · It is a noise-conditioned score network with a set of trainable parameters θ.

[0032] ·c skip (σ) is the noise-dependent pre-conditioning coefficient for modulation prediction of 3D layout.

[0033] ·c noise (σ) is a noise dependence coefficient that modulates the noise level within the scoring network, and

[0034] ·c in (σ) and c out (σ) are two values ​​scaled by x respectively. σ and S θ (c in (σ)x σ ;y,c noise The noise dependence coefficient of (σ)).

[0035] A method for using machine learning functions in this application is further provided, and is referred to hereinafter as the "machine learning method". The machine learning method includes obtaining a dataset of ground-based 3D layouts. Each ground-based 3D layout represents a corresponding scene and includes a corresponding plan view, a corresponding 3D arrangement of one or more 3D bounding boxes, and a corresponding semantic category for each 3D bounding box, each bounding box being defined by values ​​of a predetermined set of one or more bounding box parameters. The machine learning method also includes obtaining a probability distribution of noise levels. The machine learning method further includes obtaining a corresponding perturbed 3D layout for each ground-based 3D layout, the perturbed 3D layout being computeable by at least one bounding box parameter of at least one 3D bounding box of the perturbed ground-based 3D layout. The perturbation includes sampling the corresponding noise level based on the probability distribution. The perturbation also includes sampling a corresponding noise value based on the corresponding noise level for each of the at least one bounding box parameters, and applying the corresponding noise value to the corresponding bounding box parameter. The machine learning approach also includes training a function on the dataset based on a loss that penalizes a measure of dissimilarity between each ground-real 3D layout and the corresponding predicted 3D layout that can be obtained by applying the function to the corresponding perturbed 3D layout. Training is performed conditionally discarded with respect to at least one layout parameter, which includes a planar graph and / or a semantic category for each 3D bounding box.

[0036] Machine learning methods may include one or more of the following features:

[0037] - Dissimilarity measures have the following types:

[0038]

[0039] in:

[0040] · It is a set of one or more 3D bounding boxes in the ground reality candidate 3D layout.

[0041] · It is a set of one or more 3D bounding boxes in the corresponding predicted 3D layout.

[0042] ·N is and The common size, and

[0043] · It is a differentiable distance.

[0044] Optionally, the differentiable distance further penalizes the dissimilarity between 3D bounding boxes in terms of dimension and / or semantic category, and is, for example, of the following types:

[0045]

[0046] ·in:

[0047] · And x are respectively The vector of spatial bounding box parameter values ​​of o.

[0048] · It is the Euclidean norm.

[0049] K is the penalty parameter.

[0050] ·when When o has the same dimension, It is an index function that equals 1, otherwise it is 0, and

[0051] ·when When o and o have the same semantic category, It is an index function that equals 1, otherwise it is 0.

[0052] - The loss of the training function is a measure of dissimilarity. The product with the noise-dependent weighting function (λ(σ)) Expected value

[0053] A further data structure is provided, comprising: a computer program including instructions for performing the usage method and / or the machine learning method; and / or a machine learning function that has been trained by the machine learning method.

[0054] An apparatus is also provided, which includes a data storage medium on which data structures are recorded.

[0055] The device can form or serve as a non-transitory computer-readable medium, such as on a Software as a Service (SaaS) or other server, or a cloud-based platform. The device may alternatively include a processor coupled to memory having data structures recorded thereon. Therefore, the device can form, wholly or partially, a computer system (e.g., the device is a subsystem of an overall system). The system may also include a graphical user interface coupled to the processor. Attached Figure Description

[0056] A non-limiting example will now be described with reference to the accompanying drawings, in which:

[0057] - Figure 1 A flowchart illustrating an example of using the method is shown;

[0058] - Figure 2 A flowchart illustrating an example of a machine learning method is shown;

[0059] - Figure 3 An example of the architecture of a machine learning function is shown;

[0060] - Figure 4 A schematic diagram illustrating an example of the disclosed solution is shown;

[0061] - Figure 5 and Figure 6 The disclosed solution is shown; and

[0062] - Figure 7 An example of the system is shown. Detailed Implementation

[0063] Reference Figure 1 The flowchart presents a method for computer implementation using machine learning functions.

[0064] The method involves obtaining an S10 machine learning function, which is pre-configured (i.e., pre-trained / already trained) to take an input 3D layout and a given noise level. The 3D layout has a set of layout parameters including a "planar graph" parameter, a "3D arrangement of one or more 3D bounding boxes" parameter, and a "semantic category for each 3D bounding box" parameter. Each bounding box in the 3D arrangement is defined by a predetermined set of values ​​for one or more bounding box parameters. The input 3D layout includes a given planar graph (i.e., a given value for the "planar graph" parameter), a first 3D arrangement of one or more given 3D bounding boxes (i.e., a given value for the "3D arrangement of one or more 3D bounding boxes" parameter), and a given semantic category for each given 3D bounding box (i.e., a given value for the "semantic category for each 3D bounding box" parameter). Each bounding box in the first 3D arrangement is defined by a first value for a predetermined set of one or more bounding box parameters.

[0065] The function is also pre-configured to predict the output 3D layout. The output 3D layout includes a given planar graph, a second 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box in the second 3D arrangement is defined by a second value of a predetermined set of one or more bounding box parameters.

[0066] The function is configured to predict a second value for a predetermined set of one or more bounding box parameters, which differs from a first value in the predetermined set of one or more bounding box parameters. The planar graph and the semantic category for each 3D bounding box are constant variables for the function (i.e., the function does not change its values). In other words, the output of the pre-configured function, which includes the (predicted) second values ​​of the predetermined set of one or more bounding box parameters, is constrained by a given planar graph (value) and a given semantic category (value) for each given 3D bounding box; that is, the output of the pre-configured function is a conditional output. In other words, the conditional output includes a (conditionally predicted) second 3D permutation of one or more given 3D bounding boxes, each bounding box being defined in the (conditionally predicted) second 3D permutation by a (conditionally predicted) second value from the predetermined set of one or more bounding box parameters.

[0067] The output 3D layout can include the conditional output of a pre-configured function; that is, the output 3D layout can include a given planar graph, a (conditionally predicted) second 3D permutation of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box, each bounding box being defined in the second 3D permutation by (conditionally predicted) second values ​​of a predetermined set of one or more bounding box parameters. In this case, the output 3D layout is a conditional 3D layout.

[0068] The function is further pre-configured with conditional discarding regarding at least one layout parameter. In other words, the function has been trained using conditional discarding with respect to the at least one layout parameter, and then the function can be used when the at least one layout parameter is discarded (i.e., the function can be applied even if no value or null value is provided for the at least one layout parameter). The at least one layout parameter pre-configured for conditional discarding includes a planar graph and / or a semantic category for each 3D bounding box, and is, for example, composed of a planar graph, or a semantic category for each 3D bounding box, or both a planar graph and a semantic category for each 3D bounding box. The value of another layout parameter that is not discarded (if any) is a constant variable of the function. In other words, the output of the pre-configured function (including a (predicted) second value of a predetermined set of one or more bounding box parameters) is not constrained by the value of the at least one parameter that has been discarded; that is, the output of the pre-configured function is an unconditional output. In other words, the unconditional output includes a (unconditionally predicted) second 3D permutation of one or more given 3D bounding boxes, each bounding box being defined in the (unconditionally predicted) second 3D permutation by a (unconditionally predicted) second value of a predetermined set of one or more bounding box parameters.

[0069] The method of use also includes obtaining a set of conditional inputs for S20. Each conditional input includes: a distinct value of one layout parameter (labeled P*) from at least one layout parameter (i.e., exactly one layout parameter selected within a set called “at least one layout parameter”, which includes the “planar view” parameter and / or “semantic category for each 3D bounding box”); and the same value for each of the other layout parameters in the planar view and the semantic category for each 3D bounding box (i.e., each of the unselected “planar view” parameter and the “semantic category for each 3D bounding box”).

[0070] The method of use also includes: for each conditional input, determining one or more conditional candidate 3D layouts in S30. Each conditional candidate 3D layout is the result of iterative sampling (e.g., unique) using a pre-configured function.

[0071] Iterative sampling involves applying a pre-configuration function one or more times from the input 3D layout to predict the output 3D layout. In other words, iterative sampling using a pre-configuration function includes iterating the pre-configuration function one or more times.

[0072] The method also includes: for each conditional input, determining S40 perturbation-based candidate 3D layouts. Each perturbation-based candidate 3D layout is determined by adding appropriate (sampled) noise to the corresponding candidate 3D layout.

[0073] The method of use also includes: for each conditional input and each perturbed conditional candidate 3D layout, applying a pre-configured function to the S50 perturbed conditional candidate, wherein a (different / selected) layout parameter P* (i.e., a layout parameter that varies between conditional candidates is discarded, rather than a constant layout parameter) is discarded, thereby obtaining the corresponding unconditional output.

[0074] The method further includes: for each conditional input and each perturbed conditional candidate 3D layout, calculating the reconstruction error of S60 between the corresponding conditional candidate 3D layout and the corresponding unconditional output. The reconstruction error is the "difference" between the corresponding conditional output and the corresponding unconditional output in the conditional candidate 3D layout. In other words, the reconstruction error is calculated by comparing the second value of the conditionally predicted value of a predetermined set of one or more bounding box parameters with the second value of the unconditionally predicted value of a predetermined set of one or more bounding box parameters.

[0075] The method also includes: for each conditional input, averaging the reconstruction error S70 on multiple perturbated conditional candidate 3D layouts to obtain a score.

[0076] This approach leads to an improved solution for predicting (and potentially sorting) 3D layouts.

[0077] It is noteworthy that the disclosed computer-implemented method takes a pre-configured machine learning function and a set of conditional inputs as input, and outputs (i.e., assigns) corresponding scores to each conditional candidate 3D layout predicted (by the machine learning function) associated with a given conditional input. The disclosed method (equivalent to a score / diffusion-based method) provides an improved solution for ranking (i.e., classifying) conditional inputs and their corresponding one or more conditional candidate 3D layouts. In other words, the disclosed method allows determining which conditional candidate 3D layouts are most suitable based on the knowledge of a pre-configured machine learning function, which is further configured to conditionally discard information on the conditional inputs.

[0078] Furthermore, this method forms a Self-Score Evaluation (SSE) approach, which allows the use of knowledge from a pre-configured function to select a set of relevant conditional inputs to generate a 3D layout. In fact, the proposed SSE method enables the leverage of the pre-configured function's capabilities in 3D layout generation by effectively selecting conditional inputs that align with the capabilities of the pre-configured function, resulting in more realistic and plausible 3D layouts. Therefore, the disclosed SSE allows the use of a single pre-configured function (i.e., a single trained model) to select the conditional inputs that lead to the most realistic 3D layout.

[0079] Furthermore, this approach allows for different ways of obtaining conditional inputs, thereby demonstrating improved flexibility and synergy for real-world user-driven applications. In the example, the set of conditional inputs obtained at S20 can be provided by the user, who wants to determine what the optimal set of conditional inputs is for solving certain real-world problems, such as determining the best set of objects to be arranged in a given planar graph and / or the best planar graph in which the given set of objects is arranged and / or the most relevant objects to be inserted into or removed from a given 3D layout to optimize (i.e., obtain a lower score) the rearrangement task. Obtaining the set of conditional inputs (S20) can include using an external third-party source such as a Large Language Model (LLM) to obtain a set of conditional inputs generated by the LLM. In other words, obtaining the set of conditional inputs (S20) can include a set of conditional outputs generated by the LLM so that a pre-configured function can be combined with (i.e., can be used as input) the set of conditional inputs generated by the LLM, which can optionally be selected via an SSE method. Generating a set of conditional inputs via LLM is particularly useful in applications where a large number (e.g., at least 20) of conditional inputs should be provided (i.e., obtained), thus avoiding the tedious operation of providing a set of conditional inputs to the user of the method and improving the ergonomics of the method.

[0080] In the usage method, the function obtained (and used) at S10 has been pre-configured as a denoiser. In other words, the pre-configured function (i.e., the denoiser) is constrained by the noise level used to perturb the input 3D layout. This noise conditionalization (during training) gives the pre-configured function a significant ability to denoise the input and predict realistic and diverse 3D scene layouts.

[0081] The function obtained (and used) at S10 is pre-configured with a machine learning method. Details of the machine learning method are presented in the description thereafter.

[0082] This configured function is applied once or multiple times; that is, a pre-configured function is used during iterative sampling. It should be noted that iterative sampling involves applying the pre-configured function once or multiple times from the input 3D layout to predict the output 3D layout; i.e., iterative sampling using the pre-configured function includes iterating the pre-configured function once or multiple times. Since the pre-configured function is a denoiser, iterative sampling is equivalent to iterative denoising, which allows the pre-configured (i.e., trained) function to improve the realism (e.g., to look natural) and diversity of the predicted 3D arrangement. Thus, the pre-configured function can form a denoised diffusion model.

[0083] Furthermore, the denoising-based approach improves the 3D arrangement of densely packed scenes (such as real-life scene arrangements), i.e., the training function predicts more realistic and diverse 3D scenes containing many (e.g., at least 20) objects (e.g., compared to autoregressive methods that predict the sequential insertion of objects, i.e., one at a time). In the example, the proposed method generates plausible (e.g., realistic and diverse) 3D arrangements for well-arranged scenes containing at least 20 objects, demonstrating the advantage of scalability to at least 50 objects.

[0084] Moreover, the disclosed method can be equivalent to a time-efficient iterative sampling process, where the balance between sampling time and the quality of predicted 3D layout can be set by the user. In the examples, the method can support efficient batch processing techniques and / or parallelization capabilities (e.g., on a GPU) to generate 3D scene permutations of multiple scenes and / or multiple permutations of a single scene in a single iterative sampling process.

[0085] The functions configured in this way are also flexible, meaning that the pre-configured functions can be used to perform different tasks through the same publicly disclosed usage methods, such as, but not limited to, (partial) 3D arrangement generation and 3D rearrangement of a scene, for example, containing at least 20 objects to be arranged in a given planar diagram.

[0086] For example, in 3D arrangement generation, the positions of 3D objects are arbitrarily initialized at the center of the room, and the rotations and / or dimensions of the 3D objects are randomly initialized. In this example, iterative denoising starts with a sufficiently high noise level and performs at least 30 steps, thus creating a fair trade-off between the quality of the predicted 3D arrangement and the sampling time.

[0087] In some 3D arrangement generation applications, some 3D objects already have known positions and / or dimensions and / or rotation values. Therefore, these objects are initialized to their known values, while the objects to be arranged have positions initialized to the center of the room and rotations randomly initialized. In each denoising step, the model output for the known 3D objects (i.e., 3D objects with known position and rotation parameter values) can be replaced with their original position and orientation (i.e., rotation) values. Note that the model output for the known 3D objects can be replaced with their perturbed position and rotation values, which are noisier at the level corresponding to the current sampling step. In either case, these known objects will eventually converge to their initial values ​​throughout the sampling process.

[0088] In another application, such as 3D rearrangement, the positions and rotations of 3D objects are initialized to their noise (i.e., perturbed) values. The denoising process can begin with values ​​that are lower in noise than those generated by the 3D rearrangement task.

[0089] Alternatively, the method may also include arranging real-world rooms according to a predicted layout; that is, each 3D object ultimately has a corresponding real-world physical object positioned and oriented within the real-world room according to the predicted layout. Thus, this method can reproduce and sort (i.e., create and store an ordered list of scores) different and realistic 3D scene arrangements that can be achieved in a user's real-world home / apartment. In other words, the method is user-driven, facilitating real-life user interaction to generate and sort 3D layouts similar to real-world 3D scenes (e.g., in a design planner application).

[0090] The method of use includes: for each conditional input, averaging the reconstruction error S70 over multiple perturbation-based conditional candidate 3D layouts to obtain a (corresponding) score. The obtained score can be a real number. The method of use may also include comparing the obtained scores with the corresponding conditional input. Comparing the obtained scores may further include: sorting the obtained scores (i.e., sorting them according to criteria, such as from lowest to highest score), creating a (numerical) list of the obtained sorted scores, and outputting the created list (i.e., sorted). The output list may include a list of scores, each score associated with a conditional input.

[0091] For example, a user using the method obtains a set of S20 conditional inputs, each of which may include a floor plan designed by the user (e.g., from their apartment) and / or a list of 3D objects listed by the user (e.g., a list of 3D objects in their room, i.e., a real-world room), to generate and sort several 3D layouts and ultimately select an output 3D layout (e.g., the output with the lowest score, such as the output with the 3D arrangement that best suits the interior design purpose).

[0092] In another example of the usage method, the user receives a set of conditional inputs in S20, each of which may include a different list of 3D objects (e.g., one or two 3D objects that are different from each other) to obtain a ranking of the best choices of the conditional inputs (and thus receive feedback from the usage method).

[0093] The pre-configuration function is configured to take a 3D layout and a given noise level as input. The input 3D layout includes a given planar graph, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box in the first 3D arrangement is defined by a first value of a predetermined set of one or more bounding box parameters. In other words, in a given planar graph, each 3D bounding box (labeled by a semantic category) can be defined by a first value of its spatial attributes, which define its position, dimension, and orientation in the scene.

[0094] The function is configured to predict the output 3D layout. The output 3D layout includes a given planar view, a second 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box in the second 3D arrangement is defined by a second value of a predetermined set of one or more bounding box parameters.

[0095] The function is further configured to predict a second set of predetermined values ​​for one or more bounding box parameters, which differs from a first set of predetermined values ​​for the one or more parameters. The given planar graph and the given semantic category for each given 3D bounding box can be constants for the function; that is, the function does not change their values. A “second 3D permutation of one or more given 3D bounding boxes” means that one or more given 3D bounding boxes can be arranged within the same given planar graph in such a way that the predicted second values ​​of their spatial attributes differ from the first values ​​of those spatial attributes (i.e., the input). In other words, the function is trained to predict second values ​​for the position, dimension, and orientation of the 3D bounding boxes, thereby predicting a second 3D permutation of one or more 3D bounding boxes. Other variables of the function can remain constant, meaning that the predicted 3D permutations can be performed within the same given planar graph and with the same given list of semantic categories.

[0096] In other words, the pre-configuration function is trained to simply rearrange (i.e., use repositioning and / or resizing and / or reorienting) the 3D bounding boxes of the input 3D layout.

[0097] The preconfiguration function is also configured with conditional discarding of conditional inputs. Since the preconfiguration function is trained with conditional discarding, it acquires a significant ability to predict 3D layouts both conditionally and unconditionally—that is, predicting 3D layouts with and without (i.e., discarding) conditional inputs provided to the preconfiguration function, respectively. This ability of the preconfiguration function to predict 3D layouts conditionally and unconditionally allows for the computation of scores for conditional candidate 3D layouts in the SSE method.

[0098] The term "conditional drop-out" is a technique used to discard (i.e., ignore) a set (e.g., a subset) of the inputs to a machine learning function during the training phase with a certain drop-out probability p. In other words, in each training iteration, the conditional inputs (or a selected subset of the conditional inputs) are discarded with probability p and replaced with a generic input (such as an empty vector). Therefore, configuring (i.e., training) a machine learning function with conditional drop-out enables it to predict the output both conditionally (i.e., the trained machine learning function takes into account the conditional inputs used to predict the conditional output) and unconditionally (i.e., the machine learning function ignores the conditional inputs used to predict the unconditional output).

[0099] The machine learning method, which was previously trained using a computer-implemented approach, precedes the pre-configured machine learning function obtained in S10 and will be discussed later. The machine learning method is characterized by conditional discarding of at least one layout parameter, where the at least one layout parameter includes a given planar graph and / or a semantic category for each bounding box. This means that the function is pre-configured to conditionally and unconditionally take the input 3D layout and a given noise level and predict the output 3D layout.

[0100] Conditional means that the pre-configuration function takes an input 3D layout and predicts a conditional output. The input 3D layout includes a given planar graph, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box, each bounding box being defined in the first 3D arrangement by a first value of a predetermined set of one or more bounding box parameters.

[0101] Unconditional means that the pre-configuration function takes the input 3D layout without at least one layout parameter, which includes a planar graph and / or the semantic category of each 3D bounding box in a first 3D arrangement of one or more given 3D bounding boxes, and predicts an unconditional output.

[0102] This method of use also includes obtaining a set of conditional inputs for S20. The conditional inputs include: distinct values ​​for one of the at least one layout parameter; and identical values ​​for each of the other layout parameters, including the planar graph and the semantic category for each 3D bounding box. In the example, at least one layout parameter (i.e., a parameter that can be discarded) can be the planar graph and / or the semantic category for each 3D bounding box in a first 3D permutation of one or more given 3D bounding boxes. The set of conditional inputs can be, for example, one or more lists of semantic categories associated with each 3D bounding box in the first 3D permutation. The set of conditional inputs can be, for example, provided by the user and / or generated by an LLM and / or generated by a separately trained model for 3D layout generation.

[0103] The method of use also includes: for each conditional input, determining one or more conditional candidate 3D layouts for S30.

[0104] This method utilizes the pre-configuration function obtained in S10 to employ iterative sampling. It should be noted that iterative sampling involves applying the pre-configuration function one or more times from the input 3D layout to predict the output 3D layout; that is, iterative sampling using the pre-configuration function includes iterating the pre-configuration function one or more times. Since the pre-configuration function is a denoiser, iterative sampling is equivalent to iterative denoising, which allows the pre-configuration (i.e., training) function to improve the realism (e.g., to look natural) and diversity of the predicted 3D arrangement. Thus, the pre-configuration function can form a denoised diffusion model.

[0105] Each conditional candidate 3D layout is the result of iterative sampling using a pre-configured function (e.g., unique). The result of iterative sampling using a pre-configured function (e.g., unique) is the predicted 3D layout, which is the conditional candidate 3D layout.

[0106] In the example, one or more conditional candidate 3D layouts include the final result of iterative (conditional) sampling, and / or one or more conditional candidate 3D layouts include one or more intermediate (e.g., consecutive) results of iterative (conditional) sampling (i.e., the result of the steps of iterative sampling). In other words, the result of iterative sampling using a pre-configured function (e.g., unique) includes: the final result of iterative sampling (i.e., the 3D layout) and / or one or more intermediate results (i.e., one or more intermediate 3D layouts), each intermediate result being an intermediate (e.g., consecutive, e.g., between steps 10 and 40) output of the pre-configured function in each corresponding application of the pre-configured function in the (e.g., unique) iterative sampling. Therefore, for each conditional input, the method involves using (e.g., unique) iterative sampling to determine one or more conditional candidate 3D layouts, the iterative sampling including one or more applications of the pre-configured function. Iterative sampling can be iterative conditional sampling, i.e., iterative sampling with conditional inputs. One or more conditional candidate 3D layouts can be the final result of a full (i.e., complete) iterative sampling using a preconfiguration function, and / or one or more conditional candidate 3D layouts can be intermediate (e.g., consecutive) results of each application of the preconfiguration function in one or more applications during iterative sampling.

[0107] The method further includes: for each conditional input, determining S40 perturbed conditional candidate 3D layouts. Each perturbed conditional candidate 3D layout is determined by adding corresponding (sampled) noise to the corresponding conditional candidate 3D layout. Each perturbed conditional candidate 3D layout corresponds to a given conditional candidate 3D layout, wherein the values ​​of one or more bounding box parameters are perturbed by adding corresponding noise.

[0108] In the example, adding appropriate noise for the (same) final result of iterative sampling includes sampling the noise level and sampling the appropriate noise based on the sampled noise level. Alternatively, for each (different) intermediate result of iterative sampling, adding appropriate noise includes sampling the appropriate noise based on the noise level of the intermediate iteration of the iterative sampling corresponding to the intermediate result.

[0109] The method further includes applying a pre-configuration function to the perturbed conditional candidate 3D layout for each conditional input and for each perturbed conditional candidate 3D layout, wherein one layout parameter is discarded to obtain the corresponding unconditional output. In other words, the pre-configuration function takes the perturbed conditional candidate 3D layout corresponding to the corresponding conditional input (e.g., the final perturbed result of iterative sampling and / or the intermediate perturbed result of iterative sampling) as input, discards at least one layout parameter (e.g., a list of semantic categories for each 3D bounding box in a 3D arrangement of one or more 3D bounding boxes), and outputs the corresponding unconditional output. Therefore, the method includes: one-step (i.e., once) unconditional denoising for each conditional input and each perturbed conditional candidate 3D layout, i.e., applying the pre-configuration function by conditional discarding to obtain the corresponding unconditional output.

[0110] The method also includes: for each conditional input and for each perturbed conditional candidate 3D layout, calculating the reconstruction error between the corresponding conditional candidate 3D layout and the corresponding unconditional output in S60. The reconstruction error is the "difference" (e.g., the distance between the outputs of a pre-configured function) between the corresponding conditional output and the corresponding unconditional output in the conditional candidate 3D layout. In other words, the reconstruction error is calculated by comparing the second value of the conditionally predicted values ​​of a predetermined set of one or more bounding box parameters with the second value of the unconditionally predicted values ​​of a predetermined set of one or more bounding box parameters. In the example, such a difference can be calculated using a dissimilarity metric. Still in the example, depending on whether the one or more conditional candidate 3D layouts include the final result of iterative sampling or one or more conditional candidate 3D layouts include one or more intermediate results of iterative sampling, the reconstruction error can be calculated after iterative sampling or during iterative sampling. In the first example, when the conditional candidate 3D layout has been generated using iterative conditional sampling and the corresponding unconditional output has been generated, the reconstruction error is calculated using a dissimilarity metric between the corresponding conditional output and the corresponding unconditional output in the conditional candidate 3D layout. In the second example, alternatively, at each iteration of iterative conditional sampling (e.g., between steps 10 and 40), each corresponding intermediate (perturbed) conditional candidate 3D layout is treated as input to a step of unconditional denoising, thus generating an intermediate unconditional output. In each iteration of iterative conditional sampling, the reconstruction error between the intermediate conditional output and the corresponding (intermediate) unconditional output in the conditional candidate 3D layout is calculated using a dissimilarity metric.

[0111] The method also includes: for each conditional input, averaging the reconstruction error S70 on multiple perturbed conditional candidate 3D layouts to obtain a score (e.g., a real number).

[0112] In the example, the method of use could include: sorting the conditional candidate 3D layouts based on their respective scores, starting from the lowest score, i.e., the best conditional candidate 3D layout is the one with the lowest score.

[0113] We will now discuss additional optional features for iterative sampling using pre-configured functions.

[0114] Iterative sampling using a pre-configuration function, i.e., applying the pre-configuration function once or multiple times, may include: injecting noise into the input 3D layout to obtain a perturbed input 3D layout; applying the pre-configuration function to the perturbed input 3D layout at least once to obtain an output 3D layout (e.g., (i) as a direct result of a single application of the pre-configuration function, or (ii) as a result obtained by applying the pre-configuration function once and then processing the obtained output, or (iii) as a result obtained by applying the pre-configuration function multiple times, each time starting from the direct result of the previous application of the pre-configuration function or from the result obtained by applying the pre-configuration function once and then processing the obtained output); and using the output 3D layout as the input for the next iteration.

[0115] Iterative sampling using a pre-configuration function can include noise with a level that decreases with the depth of the iteration. Specifically, injecting noise into the input 3D layout at each iteration can include noise level scheduling, where the injected noise has a level that can decrease with the depth of the iteration. This noise level scheduling allows for improved quality of the predicted 3D layout. Therefore, during the inference phase, the pre-configuration function is optimally optimized to denoise the input 3D layout, which has already been perturbed with different noise levels. In other words, the noise-conditional pre-configuration function can denoise the input 3D layout to generate (i.e., predict) a realistic 3D layout.

[0116] Furthermore, in each iteration of iterative sampling, applying the pre-configuration function at least once may include the following optional steps. First, the pre-configuration function can be applied to the perturbed input 3D layout to obtain a first output 3D layout. Second, a first intermediate (e.g., middle) 3D layout can be obtained by calculating the gradient step size between the perturbed input 3D layout and the first output 3D layout. Next, the pre-configuration function can be applied to the first intermediate 3D layout to obtain a second output 3D layout. Finally, a second intermediate (e.g., middle) 3D layout can be obtained by calculating the gradient step size between the perturbed input 3D layout and the second output 3D layout to obtain the final 3D layout. These optional steps are collectively referred to as second-order sampling steps because they are performed at each iteration within iterative sampling. The implementation of second-order sampling steps improves the generation of accurate and natural-looking 3D scenes while reducing the number of computationally expensive neural evaluations (i.e., the application of the pre-configuration function). Starting with a 3D object placed at an initial position, a perturbed 3D object can be obtained by applying a noise step (i.e., applying noise). The perturbed 3D object is placed at its corresponding noise position. First, a function can be applied to a perturbated 3D object to obtain a first model prediction, where the 3D object can be placed at a first predicted position. Second, a first intermediate (e.g., middle) position can be calculated by applying a gradient step between the noise position and the first predicted position. Next, a training function can be applied to the 3D object placed at the calculated intermediate position to obtain a second model prediction, where the 3D object is placed at the second predicted position. Finally, a second intermediate (e.g., middle) position can be calculated by applying a gradient step between the noise position and the second predicted position to obtain the final predicted position.

[0117] The pre-configured function can be parameterized by a noise-conditionalized denoiser. Parameterization can have the following types:

[0118]

[0119] in:

[0120] ·x σ This represents the first 3D permutation of one or more given 3D bounding boxes (in a perturbed 3D layout that is treated as a function input during training).

[0121] · Represents a given plan view.

[0122] ·c i It is a list of given semantic categories (each given semantic category is associated with a given 3D bounding box).

[0123] σ is a given noise level.

[0124] · It is a noisy conditional scoring network with a set of trainable parameters θ.

[0125] ·c skip (σ) is the noise-dependent pre-adjustment coefficient for modulation prediction of 3D layout.

[0126] ·c noise (σ) is a noise dependence coefficient that modulates the noise level within the scoring network, and

[0127] ·c in (σ) and c out (σ) are two values ​​scaled by x respectively. σ and S θ (c in (σ)x σ ;y,c noise The noise dependence coefficient of (σ)).

[0128] This parameterization helps in training the function and helps the function learn (i.e., capture) the relationship between perturbation configurations and clean configurations.

[0129] Parameterized noise conditioning and denoising for pre-configured functions Vector x σ The spatial properties of the disturbance are taken as input. The noise conditioning denoiser can also take a given noise level and additional conditional inputs as inputs. Conditional inputs c i A given semantic category can be represented by a given planar view and each of one or more given 3D bounding boxes. A noise-adjusted denoiser can output a predicted 3D layout. The denoiser network can be noise-adjusted because it possesses the knowledge to predict a "clean" layout (i.e., an accurate and diverse scene layout) from a "messy" layout (i.e., a perturbation after perturbation) by performing either a primary object permutation (for perturbations with high noise levels) or a secondary object permutation (for perturbations with low noise levels). Such features provide the function, trained in this way, with the ability to denoise the perturbation after perturbation at any (e.g., arbitrary) noise level.

[0130] The parameterization of a noise conditioning denoiser can include a noise conditional score network S with a set of trainable parameters θ. θScore-based generative models are a class of deep generative models that employ neural networks to learn scores (i.e., gradients of the log probability density function) of a noise-level-dependent marginal distribution obtained by perturbing training samples with different levels of noise (e.g., Gaussian). In other words, score-based generative models are trained to model the data distribution by denoising samples that have already been perturbed at different noise levels. The parameterization of the score-based objective can be carefully designed to facilitate learning the underlying data distribution. This parameterization can involve noise-dependent modulation of the output of the scoring network, as well as scaling of the model's input, output, and noise level. In practice, a noisy-conditional scoring network can rescale the vector c... in (σ)x σ As input, where c in (σ) is the noise dependence coefficient. The noise conditional scoring network can also use the noise dependence coefficient c, which adjusts the noise level within the network. noise (σ) is used as input. Furthermore, the noisy conditional scoring network can use conditional inputs representing a given planar graph and a given semantic category. c i To adjust. Noise conditional scoring network. The output can also be determined by another noise dependence coefficient c. out (σ) Rescaling. In the example, the coefficient can be set as follows:

[0131]

[0132] In the above formula, σ data It is the standard deviation of the training dataset and is calculated before the training phase.

[0133] Reference Figure 2 The flowchart presents a computer-implemented method for machine learning of the machine learning function, specifically a machine learning method for training the function through conditional discarding, and thus obtaining a pre-configured function at step S10 of the computer-implemented method. The machine learning method includes obtaining a dataset of ground-based 3D layouts at step S80. Each ground-based 3D layout represents a corresponding scene. Each ground-based 3D layout includes a corresponding plan view, a corresponding 3D arrangement of one or more 3D bounding boxes, and a corresponding semantic category for each 3D bounding box. Each bounding box is defined by a predetermined set of values ​​for one or more bounding box parameters.

[0134] The machine learning approach also includes obtaining the probability distribution of the S90 noise level. In the example, the probability distribution obtained at S90 could be a Gaussian distribution.

[0135] The machine learning method further includes obtaining a corresponding perturbed 3D layout for each ground-based 3D layout, S100. The corresponding perturbed 3D layout is a 3D layout that can be computed (e.g., already computed, e.g., by a machine learning method including such computation) by perturbing (e.g., the machine learning method includes such perturbing) at least one bounding box parameter of at least one (e.g., each) 3D bounding box of the ground-based 3D layout. In other words, the machine learning method may include computed at least one (e.g., each) corresponding perturbed 3D layout, and / or retrieved (e.g., on local or remote memory) or received (e.g., from a remote third-party computer system) at least one (e.g., each) corresponding perturbed 3D layout, whereby the retrieved or received at least one corresponding perturbed 3D layout has been pre-computed. The perturbation includes sampling a corresponding noise level based on a probability distribution, S100a. The perturbation also includes sampling a corresponding noise value based on the corresponding noise level for each of the at least one parameter, S100b, and applying the corresponding noise value to the corresponding bounding box parameter, S100c.

[0136] Furthermore, the machine learning method involves training the S110 (and outputting) function. This function is configured (after training the S110) to take an input 3D layout and a given noise level, and to predict (i.e., output or generate) an output 3D layout.

[0137] Training S110 is performed on the dataset based on a loss that penalizes a measure of dissimilarity between each ground-based real 3D layout and its corresponding predicted 3D layout, wherein the corresponding predicted 3D layout can be (i.e., can be) obtained by applying a function to the corresponding perturbed 3D layout.

[0138] Training S110 is further performed via conditional discarding of at least one layout parameter, which includes a planar graph and / or a semantic category for each 3D bounding box.

[0139] Therefore, the function is configured to become a denoiser capable of denoising the input 3D layout, thereby transforming it into a more realistic output predicted 3D layout. The configured function is also flexible, meaning the training function can be used to perform different tasks, as previously described.

[0140] Due to dropouts during training, the function trained at S110 can also output predicted 3D layouts conditionally or unconditionally. Furthermore, utilizing conditional dropouts for machine learning / training reduces overfitting of the trained layouts.

[0141] The neural network architecture for the pre-configured function can be as described in European Patent Application No. EP24306557.0, filed on 23 September 2024, which is incorporated herein by reference. In particular, the neural network architecture can be based on any example described in European Patent Application No. EP 24306557.0.

[0142] For example, pre-configured functions may include the architecture of a (noise-aware) transformer. The transformer can cascade (i.e., embed) equal-length representations. The transformer can take the following as input: a representation of a given noise level, a first representation of each given 3D bounding box, and a representation of a given planar graph. The transformer can output a representation of the predicted 3D layout, including a second representation of each given 3D bounding box.

[0143] The pre-configuration function may also include an encoder to generate a (e.g., equal-length) representation that takes the transformer as input.

[0144] In the example, the function could include a noise encoder that generates a representation of a given noise level.

[0145] The pre-configured function may include a 3D object encoder that generates a first representation for each given 3D bounding box. The 3D object encoder may optionally be configured to generate a corresponding representation for each parameter and a representation for the semantic category, and may be configured to cascade all generated representations.

[0146] The pre-configured function may also include a ground encoder that generates a representation of a given planar map. The ground encoder may include a sampling module for generating samples of the given planar map. Sampling of the given planar map may include sampling, for example, a fixed number of 3D points (e.g., 250 evenly spaced 3D points) on the planar map outline (i.e., perimeter). This sampling allows obtaining a 3D cloud representation of the planar map as input to a point cloud encoder. The ground encoder may also include a point cloud encoder for processing the samples.

[0147] The pre-configuration function may also include an MLP, which takes a representation of the predicted 3D layout as input and outputs a third representation for each given 3D bounding box.

[0148] Additional optional features of the neural network architecture and pre-configured functions are provided on page 20, line 3 to page 24, line 30 of the specification of European patent application number EP24306557.0.

[0149] Machine learning methods use a data-driven approach, which allows functions to learn patterns and relationships between objects and between objects and their constraints to predict realistic 3D layouts. In other words, the data-driven approach of machine learning allows trained functions to learn interactions (i.e., relationships) between 3D objects (i.e., semantic consistency) and between 3D objects and their constraints (i.e., spatial inference) solely from the training dataset.

[0150] In the example, the training dataset obtained in S80 can include realistic and diverse ground-based 3D layouts, each representing a 3D scene. The 3D scenes can be obtained from digital 3D scene datasets and / or real-world 3D scenes. An example of a digital 3D scene dataset could be... The dataset or any subset thereof may contain at least more than one thousand densely arranged scenes (e.g., 10K scenes), such as containing at least 20 objects.

[0151] Furthermore, a machine learning method is trained on the dataset based on a loss that penalizes the dissimilarity between the ground-real 3D layout and the corresponding predicted 3D layout. In particular, this loss can be invariant under permutations of identical 3D objects. This option facilitates training and avoids penalizing the predicted 3D layout, where identical objects are interchanged relative to the ground-real 3D layout, thereby enhancing the diversity of 3D scene generation.

[0152] Furthermore, machine learning methods employ denoising techniques to generate predicted outputs. These denoising methods outperform existing generative models of other classes, such as GAN models. Machine learning methods involve injecting different noise levels into the training dataset samples based on the probability distribution of the obtained S90. Additionally, the model (i.e., the denoiser) is "noise-conditional," as it is configured to apply a given noise level value to the input samples (i.e., the noise level is given as a "condition" to the model's input). This noise-based approach allows the training function to learn from the perturbed (i.e., noisy) dataset, thus providing the training function with a significant ability to denoise the input and predict realistic and diverse 3D scene layouts. In other words, machine learning methods can train a function that optimally "denoises" the input 3D layout at any (i.e., arbitrary) noise level to predict realistic (i.e., natural-looking) 3D layouts.

[0153] Furthermore, a denoising-based approach improves the 3D arrangement of densely packed scenes (such as real-life scene layouts), where the training function predicts more realistic and diverse 3D scenes containing many (e.g., at least 20) objects (e.g., compared to autoregressive methods that predict arrangements of objects inserted sequentially (i.e., one at a time). In the example, the proposed method generates plausible (e.g., realistic and diverse) 3D arrangements for well-arranged scenes containing at least 20 objects, demonstrating the advantage of scalability to at least 50 objects. The dataset obtained in S80 can include ground-based real-world 3D layouts containing at least 20 objects and / or ground-based real-world 3D layouts containing at least 40 objects. Correspondingly, the input 3D layout obtained in S10 can contain at least 20 objects or at least 40 objects. In fact, the proposed solution achieves better (i.e., more accurate) results in predicting the physical consistency and realism of 3D layouts because the denoising method allows the training function to simultaneously learn 3D object relationships, i.e., to acquire, for example, nontrivial interdependencies between 3D objects and between each 3D object and a given planar graph using a self-attention mechanism. Simultaneously (i.e., together) means that during the training of function S100, the first values ​​of a predetermined set of one or more black-box parameters of each 3D bounding box in the first 3D arrangement can be input simultaneously. In other words, the function can simultaneously take all 3D objects in the input 3D layout as input. In other words, the training function captures all spatial and semantic relationships to obtain realistic and diverse scene arrangements. Similarly, the second values ​​(i.e., values ​​predicted by the function) of a predetermined set of one or more parameters of each 3D bounding box in the second 3D arrangement can also be output simultaneously (instead of one object after another, e.g., one object at a time). This simultaneous processing of one or more parameters of each 3D bounding box corresponds to better object grouping, i.e., the function's ability to identify objects that can be associated together in the predicted 3D arrangement.

[0154] The pre-configuration function trained using the aforementioned machine learning method is obtained in S10 and applied to the computer-implemented method, taking the 3D layout and the given noise level as inputs.

[0155] Each 3D layout is a set of data, including a given plan view, a 3D arrangement of one or more 3D bounding boxes, and a corresponding semantic category for each 3D bounding box. In other words, a 3D layout represents an arrangement of one or more 3D bounding boxes within a given plan view. The 3D bounding box of a 3D object is the smallest rectangular cuboid that encloses the 3D object; it may or may not have orientation constraints (such as the constraint that a cuboid must have faces parallel to a horizontal plane). Therefore, a 3D bounding box is characterized by its spatial properties (i.e., its position, its dimensions, and optionally its unconstrained orientation parameters) and its semantic category (i.e., a class of objects, such as books, chairs, etc., having the same function). A predetermined set of one or more bounding box parameters can describe the spatial properties of a 3D object. Each object's spatial property can have a separate real-world interpretation. In the example, the predetermined set of one or more bounding box parameters may include 3D position coordinates, three dimensions (i.e., height, depth, and length), and at least one parameter representing the object's orientation (e.g., the cosine and sine of the angle about a vertical axis). Therefore, the predetermined set of one or more bounding box parameters may include eight parameters or consist of eight parameters. The use of 3D bounding boxes captures the 3D localization of 3D objects. Therefore, by using 3D bounding boxes, the training function predicts accurate and realistic 3D localization and size of 3D objects. Specifically, the training function, and thus this method of use (i.e., the computer implementation of this pre-configured function obtained at S10), predicts a 3D layout that displays physically consistent localization in 3D, thereby avoiding subtle defects that compromise the perceptual effectiveness of the overall scene, such as overlapping, floating, or out-of-bounds objects, inaccessible areas, and inconsistent object localization.

[0156] A floor plan is data describing a plan of a scene where 3D objects can be arranged (i.e., it represents the corners of a room). Therefore, the floor plan sets the boundaries of the 3D scene layout and regulates the 3D output layout. The floor plan can be obtained from an external 3D database in S80 using a machine learning method. During training, the floor plan can be rotated by random angles along the vertical axis. The floor plan input in S10 in this usage method can be imported from the real world using 3D scanning technology.

[0157] Similarly, in machine learning methods, 3D objects can be obtained from external databases and / or online directories in S80. During inference, the 3D objects input in S10 can be imported from the real world.

[0158] In machine learning methods, a given noise level is sampled S100a based on the obtained probability distribution. In the example, the probability distribution obtained at S90 can be a Gaussian distribution.

[0159] During training, noise levels are introduced to perturb the dataset; that is, a corresponding perturbed 3D layout is obtained and / or computed for each ground-based 3D layout. A high level of noise means the perturbed 3D layout is “far” from the ground-based 3D layout; a low level of noise means the perturbed 3D layout is “close” to the ground-based 3D layout. In other words, for each ground-based 3D layout, the machine learning method involves obtaining the corresponding perturbed 3D layout S100 by: sampling different noise levels based on a probability distribution S100a, and for each corresponding parameter in at least one parameter, sampling different noise values ​​based on the corresponding noise level at S100b, and applying the corresponding noise values ​​S100c to the corresponding parameters of each of one or more 3D bounding boxes in the 3D arrangement. Therefore, the function acquires the ability to predict the 3D scene that has been perturbed at different noise levels.

[0160] Obtaining at least one (e.g., each) of the corresponding perturbed 3D layouts in S100 may include: at least one parameter of at least one (e.g., each) 3D bounding box of the perturbed ground reality 3D layout, or retrieving (e.g., on local or remote memory) or receiving (e.g., from a remote computer) the result of such perturbation. The perturbation includes sampling the corresponding noise level S100a based on a probability distribution within the real interval. The noise level is a positive (e.g., real) number. The noise level may be the magnitude by which the parameters (i.e., spatial properties) of the 3D bounding box are perturbed. The noise level may be the absolute value of a scalar drawn from the probability distribution. In the example, it may be drawn from a Gaussian distribution σ = abs[N(0,σ... s The noise level is extracted from ]], where σ s It can be, for example, higher than 0.1 and / or lower than 0.5 (e.g., σ). s =0.23) and can be experimentally configured such that during the inference phase, the trained model generates 3D permutations from configurations perturbed by both sufficiently low and high noise levels. The perturbation also includes: for each corresponding parameter of at least one parameter, sampling the corresponding noise value based on the corresponding noise level (e.g., ∈ = N(0,1)), S100b, and applying the corresponding noise value (e.g., ∈σ) to the corresponding parameter (e.g., the corresponding parameter p is perturbed to p). σ =p+∈σ).

[0161] The obtained ground-based scene dataset S80 can be further augmented by random rotation of the scene along the vertical axis; this random data augmentation can help improve training to predict arrangement scenes where walls are not aligned with at least one coordinate axis.

[0162] The function is trained on a training dataset of ground-based 3D layouts (e.g., The training is performed on a dataset and is based on a loss that penalizes a measure of dissimilarity between each ground-real 3D layout and its corresponding predicted 3D layout, which can be obtained by applying a function to a corresponding perturbed 3D layout. The corresponding perturbed 3D layout can be fed as input to the training function to obtain the corresponding predicted 3D layout. Therefore, the training loss can evaluate the "distance" between each ground-real 3D layout and its corresponding predicted 3D layout. The training loss may preferentially prioritize making the predicted 3D layout "closer" to the ground-real 3D layout.

[0163] In the example, dissimilarity measures can have the following types:

[0164]

[0165] in:

[0166] · It is a collection of one or more 3D bounding boxes in the ground-based 3D layout.

[0167] · It is a set of one or more 3D bounding boxes in the corresponding predicted 3D layout.

[0168] ·N is and The common size, and

[0169] ·l(.,.) is a differentiable distance (e.g., Euclidean distance).

[0170] Therefore, this dissimilarity metric is equivalent to chamfer distance, which measures the dissimilarity between sets of bounding boxes. In such an example, the dissimilarity metric measures a set of one or more 3D bounding boxes in a ground-based 3D layout. A set of one or more 3D bounding boxes in the corresponding predicted 3D layout The distance between them. In the example, the set and A common size N can contain at least 20 objects (e.g., 50 objects to be arranged). Therefore, the proposed machine learning method allows for realistic and accurate arrangement of densely packed scenes. In other words, the proposed solution is equivalent to a scalable method suitable for arranging several items (e.g., at least 20 objects). Furthermore, chamfer distance is more efficient than other set comparison methods (e.g., more efficient than Earth Mover's distance).

[0171] The dissimilarity metric can be characterized by a computationally efficient differentiable distance l(.,.). In fact, training S110 can include the evaluation of the differentiable distance, and each evaluation can be parallelized on a GPU. Specifically, training S100 can include executing parallel sessions, where each session includes: a parallel evaluation of the differentiable distance between each 3D bounding box in a set of one or more 3D bounding boxes in the ground reality 3D layout and a set of one or more 3D bounding boxes in the corresponding predicted 3D layout. Such parallelization allows for time and resource efficiency.

[0172] Differentiable distance can penalize the dissimilarity of dimensions and / or semantic categories between 3D bounding boxes, and can be, for example, the following types:

[0173]

[0174] in:

[0175] · And x are respectively And a vector of spatial parameter values ​​of o,

[0176] · It is the Euclidean norm.

[0177] K is the penalty parameter.

[0178] When its independent variable When o has the same dimension, It is an index function that equals 1, otherwise it is 0, and

[0179] When its independent variable When o and o have the same semantic category, It is an index function that equals 1, otherwise it is 0.

[0180] In the example, for each pair of 3D bounding boxes (one within a set of one or more 3D bounding boxes in the ground reality 3D layout, and the other within a set of one or more 3D bounding boxes in the corresponding predicted 3D layout), the differentiable distance can compute the Euclidean norm between the values ​​of their spatial parameters (e.g., spatial properties such as position and orientation). The differentiable distance can also evaluate the dissimilarity of dimensions and / or semantic categories between the pair of 3D bounding boxes. Therefore, dissimilarity distance can be called “semantic-aware dissimilarity distance” (e.g., semantic-aware chamfer distance) because when evaluating the dissimilarity between 3D bounding boxes, it is aware (i.e., it considers) the semantic category associated with each 3D bounding box. Therefore, if a pair of 3D bounding boxes does not share the same spatial dimensions and the same semantic category, it may be penalized. In the example, the penalty parameter K can be set to be greater than 10. 4 Or 106 For example, K=10 8 .

[0181] The loss on which the function S110 is trained on the dataset can be the expected value of the type. That is, the dissimilarity measure The expected value of the product between the ground reality 3D arrangement and the noise-dependent weighting function λ(σ). The presence of the weighting function facilitates training phases across different noise levels. In the example, the noise-dependent weighting function λ(σ) can be chosen, for example, to obtain uniform weighting across noise levels. More precisely, the training loss can evaluate the difference between the ground reality 3D arrangement and the predicted second 3D arrangement of one or more 3D bounding boxes. The differences in spatial properties and semantic categories between the 3D bounding boxes in the ground reality scene and the predicted scene can be calculated. The dissimilarity metric can calculate the Euclidean distance between the positions and orientations of the 3D bounding boxes in the ground reality scene and the predicted scene, and can also compare the dimensions and semantic categories of the 3D bounding boxes in the ground reality scene and the predicted scene. Therefore, the training loss can avoid penalizing predictions that swap objects with the same dimensions and share the same semantic categories compared to the ground reality scene.

[0182] Now let's discuss how to implement the proposed solution.

[0183] Figure 3 An example of the architecture of a machine learning function is shown.

[0184] Reference Figure 3 The machine learning function can be characterized by a transformer encoder denoiser network, which takes a learned encoded representation of the noise level (i.e., amplitude) σ of the input scene as input (i.e., conditioned on this representation) (thus qualifying the denoiser as noise-aware). The 3D objects of the input scene, having some of their characteristics (e.g., position and rotation attributes or position, rotation, and bounding box dimensions, etc.), and additional scene-level conditioned features such as the floor plan / shape of a room, can be perturbed. It can output a predicted clean layout of 3D objects.

[0185] Reference Figure 3 Now let's discuss a design example of a noise reduction architecture.

[0186] The implementation of a deep architecture can consist of multiple trainable components: a noise encoder (i), a 3D object encoder (ii), a ground encoder (iii), a noise-aware transformer encoder (iv), and / or a final MLP (v) that outputs predicted object position and rotation values.

[0187] The options for the noise encoder (i) are discussed below.

[0188] The sampling noise level scalar value σ can be passed through the Positional Encoding (PE) module, which deterministically increases the dimensionality of the scalar value, for example from... arrive Positional encoding can generate different representations of the same scalar value, allowing deep learning models to capture more nuanced information when necessary. In implementation, PE can be of the following types:

[0189]

[0190] The PE module can be followed by a trainable MLP, which will encode, for example, from... Mapped to The resulting vector can be a token representing the noise level at which the spatial properties of a 3D object are disturbed.

[0191] The options for the 3D object encoder (ii) are discussed below.

[0192] The scalar values ​​(p, r, d) describing each 3D bounding box in the scene can be passed through the PE module, thus allowing them to be freely defined, for example... Vector representation in .

[0193] Following the PE module, the position and dimension of the bounding box, initially described by three scalar values, can be described by a 192-dimensional vector (3 × 64 = 192). Conversely, the rotation, initially described by a pair of scalar values, can be described by a 128-dimensional vector after position encoding. To ensure the model applies similar weights to position, dimension, and rotation, a higher-dimensional version of the rotation can be passed to, for example, a vector from... Mapped to The multilayer perceptron improves the model's convergence.

[0194] One-hot encoding category c can be from {0,1} 174 The class vector. To ensure that the weighting of the classes is similar to the position, dimensions, and rotation of the bounding box, the class vector can be passed to map it to, for example, a vector. The multilayer perceptron. Applying this transformation to one-hot encodings is common practice because they are not considered meaningful rich representations. During training, regulated dropout can also be performed on the encoding categories, meaning that in each iteration, c is, for example, 0. 174 Empty token replacement in the training space. Empirically, this mechanism has been found to help reduce the training space configuration p. data Overfitting of (x). It also allows the model to generate new layouts with and without specifying object class c at sampling time.

[0195] All previously computed vectors can be concatenated as follows: A single vector within the annotation. This vector can be a token representing the 3D bounding box of the annotated object.

[0196] The following discusses one option (iii) for the ground encoder.

[0197] As a reminder, the 3D layout generation is adjusted by encoding the ground points of the room so that the resulting 3D objects are within the ground constraints.

[0198] Sampling ground points The 3D point cloud can be fed into the PointNet module, which can output, for example... The embedded vector in the vector. This embedding itself can be fed into a vector that maps to, for example... A multilayer perceptron. The final vector can be a token representing a planar graph.

[0199] The options for the encoder in the noise-aware converter are discussed below (iv).

[0200] Noise level tokens, 3D object tokens, and ground tokens can all be concatenated to form a token sequence. These tokens can be independent of each other. To capture the relationships between the different elements of this sequence, a transformer module can be used. Due to its inherent architecture, the transformer module requires a fixed input size. However, since the number of 3D bounding boxes in a scene varies from scene sample to scene, the sequence constructed by output concatenation may have a variable length. To be compatible with the transformer architecture, the sequence can use "zero" tokens (e.g., ...). Padding is used to give the sequence a fixed length. Scenes with fewer than 50 objects are padded accordingly. Therefore, the sequence can be represented as, for example... A tensor. This tensor can be fed into the transformer module, which outputs a final scene embedding tensor of the same dimension. A Boolean padding mask indicating the index of the sequence of padding "zero" tokens can also be passed to the transformer as additional input. This mask can be used for a self-attention mechanism to zero out the attention score of the padding tokens.

[0201] The following section discusses one option for the final MLP(v).

[0202] The new representation computed by the transformer for each 3D object can eventually be passed to the MLP, whose output is, for example... tensors in Indicates the predicted "clean" location for each 3D object. and rotation

[0203] The resulting architecture has a total of 12.2 million trainable parameters.

[0204] The pre-configured function can be a noise conditioning denoising network D. θ It will include the noisy (i.e., perturbed) 3D spatial attributes x+σ∈ (e.g., object position, rotation, and dimension), and the planar graph. Take a list of object semantic categories c as input and output clean 3D spatial properties. It can be expressed by the following formula

[0205]

[0206] Pre-configuration function D θ Output It is the (predicted) second value of a predetermined set of one or more bounding box parameters.

[0207] In such an implementation example, during the inference phase, a method is used to evaluate a candidate set of semantic categories of objects in a given planar graph.

[0208] However, other use cases for using the method can be implemented and carried out similarly. For example, another use case for using the method might be to evaluate a set of candidate planar graphs using a given set of object semantic categories.

[0209] Training with conditional discarding can be performed on semantic category inputs so that the candidate category set can be evaluated using a computer-implemented approach during the inference phase. More specifically, in each training iteration, the probability of discarding is set to p = 0.2, thus the candidate category set... Discard with probability p; otherwise, it will affect the denoiser's output. Thus, the model (i.e., the pre-configured function) can be conditionally (given c) and unconditionally (c set to empty input). Restore clean properties.

[0210] The training phase can be performed based on a loss that penalizes a measure of dissimilarity between each ground-based 3D layout and the corresponding predicted 3D layout obtained by applying this function to the perturbed 3D layout. In the current implementation, the loss can be the chamfer distance. (That is, the dissimilarity metric is considered to be the chamfer distance) ).

[0211] Reference Figure 4 Now, let's describe the inference phase. Figure 4 In the context of machine learning functions (refer to...) Figure 1 (Obtained in S10) is denoted as DeBaRA.

[0212] The set of conditional inputs can be a set C of semantic category candidates S120 that have the same given planar graph.

[0213] In other examples of implementations of the proposed solution, the set of conditional inputs can be a set of planar graphs with the same given semantic category.

[0214] For each conditional input, the method can determine one or more conditional candidate 3D layouts. Each conditional candidate 3D layout is the result of iterative (conditional) sampling S130 using DeBaRa. In other words, each conditional candidate 3D layout is a 3D layout sampled from the learned conditional density, i.e.,

[0215]

[0216] Where C is the number of conditional candidates, p θ The density is represented by a neural network with parameter θ. The method may also include determining multiple perturbated conditional candidate 3D layouts, each layout being determined by feeding data to the corresponding conditional candidate 3D layout x. i The appropriate noise is added to determine this. In this implementation example, each of the conditional candidate 3D layouts obtained at S130 is first perturbed, and then at S140, a one-step unconditional denoising (i.e., unconditional DeBaRa model) at level σ is applied to each perturbed conditional candidate 3D layout. The noise level σ used to perform the unconditional denoising step is also the noise level at which the perturbed 3D layout has been perturbed.

[0217] As a result of the inference phase (i.e., using the method), for each conditional candidate 3D layout x i The average reconstruction error is equal to the score evaluation S150. The proposed method is called the Self-Score Evaluation (SSE) method because it uses a score / diffusion generation model to evaluate (by performing unconditional density estimation) the samples x conditionally generated by the model itself. i .

[0218] The method of use may include ranking the candidate 3D layouts based on their corresponding scores, for example, sorting the candidate 3D layouts S160 starting from the lowest score, i.e., the best candidate 3D layout is the one with the lowest score. Therefore, the optimal (i.e., best) candidate 3D layout x * A conditional candidate 3D layout is one that makes itself (i.e., x) i ) and by applying the unconditional DeBaRa model to the perturbation of the conditional candidate 3D layout (i.e., This minimizes the average reconstruction error between the corresponding outputs. In the equation:

[0219]

[0220] In the implementation example of the proposed solution, conditional output x is used. i and corresponding unconditional output The reconstruction error is calculated using the chamfer distance between the two sides.

[0221] In other words, the optimal candidate is derived from the density estimate of its corresponding 3D spatial layout provided by the unconditional network.

[0222] Here are a few examples illustrating how SSE is actually implemented.

[0223] a. Monte Carlo estimation.

[0224] exist Figure 4 The detailed pipeline of the first inference process is shown in the figure.

[0225] The first method of this SSE is equivalent to conditionally obtaining one or more conditional candidate 3D spatial layouts for each obtained conditional input S120 by means of an iterative conditional sampling S130 with 50 sampling steps. The next step in the pipeline is to apply a pre-configured function to the determined perturbation conditional candidate 3D layouts (thus obtaining the corresponding unconditional outputs) using conditional discarding; that is, performing a one-step unconditional denoising (e.g., unbiased Monte Carlo estimation) at level σ for each conditional candidate 3D layout in step S140. This one-step unconditional denoising at level σ can be performed multiple times, for example, for T denoising trials (i.e., evaluation iterations, e.g., T = 50), and for each trial, a pair (σ, ∈) consisting of the noise level and noise value can be sampled, and the corresponding noise perturbation σ ∈ can be applied to each conditional candidate 3D layout. In other words, in each trial (i.e., each evaluation iteration), the pair (σ, ∈) consisting of the noise level and noise value is fixed in a candidate-wise manner, as shown in the following example algorithm for conditional candidate evaluation. As a result, when calculating the reconstruction error between the conditional candidate 3D layout and the corresponding unconditional output, and further averaging the reconstruction errors over multiple perturbations on the conditional candidate 3D layout, the score evaluation S150 is performed using the ranking of candidates S160.

[0226] In other words, in this first method, the conditional layout generation stage S130 is performed separately (i.e., obtaining the conditional input c for each conditional input using the conditional model via iterative sampling). i Conditional candidate layouts x i The unconditional score evaluation phase S140 (i.e., when using an unconditional model to evaluate candidate layouts under perturbation conditions) and the unconditional score evaluation phase S140 (i.e., when using an unconditional model to evaluate candidate layouts under perturbation conditions) (Calculate the average reconstruction error during denoising).

[0227] The following details an example of an algorithm used for conditional candidate evaluation (i.e., the SSE method of the disclosed solution):

[0228]

[0229] The original English translation of the above algorithm is:

[0230]

[0231] Also note that the loss used to measure the best candidate during this inference phase (e.g., chamfer distance) The loss used in the inference phase may differ from the loss used during training. For example, experiments have shown that using the noise dependency weighting term λ(σ) in the loss during the inference phase can impair the model's accuracy. Therefore, to improve accuracy during inference, the noise dependency weighting term can be set to λ(σ) = 1, while adjusting the noise scale in each evaluation trial. Perform sampling.

[0232] Furthermore, it is worth noting that the chamfer distance loss can be predicted. Instead, the mean squared error (MSE) between the ground reality spatial features x and the mean squared error (MSE) is used. This has been tested and achieved similar performance. This highlights the robustness of the published solution, since the reconstruction loss (i.e., reconstruction error) calculated during the inference method may not necessarily be the same as the loss calculated during the training of the diffusion model.

[0233] Furthermore, this method differs from existing diffusion classifiers, which assume that the conditional probabilities have uniform prior probabilities. In other words, in this method, since some conditional inputs may not lead to a reasonable permutation at all, the conditional probability p(c) is not assumed. i The prior probabilities of the samples are uniform. Therefore, the density estimate of the observed samples generated by the class-conditioned model is calculated using the unconditional model, while the diffusion classifier uses the class-conditioned model to calculate the density estimate of a single observed sample.

[0234] b. Score evaluation during sampling

[0235] This second approach to SSE is equivalent to conditionally obtaining one or more conditional candidate 3D spatial layouts for each conditional input S120 at each sampling step of the iterative conditional sampling. In other words, this second approach jointly executes the conditional layout generation phase S130 and the unconditional score evaluation phase S140.

[0236] In each sampling step and for each conditional input c iCalculate the (intermediate) conditional outputs (i.e., the outputs used to iteratively generate the conditional layout x) in the conditional candidate 3D layouts predicted based on the current intermediate iteration of the conditional model. i The reconstruction error between the network output and the unconditional output obtained by applying the unconditional model to the perturbed intermediate conditional candidate 3D layout. Compared to the conditional iterative sampling performed in the first method a), this means performing an additional unconditional network application at each step to evaluate the reconstruction error for the conditional candidate 3D layout being generated, i.e.:

[0237]

[0238] Where σ is the noise level associated with the current sampling step, and It is for the conditional candidate c i The layout is conditionally sampled in this step.

[0239] In the example of the implementation of the proposed solution, intermediate conditions are used for output. and corresponding unconditional output The reconstruction error is calculated using the chamfer distance between the two sides.

[0240] Again, the conditional candidate that minimizes the average reconstruction error over several sampling steps is the best candidate for ranking. In practice, during iterative sampling with 50 steps, the reconstruction error is only calculated between, for example, steps 10 and 40, because high and low noise levels tend to bias the estimation and severely impair accuracy. It should also be noted that the MSE loss can be applied instead of the training loss, i.e., the chamfer distance. loss.

[0241] Additionally, it should be noted that the diffusion loss can be calculated per object. In this implementation of the method, the method may include identifying one or more elements of the obtained set of conditional inputs that maximize and / or minimize the reconstruction error of the obtained set of conditional inputs. One or more elements of the set of conditional inputs may be a planar graph and / or one or more semantic categories. For example, the method may include identifying a 3D object as a conditional candidate that maximizes the reconstruction error, so as to remove or replace it. Similarly, the method may include identifying a 3D object as a conditional candidate that minimizes the reconstruction error, so as to insert it into the current scene configuration.

[0242] We will now discuss some quantitative and qualitative experiments of the methods disclosed.

[0243] publicly available Quantitative and qualitative experiments of the published method were conducted on the dataset. The following results were obtained by applying the Monte Carlo estimation method of SSE as described above (a).

[0244] Binary classification.

[0245] The goal of binary classification is to distinguish between perturbation condition candidates and ground condition candidates.

[0246] For this purpose, already The denoiser was trained on a subset of bedrooms to evaluate the efficiency of the method on a toy binary classification task. More specifically, the binary classification tested the inference method's ability to distinguish between a ground truth (good) set of object semantic categories and a (damaged or adversarial) set that has been randomly perturbed by replacing one or more object categories with random object categories. The binary classification was performed on each of 162 scenes in the test set to calculate the percentage accuracy for several settings based on the degree of perturbation of the adversarial set. The number of trials for evaluation was set to T=100. The experiments were repeated more than ten times for each setting.

[0247] Figure 5 The experimental results of binary classification are shown.

[0248] The binary classification was tested on six different settings. In the "None" setting, the adversarial set is the ground truth set, meaning no object class is replaced by a random object class. Therefore, the "None" setting corresponds to classifying the ground truth set and itself. The expected accuracy is clearly around 50%. In the "Single" setting, only one object class is replaced by a random object class from the perturbed adversarial set, while in the "All" setting, all object classes are replaced by random object classes. The significant difference in accuracy percentage between the "None" and "Single" settings indicates that the disclosed SSE solution can recognize subtle variations in well-conditioned input. It is evident that the accuracy percentage improves when the corrupted adversarial set contains more randomly replaced object classes compared to the ground truth set, suggesting that the disclosed SSE solution is better able to identify more corrupted adversarial sets.

[0249] 3D scene compositing.

[0250] Quantitative comparisons in the context of 3D scene compositing are now discussed. These comparisons consider quantitative metrics obtained by combining the method with four different approaches for obtaining sets of semantic categories (LayoutGPT, DataSetRandom, LLM, and LLM+SSE) and three other known 3D scene compositing methods in the art (LayoutGPT, ATISS, and DiffuSecene).

[0251] The aim is to use this method to select conditional candidates (i.e. conditional inputs) obtained from the Large Language Model (LLM), each conditional candidate being associated with one or more 3D layouts generated through this method.

[0252] Therefore, noise reduction devices have already been used. The training was performed on a subset of the living room and dining room. The training of the 3D layout generation method was conditionally discarded on a set of semantic categories.

[0253] The method can use different sources to obtain a set of conditional inputs (in this specific case, a set of semantic categories) to generate a 3D layout. In quantitative comparisons, the method can use a set of semantic categories generated through the following process as input:

[0254] LayoutGPT.

[0255] The method can take the set of semantic categories generated by the LayoutGPT method as input. The LayoutGPT method is available at the following URL as of the priority date of this patent application: https: / / layoutgpt.github.io / .

[0256] Dataset Random.

[0257] The method allows you to randomly select a set of semantic categories from the training set to generate a test 3D layout.

[0258] Large Language Model (LLM).

[0259] The usage method can randomly select a set of semantic categories from the collection generated by the LLM. More precisely, the usage method can include the LayoutGPT method to generate the set of semantic categories, and can randomly select a set from those sets that have the same number of objects as the ground reality test scenario under consideration.

[0260] Large Language Model and Self-Score Evaluation (LLM+SSE).

[0261] This setup is similar to the LLM setup above, but instead of randomly selecting the set of semantic categories as above, it first employs a usage method (i.e., a self-scoring evaluation method) to select the most appropriate set of semantic categories from the set of semantic categories (candidates) that have the same number of objects as the ground reality scene. Therefore, this setup is directly equivalent to the previous one (LLM) and evaluates the effectiveness of the usage method (SSE).

[0262] The LLM-based method is implemented using the publicly available Llama-3-8B model (available at the following URL on the priority date of this patent application: https: / / huggingface.co / meta-llama / Meta-Llama-3-8B).

[0263] Since LLMs often produce out-of-distribution sets, using DeBaRA (i.e., pre-configured functions) on the generated semantic categories, along with the disclosed SSE process, consistently improves the realism and effectiveness of the synthesized indoor scenes.

[0264] Quantitative metrics used to evaluate the realism and diversity of the generated 3D layouts can be 256. 2 Fréchet Inception Distance (FID), Kernel Inception Distance (KID x 1000), and Scene Classification Accuracy (SCA) calculated on top-down orthographic rendering can be reported. The accumulated out-of-bounds object area (OBAm) can also be used to calculate these values. 2 To further evaluate the effectiveness of the generated space, all metrics can be computed across each test subset. FID and KID compare the distribution of visual features extracted from the pre-trained convolutional neural network. SCA measures how well the convolutional neural network distinguishes between real-world scenes (i.e., ground-based test scenes) and generated scenes in a binary classification task. Therefore, an SCA score closer to 50% is better, meaning the generated scene is indistinguishable from the real scene.

[0265] The following table presents a quantitative comparison between three known methods for 3D scene composition (e.g., LayoutGPT, ATISS, DiffuSecene) and the pre-configured function DeBaRa, combining four different methods to obtain the semantic category set as outlined above in the specification:

[0266]

[0267] Figure 6 A top-down view of a 3D scene generated by DeBaRa from LLM and their associated SSE values ​​is shown. Clearly, result S170, with a lower score, corresponds to a more natural and realistic 3D layout compared to results S180 and S190, which correspond to higher SSE values.

[0268] For completeness, the generation times with and without the disclosed condition evaluation methods are also provided. The generation time was averaged across the living room test subset. The result is:

[0269] - Generate a single layout using 50 sampling steps: 0.488 seconds;

[0270] - Generate and evaluate 16 candidates using 50 sampling steps and 100 evaluation trials: 0.894 seconds (performed in batches);

[0271] - Generating a single layout using DiffuSecene: 32.796 seconds.

[0272] The computation time is based on a single GPU (NVIDIA RTX A6000). Although the proposed solution introduces additional network application steps through the use of the disclosed conditional evaluation method, it provides fast, real-time generation of 3D layouts in less than one second.

[0273] The learning method is a machine learning approach to a model, specifically a deep generative model. As is known in the field of machine learning itself, a model's processing of inputs involves applying operations to those inputs, defined by data including weight values ​​or parameters. Therefore, learning a model (e.g., a neural network or regressor) involves determining the values ​​of the weights / parameters based on a dataset configured for this learning; this dataset may be referred to as a learning dataset or training dataset. For this purpose, a dataset comprises data pieces, each forming a corresponding training sample or training example. Training samples / training examples represent the diversity of scenarios on which the model will be used after learning. Any training dataset in this document may include multiple training samples / training examples exceeding 1000, 10000, 100000, or 1000000. In the context of this disclosure, "performing training on a dataset" means that the dataset is the model's learning / training dataset, and the values ​​of the weights / parameters are set based on this learning / training dataset. In this disclosure, the training dataset is a dataset of obtained training examples on which the deep generative model is learned / trained. In some implementations, the training dataset consists of several hundred examples, each corresponding to a different HPP configuration.

[0274] As is known from machine learning, a neural network can be defined by its architecture, parameters, and hyperparameters. The architecture consists of layers, starting with an input layer whose neuron count can be determined by the dimension of the input data. Following this layer are several hidden layers with a given number of neurons and activation functions. These layers and neurons define the depth and width of the network, while the activation functions introduce non-linearity into the model. The output layer can have as many neurons as there are variables in the output data. The interconnections between these layers define the topology of the neural network. The parameters of the neural network are learnable weights and biases that are determined during training. Conversely, hyperparameters are predefined settings that are not learned from the training data. These include the number of hidden layers, the number of neurons per layer, and so on. To train a neural network, at least two settings can be defined. First, the loss function, which is a measure of the error (such as mean squared error, MSE) between the training data and the model's predictions. Second, the optimizer, which modifies the model's weights and biases during training to minimize the loss function. Each optimizer has its own set of hyperparameters.

[0275] The method is implemented by a computer. This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the execution of the steps by a computer may be fully automatic or semi-automatic. In the example, the triggering of at least some steps of the method can be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the need to fulfill the user's wishes. In the example, this level may be user-defined and / or predefined.

[0276] A typical example of a computer implementation of this method is to execute it using a system suitable for this purpose. This system may include a processor coupled to memory and a graphical user interface (GUI), on which a computer program containing instructions for executing the method is stored. The memory may also store a database. The memory is any hardware suitable for such storage and may comprise several physically distinct parts (e.g., one for the program, and perhaps one for the database).

[0277] Figure 7 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0278] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus (BUS) 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer also includes a graphics processing unit (GPU) 1110, which is associated with video RAM 1100 connected to the bus. Video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices (such as hard disk drives 1030). Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; and magneto-optical disks. Any of the above may be supplemented or incorporated by a specially designed application-specific integrated circuit (ASIC). A network adapter 1050 manages access to a network 1060. The client computer may also include a tactile device 1090, such as a cursor control device, a keyboard, etc. The cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0279] A computer program may include computer-executable instructions, which include means for causing the system to perform the method. The program may be recorded on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as means, for example, tangibly embodied in a machine-readable storage device for use in a product executed by a programmable processor. The method steps may be executed by a programmable processor that executes the program of instructions to perform the method by manipulating input data and generating output. Therefore, the processor may be programmable and coupled to receive data and instructions from the data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or, if necessary, in assembly or machine language. In any case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. The application of the program on the system in any case results in instructions for performing the method. Alternatively, the computer program can be stored and executed on a server in a cloud computing environment, where the server communicates with one or more clients across a network. In this case, the processing unit executes the instructions included in the program, thereby enabling the method to execute on the cloud computing environment.

Claims

1. A computer-implemented method, comprising: - Obtain (S10) the machine learning function, which is pre-configured as follows: • Using an input 3D layout and a given noise level, the 3D layout has a set of layout parameters including a planar view, a 3D arrangement of one or more 3D bounding boxes, and a set of layout parameters for the semantic category of each 3D bounding box, each bounding box being defined in the 3D arrangement by a predetermined set of values ​​for one or more bounding box parameters, the input 3D layout comprising: Given a floor plan, a first 3D arrangement of one or more given 3D bounding boxes, each bounding box in the first 3D arrangement being defined by a first value of a predetermined set of one or more bounding box parameters, and For each given semantic category of a given 3D bounding box, and • Predict the output 3D layout, the output 3D layout including: The given plan, The second 3D arrangement of one or more given 3D bounding boxes, each bounding box in the second 3D arrangement being defined by a second value of the predetermined set of one or more bounding box parameters, and For each given semantic category of a given 3D bounding box, The function is configured to predict a second value of a predetermined set of one or more bounding box parameters, the second value being different from a first value in the predetermined set of one or more bounding box parameters; and The function is also pre-configured with conditional discarding for at least one layout parameter, which includes a planar graph and / or semantic category for each 3D bounding box; - Obtain (S20) a set of conditional inputs, each conditional input including a different value of one of the at least one layout parameters and the same value of each of the other layout parameters in the planar graph and the semantic category for each 3D bounding box; - Enter the following for each condition: • Determine (S30) one or more conditional candidate 3D layouts, each conditional candidate 3D layout being the result of iterative sampling using the pre-configuration function; • Determine (S40) multiple perturbation-induced conditional candidate 3D layouts, each by adding corresponding noise to the corresponding conditional candidate 3D layout; • For each conditional candidate 3D layout after perturbation: The pre-configured function is applied (S50) to the perturbed conditional candidate 3D layout. One of the layout parameters is discarded, thereby obtaining the corresponding unconditional output; and οCalculate (S60) the reconstruction error between the corresponding conditional candidate 3D layout and the corresponding unconditional output; and • The reconstruction error is averaged over the multiple perturbated conditional candidate 3D layouts (S70) to obtain a score.

2. The method according to claim 1, wherein, Iterative sampling using the pre-configured function includes iteration: • Noise is injected into the input 3D layout to obtain a perturbed input 3D layout; • The pre-configured function is applied at least once to the perturbed input 3D layout to obtain the output 3D layout; as well as • Use the output 3D layout as input for the next iteration; Optionally, the noise has a level that decreases with the depth of the iteration.

3. The method according to claim 2, wherein, Applying the pre-configured function at least once in each iteration includes: • The pre-configured function is applied to the perturbed input 3D layout to obtain the first output 3D layout; • The first intermediate 3D layout is obtained by calculating the gradient step size between the perturbed input 3D layout and the first output 3D layout; • Apply the pre-configured function to the first intermediate 3D layout to obtain the second output 3D layout; and • The second intermediate 3D layout is obtained by calculating the gradient step size between the perturbed input 3D layout and the second output 3D layout, thereby obtaining the final 3D layout.

4. The method according to any one of claims 1 to 3, wherein, The one or more conditional candidate 3D layouts include the final result of the iterative sampling.

5. The method according to claim 4, wherein, Adding the corresponding noise to the final result of the iterative sampling includes: sampling the noise level and sampling the corresponding noise based on the sampled noise level.

6. The method according to any one of claims 1 to 5, wherein, The one or more conditional candidate 3D layouts include one or more intermediate results from the iterative sampling.

7. The method according to claim 6, wherein, Adding the corresponding noise for each intermediate result of the iterative sampling includes: sampling the corresponding noise according to the noise level of the intermediate iteration of the iterative sampling corresponding to the intermediate result.

8. The method according to any one of claims 1 to 7, wherein, The method further includes: starting with the lowest score, sorting the candidate 3D layouts according to their corresponding scores.

9. The method according to any one of claims 1 to 8, wherein, The pre-configured function is parameterized as follows: in: ·x σ The first 3D arrangement represents one or more given 3D bounding boxes. · This represents the given plan view. ·c i It is a list of the given semantic categories. σ is the given noise level. · It is a noisy conditional scoring network with a set of trainable parameters θ. ·c skip (σ) is the noise-dependent pre-adjustment coefficient for modulation prediction of 3D layout. ·c noise (σ) is a noise dependence coefficient that modulates the noise level within the scoring network, and ·c in (σ) and c out (σ) are two values ​​scaled by x respectively. σ and S θ (c in (σ)x σ ;y,c noise The noise dependence coefficient of (σ)).

10. A computer-implemented method for machine learning of a machine learning function used in the method according to any one of claims 1 to 9, wherein, The machine learning methods include: - Obtain a dataset of (S80) ground reality 3D layouts, each ground reality 3D layout representing the corresponding scene and including: • Corresponding floor plan • A corresponding 3D arrangement of one or more 3D bounding boxes, each bounding box being defined by a predetermined set of values ​​for one or more bounding box parameters, and • For each 3D bounding box, there is a corresponding semantic category; - Obtain the probability distribution of the (S90) noise level; - For each ground-based 3D layout, obtain (S100) a corresponding perturbed 3D layout, wherein the perturbed 3D layout can be calculated by perturbing at least one bounding box parameter of at least one 3D bounding box of the ground-based 3D layout, wherein the perturbation includes: • Sample the corresponding noise level based on the probability distribution (S100a); and • For each corresponding bounding box parameter of the at least one bounding box parameter: o Sample the corresponding noise value based on the corresponding noise level (S100b); and Apply the corresponding noise value (S100c) to the corresponding bounding box parameter; and - The function is trained on the dataset based on a loss (S100), the loss penalizing a dissimilarity metric between each ground-based real 3D layout and its corresponding predicted 3D layout, the corresponding predicted 3D layout being obtainable by applying the function to the corresponding perturbed 3D layout. The training is performed with regard to at least one layout parameter in a conditional discard manner, the at least one layout parameter including a planar graph and / or a semantic category for each 3D bounding box.

11. The method according to claim 10, wherein, The dissimilarity metric is of the following type: in: · It is a set of one or more 3D bounding boxes in the candidate 3D layout of the ground reality. It is a set of one or more 3D bounding boxes in the corresponding predicted 3D layout. ·N is and The common size, and · It is a differentiable distance. Optionally, the differentiable distance further penalizes the dissimilarity between 3D bounding boxes in terms of dimension and / or semantic category, and is, for example, of the following types: in: · And x are respectively The vector of spatial bounding box parameter values ​​of o. · It is the Euclidean norm. K is the penalty parameter. ·when When o has the same dimension, It is an index function that equals 1, otherwise it is 0, and ·when When o and o have the same semantic category, It is an index function that equals 1, otherwise it is 0.

12. The method according to any one of claims 10 to 11, wherein, The loss of the training function is the dissimilarity metric. The product with the noise-dependent weighting function (λ(σ)) Expected value 13. A data structure, comprising: A computer program comprising instructions for performing the method according to any one of claims 1 to 9 and / or the method according to any one of claims 10 to 12; and / or A machine learning function, which has been trained by the method according to any one of claims 10 to 12.

14. A computer-readable storage medium having the data structure of claim 13 recorded thereon.

15. A system comprising a processor coupled to a memory, the memory storing the data structure of claim 13.

Images