Noise-based 3D layout prediction

Abstract

This document provides methods, data structures, and systems related to 3D layout prediction. [Solution] The method includes obtaining a dataset of ground truth 3D layouts, obtaining a probability distribution of noise levels, obtaining a perturbed 3D layout for each ground truth 3D layout, and training a function that takes the input 3D layout and a given noise level and predicts the output 3D layout. Training is performed on the dataset based on a loss that penalizes the dissimilarity metric between each ground truth 3D layout and its predicted 3D layout.

Description

[Technical Field]

[0001] This disclosure relates to the field of computer programs and systems, and more specifically to methods, data structures, and systems related to 3D layout prediction. [Background technology]

[0002] Some available solutions for 3D scene generation involve machine learning techniques such as generative adversarial networks (GANs).

[0003] Current prior art presents significant limitations when attempting to achieve realistic and diverse scene arrangements. A realistic scene arrangement means that the scene composition is semantically consistent between objects, and between objects and the floor plan, and is also physically meaningful. In other words, realistic scene arrangements tend to promote groups of objects that have strong semantic relationships with each other and with the floor plan, and prefer to place objects in a consistent physical manner. Therefore, realistic scene arrangements tend to discourage, for example, placing a bed in the kitchen (an object that is semantically contradictory to the floor plan), placing an oven next to a bed (objects that are semantically contradictory to each other), and placing objects without the necessary physical support (e.g., a teacup floating instead of being placed on a table). [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Thus, improved solutions for predicting 3D layouts are still needed. [Means for solving the problem]

[0005] Accordingly, a computer implementation method for machine learning is provided. The machine learning method includes obtaining a dataset of ground truth 3D layouts. Each ground truth 3D layout represents a scene. Each ground truth 3D layout includes a floor plan, a 3D arrangement of one or more 3D bounding boxes, and a semantic category for each 3D bounding box. Each bounding box is defined by a predetermined set of values ​​of one or more parameters. The machine learning method further includes obtaining a probability distribution of noise levels. The machine learning method also includes obtaining a perturbed 3D layout for each ground truth 3D layout. Each perturbed 3D layout is a 3D layout that can be computed by perturbing at least one parameter of at least one 3D bounding box of the ground truth 3D layout. Perturbing includes sampling each noise level based on the probability distribution. Perturbing also includes sampling each noise value for each of the at least one parameter based on the noise level, and applying each noise value to each parameter. The machine learning method further includes training a function. The function is configured to take in an input 3D layout and a given noise level. The input 3D layout includes a given floor plan, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters. The function is also configured to predict an output 3D layout. The output 3D layout includes a given floor plan, a second 3D arrangement of the one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters.The function is further configured to predict a second value of a predetermined set of one or more parameters that is different from a first value of the predetermined set of one or more parameters. Training is performed on a dataset based on a loss that penalizes a dissimilarity metric between each correct 3D layout and each predicted 3D layout obtainable by applying the function to the respective perturbed 3D layout.

[0006] The machine learning method may include one or more of the following features. The dissimilarity metric is of the following type. [Number] Here, [Number] is a set of one or more 3D bounding boxes in the correct 3D layout, [Number] is a set of one or more 3D bounding boxes in each predicted 3D layout, N is [Number] and [Number] is the common size of [Number] is a differentiable distance; The differentiable distance penalizes dissimilarities in dimensions and / or semantic categories between 3D bounding boxes and is, for example, of the following type. [Number] Here,

number

number

number

number

number

number

number

number

number

number

number

number

number

[0007] Further, a computer implementation method for using a function trained according to this machine learning method is provided. This method includes obtaining an input 3D layout and a given noise level. The input 3D layout includes a given floor plan, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters. The method also includes applying the function trained starting from the input 3D layout one or more times to predict an output 3D layout. The output 3D layout includes a given floor plan, a second 3D arrangement of the one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters.

[0008] This method of use may include one or more of the following features: Applying a trained function one or more times includes the following iterations: Injecting noise into the input 3D layout and thereby obtaining a perturbed input 3D layout; Applying a trained function at least once to a perturbed input 3D layout, thereby obtaining an output 3D layout; and Use the output 3D layout as input for the next iteration; The noise has a level that decreases with the depth of the repetitions; and / or Applying the trained function at least once means that in each iteration, the following is performed: Applying a trained function to a perturbed input 3D layout to obtain a first output 3D layout; Obtain a first intermediate 3D layout by calculating the gradient step between the perturbed input 3D layout and the first output 3D layout; Applying a trained function to the first intermediate 3D layout to obtain a second output 3D layout; and Obtain a second intermediate 3D layout by calculating the gradient step between the perturbed input 3D layout and the second output 3D layout, thereby obtaining the final 3D layout.

[0009] Further provided are computer programs containing instructions for performing machine learning methods and / or usage methods, and / or data structures containing functions trained by machine learning methods.

[0010] A device including a data storage medium on which a data structure is recorded is further provided.

[0011] The device may form or function as a non-temporary computer-readable medium, for example, in a Software as a Service (SaaS) or other server or cloud-based platform. Alternatively, the device may include a processor coupled to a data storage medium. The device may therefore form a computer system, either entirely or partially (for example, the device is a subsystem of the overall system). The system may further include a graphical user interface coupled to the processor. The following is a non-limiting example, with reference to the attached diagram. [Brief explanation of the drawing]

[0012] [Figure 1] An example flowchart of this machine learning method is shown. [Figure 2A] A flowchart illustrating an example of this usage is shown below. [Figure 2B] A flowchart illustrating an example of this usage is shown below. [Figure 2C] A flowchart illustrating an example of this usage is shown below. [Figure 3] This is a conceptual diagram of an example solution. [Figure 4] Let's illustrate the solution. [Figure 5] Let's illustrate the solution. [Figure 6] Let's illustrate the solution. [Figure 7] Let's illustrate the solution. [Figure 8] Let's illustrate the solution. [Figure 9] Let's illustrate the solution. [Figure 10] Let's illustrate the solution. [Figure 11] Let's illustrate the solution. [Figure 12] Let's illustrate the solution. [Figure 13] Let's illustrate the solution. [Figure 14] Let's illustrate the solution. [Figure 15] Let's illustrate the solution. [Figure 16] Let's illustrate the solution. [Figure 17] Let's illustrate the solution. [Figure 18] Let's illustrate the solution. [Figure 19] An example of this system is shown. [Modes for carrying out the invention]

[0013] Referring to the flowchart in Figure 1, a computer implementation method of machine learning that may be useful for noise-based prediction of scene placement is proposed. The machine learning method includes obtaining a dataset of ground truth 3D layouts (S10). Each ground truth 3D layout represents a respective scene. Each ground truth 3D layout includes a floor plan, a 3D placement of one or more 3D bounding boxes, and a semantic category for each 3D bounding box. Each bounding box is defined by a predetermined set of values ​​of one or more parameters. The machine learning method further includes obtaining a probability distribution of noise levels (S20).

[0014] The machine learning method also includes obtaining each perturbed 3D layout for each ground truth 3D layout (S30). Each perturbed 3D layout is a 3D layout that is computable (e.g., computed, e.g., the method includes such computation) by perturbing at least one parameter of at least one 3D bounding box of the ground truth 3D layout (e.g., the method includes such perturbation). In other words, the machine learning method may include computed at least one (e.g., each) each perturbed 3D layout, and / or retrieved (e.g., on local or remote memory) or received (e.g., from a remote third-party computer system) at least one each perturbed 3D layout retrieved or received, which is pre-computed. Perturbation includes sampling each noise level based on a probability distribution (S30a). Perturbation also includes sampling each noise value for each of the at least one parameter based on its respective noise level (30b), and applying each noise value to each parameter (30c). The machine learning method further includes training (S40) (and outputting) a function. The function (after training S40) is configured to take an input 3D layout and a given noise level and predict (i.e., output or generate) an output 3D layout.

[0015] The input 3D layout includes a given floor plan, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters. The output 3D layout includes the (same) given floor plan, a second 3D arrangement of the (same) one or more given 3D bounding boxes, and a (same) given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters. The function is configured to predict a second value for a given set of one or more parameters, which is different from a first value for a given set of one or more parameters (a given floor plan, a given noise level, and a given semantic category for each given 3D bounding box are constants of the function, i.e., the function does not change their values, but a given set of one or more parameters in the 3D arrangement of one or more given 3D bounding boxes is allowed to change, i.e., the function may change their values).

[0016] Training is performed on the dataset based on a loss that penalizes the (same) dissimilarity metric between each ground truth 3D layout and each predicted 3D layout obtainable (i.e., obtainable) by applying a function to each perturbed 3D layout.

[0017] Such machine learning methods form an improved solution for predicting 3D layouts.

[0018] In particular, the function is trained in S40 to become a denoiser capable of transforming the input 3D layout into a more realistic output predicted 3D layout by denoising it. The function trained in S40 is also flexible, meaning that the trained function can be used to perform different tasks, as will be discussed later.

[0019] Referring to the flowchart in Figure 2a, the function can therefore be used in a computer implementation, which includes obtaining an input 3D layout and a given noise level (S50). The input 3D layout includes a given floor plan, a first 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters. The usage also includes applying the trained function, starting from the input 3D layout, one or more times to predict the output 3D layout (S60). In other words, the trained function is first applied to the input 3D layout, and then the trained function may optionally be applied one or more additional times, each time applied to the input obtained based on the previous output of the trained function. The output 3D layout (finally predicted, i.e., after the application of the function or all applications) includes a given floor plan, a second 3D arrangement of the one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters.

[0020] Thanks to the function trained according to the machine learning method shown in Figure 1, the output 3D layout obtained from S60 is a more realistic version of the input 3D layout provided by S50.

[0021] In fact, machine learning methods employ a data-driven approach that allows a function to learn placement patterns and relationships between objects and between objects and a constrained environment in order to predict realistic 3D layouts. In other words, the data-driven approach of machine learning methods allows a trained function to learn interactions between 3D objects (i.e., semantic consistency) and interactions between 3D objects and a constrained environment (i.e., spatial inference) solely from the training dataset.

[0022] For example, the training dataset acquired in S10 may include realistic and diverse ground truth 3D layouts, each representing a 3D scene. 3D scenes may be acquired from digital 3D scene datasets and / or real-world 3D scenes. An example of a digital 3D scene dataset is the HomeByMe® dataset, or any subset thereof that may include at least 1000 scenes (e.g., 10K scenes) containing, for example, at least 20 objects with densely arranged furniture.

[0023] The machine learning method is further trained on the dataset based on a loss that penalizes the dissimilarity between the ground truth 3D layout and each predicted 3D layout. In particular, such a loss may be invariant under permutations of identical 3D objects. Such an option can facilitate training and avoid penalizing predicted 3D layouts where identical objects are swapped with the ground truth 3D layout, thus forcing diversity in 3D scene generation.

[0024] In addition, machine learning methods employ a denoising approach to generate predicted outputs. The denoising approach performs better than other classes of existing generative models, such as GAN models. Machine learning methods involve injecting different noise levels based on acquired probability distributions to perturb samples in the training dataset. Furthermore, the model (i.e., the denoiser) is "noise-conditional," meaning it is configured to be applied to input samples with a given value of the noise level (i.e., the noise level is given as input to the model as a "condition"). Such a noise-based approach allows the trained function to learn from a perturbed (i.e., noisy) dataset, and thus provides the trained function with a remarkable ability to denoise the input and predict realistic and diverse 3D scene layouts. In other words, a machine learning method may train a function that best "denoses" an input 3D layout at an arbitrary (i.e., arbitrary) noise level to predict a realistic (i.e., natural-looking) 3D layout.

[0025] Furthermore, the denoising-based approach improves 3D placement of densely furnished scenes, such as real-life scene arrangements; that is, the trained function predicts more realistic and diverse 3D scenes containing a large number of objects (e.g., at least 20) (compared to autoregressive methods that would predict arrangements where objects are inserted sequentially, i.e., one at a time). For example, the proposed method generates plausible (e.g., realistic and diverse) 3D placements for well-furnished scenes containing at least 20 objects and has the advantage of being scalable up to at least 50 objects. The dataset acquired in S10 may include ground truth 3D layouts containing at least 20 objects and / or ground truth 3D layouts containing at least 40 objects. The input 3D layouts acquired in S50 may contain at least 20 objects or at least 40 objects, respectively. The proposed method actually achieves better (i.e., more accurate) results in terms of the physical consistency and realism of the predicted 3D layouts. This is because the denoising approach allows the trained function to simultaneously learn the relationships between 3D objects, that is, to acquire non-trivial interdependencies between 3D objects and between each 3D object and a given floor plan, for example, using a self-attention mechanism. Simultaneously (i.e., all at once) means that during the training of the function S40, a first value of a given set of one or more parameters of each 3D bounding box in the first 3D arrangement may be input simultaneously. In other words, the function may take all 3D objects in the input 3D layout as input simultaneously. To put it another way, the trained function captures all spatial and semantic relationships in order to obtain realistic and diverse scene arrangements. Similarly, a second value (i.e., a value predicted by the function) of a given set of one or more parameters of each 3D bounding box in the second 3D arrangement may also be output simultaneously (not one object at a time, e.g., one object after the other).Such simultaneous processing of one or more parameters of each 3D bounding box corresponds to better object grouping, i.e., the ability of a function to identify objects that can be associated together in a predicted 3D arrangement.

[0026] As shown in the flowchart of Figure 2b, one or more applications of the function (trained in S40) in S60 may optionally include obtaining a perturbed input 3D layout by injecting noise into the input 3D layout (S60a), obtaining an output 3D layout by applying the trained function to the perturbed input 3D layout at least once (S60b) (for example, (i) as the direct result of one application of the trained function, or (ii) as the result of processing the result of one application of the trained function and the output obtained thereafter, or (iii) as the result of multiple applications of the trained function, each time starting from the direct result of the previous application of the trained function, or from the result of processing the result of the previous application of the trained function and the output obtained thereafter), and using the output 3D layout as input to the next iteration (S60c).

[0027] Therefore, the method of use may form a denoising diffusion model. Such a model allows for the arrival of an accurate and realistic 3D layout through an iterative sampling process (i.e., iterative denoising). Iterative sampling corresponds to one or more applications S60 of a trained function starting from the input 3D layout to predict the output 3D layout. The iterative sampling approach allows the trained function to improve the quality of the predicted 3D layout. In other words, the predicted 3D layout corresponds to a more natural-looking 3D layout that exhibits improved quality of realism and diversity in the 3D scene.

[0028] Furthermore, the usage may correspond to a time-efficient iterative sampling process involving a trade-off between sampling time and the quality of the predicted 3D layout, which can be configured by the user. For example, the usage may support efficient batch processing techniques and / or parallelization capabilities (e.g., on a GPU) for generating 3D scene layouts for multiple scenes and / or multiple arrangements of a single scene in a single iterative sampling process.

[0029] Furthermore, the trained function may be applied one or more times within an iterative process with a decreasing noise level in each iteration. In particular, the injection of noise into the input 3D layout (S60a) in each iteration may include noise level scheduling such that the injected noise has a level that can decrease with depth in the iteration. Such noise level scheduling allows for an improvement in the quality of the predicted 3D layout. Thus, the trained function is optimized to denoise the input 3D layout perturbed with different noise levels during the inference phase. In other words, because the trained function is noise-conditional, it has the ability to denoise the input 3D layout to generate (i.e., predict) a realistic 3D layout.

[0030] The usage method should take full advantage of the flexibility of the trained function, which can be used in different applications. For example, it may be implemented for 3D placement generation, partial 3D placement generation, 3D rearrangement, and / or grouping of 3D objects in a given floor plan. In particular, iterative denoising may be adapted according to the specific application.

[0031] For example, in 3D placement generation, the position of a 3D object is arbitrarily initialized at the center of the room, and the rotation and / or dimensions of the 3D object are randomly initialized. In such an example, iterative denoising is performed starting from a sufficiently high noise level and run for at least 30 steps, thus creating a fair compromise between the quality of the predicted 3D placement and the sampling time.

[0032] In a partial 3D placement generation application, some 3D objects have their position and / or dimensions and / or rotation values ​​already known. Therefore, these objects are initialized to their known values, while the objects to be placed have a position initialized to the center of the room and a randomly initialized rotation. At each denoising step, the model output for known 3D objects (i.e., 3D objects with known values ​​for their position and rotation parameters) may be replaced with their original position and orientation (i.e., rotation) values.

[0033] Note that the model output for known 3D objects may be replaced with their perturbed position and rotation values, which are noised to a level corresponding to the current sampling step. In either case, these known objects will eventually converge to their initial values ​​throughout the entire sampling process.

[0034] In other applications such as 3D relocation, the position and rotation of 3D objects are initialized to their noisy (i.e., perturbed) values. The denoising process may be performed starting from a lower noise level value than that used for the 3D placement generation task.

[0035] A function trained according to S40 takes a 3D layout and a given noise level as input. Each 3D layout is a set of data containing a given floor plan, the 3D arrangement of one or more 3D bounding boxes, and, for each 3D bounding box, its respective semantic category. In other words, a 3D layout represents the arrangement of one or more 3D bounding boxes within a given floor plan. A 3D bounding box of a 3D object is the smallest rectangular prism surrounding the 3D object, with or without orientation constraints (such as the constraint that the rectangular prism must have faces parallel to the horizontal plane). Thus, a 3D bounding box is characterized by its spatial attributes (i.e., its position, its dimensions, and optionally its (unconstrained) orientation parameter) and its semantic category (i.e., the class of the object, e.g., those with the same function, e.g., books, chairs, etc.). A predetermined set of one or more parameters may describe the spatial attributes of a 3D object. Each object spatial attribute may have a separate real-world interpretation. For example, a given set of one or more parameters may include 3D position coordinates, three dimensions (i.e., height, depth, and length), and at least one parameter representing the object's orientation (e.g., cosine and sine of the angle around the vertical axis). Thus, a given set of one or more parameters may include eight parameters or consist of eight parameters. The use of a 3D bounding box captures the three-dimensional positioning of a 3D object. Therefore, thanks to the use of a 3D bounding box, the trained function predicts accurate and realistic 3D positioning and sizing of a 3D object. In particular, the trained function, and its resulting use, predicts a 3D layout that exhibits physically consistent positioning in three dimensions, and thus avoids subtle defects that impair the perceived validity of the overall scene, such as overlapping, floating or out-of-bounds objects, inaccessible areas, and inconsistent object positioning.

[0036] A floor plan is data that describes the plane of a scene where 3D objects can be placed; that is, it represents the corners of rooms. Therefore, the floor plan sets the boundaries of the 3D scene placement and conditions the 3D output layout. The floor plan may be obtained in S10 from an external 3D database in the machine learning method. During training, the floor plan may be rotated by a random angle along the vertical axis. In the usage method, the floor plan input in S50 may be imported from the real world by 3D scanning technology.

[0037] Similarly, 3D objects may be obtained in S10 from an external database and / or online catalog in the machine learning method. During inference, 3D objects input in S50 in the usage method may be imported from the real world.

[0038] Additionally or alternatively, the method of use may further include arranging real-world rooms according to a predicted layout, i.e., each 3D object ultimately has a corresponding real-world physical object positioned and oriented in the real-world room according to the predicted layout. Thus, the method of use may reproduce a variety of realistic 3D scene arrangements that are feasible in the user's real-world home / apartment. In other words, the method of use is user-driven, i.e., it facilitates real-life user interaction to generate 3D layouts similar to real-world 3D scenes (e.g., in a design planner application). For example, the user may design a floor plan, select a list of 3D objects to generate several 3D layouts, and finally select an output (e.g., an output with the 3D arrangement best suited to the purpose of interior design). As an example of the method of use, the user may input their own floor plan (e.g., from their own home or apartment) along with a list of furniture (i.e., a list of semantic categories of 3D objects to be placed) in order to obtain various 3D arrangements. In another scenario, a user might want to rearrange a 3D scene that already has furniture in place by selecting several 3D objects that need to be repositioned, while leaving the others fixed in their positions.

[0039] The given noise level is sampled from the acquired probability distribution in the machine learning method (S30a) and obtained (i.e., provided) in the usage method (S50). For example, the probability distribution obtained in S20 may be a Gaussian distribution.

[0040] During training, noise levels are introduced to perturb the dataset, i.e., for each ground truth 3D layout, each perturbed 3D layout is obtained and / or calculated. A high level of noise means the perturbed 3D layout is "far" from the ground truth 3D layout, and a low level of noise means the perturbed 3D layout is "close" to the ground truth 3D layout. In other words, the machine learning method obtains each perturbed 3D layout (S30) by sampling different noise levels based on a probability distribution for each ground truth 3D layout (S30a), sampling different noise values ​​based on their respective noise levels for each parameter of at least one of the parameters (S30b), and applying their respective noise values ​​to each parameter of one or more 3D bounding boxes in the 3D arrangement (S30c). Thus, the function acquires the ability to predict 3D scenes perturbed at different noise levels.

[0041] Acquisition of at least one (e.g., each) perturbed 3D layout (S30) may include perturbing at least one parameter of at least one (e.g., each) 3D bounding box of the ground truth 3D layout, or retrieving the results of such perturbations (e.g., on local or remote memory) or receiving them (e.g., from a remote computer). Perturbation includes sampling each noise level (S30a) based on a probability distribution in a real interval. The noise level is a positive (e.g., real) number. The noise level may be the magnitude to which the parameters (i.e., spatial attributes) of the 3D bounding boxes are generally perturbed. The noise level may be the absolute value of a scalar drawn from a probability distribution. For example, the noise level may be a Gaussian distribution.

number

Number

[0042] The acquired dataset of correct scenes (S10) may be additionally augmented by random rotation of the scenes along the vertical axis. This random data augmentation can help improve the training for predicting layout scenes having walls not aligned with at least one coordinate axis.

[0043] The function is trained in S40 to be configured to take as input a 3D layout and a given noise level. The input 3D layout includes a given floor plan, a first 3D layout of one or more given 3D bounding boxes, and for each given 3D bounding box, a given semantic category. Each bounding box is defined in the first 3D layout by a first value of a predetermined set of one or more parameters. In other words, in a given floor plan, each 3D bounding box (labeled by a semantic category) may be defined by a first value of its spatial attributes that define its position, dimensions, and orientation within the scene.

[0044] The function is configured to predict an output 3D layout. The output 3D layout includes a given floor plan, a second 3D arrangement of one or more given 3D bounding boxes, and a given semantic category for each given 3D bounding box. Each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters. The function is further configured to predict a second value of a predetermined set of one or more parameters that is different from a first value of the predetermined set of one or more parameters. The given floor plan and the given semantic category for each given 3D bounding box may be constants of the function, i.e., the function does not change their values. "A second 3D arrangement of one or more given 3D bounding boxes" means that one or more given 3D bounding boxes may be arranged within the same given floor plan in such a way that the predicted second value of its spatial attribute is different from the first value of its spatial attribute (i.e., the input). In other words, the trained function predicts a second 3D arrangement of one or more 3D bounding boxes by predicting second values ​​for the position, dimensions, and orientation of the 3D bounding boxes. Other variables of the function may remain constant, i.e., the predicted 3D arrangement may be performed on the same given floor plan and using the same given list of semantic categories.

[0045] In other words, the function is trained at S40 to do nothing but rearrange the 3D bounding box of the input 3D layout (i.e., using repositioning and / or resizing and / or reorienting).

[0046] The function is trained on a training dataset of ground truth 3D layouts (e.g., the HomeByMe® dataset) and is based on a loss that penalizes the dissimilarity metric between each ground truth 3D layout and each predicted 3D layout obtainable by applying the function to each perturbed 3D layout. Each perturbed 3D layout may be supplied as input to the trained function to obtain its respective predicted 3D layout. Thus, the training loss may evaluate the "distance" between each ground truth 3D layout and its respective predicted 3D layout. The training loss may favor predicted 3D layouts that are "closer" to the ground truth 3D layout.

[0047] For example, the dissimilarity metric may be of the following types:

number

number

number

number

number

number

[0048] Such a dissimilarity metric would therefore correspond to the Chamfer distance, which measures the dissimilarity between sets of bounding boxes. In such an example, the dissimilarity metric is the set of one or more 3D bounding boxes in the ground truth 3D layout.

number

number

number

number

[0049] The dissimilarity metric is a computationally efficient differentiable distance.

number

[0050] Differentiable distances may penalize dissimilarity in dimensions and / or semantic categories between 3D bounding boxes, for example, of the following types:

number

number

number

number

number

number

number

number

[0051] For example, for each pair of 3D bounding boxes—one in a set of one or more 3D bounding boxes in the ground truth 3D layout and another in a set of one or more 3D bounding boxes in each predicted 3D layout—the differentiable distance may calculate the Euclidean norm between the values ​​of their spatial parameters (e.g., spatial attributes such as position and orientation). The differentiable distance may also assess the dissimilarity between pairs of 3D bounding boxes in terms of dimensions and / or semantic categories. Thus, the dissimilarity distance may be named the "semantic-aware dissimilarity distance" (e.g., semantic-aware Chamfer distance) because it recognizes (i.e., takes into account) the semantic categories associated with each 3D bounding box when assessing the dissimilarity between them. As a result, a penalty may be applied if pairs of 3D bounding boxes do not share the same spatial dimensions and the same semantic categories. The penalty parameter K may be higher than, for example, 10^4 or 10^6, and may be set to, for example, K = 10^8.

[0052] Based on that, the loss S40 is used to train the function on the dataset, which is of the type

number

number

[0053] The trained function may include an architecture that incorporates a (noise-aware) transformer. A transformer is a type of deep neural network architecture that has a remarkable ability to recognize relationships between elements in an input sequence. Thanks to a mechanism called self-attention, transformers allow the function to learn the relevance of each element to others and appropriately weight contextual information. The transformer module takes a sequence as input and outputs a new vector representation of the input data with relationships in the input sequence emphasized.

[0054] The use of transformers allows a function to learn how to capture the physical and semantic relationships between 3D objects in a scene, and between each 3D object and a given floor plan. This allows for the prediction of a scene with high realism and physical plausibility. For example, a trained function calculates attention scores between each 3D bounding box in a predicted 3D layout. For example, a transformer can handle representations of equal length (i.e., embeddings, e.g.,

number

[0055] The function may further include an encoder for generating representations (e.g., of equal length) that are taken as input by the transformer. For example, the function may include a noise encoder that generates a representation of a given noise level. The following details how the noise encoder generates the representation. The scalar value of a given (e.g., sampled) noise level is deterministically determined by the dimensionality of the scalar value.

number

number

number

number

number

[0056] The function may include a 3D object encoder that generates a first representation of each given 3D bounding box. The 3D object encoder may optionally be configured to generate a representation of each parameter and a representation of a semantic category, and to concatenate all the generated representations. For example, the values ​​of one or more parameters of each 3D bounding box in the scene are such that each value

number

number

number

[0057] The function may further include a floor encoder that generates a representation of a given floor plan. The floor encoder may include a sampling module for generating samples of the given floor plan. A floor (i.e., a floor plan) is a representation of the corners of a room. The sampling module may additionally compute points (e.g., at least 100 points) along the walls of the floor plan (i.e., rooms). This ensures that the boundaries of the floor plan are represented by a 3D point cloud, which is a standard data type that is easier to encode in a deep learning network.

[0058] During training, data augmentation may be performed by rotating the scene by a random angle along the vertical axis. Data augmentation helps reduce overfitting of the training data and results in a more robust model for learning the shapes of complex floor plans.

[0059] A floor encoder may also include a point cloud encoder. For example, the point cloud encoder may be PointNet. The 3D point cloud of a sampled floor plan is then fed to a point cloud module (e.g., a PointNet module), which outputs an embedding vector representing the given floor plan. The floor encoder takes the embedding vector of the given floor plan as input and generates a representation of the given floor plan (e.g.,

number

[0060] The function may further include an MLP that takes a representation of the predicted 3D layout (i.e., the aforementioned -higher-dimensional-representation which is the output of the transformer) as input and outputs a third representation of each given 3D bounding box. Any further processing of the third representation of each given 3D bounding box may be deterministic, i.e., it does not involve neural training. The third representation of each given 3D bounding box is the predicted position, dimensions, and orientation of each 3D bounding box in the predicted scene (e.g.,

number

[0061] During the training phase, the function may be parameterized by a noise-conditional denoiser. The parameterization may be of the following types:

number

number

number

number

[0062] Such parameterization facilitates training the function and helps it learn (i.e., capture) the relationship between perturbed and clean constructions.

[0063] Noise-conditional denoiser that parameterizes the trained function

number

number

[0064] Parameterization of a noise-conditional denoiser is performed using a noise-conditional score network S with a set of trainable parameters θ. θ This may include: Score-based generative models are a class of deep generative models that employ neural networks to learn scores (i.e., gradients of the log-probability density function) of noise-level dependent marginal distributions obtained by perturbing training samples with different levels of noise (e.g., Gaussian noise). In other words, score-based generative models are trained to model data distributions by denoising samples perturbed with different noise levels. The parameterization of the score-based objective function may be carefully designed to facilitate learning of the underlying data distribution. This parameterization may involve noise-dependent modulation of the score network's output, as well as scaling of the model's inputs, outputs, and noise levels. In fact, a noise-conditional score network is a rescaled vector c in (σ)x σ You can also take this as input, and here c in(σ) is the noise dependency coefficient. A noise-conditional score network also has a noise dependency coefficient c that conditions the noise level within the score network. noise (σ) may be taken as input. Additionally, the noise-conditioned score network is a conditioned input representing a given floor plan, a given semantic category, and the respective spatial dimensions of one or more given 3D bounding boxes.

number

number

number

[0065] In the above formula, σ data This is the standard deviation of the training dataset and is calculated before the training phase.

[0066] Returning to the flowchart in Figure 2b, at least one application of the trained function (S60b) may include an implementation of quadratic sampling in each application of the trained function, as described below.

[0067] Referring to the flowchart in Figure 2c, the trained function may obtain a first output 3D layout by applying it to a perturbed input 3D layout (S70a). Secondly, a first intermediate (e.g., middle) 3D layout may be obtained by calculating the gradient step between the perturbed input 3D layout and the first output 3D layout (S70b). Next, the trained function may obtain a second output 3D layout by applying it to the first intermediate 3D layout (S70c). Finally, the final 3D layout may be obtained by calculating the gradient step between the perturbed input 3D layout and the second output 3D layout, thereby obtaining a second intermediate (e.g., middle) 3D layout (S70d). Steps S70a through S70d are called quadratic sampling steps because these steps are performed in each iteration within iterative sampling. The implementation of a second-order sampling step improves the generation of accurate and natural-looking 3D scenes while reducing the number of computationally expensive neural evaluations (i.e., function applications). A perturbed 3D object may be obtained by applying a noise step (i.e., applying a perturbation of a machine learning method) to a 3D object placed at an initial position. The perturbed 3D object is placed at each of its noisy positions. Firstly, a function may be applied to the perturbed 3D object. This yields a first model prediction in which the 3D object may be placed at a first predicted position. Secondly, a first intermediate position (e.g., midpoint) may be computed by applying a gradient step between the noisy position and the first predicted position. Next, the trained function may be applied to the 3D object placed at the computed intermediate position. This yields a second model prediction in which the 3D object is placed at a second predicted position. Finally, the final predicted position may be obtained by computing a second intermediate position (e.g., midpoint) by applying a gradient step between the noisy position and the second predicted position. Examples of visual representations of steps S70a to S70d are shown in Figure 6.

[0068] Figure 3 shows an example of a training solution and neural network architecture.

[0069] In particular, Figure 3 features a transformer encoder-denoiser network that takes as input (i.e., is conditioned) trained encoded representations of a noise level (i.e., magnitude) σ used to perturb the input scene (and thus the denoiser certifies as noise-aware), 3D objects with some of their features perturbed (e.g., position and rotation attributes, or position, rotation, and bounding box dimensions), and additional scene-level conditioned features such as a room layout / shape. It may output a predicted clean 3D object layout. Like other deep learning models, the proposed model features an offline stage (training) and an online stage (also called generation, sampling, or inference).

[0070] The objective of the offline training stage is to simultaneously train individual modules so that the denoiser / score network efficiently reconstructs valid 3D object-space features from perturbations of varying magnitudes. Since the model can be conditioned on a noise scale, it knows how "far" the input noisy spatial configuration is from the target, and whether the model predicts a "clean" configuration from a completely degraded signal (high noise scale) or merely models slight discrepancies to fine-tune the input object layout (low noise scale). This training phase can be facilitated by carefully designed preconditioning of the denoising pipeline and an efficient semantically aware Chamfer distance training objective. Random data augmentations, such as scene rotation, may also be dynamically applied during training to enforce the robustness of the resulting model.

[0071] More precisely, a noise encoder may take the level of noise added to the ground truth 3D object layout as input and return noise encoding tokens. A 3D object encoder may take features representing 3D objects, such as noisy spatial attributes (e.g., their position, rotation, and dimensions), as well as other object-level attributes (e.g., semantic categories), as input and output 3D object encoding tokens for each object. A room encoder may take features relating to a 3D room (e.g., its floor plan / shape) as input and return room encoding tokens. A noise-aware transformer encoder network may then take a sequence of the above tokens as input and compute a new representation for each input token. The output tokens representing 3D objects may be passed to a final MLP, which may return denoised 3D object spatial features (e.g., the position and rotation of each 3D object in the sequence). This training is end-to-end, and a single loss value may be computed and backpropagated to adjust the weights of all trainable modules. The training stage setup may involve the following subtasks:

[0072] The first subtask is data preprocessing. Annotated 3D scene data samples may be processed so that they can be conveniently passed to the relevant modules. This generally involves extracting different features (e.g., 3D room layouts, 3D object bounding boxes, etc.) from the raw scene annotation data and rescaling them to a given range of values.

[0073] The second subtask is architecture design. Individual neural network modules may be designed to effectively compute rich representations of different input features that help denoise the perturbed samples. It should also be noted that the transformer encoder-denoiser takes fixed-size tensor embeddings as input. This may be followed by an MLP to output clean predictions of noisy 3D object-space features.

[0074] The third subtask is the preconditioning of the score network. Parameterization of the score-based objective may be carefully designed to facilitate learning of the underlying data distribution. This may involve noise-dependent modulation of the score network's output, as well as scaling of the model's inputs, outputs, and noise levels.

[0075] The fourth subtask is the selection of the training loss. The training loss function may measure the distance between the predicted 3D object layout from noisy data and the clean, ground truth layout. This loss function may avoid penalizing predictions where identical objects are swapped (for example, a predicted layout where identical pairs of chairs around a table are swapped compared to the ground truth).

[0076] Finally, the last subtask is the training process. Training may be performed by iterating multiple times on a dataset (pairs of randomly noisy and clean layouts at various scales).

[0077] The online stage (also called generation, sampling, or inference) aims to generate a natural and accurate 3D object layout through iterative denoising, given a set of user-provided conditioned information, such as a room floor plan or a list of object categories. To do so, the objects to be placed within the input floor plan may be initialized arbitrarily (e.g., at the center of the room with random rotation values ​​and arbitrary dimensions). Since the trained model is noise-conditioned, a state-of-the-art diffuse sampling framework may be employed to reconstruct a valid object layout by iteratively applying a model conditioned on a decreasing noise scale in a fixed and reasonable number of steps. A probabilistic sampling framework also allows for diverse placement results from one generation to another. This noise-awareness also makes it easier for the trained model to perform other tasks during sampling. To perform repositioning where the position of the target object is known not to be far from the noisy one with fixed dimensions, iterative denoising may be performed starting from a lower noise scale.

[0078] The online stage may include the following subtasks:

[0079] The first subtask is to obtain the conditioned input and initialize the noisy features. User-provided conditioned information, such as a target floor plan, may be appropriately preprocessed so that it can be passed to the relevant encoder. The spatial object features output by the score network may be initialized with random or arbitrary values.

[0080] The second subtask is to define a sampling algorithm (i.e., a sampler). An efficient sampler may be implemented to generate new samples from the trained score network. This may involve adjusting the number of denoising steps, selecting a noise discretization scheme (value of the decreasing noise level), and selecting other relevant sampling hyperparameters, which are widely discussed in the literature.

[0081] The third subtask is the sampling process (also known as iterative denoising). The defined sampling process may run for a fixed number of steps (e.g., at least 30 steps, where fixed means set by the user) by iteratively applying the score network to its own denoised spatial object feature output, starting from a randomly / arbitrarily initialized set. In each iteration, the current noise level and user-provided conditioning features may also be passed to the relevant encoder module. At the end of the sampling process, the 3D object space features converge to their clean predicted values.

[0082] The fourth subtask is flexible adaptation to multiple tasks. The iterative denoising process may be adjusted at inference time depending on the target task. For example, some spatial object features that are normally predicted by the model may be fixed to predetermined values ​​(e.g., by fixing the objects in specific locations or by setting their desired dimensions). At each sampling step, the model's predictions may be replaced with predetermined values. Thus, these object features may eventually converge to predetermined values, and the smoothly denoised unknowns may be fitted to predetermined values ​​throughout the entire sampling process. Other tasks may also be performed by fine-tuning the sampler's parameters. For example, 3D relocation may be performed by applying iterative sampling starting from a lower noise level, since it is known that the objects are not far from their target locations. The attention weights of the trained transformer may also be utilized to perform object grouping.

[0083] Finally, the last subtask is object retrieval. Generating a 3D indoor layout typically involves a retrieval phase to actually place furniture in the room using assets from an external 3D database. This retrieval may, in most cases, be performed based on the dissimilarity between predicted (or provided) dimensions and object dimensions from the external database for a given object category. This process can be sub-optimal due to discrepancies between predicted / provided object dimensions and object dimensions available in the database. To circumvent this limitation, the proposed method may perform an additional denoising / sampling "repositioning" step, starting from a noise level derived from the discrepancies between predicted and retrieved object dimensions, in order to fine-tune the generated layout to the actual dimensions of the furniture objects.

[0084] The implementation of the solution (for example, as shown in Figure 3) will be explained with reference to Figures 4-18.

[0085] The implementation example focuses on generating 3D indoor room layouts. It outputs the position and rotation of 3D objects from a list of objects represented by semantic classes and 3D dimensions, as well as a floor plan of the target room.

[0086] The acquisition (i.e., obtaining) of the ground truth 3D layout dataset S10 and the contents of the dataset are described in detail. The implementation may use data extracted from furniture-placed digital 3D scenes created by various HomeByMe® users. HomeByMe® is a free interior design application that allows users to model their homes in 3D by selecting and accurately placing furniture from an extensive object catalog. Each HomeByMe® project is associated with a rich annotation file containing various information about the 3D scene (typed object 3D bounding boxes, room shapes, etc.). More formally, the following data elements may be considered: i. 3D bounding box. An annotation file may contain a list of various features that describe the objects in each scene. Two of these attributes, namely the object's class and its oriented 3D bounding box, may be of interest to us. The raw data obtained from this file defines the 3D bounding box by two 3D points corresponding to two opposite vertices of the bounding box. The HomeByMe® dataset has a total of 174 possible classes. ii. Room shape. The room shape may be saved in the annotation file as a list of 2D points representing the corners of the room.

[0087] Following such an implementation, the function has been trained on at least tens of thousands (e.g., 50K scenes) and evaluated on at least several thousand (e.g., 5K scenes) from the Bedroom subset of the HomeByMe® dataset. The model holds rooms containing up to 50 objects, which can be considered densely furnished rooms. Note, however, that the training subset may scale to a much larger dataset featuring other types of rooms.

[0088] The offline training S40 stage of such an implementation is described in detail below.

[0089] During data processing, each scene may be represented as an unordered set of objects (each represented by a typed 3D bounding box) and a floor plan. Data processing may include 3D bounding box processing, object class processing, floor point sampling, ground truth scene perturbation, and random data augmentation.

[0090] A raw 3D bounding box is a representation based on two opposite vertices, and their positions

number

number

number

number

[0091] For example, in the implementation, each 3D object from the HomeByMe® dataset is described by a class that provides a wide range of descriptions (chair, table, door, ...). The HomeByMe® dataset has a total of 174 classes. To be fed into a deep learning model, the object classes are:

number

[0092] This implementation uses a 3D point cloud to represent the boundaries of a room.

number

[0093] Figure 4 shows the correspondence between a scene in the HomeByMe® dataset and its preprocessed representation using sampled floor points and typed 3D bounding boxes.

[0094] The implementation of the machine learning method includes obtaining a normally distributed noise level (S20). For each training iteration and for each training sample (i.e., each ground truth 3D layout), the implementation includes obtaining each perturbed 3D layout, which can be calculated by perturbing at least one parameter of at least one 3D bounding box of the ground truth 3D layout (S30). Perturbing includes sampling each noise level σ (S30a), which defines the magnitude by which the position and rotation values ​​of the 3D object are perturbed. Each noise level may be obtained by taking the absolute value of a scalar drawn from the normally distributed.

number

number

number

[0095] To facilitate the generation of placements for scenes presenting walls that are not aligned with the coordinate axes, which may be rarely represented in the dataset, data augmentation may be performed during training by rotating the scene by a random angle along the vertical axis. Data augmentation reduces overfitting of the training data and allows for a more robust model to complex floor plan shapes.

[0096] Figure 5 shows a 2D top view visualization of pairs of scenes and their rotated versions from the dataset.

[0097] We will discuss the design of denoising architectures here.

[0098] The implementation of the deep architecture may consist of multiple trainable components: a noise encoder (i), a 3D object encoder (ii), a floor encoder (iii), a noise-aware transformer encoder (iv), and a final MLP (v) that outputs predicted object position and rotation values.

[0099] The following provides a description of the noise encoder (i).

[0100] The scalar value of the sampled noise level σ is deterministically determined by the dimension of the scalar value.

number

number

number

number

number

[0101] The following provides a description of the 3D object encoder (ii).

[0102] The scalar values ​​(p, r, d) that describe each 3D bounding box in the scene are, respectively

number

[0103] After the PE module, the position and dimensions of the bounding box, originally described by three scalar values ​​each, are now described by a 192-dimensional vector (3 × 64 = 192). On the other hand, the rotation, originally described by a pair of scalar values, is now described by a 128-dimensional vector after position encoding. To ensure that position, dimensions, and rotation are similarly weighted by the model, the higher-dimensional version of rotation is used to represent it.

number

number

[0104] One-hot encoded category c is,

number

number

number

number

[0105] All previously calculated vectors are,

number

[0106] The following is a description of the floor encoder (iii). For the record, encoding the floor points of a room conditions the 3D layout generation so that the resulting 3D object is within the limits of the floor.

[0107] The 3D point cloud of the sampled floor points F may be supplied to a PointNet module that outputs an embedding vector. This embedding itself may be supplied to a multi-layer perceptron that maps the vector to [Number] . This final vector may be a token representing the floor plan. [Number] .

[0108] The following provides an explanation of the noise-aware transformer encoder (iv).

[0109] The noise level token, the 3D object token, and the floor token may all be concatenated to form a sequence of tokens. These tokens may be independent of each other. A transformer module may be used to capture the relationships between different elements of this sequence. The transformer module requires a fixed input size due to its specific architecture. However, the sequence constructed through the concatenation of outputs may have a variable length because the number of 3D bounding boxes per scene sample is different. To make it compatible with the transformer architecture, the sequence may be padded with "zero" tokens ( [Number] ). Scenes with less than 50 objects are padded accordingly. Thus, the sequence [Number] It may be represented as a tensor. This tensor may be supplied to a transformer module, which outputs a final scene embedding tensor of the same dimension. A Boolean padding mask indicating the index of the sequence that is a padding "zero" token may also be passed to the transformer as an additional input. This mask may be used in a self-attention mechanism to set the attention score of the padding tokens to zero.

[0110] The following provides a description of the final MLP(v).

[0111] The new representation calculated by the transformer for each 3D object ultimately shows the predicted "clean" position p and rotation r for each 3D object.

number

number

[0112] The resulting architecture has a total of 12.2 million trainable parameters.

[0113] The noise-dependent preconditioning of score networks is described below.

[0114]

number

number

number

number

number

number

Number

[0115] Such parameterization in this implementation corresponds to a trained function that is flexible enough to predict a realistic and plausible 3D layout and adapt to different usage scenarios (e.g., 3D placement generation, partial 3D placement generation, 3D re - placement, etc.).

[0116] The instance - level Chamfer distance loss is described in detail below.

[0117] It may be required that the objective of the training loss function measures the distance between the predicted 3D object spatial attributes (position and rotation) and the correct ones. Also, this loss may be required to be invariant to permutations of the same 3D object, i.e., objects with the same dimensions and the same semantic category. Thus, a new semantic - aware Chamfer distance loss function that measures the distance between the set of objects of the predicted scene

Number

Number

Number

Number

[0118] If a couple of objects does not share the same category and dimensions, a significant penalty is applied, and they are therefore excluded from being returned by the min operator. If object dimensions are predicted, the distance between the predicted dimensions and the correct dimensions may be included, or the penalty may be applied based solely on the category of the object pair.

[0119] This loss is far more computationally efficient than Earth Mover's Distance used in previous studies, is intrinsically differentiable, and can be parallelized for high-speed computation on GPUs.

[0120] Just to be clear, during training S40, this distance is noise.

number

number

number

number

number

[0121] Here, a description of the placement sampling / inference stage is provided.

[0122] Once a score-based denoiser model is trained according to the proposed machine learning method (after S40), it can be used to perform various online tasks at inference by taking an input 3D layout and a given noise level (S50) and setting up a sampling procedure (i.e., applying the trained function one or more times, starting from the input 3D layout, to predict the output 3D layout S60) that can be flexibly fine-tuned to adapt to different scenarios. The sampler defines how the trained score-based model is iteratively applied to generate new, realistic samples similar to the training distribution. In particular, setting up a sampler generally involves defining noise level scheduling (i.e., decreasing noise levels at which the model is evaluated). A second-order sampler may involve applying the model twice at each scheduled noise level. A probabilistic sampler may also involve injecting fresh noise (new noise) at each iteration of the iterative denoising process to better explore the space of the learned distribution. Several sampling strategies can be implemented, but the one adopted is described here.

[0123] Implementing a quadratic probabilistic sampler helps obtain accurate generated results while reducing the number of computationally expensive neural evaluations (i.e., model applications). The following algorithm forms an example of implementing this sampling procedure. [Table 1]

[0124] Here, fresh noise may be added first in each iteration, and then the gradient step may be calculated from the intermediate position. A second-order correction may be employed to achieve faster convergence.

number

number

number

[0125] In reality, ρ = 7, S noise =1.003, and S churn Set to =40.

[0126] Figure 6 shows a visual representation of the sampling step applied to a single object (e.g., a chair).

[0127] The proposed solution may be used for a variety of inference tasks, including 3D placement generation (i), 3D repositioning (ii), partial 3D placement generation (iii), and attention-based object grouping (iv).

[0128] Figure 7 shows typical qualitative results obtained by applying the usage to the 3D placement generation (i) task. The 3D placement generation task aims to generate a realistic scene placement from an initialized configuration. To perform 3D placement generation from a given floor plan and list of 3D objects, the positions of the 3D objects may be arbitrarily initialized at the center of the room, and the rotation (i.e., orientation) of the 3D objects may be initialized randomly. The denoising process is performed using a sufficiently high noise level (e.g., σ). max = 1.0 or σ max It may be executed starting from =0.5). The number of steps in iterative sampling may be set to N=50 steps. Setting an appropriate number of steps in iterative sampling creates a good trade-off between the quality and accuracy of the predicted 3D scene placement and the sampling time.

[0129] Figure 8 shows an example of 3D relocation (ii). The task of 3D relocation aims to restore a realistic placement configuration from a messy initial configuration. To perform 3D relocation from a given list of floor plans and 3D objects, the position and rotation of the 3D objects may be initialized to their messy (i.e., perturbed) values. The denoising process may be performed starting from a lower noise level value than that used for the 3D placement generation task.

[0130] Figure 9 shows an example of partial 3D placement generation. The task of partial 3D placement generation aims to generate a realistic scene placement from an initialized configuration in which several 3D objects have their positions and rotations determined. These objects are therefore initialized to their known position and rotation values, while the object to be placed is again initialized to the center of the room with random rotation values. At each denoising step, the output of the model for known 3D objects (i.e., 3D objects with known values ​​for position and rotation parameters) may be replaced with their original position p and orientation (i.e. rotation) r values. Alternatively, the output of the model for known 3D objects may be their perturbed position p, noised at a level σ corresponding to the current sampling step. σ and rotation r σ Note that these values ​​may be replaced by the known values. In either case, these known objects will eventually converge to their initial values ​​throughout the entire sampling process. Figure 10 shows an example of partial 3D placement generation where more 3D objects are fixed to determined values ​​for position and rotation.

[0131] Figure 11 shows the attention map of the selected electronic device 110 (on the table 112). Since the trained function features a transformer encoder module that processes 3D object tokens, an attention score can be calculated between a given 3D object and each other object in the scene. The score may be calculated based on the last transformer layer and averaged across its entire attention head. 3D objects 112, 114 that are close to and related to the 3D object of interest have higher attention scores. For example, still referring to Figure 11, the selected 3D object 110 is located on the table 112, and the other 3D objects 114 are grouped near the selected object. This demonstrates that the transformer model has successfully learned the relationships between 3D objects, and that a threshold can be dynamically defined to determine which set of 3D objects are grouped with the selected 3D object, thereby allowing them to be moved together, for example. Note that calculating a score for a given 3D object based on another transformer layer may generate other meaningful groups.

[0132] Figure 12 shows a comparison between results obtained using a function trained without data augmentation (left panel) and results obtained using a function trained with data augmentation (right panel). The 3D scene placements generated with data augmentation are more realistic and plausible than those obtained using a function trained without data augmentation. For example, the 3D scene placements obtained without data augmentation during training clearly predict a bed that is partially outside the given floor plan and a wall shelf in the center of the room. These inconsistencies are better resolved by data augmentation.

[0133] The proposed method is qualitatively compared here with solutions developed based on the state of the art. A quantitative comparison will be discussed later.

[0134] For comparative purposes, an adaptation of an existing state-of-the-art method (originally developed for a different task) was constructed. This state-of-the-art method is LEGO-Net, which was originally proposed for the task of 2D relocation. Therefore, for qualitative comparison, the LEGO-Net model may be trained on the same dataset (e.g., the HomeByMe dataset) and adapted for the task of 3D placement generation. In particular, the original LEGO-Net model may be fine-tuned to set a high initial noise in the sampler.

[0135] Figure 13 shows a qualitative comparison between an adapted version of LEGO-Net and the proposed method. The results are exported to the HomeByMe application solely for the purpose of obtaining better 3D visualization. For the task of 3D placement generation, 3D objects may be initialized to the center of the room with random rotations. The adapted LEGO-Net model may perform sampling with high initial noise, while the proposed method may perform iterative sampling with 50 denoising steps. The comparison of the results shows that the proposed method achieves a more realistic and natural-looking 3D scene placement. For example, the proposed method (second row in Figure 13) generates a realistic and natural-looking (i.e., plausible) 3D placement, while the adapted LEGO-Net (first row in Figure 13) generates a 3D scene placement with clearly unrealistic and inconsistent results. For example, the first result in the first row shows a bed floating above a wardrobe. Furthermore, the generation time for LEGO-Net is much longer than that of the proposed method. This is because the sampling procedure implemented in LEGO-Net does not guarantee convergence for a fixed number of iterations. In other words, iterative denoising is performed until the 3D object can no longer move, thus making LEGO-Net inefficient compared to the proposed method.

[0136] Referring to Figure 14, another qualitative comparison is shown between the results predicted by LEGO-Net and the proposed method for a 3D rearrangement task. The 2D rearrangement task is the original case for LEGO-Net. Figure 14 shows the same ground truth scene arrangement, which can be initialized to random (i.e., perturbed) values ​​for position and rotation. LEGO-Net may perform its original sampling, while the proposed method may perform iterative sampling with 50 denoising steps. The respective predictions are quite different. The proposed method predicts a tidy arrangement where objects are consistently (semantically and physically) grouped. For example, the bed is properly aligned with the wall, the two nightstands are placed on either side of the bed frame, and the dressing table is located in the corner of the floor plan. In contrast, the LEGO-Net prediction shows significant inconsistencies and does not look natural. The bed is still randomly oriented, and the nightstands are not well-placed.

[0137] A more quantitative comparison will be performed below.

[0138] Generative models are inherently difficult to evaluate quantitatively. However, proprietary metrics can assess the quality of generated placements with respect to several indicators that measure the validity and ergonomics of the generated scenes. Computing these indicators (i.e., metrics) in both the training set and a significant number of generated placements provides insights into the practical performance of the proposed model and assesses the realism and diversity of the generated placements.

[0139] The metric may be an opening clearance (clearance metric) that determines whether there is sufficient space in front of a door or window to open or pass through. To determine this, a virtual 3D bounding box with the dimensions of the door or window is placed in front of it. Its depth may be set to the width of the relevant door or window. The Intersection over Union (IoU) between the virtual box and the 3D bounding box of the object in the scene may then be calculated. The diversity of generated placements obtained using the proposed model may also be utilized to automatically select the best performing one or more metrics for a given scene.

[0140] Figure 15 shows the best-performing door and window clearance metric values ​​for two different trial configurations (generated configurations of a given scene) as a function of the number of trials, and for two different settings. Doors and windows may be fixed or moved like other objects. The metric is calculated for 2048 scenes each time and may be compared to a baseline value for the dataset. The results in Figure 15 provide two observations. First, they demonstrate the diversity of the output domain of the proposed model. Since scaling the number of trials can have a clear impact on the metric, it shows that the generated configurations differ across several generation for a given scene. Second, generating several trial configurations may help to achieve comparable performance to the training data with respect to this metric. Thanks to the efficient batching and parallelization capabilities of the proposed model, this can also be done at little cost.

[0141] Another quantitative indicator may be the presence of objects outside the boundaries (e.g., outside the given floor plan). To assess whether the geometric shape of the floor plan has been correctly considered, it is appropriate to calculate the average area of ​​3D objects outside the boundaries (e.g., outside the given floor plan) and the frequency with which 3D objects are at least partially outside the floor plan. Formulated, these two indicators may be given by:

number

number

[0142] A useful quantitative metric may be the number of colliding objects. Within a floor plan, competition between objects (e.g., collisions, penetrations, overlaps) can impair the perceived validity of the scene. To estimate their importance, 3D IoU may be of the following types:

number

number

number

[0143] Finally, another quantitative metric may be the diversity of scene arrangements, which can be evaluated by the Chamfer distance between pairs of generated layouts. This evaluation may be performed with a considerable number of samples (e.g., 2048 scenes). The higher the Chamfer distance, the more diverse the generated layouts may be.

[0144] Figure 16 shows the results of the above metrics, evaluated across 2048 scenes for a single generation trial, compared to the values ​​in the dataset.

[0145] The proposed method may also be trained to predict the position, rotation, and dimensions of 3D objects from a given list of floor plans and semantic objects on the publicly available 3D-Front dataset (available at the following URL as of the priority date of this patent application: https: / / tianchi.aliyun.com / specials / promotion / alibaba-3d-scene-dataset) and may be compared with established baseline methods such as LayoutGPT, ATISS, and DiffuScene. During training, the set of conditioned inputs y may consist only of a given list of floor plans and a given list of semantic categories, i.e., y=(F,c). Conditioned dropout on semantic categories may be implemented for training these models. Conditioned dropout consists of replacing a given list of semantic categories with a list of zeros (i.e., a tensor whose entries contain zeros) with a predetermined probability (e.g., equal to 0.2). Of the available room types in the 3D-Front dataset, only living rooms and dining rooms are considered because they have more densely arranged furniture and feature complex floor plan shapes. The ATISS preprocessing step may result in 2338 / 587 train / test splits for the living room and 2071 / 516 for the dining room. Metrics for evaluating the realism and diversity of the generated 3D layouts may include 2562 Frechet Inception Distances (FID), Kernel Inception Distances (KID × 1000), and Scene Classification Accuracy (SCA) calculated on top-down orthographic rendering. The spatial validity of the generation may be further evaluated by reporting the accumulated out-of-bounds object area (OBA m2). All metrics may be calculated across each test subset. FID and KID compare the distribution of visual features extracted from pre-trained convolutional neural networks.SCA measures how well a convolutional neural network distinguishes a real scene (i.e., the correct test scene) from a generated scene in a binary classification task. Therefore, an SCA score close to 50% is good, meaning that the generated scene is indistinguishable from the real scene.

[0146] Table 1 shows a quantitative comparison of constrained 3D layout generation (providing a given floor plan and a given list of semantic categories for 3D objects). The comparison characterizes the results of the proposed method against results obtained from training-free LayoutGPT and other learning-based approaches (ATISS and DiffuScene). The comparison shows that the proposed method consistently generates more realistic and diverse scene arrangements, significantly outperforming the other baseline methods.

[0147] [Table 2]

[0148] Figure 17 shows a comparison of results obtained by ATISS, DiffuScene, and the proposed method, which feature the generated arrangements with 3D objects by scene layout and semantic categories. The proposed method generates the most realistic and diverse 3D arrangement layouts.

[0149] Figure 18 shows the results obtained by the proposed method for performing the task of 3D rearrangement from a messy configuration (shown in the left panel) and the completion of partial rearrangement (shown in the right panel).

[0150] Table 2 reports the number of network parameters for each method and the average generation time across the 3D-FRONT living room test subset. [Table 3]

[0151] The proposed method's architecture is lighter and, when combined with an efficient sampling procedure, enables the generation of novel layouts faster than prior art denoising-based methods (such as DiffuScene).

[0152] This learning method is a machine learning method for a model that is a deep generative model. As is known by itself from the field of machine learning, the processing of input by a model involves applying an operation to the input, and such operation is defined by data containing weight values ​​or parameters. Learning a model (e.g., a neural network or a regressionr) therefore involves determining the values ​​of weights / parameters based on a dataset configured for such learning, and such a dataset may be called a learning dataset or training dataset. For that purpose, a dataset contains data pieces, each forming its own training sample or example. The training samples / examples represent the variety of situations in which the model will be used after it has been learned. Any training dataset here may contain 1,000, 10,000, 1,000,000, or more than 1,000,000 training samples / examples. In the context of this disclosure, “training is performed on a dataset” means that the dataset is the learning / training dataset of the model on which the weight / parameter values ​​are set. In this disclosure, the training dataset is the dataset of acquired training examples on which the deep generative model is learned / trained. For example, the training dataset consists of hundreds of examples, each corresponding to a different HPP configuration.

[0153] As is well known from machine learning, a neural network may be defined by its architecture, parameters, and hyperparameters. The architecture consists of layers, beginning with an input layer where the number of neurons can be determined by the dimensions of the input data. This layer is followed by several hidden layers with a given number of neurons and activation functions. These layers and neurons define the depth and width of the network, while the activation functions may introduce nonlinearity into the model. The output layer may have the same number of neurons as the variables in the output data. The interconnections between these layers define the topology of the neural network. The parameters of a neural network are learnable weights and biases, which are determined during the training process. In contrast, hyperparameters are predefined settings that are not learned from the training data. These include the number of hidden layers, the number of neurons per layer, and many others. To train a neural network, at least two settings may be defined. First, a loss function, which is a metric that measures the error between the training data and the model's predictions, such as the mean squared error (MSE). Second, an optimizer, which modifies the model's weights and biases during the training process to minimize the loss function. Each optimizer has its own set of hyperparameters.

[0154] This method is computer-implemented. This means that the steps (or substantially all steps) of the method are performed by at least one computer, or any similar system. Thus, the steps of the method are performed by the computer, possibly fully automatically or semi-automatically. For example, at least some triggers of the steps of the method may be performed through user-computer interaction. The required level of user-computer interaction may depend on the anticipated level of automation and may be balanced with the need to implement user preferences. For example, this level may be user-defined and / or predefined.

[0155] A typical computer implementation of this method is to perform the method using a system adapted for this purpose. This system may include a processor coupled with memory and a graphical user interface (GUI), where the memory stores a computer program containing instructions for performing the method. The memory may also store a database. The memory is any hardware adapted for such storage and may include several physically separate parts (e.g., one for the program and possibly one for the database).

[0156] Figure 19 shows an example of this system, which is a client computer system, such as a user's workstation.

[0157] An example client computer includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphical processing unit (GPU) 1110 associated with video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass memory devices such as a hard drive 1030. Mass memory devices suitable for tangibly realizing computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; and all forms of non-volatile memory, including magneto-optical disks. Any of the above may be complemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include haptic devices 1090 such as a cursor control device and a keyboard. A cursor control device is used in a client computer to allow the user to selectively position the cursor at any desired location on the display 1080. In addition, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for input control signals to the system. Typically, the cursor control device may be a mouse, with the mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0158] A computer program may include instructions that can be executed by a computer, and such instructions include means for causing the system to execute a method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in a digital electronic circuit, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, for example, as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be executed by a programmable processor that executes a program of instructions to perform the function of the method by manipulating input data and producing an output. The processor may therefore be programmable and may be coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. The application program may be implemented in a high-level procedural or object-oriented programming language, or, if desired, in assembly language or machine language. In any case, the language may be a compiled language or an interpreted language. The program may be a full installation program or an update program. In any case, the application of the program on the system results in instructions for executing the method. Alternatively, this computer program may be stored and executed on a server in a cloud computing environment, which communicates with one or more clients over a network. In such a case, the processing unit causes the method to be executed on the cloud computing environment by executing instructions contained in the program.

Claims

1. A computer implementation method for machine learning, wherein the method is The process involves obtaining a dataset of correct 3D layouts (S10), where each correct 3D layout represents a scene. Each floor plan, - A 3D arrangement of one or more 3D bounding boxes, where each bounding box is defined by a predetermined set of values ​​of one or more parameters, and • Each 3D bounding box has a corresponding meaning category. Obtaining a dataset that includes, Obtaining the probability distribution of the noise level (S20), For each ground truth 3D layout, a perturbed 3D layout that can be calculated is obtained by perturbing at least one parameter of at least one 3D bounding box of the ground truth 3D layout (S30), wherein the perturbation is - Sampling each noise level based on the aforementioned probability distribution (S30a) and For each of the parameters of the at least one parameter mentioned above, ...Sampling each noise value based on the respective noise levels (S30b) ...Applying the respective noise values ​​to each of the aforementioned parameters (S30c) Includes, Training a function (S40), The function is configured to capture an input 3D layout and a given noise level, and the input 3D layout is, ...The given floor plan, ...a first 3D arrangement of one or more given 3D bounding boxes, wherein each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters, and ...For each given 3D bounding box, a given semantic category, Includes, The function is configured to predict the output 3D layout, and the output 3D layout is, ...The aforementioned floor plan, ...a second 3D arrangement of one or more given 3D bounding boxes, wherein each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters, and For each given 3D bounding box, the given semantic category, Includes, The function is configured to predict a second value of the predetermined set of one or more parameters, which is different from the first value of the predetermined set of one or more parameters. The training is performed on the dataset based on a loss that penalizes the dissimilarity metric between each ground truth 3D layout and each predicted 3D layout obtainable by applying the function to each perturbed 3D layout. method.

2. The method according to claim 1, wherein the dissimilarity metric is of the following type. [Math 1] Here, [Math 2] This is a set of one or more 3D bounding boxes in the correct 3D layout, [Math 3] This is a set of one or more 3D bounding boxes in each predicted 3D layout, N is [Math 4] and [Math 5] It is a common size, [Math 6] This is the differentiable distance.

3. The said differentiable distance penalizes the dissimilarity in dimensions and / or semantic categories between 3D bounding boxes, and is of the following types: 【Number 7】 Here, [Number 8] and x are, respectively [Number 9] and a vector of the spatial parameter values ​​of o, [Number 10] is the Euclidean norm, K is a penalty parameter, [Math 11] is the argument [Math 12] An indicator function that is equal to 1 if and o have the same dimensions, and 0 otherwise. [Number 13] is the argument [Number 14] And o is an indicator function that is equal to 1 if they have the same semantic category, and 0 otherwise. The method according to claim 2.

4. The loss of the trained function is the dissimilarity metric [Number 15] and the product of the noise-dependent weighting function λ(σ) [Number 16] Expected value [Number 17] That is, The method according to any one of claims 1 to 3.

5. The aforementioned function includes a transformer, The Transformer in question is, - The concatenation of the given noise level representation, the first representation of each given 3D bounding box, and the given floor plan representation is taken as input. - Output a representation of the predicted 3D layout, including a second representation of each given 3D bounding box. The method according to any one of claims 1 to 4.

6. The aforementioned function further, - A noise encoder that generates a representation of the given noise level, A 3D object encoder that generates the first representation of each given 3D bounding box, wherein the 3D object encoder is configured to generate the respective representations of each parameter and the representation of the semantic category, and to concatenate all the generated representations, and - A floor encoder for generating a representation of the given floor plan, the floor encoder comprising a sampling module for generating samples from the given floor plan, and a point cloud encoder for processing the samples. Includes at least one of the following: The method according to claim 5.

7. The function further includes a multilayer perceptron that takes the predicted 3D layout representation as input and outputs a third representation of each given 3D bounding box. The method according to claim 5 or 6.

8. The method according to claim 5, wherein the function is parameterized as follows. [Number 18] Here, x σ is the first 3D arrangement of one or more given 3D bounding boxes, y represents the given floor plan, the given semantic category of each of the one or more given 3D bounding boxes, and the dimensions of each of the one or more given 3D bounding boxes, and σ is the given noise level. [Number 19] This is a noise-conditional score network having a set of trainable parameters θ, and c skip (σ) is a noise-dependent preconditioning coefficient that modulates the predicted 3D layout, and c noise (σ) is a noise dependency coefficient that conditions the noise level in the score network, and c in (σ) and c out (σ) is x σ and [Number 20] These are two noise-dependent coefficients that scale the result.

9. A computer implementation method for using a function trained according to any one of claims 1 to 8, wherein the method of use is (S50) to acquire the input 3D layout and a given noise level, wherein the input 3D layout is - The given floor plan, - A first 3D arrangement of one or more given 3D bounding boxes, wherein each bounding box is defined in the first 3D arrangement by a first value of a predetermined set of one or more parameters, and For each given 3D bounding box, a given semantic category This includes obtaining the input 3D layout and the given noise level, To predict the output 3D layout, the trained function is applied one or more times, starting from the input 3D layout (S60), wherein the output 3D layout is - The given floor plan, - A second 3D arrangement of one or more given 3D bounding boxes, wherein each bounding box is defined in the second 3D arrangement by a second value of a predetermined set of one or more parameters, and For each given 3D bounding box, the given semantic category including, Starting from the input 3D layout, apply the trained function one or more times. A method of having.

10. Applying the aforementioned trained function one or more times means - By injecting noise into the input 3D layout (S60a), a perturbed input 3D layout is obtained, - Obtain an output 3D layout by applying the trained function to the perturbed input 3D layout at least once (S60b), - Using the output 3D layout as input for the next iteration (S60c) Including the repetition of The method according to claim 9.

11. The noise has a level that decreases with the depth of the repetitions. The method according to claim 10.

12. Applying the aforementioned trained function at least once is required in each iteration. - A first output 3D layout is obtained by applying the trained function to the perturbed input 3D layout (S70a), - Obtain a first intermediate 3D layout by calculating the gradient step between the perturbed input 3D layout and the first output 3D layout (S70b). By applying the trained function to the first intermediate 3D layout (S70c), a second output 3D layout is obtained, and - Obtain the final 3D layout by obtaining a second intermediate 3D layout (S70d) by calculating the gradient step between the perturbed input 3D layout and the second output 3D layout. including, The method according to any one of claims 9 to 11.

13. A computer program comprising instructions for performing any method of claims 1 to 8 and / or any use of any method of claims 9 to 12, and / or a data structure comprising a function trained by the method of any one of claims 1 to 8.

14. A computer-readable storage medium on which the data structure described in claim 13 is recorded.

15. A system comprising a processor coupled to memory, wherein the memory records the data structure described in claim 13.

Images