Indoor three-dimensional scene automatic generation method based on stream matching

Abstract

The invention discloses an indoor three-dimensional scene automatic generation method based on stream matching, and the method comprises the following steps: 1, obtaining the boundary information of a plurality of indoor scenes, and extracting the boundary features of the room layout of the corresponding indoor scenes; each indoor scene is expressed as an object set, each object in the object set is formed by splicing attributes, and a scene code of each indoor scene is obtained; forming sample points by boundary features and scene codes of each indoor scene, and forming a real scene sample; 2, generating a stream matching model, and inputting the real scene sample obtained in the step 1 into the stream matching model to train the stream matching model to obtain a trained stream matching model; and step 3, inputting information of an indoor scene to be generated into the trained flow matching model, and predicting and outputting a three-dimensional layout of the indoor scene by the flow matching model. According to the method, stream matching is adopted as a generation framework, a shorter sampling path and higher reasoning efficiency are achieved, and the method is suitable for real-time or interactive application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of indoor scene generation methods, in particular to an indoor three-dimensional scene automatic generation method based on flow matching. BACKGROUND

[0002] High-quality indoor scene generation has important value in virtual reality, video games, and architectural design. Traditional methods rely on manual arrangement by designers or the use of interactive tools, which is tedious and time-consuming. In recent years, data-driven methods based on deep learning have gradually become a research hotspot, mainly including autoregressive models, diffusion models, and flow matching models.

[0003] Autoregressive models (such as ATISS and SceneFormer) generate scenes by generating each object step by step, which can ensure local rationality but is difficult to model global dependencies, leading to inconsistent functions and boundary violations. Diffusion models (such as DiffuScene and MiDiffusion) generate scenes through multiple denoising steps, which can generate high-quality scenes but are slow in inference and difficult to meet real-time application requirements. In addition, existing methods generally have geometric inconsistency problems such as object overlap and boundary violation, affecting the practicality and realism of generated scenes.

[0004] Flow matching, as a new generation model, defines a continuous transformation from a simple distribution to a complex distribution through an ordinary differential equation, with the advantages of short sampling path and high inference efficiency. However, existing flow matching methods still face challenges in complex three-dimensional scene generation, such as difficulty in ensuring geometric constraints and unreasonable object layout.

[0005] Therefore, there is an urgent need for an indoor scene automatic generation method that can ensure generation efficiency and effectively maintain geometric consistency. SUMMARY The present application provides an indoor three-dimensional scene automatic generation method based on flow matching to address the deficiencies of existing technology in generation efficiency, geometric consistency, and scene rationality.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] An indoor three-dimensional scene automatic generation method based on flow matching, the process is as follows:

[0008] Step 1, obtain the boundary information of multiple indoor scenes, and encode the boundary information of each indoor scene to extract the geometric features of the corresponding indoor scene room layout as boundary features;

[0009] And each indoor scene is represented as an object set, each object in the object set is spliced by semantic attributes and geometric attributes respectively, and each object in the object set is encoded according to the attributes of the object to obtain the scene coding of each indoor scene;

[0010] Finally, the sample points are composed of the boundary features and the scene coding of each indoor scene, and the real scene samples are composed of the sample points of each indoor scene;

[0011] Step 2, a flow matching model is constructed, the flow matching model is composed of an 8-layer Transformer network as a backbone network, and a separate Transformer network is introduced as a geometric refinement module;

[0012] The real scene samples obtained in step 1 are input into the flow matching model to train the flow matching model;

[0013] During training, an initial noise scene is sampled from a prior distribution, a probability path is constructed between the initial noise scene and the real scene sample through linear interpolation, a conditional vector field is predicted by the backbone network, and smooth conversion from the initial state to the target state is realized by solving ordinary differential equations;

[0014] At the same time, the geometric refinement module adds Gaussian noise to the geometric attributes of the real scene sample to construct training data for training the geometric refinement module to optimize the geometric attributes of the generated scene;

[0015] The loss function used when training the flow matching model includes a conditional flow matching loss and a non-overlapping constraint loss, wherein the non-overlapping constraint loss is the sum of the intersection volume to the union volume ratio of all object pairs in the scene;

[0016] Step 3, the boundary information of the indoor scene to be generated is input into the trained flow matching model, and the three-dimensional layout of the indoor scene is predicted and output by the flow matching model.

[0017] Further, in step 1, the boundary information of each indoor scene is encoded by a PointNet network;

[0018] During encoding, the PointNet network uniformly samples a point set from the room boundary, each sampled point includes two-dimensional position and normal vector information, and the PointNet network encodes and outputs a 512-dimensional geometric feature vector based on the sampled point set, which is the boundary feature.

[0019] Further, in step 2, the flow matching model uses a conditional flow matching method, constructs a probability path between an initial noise sample and a real scene sample through linear interpolation, and trains the backbone network to predict a conditional vector field;

[0020] where the initial noise samples are sampled from a Gaussian distribution, and the real scene samples are obtained from step 1;

[0021] The conditional vector field is predicted by the backbone network based on the encoded scene, time step and boundary features, and is used to guide the evolution of the samples along the probability path.

[0022] The conditional vector field is combined with the solution of the ordinary differential equation to update the scene state through iteration, realizing the conversion from the initial noise distribution to the target scene distribution.

[0023] Further, in step 2, the non-overlapping constraint loss calculates the three-dimensional bounding box intersection over union between all object pairs in the scene, and optimizes the object position and size through gradient backpropagation to reduce object overlap.

[0024] Further, in step 2, when training the flow matching model, it is realized by iteratively solving the ordinary differential equation, and at each step, the scene state is updated according to the predicted vector field, and finally the indoor scene three-dimensional layout conforming to the boundary constraint and object compatibility is output.

[0025] Where the boundary constraint is provided by the boundary feature to ensure that the object does not exceed the room boundary; and the object compatibility is jointly ensured by the scene encoding and the non-overlapping constraint to ensure that the object layout is reasonable and non-overlapping.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] 1. Adopting flow matching as the generation framework, it has a shorter sampling path and higher inference efficiency than diffusion models, and is suitable for real-time or interactive applications.

[0028] 2. Introducing non-overlapping constraints and geometric refinement modules significantly improves the geometric consistency and visual rationality of the generated scene.

[0029] 3. Support for completing the whole generation from a partial scene, enhancing the practicality and flexibility of the method.

[0030] 4. The model structure is simple, stable in training, and easy to deploy on existing deep learning platforms. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the overall workflow diagram of the present application.

[0032] Figure 2 is a schematic diagram of indoor scene object representation.

[0033] Figure 3 is a schematic diagram of the flow matching generation process.

[0034] Figure 4 is a structure diagram of the geometric refinement module. DETAILED DESCRIPTION

[0035] The application will be further described below in conjunction with the accompanying drawings and examples.

[0036] As shown in the Figure 1 , the embodiment discloses a flow matching based indoor three-dimensional scene automatic generation method, and the process is as follows:

[0037] Step 1, obtaining the boundary information of multiple indoor scenes, the boundary information is represented as a planar layout of the indoor scene, the planar layout is represented in the form of a gray image, and the image size is 256x256, wherein the pixel value of the indoor area of the room is 1, and the pixel value of the outdoor area of the room is 0.

[0038] A PointNet network is used as an encoder to encode the boundary information of each indoor scene. When encoding, the PointNet network uniformly samples 250 points from the room boundary to form a point set, each sampled point includes a two-dimensional position on the XZ plane and a unit normal vector information, and the PointNet network encodes and outputs a 512-dimensional geometric feature vector based on the sampled point set, thereby extracting the geometric features of the room layout of the corresponding indoor scene as the boundary features. Thus, the geometry and topology structure of the indoor scene boundary is effectively captured.

[0039] Furthermore, the objects in each indoor scene are uniformly represented to form an object set. Each object is spliced from semantic attributes (object category label) and geometric attributes (position, size, angle) (as shown in Figure 2 ), wherein the angle is represented by sine and cosine values. The entire indoor scene is composed of at most N object vectors, and the insufficient part is filled with an "empty" label, and the geometric attribute is set to zero. The attributes of each object in the object set are encoded according to the attributes of the object: a multi-layer perceptron (MLP) is used to encode the semantic attributes and geometric attributes respectively, and a 512-dimensional vector is output, and then spliced into a unified attribute embedding to form a scene encoding.

[0040] Finally, the sample points are composed of the boundary features and the scene encoding of each indoor scene, and the real scene samples are composed of the sample points of each indoor scene.

[0041] Step 2, constructing a flow matching model, as shown in Figure 3 , the flow matching model is composed of an 8-layer Transformer network as the backbone network. And a separate Transformer network is introduced as a geometric refinement module as shown in Figure 4 .

[0042] The real scene samples obtained in step 1 are input into the flow matching model to train the flow matching model. The training process is as follows:

[0043] ①, sampling an initial noise scene from a Gaussian distribution .

[0044] ② Random sampling time Intermediate samples are constructed using linear interpolation. ,in These are real-world scenario samples.

[0045] ③, will The time step t and boundary features are input into the backbone network, and the backbone network predicts the conditional vector field. .

[0046] ④ The loss function is the conditional flow matching loss. Non-overlapping constraint loss Weighted sum: .in, Calculate the sum of the 3D bounding box intersection-union ratios (IoU) between all object pairs; These are the weighting coefficients for the non-overlapping constraint loss, used to balance the importance of the two losses.

[0047] ⑤ Optimize backbone network parameters through backpropagation.

[0048] Simultaneously, train the geometry refinement module:

[0049] ① Add Gaussian noise to the geometric attributes (position, size) of real-world scene samples to construct noise data. .in Geometric properties of real-world scene samples (including position and size); This is the noise scaling factor, used to control noise intensity; It is standard Gaussian noise with a mean of 0 and a variance of . ; Noise standard deviation.

[0050] ② Input the noise data into the Refinement Transformer module and output the adjusted geometric properties.

[0051] ③ The loss function is the mean square error between the predicted geometric attributes and the true geometric attributes.

[0052] Flow matching models use time-dependent vector fields Definition: By solving ordinary differential equations Achieve from initial state To the target state A smooth transition. Among them... It represents the flow transformation at time t, which describes the evolution path of the sample from the initial state to the target state.

[0053] Step 3, input the boundary information of the indoor scene to be generated into the trained flow matching model, and output the three-dimensional layout of the indoor scene by the flow matching model. Specifically:

[0054] ①, sampling an initial scene from a Gaussian distribution .

[0055] ②, iteratively solve ODE: , update the scene state step by step. Wherein is the scene state at time step , and is the scene state at time step .

[0056] ③, finally output the indoor scene three-dimensional layout conforming to the boundary constraint (objects do not exceed the room boundary) and object compatibility (object layout is reasonable and there is no overlap).

[0057] The present application is verified on the 3D-FRONT dataset, and the experiment shows that the present method is superior to the existing autoregressive and diffusion model in terms of generation quality, geometric consistency and reasoning efficiency.

[0058] The preferred embodiments of the present application are described in detail above in combination with the drawings, and the embodiments described by the present application are only a description of the preferred embodiments of the present application, and do not limit the concept and scope of the present application. In the above specific embodiments, various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction, and such combination should also be considered as disclosed by the present disclosure as long as it does not deviate from the technical concept of the present application. In order to avoid unnecessary repetition, the present application does not further describe various possible combinations.

[0059] The present application is not limited to the specific details in the above embodiments, and various modifications and improvements of the technical solutions of the present application made by those skilled in the art within the technical concept of the present application and without departing from the design idea of the present application should fall within the protection scope of the present application. The technical content claimed by the present application has been fully recorded in the claims.

Claims

1. A method for automatically generating an indoor three-dimensional scene based on flow matching, characterized in that, The process is as follows: Step 1, obtain the boundary information of multiple indoor scenes, and encode the boundary information of each indoor scene to extract the geometric features of the corresponding indoor scene room layout as boundary features; And each indoor scene is represented as an object set, and each object in the object set is spliced by semantic attributes and geometric attributes, and each object in the object set is encoded according to the attributes of the object to obtain the scene encoding of each indoor scene; Finally, the sample points are composed of the boundary features and the scene encoding of each indoor scene, and the real scene samples are composed of the sample points of each indoor scene; Step 2, construct a flow matching model, which is composed of an 8-layer Transformer network as a backbone network and introduces a separate Transformer network as a geometric refinement module; The real scene samples obtained in step 1 are input into the flow matching model to train the flow matching model; During training, an initial noise scene is sampled from a prior distribution, a probability path is constructed between the initial noise scene and the real scene sample by linear interpolation, a conditional vector field is predicted by the backbone network, and a smooth transition from the initial state to the target state is realized by solving an ordinary differential equation; At the same time, the geometric refinement module adds Gaussian noise to the geometric attributes of the real scene sample to construct training data for training the geometric refinement module to optimize the geometric attributes of the generated scene; The loss function used to train the flow matching model includes a conditional flow matching loss and a non-overlapping constraint loss, where the non-overlapping constraint loss is the sum of the intersection volume and the union volume of all object pairs in the scene. Step 3, input the boundary information of the indoor scene to be generated into the trained flow matching model, and output the three-dimensional layout of the indoor scene predicted by the flow matching model.

2. The method of claim 1, wherein, In step 1, the boundary information of each indoor scene is encoded by a PointNet network; During encoding, the PointNet network uniformly samples a point set from the room boundary, each sampled point includes two-dimensional position and normal vector information, and the PointNet network encodes and outputs a 512-dimensional geometric feature vector based on the sampled point set, which is the boundary feature.

3. The method of claim 1, wherein, In step 2, the flow matching model uses a conditional flow matching method to construct a probability path between an initial noise sample and a real scene sample by linear interpolation, and trains the backbone network to predict a conditional vector field; Wherein, the initial noise sample is sampled from a Gaussian distribution, and the real scene sample is obtained from step 1; The conditional vector field is predicted by the backbone network based on the encoded scene, time step and boundary feature, and is used to guide the evolution of the sample along the probability path; The conditional vector field is combined with the solution of the ordinary differential equation to update the scene state through iteration, realizing the transition from the initial noise distribution to the target scene distribution.

4. The method of claim 1, wherein, In step 2, the non-overlapping constraint loss calculates the three-dimensional bounding box intersection ratio between all object pairs in the scene, and optimizes the object position and size through gradient backpropagation to reduce object overlap.

5. The method of claim 1, wherein the method further comprises: In step 2, the flow matching model is trained by solving ordinary differential equations iteratively, and the scene state is updated according to the predicted vector field at each step, and finally the indoor scene three-dimensional layout conforming to the boundary constraint and object compatibility is output. Among them, the boundary constraint is provided by the boundary feature, which ensures that the object does not exceed the room boundary; the object compatibility is jointly ensured by the scene encoding and the non-overlapping constraint, which ensures that the object layout is reasonable and non-overlapping.

Images