A room reconstruction method and apparatus

Abstract

The application discloses a room reconstruction method and device, which can be applied to the field of deep learning and comprises the following steps: the soft furnishing furniture area is accurately identified and separated through a semantic segmentation technology, so that the boundary of the furniture area is clear and accurate, and an explicit reference is provided for subsequent processing. Then, the empty room image obtained after the soft furnishing furniture area is emptied retains the hard furnishing structure and background information of the room, and provides a complete background reference for generating a new furniture mask area. The empty room image is input into a diffusion model, and a parameterized noise prediction network of the diffusion model is used to generate a target furniture mask area matched with the room style and layout. Finally, the empty room image and the target furniture mask area are fused to obtain a complete reconstructed room image. The natural transition of the soft furnishing furniture and the room background is realized, the generated reconstructed room image is not only consistent in style, but also has high authenticity and naturalness, so that the demand of users for high-quality room reconstruction is met.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a room reconstruction method and apparatus. Background Technology

[0002] In the fields of interior design and home furnishing, room reconstruction technology has always been a research hotspot. Traditional room reconstruction methods typically rely on manual design and adjustments, which is not only time-consuming and labor-intensive but also struggles to meet users' demands for personalized and diverse designs. Furthermore, traditional methods often require substantial professional knowledge and experience when dealing with complex scenarios, making them quite challenging for ordinary users.

[0003] With the development of computer vision and deep learning technologies, image processing-based room reconstruction technology has gradually become a cutting-edge research direction. However, existing image-based room reconstruction methods still have some limitations. For example, existing technologies typically employ image processing and simple image synthesis techniques. These methods first manually remove existing furniture from the room image using image editing tools, and then add new furniture models to the empty room image. However, this method has a key problem: after removing the soft furnishings, the generated new furniture mask areas have a low match with the room's style and layout, and cannot blend naturally into the background. This is because existing technologies lack a deep understanding of the overall style and layout of the room, and cannot accurately predict the shape and position of the new furniture mask areas, resulting in a stiff and unnatural reconstructed room image, thus causing a significant deviation between the reconstruction result and the actual needs. This affects the quality of room reconstruction and the user experience.

[0004] Therefore, how to make the reconstructed room images more natural, improve the quality of room reconstruction and user experience is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides a room reconstruction method to make the reconstructed room image more natural, improve the quality of room reconstruction and user experience, and this application also provides a room reconstruction device.

[0006] Firstly, this application provides a room reconstruction method, including:

[0007] Obtain an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area;

[0008] Clear the area of ​​soft furnishings in the initial room image to obtain an empty room image;

[0009] The empty room image is input into a pre-trained diffusion model to obtain the target furniture mask region. The training objective of the diffusion model is to learn a parameterized noise prediction network that can predict noise given a time step and control conditions.

[0010] The empty room image is fused with the target furniture mask area to obtain a reconstructed room image.

[0011] Optionally, the diffusion model has the following structure:

[0012]

[0013] in: For the control branch, used to process empty room information, w is a learnable weight parameter, ∈ θ The original denoising network, As a new denoising prediction model, it participates in the training, where t is the time step and c includes empty room images to indicate the generation process of the diffusion model.

[0014] Optionally, the diffusion model is trained in the following manner:

[0015] The original denoising network of the diffusion model is preserved. θ The weights remain unchanged, and the control branch is trained.

[0016] By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions;

[0017] During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

[0018] Optionally, the method further includes:

[0019] The initial room image I and the soft furnishing area M are processed to generate a white image IW with the same structure as the initial room image I. Then, the soft furnishing area M is used to synthesize pixels to obtain the synthesized image I′. The formula is I′=M·I+(1-M)·IW, where the area outside the mask remains white.

[0020] Convert the synthesized image I′ into a grayscale image G;

[0021] By comparing each pixel value in the grayscale image G with the white pixel threshold Tw, a non-white region mask Mnw is generated, with the formula Mnw = G. <Tw;

[0022] Calculate the pixel brightness distribution of the non-white area mask and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness, and the formula is Td = percentile(G[Mnw], Pd).

[0023] The shadow pixel mask Ms is extracted based on the white area mask, and the formula is Ms=Mnw∩(G <Td);

[0024] Clearing the soft furnishing area from the initial room image to obtain an empty room image includes:

[0025] The initial room image after semantic segmentation and the shadow pixel mask are input into the image inpainting model to obtain an empty room image.

[0026] Optionally, the initial room image after semantic segmentation is a segmented image. The step of inputting the initial room image after semantic segmentation and the shadow pixel mask into the image inpainting model to obtain an empty room image includes:

[0027] The image restoration model is used to extract contextual information from the non-shaded regions of the segmented image;

[0028] The area that needs to be repaired is determined based on the shadow pixel mask;

[0029] Image data is generated in the area requiring repair based on the context information, and the image data is fused with the non-shaded area to obtain an image of the empty room.

[0030] Secondly, this application also provides a room reconstruction device, the device comprising:

[0031] The acquisition unit is used to acquire an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area;

[0032] The clearing unit is used to clear the soft furnishing area in the initial room image to obtain an empty room image;

[0033] The input unit is used to input the empty room image into a pre-trained diffusion model to obtain the target furniture mask region. The training objective of the diffusion model is to learn a parameterized noise prediction network that can predict noise given a time step and control conditions.

[0034] The fusion unit is used to fuse the empty room image with the target furniture mask area to obtain a reconstructed room image.

[0035] Optionally, the diffusion model has the following structure:

[0036]

[0037] in: For the control branch, used to process empty room information, w is a learnable weight parameter, ∈ θ The original denoising network, As a new denoising prediction model, it participates in the training, where t is the time step and c includes empty room images to indicate the generation process of the diffusion model.

[0038] Optionally, the diffusion model is trained in the following manner:

[0039] The original denoising network of the diffusion model is preserved. θ The weights remain unchanged, and the control branch is trained.

[0040] By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions;

[0041] During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

[0042] Optionally, the device further includes:

[0043] The initial room image I and the soft furnishing area M are processed to generate a white image IW with the same structure as the initial room image I. Then, the soft furnishing area M is used to synthesize pixels to obtain the synthesized image I′. The formula is I′=M·I+(1-M)·IW, where the area outside the mask remains white.

[0044] Convert the synthesized image I′ into a grayscale image G;

[0045] By comparing each pixel value in the grayscale image G with the white pixel threshold Tw, a non-white region mask Mnw is generated, with the formula Mnw = G. <Tw;

[0046] Calculate the pixel brightness distribution of the non-white area mask and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness, and the formula is Td = percentile(G[Mnw], Pd).

[0047] The shadow pixel mask Ms is extracted based on the white area mask, and the formula is Ms=Mnw∩(G <Td);

[0048] The clearing unit is specifically used for:

[0049] The initial room image after semantic segmentation and the shadow pixel mask are input into the image inpainting model to obtain an empty room image.

[0050] Optionally, the initial room image after semantic segmentation is a segmented image, and the input unit is specifically used for:

[0051] The image restoration model is used to extract contextual information from the non-shaded regions of the segmented image;

[0052] The area that needs to be repaired is determined based on the shadow pixel mask;

[0053] Based on the context information, image data is generated in the area that needs to be repaired, and the image data is fused with the non-shaded area to obtain an image of an empty room.

[0054] Thirdly, embodiments of this application provide an apparatus including a memory and a processor, the memory being used to store instructions or code, and the processor being used to execute the instructions or code to cause the apparatus to perform the method described in the first aspect above.

[0055] Fourthly, embodiments of this application provide a computer storage medium storing code, wherein when the code is executed, a device running the code implements the method described in the first aspect above. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart illustrating a room reconstruction method provided in this application embodiment;

[0058] Figure 2 A schematic diagram illustrating an image semantic segmentation method provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram illustrating the generation of an empty room image provided in an embodiment of this application;

[0060] Figure 4 A schematic diagram illustrating image processing using a shadow detector, provided as an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of generating a target mask region provided in an embodiment of this application;

[0062] Figure 6This is a structural schematic diagram of one specific embodiment of the room reconstruction device provided in this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0064] It should be noted that the room reconstruction method and apparatus provided in this application are for deep learning application scenarios. "Other fields" refers to any field other than deep learning, such as the field of artificial intelligence. The above is merely an example and does not limit the application fields of the method and apparatus provided in this application.

[0065] In the fields of interior design and home furnishing, room reconstruction technology has always been a research hotspot. Traditional room reconstruction methods typically rely on manual design and adjustments, which is not only time-consuming and labor-intensive but also struggles to meet users' demands for personalized and diverse designs. Furthermore, traditional methods often require substantial professional knowledge and experience when dealing with complex scenarios, making them quite challenging for ordinary users.

[0066] With the development of computer vision and deep learning technologies, image processing-based room reconstruction technology has gradually become a cutting-edge research direction. However, existing image-based room reconstruction methods still have some limitations. For example, existing technologies typically employ image processing and simple image synthesis techniques. These methods first manually remove existing furniture from the room image using image editing tools, and then add new furniture models to the empty room image. However, this method has a key problem: after removing the soft furnishings, the generated new furniture mask areas have a low match with the room's style and layout, and cannot blend naturally into the background. This is because existing technologies lack a deep understanding of the overall style and layout of the room, and cannot accurately predict the shape and position of the new furniture mask areas, resulting in a stiff and unnatural reconstructed room image, thus causing a significant deviation between the reconstruction result and the actual needs. This affects the quality of room reconstruction and the user experience.

[0067] This application proposes a technical solution that accurately identifies and separates the soft furnishing furniture area using semantic segmentation technology, ensuring clear and accurate boundaries for the furniture area and providing a clear reference for subsequent processing. Next, the empty room image obtained after clearing the soft furnishing furniture area retains the room's hard furnishing structure and background information, providing a complete background reference for generating new furniture mask areas. The empty room image is input into a diffusion model, which uses its parameterized noise prediction network to generate target furniture mask areas that match the room's style and layout. By learning noise prediction, the diffusion model can generate more natural furniture mask areas that better match the overall style of the room. Finally, the empty room image and the target furniture mask areas are fused to obtain a complete reconstructed room image. This application employs a method combining semantic segmentation and a diffusion model. Semantic segmentation can accurately identify the boundaries of soft furnishing furniture, ensuring accurate separation of the furniture area. The empty room image obtained after clearing the soft furnishing furniture completely retains the room's hard furnishing structure and background information, providing accurate context for generating new furniture mask areas. The diffusion model, through its parameterized noise prediction network, can generate highly matching target furniture mask regions based on the room's style and layout, avoiding the unnatural results caused by the lack of deep learning and style matching in traditional methods. Finally, by fusing the empty room image with the target furniture mask regions, a natural transition between the soft furnishings and the room background is achieved. The generated reconstructed room image not only has a consistent style but also possesses a high degree of realism and naturalness, thus meeting users' needs for high-quality room reconstruction.

[0068] The method provided in this application embodiment can be executed by software on a terminal device. The terminal device may be, for example, a mobile phone, tablet computer, or other similar device.

[0069] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following description uses the method provided in the embodiments of the present application, executed by a computing device, as an example.

[0070] Figure 1 A flowchart illustrating a room reconstruction method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0071] S101: Obtain an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area.

[0072] The computing device acquires the initial room image, such as Figure 2 As shown in (a), semantic segmentation can then be performed on the initial room image to obtain the soft furnishing area, such as... Figure 2As shown in (b) of the image. The initial room image can include both hard furnishings and soft furnishings. Hard furnishings refer to the fixed parts of the room that are not easily changed, such as walls, floors, and ceilings. Soft furnishings refer to furniture and decorations that can be easily replaced, such as sofas, curtains, and lighting fixtures.

[0073] For example, the computing device can utilize the segmentation algorithm Mask2Former–DCNv4, employing its FlashInternImage-L model as the core segmentation scheme. This model can parse the initial room image and generate high-precision segmentation results containing multiple classes of indoor objects (based on the ADE20K dataset). Building upon this, the computing device can further extract structural regions such as walls, floors, ceilings, doors, and window glass from the segmented image and compare them with the overall segmentation result, thereby accurately identifying the soft furnishing areas in the image.

[0074] Mask2Former is a segmentation algorithm based on the Transformer architecture, capable of effectively handling various complex segmentation tasks. DCNv4 (Deformable Convolutional Networks v4) is its upgraded version, further optimizing feature extraction and segmentation accuracy. This algorithm combines the powerful modeling capabilities of Transformer with the flexibility of deformable convolution, enabling it to better handle complex structural and semantic information in images.

[0075] The FlashInternImage-L model is a pre-trained segmentation model specifically optimized for indoor scenes. It can be trained on the ADE20K dataset, which contains a large number of indoor scene images with rich semantic category annotations. This model can parse initial room images and generate high-precision segmentation results covering multiple categories of indoor objects, such as furniture, decorations, and structural elements.

[0076] Furthermore, the computing device processes the input room image using the FlashInternImage-L model. Based on semantic information learned on the ADE20K dataset, the model segments the image into multiple distinct category regions. Each pixel is assigned a specific semantic category, such as "wall," "floor," "ceiling," "sofa," and "table." In the generated segmentation results, the computing device further extracts structural regions, such as walls, floors, ceilings, doors, and window glass. These regions typically have relatively fixed shapes and positions and are fundamental components of the interior space. The extracted structural regions are compared with the overall segmentation result. By excluding structural regions, the remaining portion represents the soft furnishings area. Soft furnishings refer to movable, replaceable, or decorative items in the interior space, such as sofas, chairs, curtains, and carpets. After these regions are extracted, a soft furnishings mask can be generated, such as... Figure 2 As shown in (c) above. The soft furnishing mask is a special image generated using image segmentation technology, used to identify and distinguish the location and extent of soft furnishings in an interior scene.

[0077] S102: Clear the soft furnishing area in the initial room image to obtain an empty room image.

[0078] The computing device can clear the area of ​​soft furnishings in the initial room image to obtain an empty room image.

[0079] In some possible implementations, a soft furnishing mask can be input into a pre-trained LAMMA restoration model for processing to obtain an image of an empty room. The core idea of ​​the LAMMA model is to use deep learning techniques, through convolutional neural networks (CNNs) or Transformer architectures, to learn the semantic and textural information of an image. Generating an empty room image using the LAMMA model essentially involves removing soft furnishings and other hard furnishing elements from the room image and filling in these areas, thereby generating an image of an empty room containing only hard furnishing elements such as walls, floors, and ceilings.

[0080] In some possible implementations, the LAMMA restoration model may generate artifacts, and the segmentation model may not accurately identify the shadow areas of furniture, potentially resulting in numerous dark shadows in the restoration result, affecting the overall naturalness. For example... Figure 3 As shown in (b) and (c) of this application, an embodiment of the application develops a shadow detector based on a graphics algorithm to accurately identify shadow regions in the original image and incorporate them into the segmentation mask. This optimization significantly improves the accuracy of the initial repair, making subsequent processing more stable and reliable.

[0081] Exemplarily, the computing device can process the initial room image I and the soft furnishing area M to generate a completely white image IW with the same structure as the initial room image I, and perform pixel synthesis through the soft furnishing area M to obtain a synthesized image I′. The formula is I′ = M·I+(1 - M)·IW, where the area outside the mask remains white; then convert the synthesized image I′ into a grayscale image G; by comparing each pixel value in the grayscale image G with the white pixel threshold Tw, generate a non-white area mask Mnw. The formula is Mnw = G < Tw; calculate the pixel brightness distribution of the non-white area mask, and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness. The formula is Td = percentile(G[Mnw], Pd), and extract the shadow pixel mask Ms according to the white area mask, as shown in Figure 4 shown in (b) of. The formula is Ms = Mnw ∩ (G < Td). Then it can be normalized and converted into a tensor format for facilitating subsequent deep learning task applications. The computing device can input the initial room image and the shadow pixel mask after semantic segmentation into the Inpainting model for processing to obtain an empty room image. Among them, the Inpainting (image inpainting) model is a deep learning technology used to fill in missing or damaged parts of an image. It learns the semantic information and texture features of the image and automatically generates the content of the missing area to make it visually consistent with the surrounding area. In this way, this application can generate natural, coherent, highly realistic and shadow-removed soft furnishing clearance effect pictures, ensuring the integrity and visual consistency of the picture, as shown in Figure 3 shown in (d) and (e) of.

[0082] S103: Input the empty room image into a pre-trained diffusion model to obtain a target furniture mask area.

[0083] The computing device inputs the empty room image into a pre-trained diffusion model to obtain a room image with furniture, as shown in Figure 5 shown in (b) of, and then perform semantic segmentation on this image to obtain an image with a soft furnishing area, as shown in Figure 5 shown in (c) of, and finally extract the soft furnishing area to obtain a target furniture mask area. The obtained target furniture mask area is as shown in Figure 5 shown in (d) of. Specifically, it is similar to the principle described in Figure 2 shown in in step S101, and no redundant description will be given here.

[0084] In some possible implementation manners, on the original denoising network U-Net structure ∈ θ of the diffusion model, an additional conditional control branch is introduced and an empty room condition input is added. The diffusion model can have the following structure:

[0085]

[0086] in: For the control branch, used to process empty room information, w is a learnable weight parameter, ∈ θ The original denoising network, As a new denoising prediction model, it participates in the training, where t is the time step and c includes empty room images to indicate the generation process of the diffusion model.

[0087] The training objective of the diffusion model is to learn a parameterized noise prediction network that, given a time step and control conditions, predicts the noise; that is, let X0 be the real data, X... t Given the data after forward diffusion at time step t, and c as the control conditions (including the empty room condition), the goal is to learn a parameterized noise prediction network ∈ θ Make:

[0088]

[0089] Where ∈ represents standard Gaussian noise, t is the time step, and c now contains empty room images to guide the generation process of the diffusion model. E represents finding the minimum value of parameter θ; x0，c，t，∈ This represents the expectation with respect to the variables x0, c, and ∈.

[0090] Let represent the square of the L2 norm of the difference between ∈ and ∈θ(xt,t,c).

[0091] In some possible implementations, the diffusion model can be trained in the following ways:

[0092] The original denoising network of the diffusion model is preserved. θ The weights remain unchanged, and the control branch is trained.

[0093] By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions;

[0094] During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

[0095] The core of this training method lies in introducing additional control branches and empty room conditions to impose specific constraints on the generation process of the diffusion model. Simultaneously, strategies such as freezing the main network and mixing training data ensure that the original model's generation capability is not compromised while introducing control signals. In other words, physical rationality constraints are applied when the diffusion model generates furniture to prevent unreasonable structural alterations. Furthermore, the diffusion model only repairs and rebuilds the soft furnishing areas, without affecting the room's structure or original hard furnishing materials, thus avoiding potential loss of hard furnishing information.

[0096] S104: Fuse the empty room image with the target furniture mask area to obtain a reconstructed room image.

[0097] The computing device fuses the image of the empty room with the masked area of ​​the target furniture to obtain a reconstructed image of the room.

[0098] For example, an image of an empty room and the masked area of ​​the target furniture can be input into an Inpainting model, an image restoration technique. Its core principle is to fill in missing areas in an image using a deep learning model. The model can extract semantic and texture features from the input image, understanding its overall structure and content. Based on the extracted features and mask information, the model generates content for the missing areas, making it visually consistent with the surrounding areas not covered by the mask. The model is trained using a large number of images with masks and their corresponding complete images, learning how to generate high-quality restoration results.

[0099] Prompt semantic understanding is a natural language processing technique that guides a generative model to generate images that conform to specific semantic content based on a user-inputted text description (Prompt). The user inputs a descriptive text (Prompt), such as "a modern minimalist living room with a comfortable sofa and a simple coffee table."

[0100] The computing device can parse the semantic information in the prompt to understand the user's desired style and layout. Then, based on the semantic information, the generative model generates a furniture layout that matches the user's settings in the repaired image, resulting in a reconstructed room image.

[0101] This application accurately identifies and separates the soft furnishing furniture area using semantic segmentation technology, ensuring clear and accurate boundaries for the furniture area and providing a clear reference for subsequent processing. Next, the empty room image obtained after removing the soft furnishing furniture area retains the room's hard furnishing structure and background information, providing a complete background reference for generating new furniture mask areas. The empty room image is input into a diffusion model, which uses its parameterized noise prediction network to generate target furniture mask areas that match the room's style and layout. By learning noise prediction, the diffusion model can generate more natural furniture mask areas that better match the overall style of the room. Finally, the empty room image and the target furniture mask areas are fused to obtain a complete reconstructed room image. This application employs a method combining semantic segmentation and a diffusion model. Semantic segmentation can accurately identify the boundaries of soft furnishing furniture, ensuring accurate separation of the furniture area. The empty room image obtained after removing the soft furnishing furniture completely retains the room's hard furnishing structure and background information, providing accurate context for generating new furniture mask areas. The diffusion model, through its parameterized noise prediction network, can generate highly matching target furniture mask regions based on the room's style and layout, avoiding the unnatural results caused by the lack of deep learning and style matching in traditional methods. Finally, by fusing the empty room image with the target furniture mask regions, a natural transition between the soft furnishings and the room background is achieved. The generated reconstructed room image not only has a consistent style but also possesses a high degree of realism and naturalness, thus meeting users' needs for high-quality room reconstruction.

[0102] The above are some specific implementations of the room reconstruction method provided in this application. Based on this, this application also provides a corresponding device. The device provided in this application will be described below from the perspective of functional modularization. This device and the room reconstruction method described above can be referred to each other.

[0103] Figure 6 The structural block diagram of the room reconstruction device provided in this embodiment of the invention is referred to as Specific Embodiment Two, and is referred to as follows. Figure 6 The device may include:

[0104] The acquisition unit 600 is used to acquire an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area;

[0105] The clearing unit 610 is used to clear the soft furnishing area in the initial room image to obtain an empty room image;

[0106] The input unit 620 is used to input the empty room image into a pre-trained diffusion model to obtain the target furniture mask region. The training objective of the diffusion model is to learn a parameterized noise prediction network that can predict noise under given time steps and control conditions.

[0107] The fusion unit 630 is used to fuse the empty room image with the target furniture mask area to obtain a reconstructed room image.

[0108] Optionally, the diffusion model has the following structure:

[0109]

[0110] in: For the control branch, used to process empty room information, w is a learnable weight parameter, ∈ θ The original denoising network, As a new denoising prediction model, it participates in the training, where t is the time step and c includes empty room images to indicate the generation process of the diffusion model.

[0111] Optionally, the diffusion model is trained in the following manner:

[0112] The original denoising network of the diffusion model is preserved. θ The weights remain unchanged, and the control branch is trained.

[0113] By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions;

[0114] During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

[0115] Optionally, the device further includes:

[0116] The initial room image I and the soft furnishing area M are processed to generate a white image IW with the same structure as the initial room image I. Then, the soft furnishing area M is used to synthesize pixels to obtain the synthesized image I′. The formula is I′=M·I+(1-M)·IW, where the area outside the mask remains white.

[0117] Convert the synthesized image I′ into a grayscale image G;

[0118] By comparing each pixel value in the grayscale image G with the white pixel threshold Tw, a non-white region mask Mnw is generated, with the formula Mnw = G. <Tw;

[0119] Calculate the pixel brightness distribution of the non-white area mask and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness, and the formula is Td = percentile(G[Mnw], Pd).

[0120] The shadow pixel mask Ms is extracted based on the white area mask, and the formula is Ms=Mnw∩(G <Td);

[0121] The clearing unit is specifically used for:

[0122] The initial room image after semantic segmentation and the shadow pixel mask are input into the image inpainting model to obtain an empty room image.

[0123] Optionally, the initial room image after semantic segmentation is a segmented image, and the input unit is specifically used for:

[0124] The image restoration model is used to extract contextual information from the non-shaded regions of the segmented image;

[0125] The area that needs to be repaired is determined based on the shadow pixel mask;

[0126] Based on the context information, image data is generated in the area that needs to be repaired, and the image data is fused with the non-shaded area to obtain an image of an empty room.

[0127] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0128] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to cause the device to perform the method described in any embodiment of this application.

[0129] The computer storage medium stores code, and when the code is run, the device running the code implements the method described in any embodiment of this application.

[0130] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0131] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0133] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A room reconstruction method, characterized in that, include: Obtain an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area; Clear the area of ​​soft furnishings in the initial room image to obtain an empty room image; The empty room image is input into a pre-trained diffusion model to obtain the target furniture mask region. The training objective of the diffusion model is to learn a parameterized noise prediction network that can predict noise given a time step and control conditions. The empty room image is fused with the target furniture mask area to obtain a reconstructed room image; The method further includes: The initial room image I and the soft furnishing area M are processed to generate a pure white image IW with the same structure as the initial room image I. Then, pixel synthesis is performed using the soft furnishing area M to obtain the synthesized image I′, with the formula I′=M. I+(1-M) IW, where the area outside the mask remains white; Convert the synthesized image I′ into a grayscale image G; By comparing each pixel value in the grayscale image G with the white pixel threshold Tw, a non-white region mask Mnw is generated, with the formula Mnw=G. <Tw; Calculate the pixel brightness distribution of the non-white area mask and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness, and the formula is Td=percentile(G[Mnw],Pd). The shadow pixel mask Ms is extracted based on the white area mask, and the formula is Ms=Mnw∩(G <Td); Clearing the soft furnishing area from the initial room image to obtain an empty room image includes: The initial room image after semantic segmentation and the shadow pixel mask are input into the image inpainting model to obtain an empty room image.

2. The method according to claim 1, characterized in that, The diffusion model has the following structure: ; in: To control the branch, used to process information about empty rooms. These are learnable weight parameters. The original denoising network, As a new denoising prediction model, X participates in the training, where t is the time step, and c includes the empty room image to indicate the generation process of the diffusion model; t This represents the data after forward diffusion at time step t.

3. The method according to claim 2, characterized in that, The diffusion model is trained in the following way: Preserve the original denoising network of the diffusion model The weights remain unchanged, and the control branch is trained. ; By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions; During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

4. The method according to claim 1, characterized in that, The initial room image after semantic segmentation is the segmented image. The step of inputting the initial room image after semantic segmentation and the shadow pixel mask into the image inpainting model to obtain the empty room image includes: The image restoration model is used to extract contextual information from the non-shaded regions of the segmented image; The area that needs to be repaired is determined based on the shadow pixel mask; Based on the context information, image data is generated in the area that needs to be repaired, and the image data is fused with the non-shaded area to obtain an image of an empty room.

5. A room reconstruction device, characterized in that, include: The acquisition unit is used to acquire an initial room image, perform semantic segmentation on the initial room image, and obtain the soft furnishing area; The clearing unit is used to clear the soft furnishing area in the initial room image to obtain an empty room image; The input unit is used to input the empty room image into a pre-trained diffusion model to obtain the target furniture mask region. The training objective of the diffusion model is to learn a parameterized noise prediction network that can predict noise given a time step and control conditions. The fusion unit is used to fuse the empty room image with the target furniture mask area to obtain a reconstructed room image; The device further includes: The initial room image I and the soft furnishing area M are processed to generate a pure white image IW with the same structure as the initial room image I. Then, pixel synthesis is performed using the soft furnishing area M to obtain the synthesized image I′, with the formula I′=M. I+(1-M) IW, where the area outside the mask remains white; Convert the synthesized image I′ into a grayscale image G; By comparing each pixel value in the grayscale image G with the white pixel threshold Tw, a non-white region mask Mnw is generated, with the formula Mnw=G. <Tw; Calculate the pixel brightness distribution of the non-white area mask and set a shadow threshold Td. The shadow threshold Td is set based on the percentile Pd of the non-white pixel brightness, and the formula is Td=percentile(G[Mnw],Pd). The shadow pixel mask Ms is extracted based on the white area mask, and the formula is Ms=Mnw∩(G <Td); The clearing unit is specifically used for: The initial room image after semantic segmentation and the shadow pixel mask are input into the image inpainting model to obtain an empty room image.

6. The apparatus according to claim 5, characterized in that, The diffusion model has the following structure: ; in: To control the branch, used to process information about empty rooms. These are learnable weight parameters. The original denoising network, As a new denoising prediction model, X participates in the training, where t is the time step and c includes the empty room image to indicate the generation process of the diffusion model; t This represents the data after forward diffusion at time step t.

7. The apparatus according to claim 6, characterized in that, The diffusion model is trained in the following way: Preserve the original denoising network of the diffusion model The weights remain unchanged, and the control branch is trained. ; By using residual learning, the control signal of the empty room is applied to the denoising process, controlling the branch. Predict the impact of empty room conditions; During training, a preset proportion of samples are randomly selected from the training samples to use the original diffusion model, and empty room condition inputs are applied to the remaining samples of the training samples.

8. The apparatus according to claim 5, characterized in that, The initial room image after semantic segmentation is the segmented image, and the input unit is specifically used for: The image restoration model is used to extract contextual information from the non-shaded regions of the segmented image; The area that needs to be repaired is determined based on the shadow pixel mask; Based on the context information, image data is generated in the area that needs to be repaired, and the image data is fused with the non-shaded area to obtain an image of an empty room.

Images