Indoor scene image rendering method and device, electronic equipment and storage medium

Abstract

The invention provides an indoor scene image rendering method and device, electronic equipment and a storage medium. The invention relates to the technical field of computers, in particular to the fields of computer vision, data processing, intelligent drawing and the like. According to the specific scheme, in response to a received indoor scene original image, a to-be-rendered area predicted for the indoor scene original image is output; in response to the received editing operation on the to-be-rendered area, calling an image segmentation model to determine a target area of the indoor scene original image; in response to the detected target material selected for the target area, outputting a preview of the target area; and in response to the detected image rendering instruction, outputting an indoor scene rendering image generated based on the target area and the scene data corresponding to the preview thereof. According to the scheme, the indoor scene image rendering effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of computer vision, data processing, and intelligent mapping. Background Art

[0002] After the emergence of artificial intelligence image generation models, there are already some artificial intelligence home decoration scene generation or home decoration scene replacement products based on their research and development on the market. The products currently available on the market can only replace the entire scene and cannot accurately control the details; or the user can only manually draw the area to be replaced and then replace it in this area, which makes the interaction complex and inconvenient, and the area shape is limited, resulting in poor results; or it can only be redrawn through rough description words, and cannot provide stable and accurate material expression and drawing effects. In summary, home decoration scene image rendering still needs to be improved. Summary of the invention

[0003] The present invention provides an image rendering method, device, electronic device and storage medium for an indoor scene.

[0004] According to a first aspect of the present disclosure, there is provided an image rendering method for an indoor scene, comprising:

[0005] In response to receiving the original image of the indoor scene, outputting a region to be rendered predicted by the original image of the indoor scene;

[0006] In response to receiving an editing operation on the area to be rendered, calling an image segmentation model to determine a target area of ​​the original image of the indoor scene;

[0007] In response to detecting the target material selected for the target area, outputting a preview image of the target area, wherein in the preview image, the target area is paved with the target material;

[0008] In response to detecting the image rendering instruction, an indoor scene rendering image generated based on the scene data corresponding to the target area and its preview image is output.

[0009] According to a second aspect of the present disclosure, there is provided an image rendering device for an indoor scene, comprising:

[0010] A first generating module is used for, in response to receiving an original image of an indoor scene, outputting a region to be rendered predicted by the original image of the indoor scene;

[0011] An adjustment module, configured to, in response to receiving an editing operation on the area to be rendered, call an image segmentation model to determine a target area of ​​the original image of the indoor scene;

[0012] A preview module, configured to output a preview image of the target area in response to detecting the target material selected for the target area, wherein the target material is laid on the target area in the preview image;

[0013] The second generating module is used for outputting an indoor scene rendering image generated based on the scene data corresponding to the target area and its preview image in response to detecting the image rendering instruction.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in the embodiments of the present disclosure.

[0015] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0016] According to the technical solution of the present disclosure, rendering of indoor scene images based on target areas and scene data helps to improve the rendering effect of indoor scene images.

[0017] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0019] Figure 1 is a flowchart of an image rendering method for an indoor scene according to an embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of an image rendering process for an indoor scene according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of a component structure for implementing image rendering of an indoor scene according to an embodiment of the present disclosure;

[0022] Figure 4 is a structural schematic diagram of an image rendering device for an indoor scene according to an embodiment of the present disclosure;

[0023] Figure 5 is a scene schematic diagram of an image rendering method for an indoor scene according to an embodiment of the present disclosure;

[0024] Figure 6 The present invention is a schematic diagram of the structure of an electronic device used to implement the image rendering method of an indoor scene according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0026] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0027] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0028] Artificial Intelligence Image Generation Model (Stable Diffusion): It can be widely used to generate detailed images based on text descriptions, and can also be applied to other tasks such as infill drawing, outfill drawing, and the transformation of image-to-image under the guidance of prompt words. As technology develops, the functions of Stable Diffusion software are constantly updated and expanded.

[0029] In the prior art, products that currently generate or replace home decoration scenes with artificial intelligence based on Stable Diffusion can usually only replace the entire scene, and cannot accurately control the details; or users can only manually draw the area to be replaced, and then replace it in this area, which is complicated and inconvenient to interact with, and the area shape is limited, and the effect is not good; or they can only redraw through rough description words, and cannot provide stable and accurate material expression and drawing effects. In summary, there is a technical problem of poor scene image rendering effect in the current prior art.

[0030] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes an image rendering method for an indoor scene. In response to receiving an original image of an indoor scene, an area to be rendered predicted by the original image of the indoor scene is output; in response to receiving an editing operation on the area to be rendered, an image segmentation model is called to determine a target area of ​​the original image of the indoor scene; in response to detecting a target material selected for the target area, a preview image of the target area is output; in response to detecting an image rendering instruction, a rendered image of the indoor scene generated based on scene data corresponding to the target area and its preview image is output, which can improve the rendering effect of the indoor scene image.

[0031] The present disclosure provides an image rendering method for an indoor scene. Figure 1 The figure is a flow chart of an indoor scene image rendering method according to an embodiment of the present disclosure. The indoor scene image rendering method can be applied to an indoor scene image rendering device. The indoor scene image rendering device is located in an electronic device. The electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. In some possible implementations, the indoor scene image rendering method can also be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 As shown, the image rendering method of the internal scene includes:

[0032] S101: In response to receiving an original image of an indoor scene, outputting a region to be rendered predicted by the original image of the indoor scene;

[0033] S102: In response to receiving an editing operation on the area to be rendered, calling an image segmentation model to determine a target area of ​​the original image of the indoor scene;

[0034] S103: In response to detecting the target material selected for the target area, outputting a preview image of the target area, wherein in the preview image, the target area is paved with the target material;

[0035] S104: In response to detecting the image rendering instruction, outputting an indoor scene rendering image generated based on scene data corresponding to the target area and its preview image.

[0036] In the embodiment of the present disclosure, the indoor scene original image may be at least one image uploaded by a user; the indoor scene original image may also be at least one image captured on-site by an image acquisition device. The above is only an exemplary description and is not intended to limit all possible situations of the indoor scene original image, but it is not exhaustive here.

[0037] In the embodiment of the present disclosure, the area to be rendered predicted by the original image of the indoor scene refers to dividing the wall area and the ground area after analyzing the original image of the indoor scene, and pre-selecting all or part of the wall area and / or all or part of the ground area for the user according to preset rules, and using the selected all or part of the wall area and / or all or part of the ground area as the area to be rendered predicted by the original image of the indoor scene.

[0038] In the disclosed embodiment, the editing operation on the area to be rendered refers to an editing operation that can be performed through the area editing mode when the user is not satisfied with the area to be rendered predicted by the original image of the indoor scene or needs to reselect it. The editing operation may include adding a selection and reducing a selection. The selection can be increased by clicking on an unselected area to select it, or by clicking on a selected area to deselect it to reduce the selection. The above is only an exemplary description and is not intended to limit all possible situations of user editing operations, nor is it intended to limit all possible situations of the operation methods of adding and / or reducing selections. It is just that this is not exhaustive.

[0039] In the disclosed embodiment, the image segmentation model is an algorithm or neural network model established to realize the process of dividing an image into multiple regions or objects. Among them, the divided regions or objects usually have similar features and have obvious boundaries between each other. Exemplarily, the image segmentation model can select the Segment Anything model, which is a segmentation model based on deep learning and prompt engineering. Similarly, other models such as the ReSeg model in the prior art, the model combining the Graph Neural Network (GNN) and the Long Short-Term Memoty (LSTM) network can also be selected for implementation. The above is only an exemplary description and is not intended to limit all possible situations for selecting an image segmentation model, but it is not exhaustive here.

[0040] In the disclosed embodiment, the target area of ​​the original image of the indoor scene refers to the whole or part of the wall area and / or the whole or part of the ground area selected after image segmentation according to the selected area determined by the user editing operation.

[0041] In the disclosed embodiment, the target material selected for the target area refers to the material selected by the user from a preset material list for the target area of ​​the original image of the indoor scene. Exemplarily, the materials in the material list may include wall area materials and ground area materials, and may also include various materials of different colors. The above is only an exemplary description and is not intended to limit all possible situations of the target material, but it is not exhaustive here.

[0042] In the disclosed embodiment, the preview image of the target area refers to the preview image displayed to the user after the target area of ​​the original image of the indoor scene is tiled with materials. In particular, the material tiling process is performed only on the selected area, without affecting the content of other scenes and / or areas.

[0043] In the embodiment of the present disclosure, the image rendering instruction refers to an instruction initiated when a user needs a rendered image of an indoor scene after approving a preview image of a target area.

[0044] In the disclosed embodiment, the indoor scene rendering image refers to an artificial intelligence rendering performed by an image generation model according to a pre-configured rendering process based on the scene data corresponding to the target area and its preview image, such as target area data, material data, material paving data. Exemplarily, the image generation model can select a graphical reasoning engine developed based on Stable Diffusion and an open source artificial intelligence rendering framework (ComfyUI), or other image generation models in the prior art can be selected. The above is only an exemplary description, and does not limit all possible situations of the indoor scene rendering image generation method, but it is not exhaustive here.

[0045] The technical solution of the disclosed embodiment is, in response to receiving the original image of the indoor scene, outputting the area to be rendered predicted by the original image of the indoor scene; in response to receiving the editing operation on the area to be rendered, calling the image segmentation model to determine the target area of ​​the original image of the indoor scene; in response to detecting the target material selected for the target area, outputting the preview image of the target area; in response to detecting the image rendering instruction, outputting the indoor scene rendering image generated based on the scene data corresponding to the target area and its preview image. In this way, the indoor scene image can be intelligently segmented and selected, the selected area can be intelligently tiled, and the selected area can be accurately redrawn with high quality according to the specific material, which helps to improve the rendering effect of the indoor scene image.

[0046] In some embodiments, the response to receiving an original image of an indoor scene, outputting an area to be rendered that is predicted by the original image of the indoor scene, includes: initializing an image output component according to the original image of the indoor scene; determining, through an image segmentation model, the wall area and the ground area included in the original image of the indoor scene according to the wall bounding box and the ground bounding box obtained during the initialization process, and pre-selecting any area from the wall area and the ground area as the area to be rendered.

[0047] In the disclosed embodiment, the image output component includes an open set target detection model. Exemplarily, the open set target detection model can select the Grounding Dino segmentation model, or other open set target detection models in the prior art such as the ViLD model and the Detic model. The above is only an exemplary description and is not intended to limit all possible situations of the open set target detection model, but it is not exhaustive here.

[0048] In the disclosed embodiment, the bounding box refers to a rectangular frame that can cover the minimum area of ​​one or more target objects in the image. This rectangular frame is usually defined by four coordinate values: the x-coordinate and y-coordinate of the upper left corner, and the width and height of the rectangular frame. In the disclosed embodiment, the wall bounding box can be used to locate and identify the wall in the image, and the ground bounding box can be used to locate and identify the ground in the image.

[0049] In the disclosed embodiment, the initialization process of the image output component is the information acquisition process. During the initialization process, the open set object detection model of the image output component, after data preprocessing, feature extraction, convolutional network generation, post-processing and other steps, implements the acquisition of wall bounding boxes and ground bounding boxes based on various deep learning frameworks and libraries in the prior art. Exemplarily, the Grounding Dino segmentation model can be used for semantic segmentation to achieve the acquisition of wall bounding boxes and ground bounding boxes.

[0050] In the disclosed embodiment, the process of determining the wall area and the ground area included in the original image of the indoor scene through the image segmentation model can be done by first obtaining the coordinates of the bounding box, then preprocessing the image, using the image segmentation model to output the segmentation result of each pixel, then extracting the corresponding area from the segmentation result according to the coordinates of the bounding box, and finally post-processing the extracted area. Exemplarily, the Segment Anything model can be selected as the image segmentation model for processing, or other image segmentation models in the prior art can be selected. The above is only an exemplary explanation, and is not intended to limit all possible situations of the image segmentation model, but it is not exhaustive here.

[0051] In the disclosed embodiment, the process of preselecting any area from the wall area and the ground area as the area to be rendered can be performed according to a preset selection rule. For example, the area with the highest probability of user selection in the wall area and / or the ground area can be selected as the area to be rendered based on tools such as historical data and user portrait analysis, or other selection specifications in the prior art can be used for selection. The above is only an exemplary description and is not intended to limit all possible situations for selecting the area to be rendered, but it is not exhaustive here.

[0052] The technical solution of the disclosed embodiment is to initialize the image output component according to the original image of the indoor scene; according to the wall bounding box and the ground bounding box obtained during the initialization process, the wall area and the ground area included in the original image of the indoor scene are determined through the image segmentation model, and any area is pre-selected from the wall area and the ground area as the area to be rendered. In this way, the image is automatically semantically segmented through the segmentation algorithm to help users pre-select the wall or the ground. After the user clicks, the segmentation algorithm can directly select a more reasonable ground or wall area, which can greatly improve the user interaction experience under the premise of accurately selecting the specified area.

[0053] In some embodiments, the response to receiving an editing operation on an area to be rendered is to call an image segmentation model to determine a target area in the original image of the indoor scene, including: when the location of the editing operation corresponds to a wall area or a ground area, determining a target bounding box, the target bounding box being able to enclose or identify the area pointed to by the editing operation; inputting the target bounding box and related intermediate files into the image segmentation model, and the image segmentation model performing inference based on the target bounding box and the intermediate files to segment and determine the target area in the original image of the indoor scene.

[0054] In the disclosed embodiment, the intermediate file refers to a file generated after the image is preprocessed by the image segmentation model, and the image segmentation model converts the image data into a low-dimensional vector file, which is used to represent the features of the image. Exemplarily, the Segment Anything model can map the discrete feature index of the image to a low-dimensional continuous vector through an internal embedding layer, and then convert the value of each feature into its index in the embedding matrix, and then obtain the corresponding embedding vector by passing the index to the embedding matrix, and then connect or aggregate multiple embedding vectors corresponding to multiple features to obtain a single embedding vector, and finally use the training and optimization of the embedding layer to generate an intermediate file, that is, an Embedding file. In particular, the intermediate file can also be obtained using other models and methods in the prior art. The above is only an exemplary description and is not intended to limit all possible situations of the intermediate file, but it is not exhaustive here.

[0055] In the embodiment of the present disclosure, when the location of the editing operation corresponds to the wall area or the ground area, the process of segmenting and determining the target area in the original image of the indoor scene can first determine the target bounding box, and then perform reasoning through the entity differentiation service of the image segmentation model based on the target bounding box and the intermediate file, so as to split the target bounding box into finer bounding boxes, and then determine the target area in the original image of the indoor scene. Exemplarily, when the location of the editing operation corresponds to the wall area, the corresponding target wall bounding box and the Embedding file are input into the Segment Anything model, and the wall entity differentiation function built into the model splits the corresponding wall into finer wall bounding boxes, and then the wall corresponding to the finer wall bounding box where the point selected by the user is located is determined as the target area in the original image of the indoor scene. Similarly, when the location of the editing operation corresponds to the ground area, the corresponding target ground bounding box and the Embedding file are input into the Segment Anything model, and the ground entity differentiation function built into the model splits the corresponding ground into finer ground bounding boxes, and then the ground corresponding to the finer ground bounding box where the point selected by the user is located is determined as the target area in the original image of the indoor scene. In particular, the target area may also be obtained using other models and methods in the prior art. The above is only an exemplary description and is not intended to limit all possible situations for determining the target area. This is just not exhaustive.

[0056] The technical solution of the embodiment of the present disclosure is to determine a target bounding box when the location of the editing operation corresponds to a wall area or a ground area, and the target bounding box can enclose or identify the area pointed to by the editing operation; the target bounding box and the related intermediate file are input to the image segmentation model, and the image segmentation model performs reasoning based on the target bounding box and the intermediate file to segment and determine the target area in the original image of the indoor scene. In this way, by using the intermediate file of the image segmentation model for reasoning, image processing can be effectively simplified, and key information can be retained, which is convenient for subsequent machine learning tasks.

[0057] In some embodiments, the response to receiving an editing operation on an area to be rendered, calling an image segmentation model to determine a target area of ​​the original image of the indoor scene, also includes: when the location of the editing operation does not correspond to a wall area and a ground area, determining coordinate information corresponding to the editing operation; inputting the coordinate information and related intermediate files into the image segmentation model, and the image segmentation model inferring based on the coordinate information and the intermediate files, and segmenting and determining the target area in the original image of the indoor scene.

[0058] In the disclosed embodiment, when the location of the editing operation does not correspond to the wall area or the ground area, the process of segmenting and determining the target area in the original image of the indoor scene can first determine the coordinate information corresponding to the editing operation, and then perform reasoning through the entity differentiation service of the image segmentation model based on the coordinate information and the intermediate file, so as to determine the target bounding box, and then split the target bounding box into finer bounding boxes, and finally determine the target area in the original image of the indoor scene. Exemplarily, when the location of the editing operation does not correspond to the wall area or the ground area, the coordinate information and the Embedding file corresponding to the editing operation are input into the Segment Anything model, and the built-in coordinate positioning function in the model projects the corresponding coordinate information to the wall area or the ground area, determines the target bounding box, and then splits the corresponding wall or ground into finer wall bounding boxes or finer ground bounding boxes, and then determines the wall or ground corresponding to the finer wall bounding box or finer ground bounding box where the point is projected as the target area in the original image of the indoor scene. In particular, the target area may also be obtained using other models and methods in the prior art. The above is only an exemplary description and is not intended to limit all possible situations for determining the target area. This is just not exhaustive.

[0059] The technical solution of the embodiment of the present disclosure determines the coordinate information corresponding to the editing operation when the location of the editing operation does not correspond to the wall area and the ground area: the coordinate information and the related intermediate file are input to the image segmentation model, and the image segmentation model performs reasoning based on the coordinate information and the intermediate file to segment and determine the target area in the original image of the indoor scene. In this way, by using the intermediate file of the image segmentation model for reasoning, the user's intention can be better understood and the user interaction experience can be improved.

[0060] In some embodiments, when the location of the editing operation corresponds to a wall area or a ground area, determining the target bounding box includes: based on the location corresponding to the editing operation, selecting one of the wall bounding box and the ground bounding box of the original image of the indoor scene that has been acquired during the initialization process of the output component as the target bounding box.

[0061] In the disclosed embodiment, the process of determining the target bounding box can first determine the coordinate information of the point based on the position corresponding to the editing operation, and then find the wall bounding box or the ground bounding box containing the coordinate information according to the coordinate information and the wall bounding box and the ground bounding box of the original image of the indoor scene, and use the wall bounding box or the ground bounding box as the target bounding box.

[0062] The technical solution of the embodiment of the present disclosure selects one of the wall bounding box and the ground bounding box of the original image of the indoor scene that has been obtained during the initialization of the image output component as the target bounding box based on the position corresponding to the editing operation. In this way, the user click is matched with the bounding box, which can better understand the user's intention and enhance the user interaction experience.

[0063] In some embodiments, the image rendering method for indoor scenes also includes: combining the current segmentation results of the image segmentation model and the image depth information, vanishing point prediction results and full image segmentation results generated by the output component during the initialization process, and paving the target area with materials so that the target area is paved with the target material.

[0064] In the disclosed embodiment, the image output component also includes a depth estimation model. Exemplarily, the depth estimation model can select the Monodepth monocular depth estimation model, or other depth estimation models in the prior art such as the Depth Anything model and the GeoNet model. The above is only an exemplary description and is not intended to limit all possible situations of the depth estimation model, but it is not exhaustive here.

[0065] In the disclosed embodiment, the image output component also includes a vanishing point prediction model. Exemplarily, the vanishing point prediction model can select the DeepVP vanishing point detection model, or other vanishing point prediction models in the prior art. The above is only an exemplary description and is not intended to limit all possible situations of the vanishing point prediction model, but it is not exhaustive here.

[0066] In the disclosed embodiment, during the initialization process of the image output component, the depth estimation model of the image output component uses the geometric constraints between the left and right views and various deep learning frameworks and libraries in the prior art to obtain image depth information after data preprocessing, feature extraction, convolutional network generation, post-processing and other steps. The above is only an exemplary description and is not intended to limit all possible situations for obtaining image depth information, but it is not exhaustive here.

[0067] In the disclosed embodiment, during the initialization of the image output component, the vanishing point prediction model of the image output component, after data preprocessing, feature extraction, convolutional network generation, post-processing and other steps, uses an end-to-end neural network to learn the mapping of image space-edge space-line space-vanishing point space, based on various deep learning frameworks and libraries in the prior art to achieve the acquisition of vanishing point prediction results. The above is only an exemplary description and is not intended to limit all possible situations for obtaining vanishing point prediction results, but it is not exhaustive here.

[0068] In the disclosed embodiment, during the initialization of the image output component, the image segmentation model performs full-image sampling on the image. After data preprocessing, feature extraction, convolutional network generation, post-processing and other steps, the full-image segmentation result can be obtained. The above is only an exemplary description and is not intended to limit all possible situations for obtaining full-image segmentation results. It is just not exhaustive here.

[0069] In the disclosed embodiment, the target area is paved with material, so that the process of paving the target area with the target material can generate or optimize the material of the target area through a diffusion model, an optimization method, and a retrieval method, and can use different lighting conditions and texture scenes. Exemplarily, it can be implemented using the Material Anything framework, or it can be implemented using the DreamMat model or Paint3D model of the prior art. The above is only an exemplary description and is not intended to limit all possible situations of material paving, but it is not exhaustive here.

[0070] The technical solution of the disclosed embodiment combines the current segmentation result of the image segmentation model and the image depth information, vanishing point prediction result and full image segmentation result generated by the image output component during the initialization process to perform material paving on the target area so that the target area is paved with the target material. In this way, by combining data information such as image depth information, vanishing point prediction result and full image segmentation result, it is possible to optimize light and shadow effects and control space in a targeted manner, improve the robustness of the prediction results, and thus improve the rendering effect of indoor scene images.

[0071] In some embodiments, the target area is paved with materials, and further includes: if there are multiple walls in the target area, paving the materials according to the wall orientation, wall size and material properties of the target material of each wall, and adjusting the width, color and texture of the seams according to the connection relationship between adjacent walls, so as to maintain the continuity and consistency of the material seams between different walls.

[0072] In the disclosed embodiment, when the material is paved, the wall orientation and wall size of each wall are determined according to the wall bounding box and the ground bounding box obtained during the initialization process of the output component, and then the material properties of the target material are obtained from the material list according to the material selected by the user, and then the conditional generation model is used to achieve image repair through the conditional generation technology after data preprocessing, feature extraction, convolutional network generation, post-processing and other steps, so that the continuity and consistency of the seams can be maintained when processing the wall material. It uses the dual generation mechanism of texture and structure to fill the missing area to ensure the natural transition of the material between different parts. Exemplarily, the conditional generation model can select the deep network-based repair method (Image Inpainting via Conditional Texture and Structure DualGeneration, referred to as CTSDG) model in the prior art, or other methods in the prior art can be selected. For the characteristics of the square walls and floors of the indoor scene being perpendicular to each other, the capabilities of the existing algorithms and models are combined and optimized, and then the material is paved. The above is only an exemplary description, and is not a limitation of all possible situations of the conditional generation model, but it is not exhaustive here.

[0073] The technical solution of the disclosed embodiment is that if there are multiple walls in the target area, the material is laid according to the wall orientation, wall size and material characteristics of each wall, and the width, color and texture of the seam are adjusted according to the connection relationship between adjacent walls, so as to maintain the continuity and consistency of the material seams between different walls. In this way, the wall and floor area can be automatically detected during paving, and the corresponding paving effect can be made according to the orientation of each surface, roughly ensuring the consistency of the seams and improving the rendering effect of indoor scene images.

[0074] In some embodiments, the response to detecting an image rendering instruction outputs an indoor scene rendering image generated based on scene data corresponding to the target area and its preview image, including: sending the area mask map, paving results and material information corresponding to each target area to the graphical reasoning engine; controlling the graphical reasoning engine to perform image rendering using a pre-configured output process to obtain an indoor scene rendering image.

[0075] In the disclosed embodiment, the region mask map is a commonly used technique in image processing, which is used to specify which parts of the image should be processed or displayed and which parts should be ignored or hidden, and is usually a binary image. The region mask map can utilize the image mask model to dynamically generate a mask according to the target area, combined with the blended latent diffusion (Blended Latent Diffusion, BLD) process and the semantic loss of contrastive language-image pre-training (Contrastive Language-Image Pre-training, CLIP) based on contrastive learning to guide the generation of the mask. Exemplarily, the Click2Mask technology in the prior art can be used to obtain the region mask map, and it can also be obtained by other methods in the prior art. The above is only an exemplary description and is not intended to limit all possible situations for obtaining the region mask map, but it is not exhaustive here.

[0076] In the disclosed embodiment, the diagram reasoning engine is a diagram reasoning engine developed based on an artificial intelligence image generation model and an open source artificial intelligence image generation framework. In particular, other image generation models in the prior art may also be selected. The above is only an exemplary description and does not limit all possible situations of the indoor scene rendering image generation method. It is just not exhaustive here.

[0077] In the disclosed embodiment, the indoor scene rendering image refers to an artificial intelligence drawing performed by a graphical reasoning engine according to a pre-configured drawing process based on data such as a regional mask map, paving results, and material information. Exemplarily, after the indoor scene rendering image is generated, the result is returned to the user for viewing, and repeated generation and secondary editing are supported. Exemplarily, secondary editing may refer to changing the clarity, or re-rendering the generated indoor scene rendering image as the original image of the indoor scene. The above is only an exemplary explanation, and does not limit all possible situations of secondary editing, but it is not exhaustive here.

[0078] The technical solution of the disclosed embodiment sends the area mask map, paving results and material information corresponding to each target area to the diagram reasoning engine; the diagram reasoning engine is controlled to use the pre-configured output process to render the image to obtain the indoor scene rendering image. In this way, the indoor scene rendering image can be obtained under reasonable light and shadow effects and space control conditions, improving the indoor scene image rendering effect.

[0079] Figure 2 A schematic diagram of an image rendering process for an indoor scene is shown. Figure 2 As shown, the process may include:

[0080] S201: The user uploads an original image of an indoor scene;

[0081] S202: Initializing the image output component to obtain vanishing point detection results, depth information, and segmentation information;

[0082] S203: preselecting the wall / ground area as the area to be rendered according to the semantic segmentation information, and outputting the area to be rendered;

[0083] S204: Determine whether the user manually clicks;

[0084] S205: If a manual click by the user is detected, the region is inferred and modified through the segmentation model according to the user's click operation, and finally the target region is obtained;

[0085] S206: If no manual selection by the user is detected, or after the target area has been obtained through segmentation model inference and modification, the user selects the material;

[0086] S207: performing material tiling according to the segmentation result, depth information and vanishing point detection result, and outputting a preview image;

[0087] S208: The user confirms the generation, and the image output component generates the corresponding mask map and material tiling map, and submits the material information to the graphical reasoning engine;

[0088] S209: The graph reasoning engine performs image rendering according to a pre-configured image output process;

[0089] S210: Outputting an indoor scene rendering image.

[0090] In this way, the image is automatically semantically segmented through the segmentation algorithm to help users pre-select walls or floors. After clicking, the user can directly select a more reasonable floor or wall area through the segmentation algorithm; the selected area can be tiled separately without affecting other scene content; the wall and floor areas can be automatically detected during paving, and the corresponding paving effect can be made according to the orientation of each surface, roughly ensuring the consistency of the joints; users can freely choose the paving material, the material characteristics will be retained to the greatest extent during drawing, and the light and shadow effects will be optimized and the space will be controlled in a targeted manner.

[0091] Figure 3 A schematic diagram of the component structure for realizing image rendering of indoor scenes is shown in FIG. Figure 3As shown, the combination component 310 is used to implement the image rendering method of the indoor scene; the back-end capability component 320 is used to provide technical support for the function of the combination component 310; and the business implementation component 330 is used for the application of specific business. For example, the combination component 310 may include components with core capabilities such as the image expansion component, the local image modification component, the magic elimination component, and the wall-to-ground integrated hard installation component. Among them, the combination component 310 can be assembled by uploading the middle platform component; the multi-language component; the smear, erase, reverse selection, and clear component; the operation undo and restore component; the mouse / finger event monitoring component; the front-end sorting component; the moving, rotating, and scaling component; the pure color image judgment component; the mutual conversion component, etc. For example, the back-end capability component 320 may include: the picture segmentation component; the area recognition component; the material paving map component; the depth information auxiliary auxiliary paste component; the mask component; the elimination component; the depth map acquisition component; the image segmentation encoding component, the image segmentation decoding component, etc. For example, the business implementation component 330 may include: furniture trial assembly component; background replacement component; furniture replacement component; wall paint color test component; magic elimination component; wall-to-floor integration component; floor material selection component, etc. It can be understood that the number and functions of the components in the framework can be increased or decreased according to demand.

[0092] It should be understood that Figure 2 and Figure 3 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 2 and Figure 3 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0093] The present disclosure provides an image rendering device for an indoor scene. Figure 4 As shown, the image rendering device for the indoor scene includes:

[0094] A first generating module 410 is configured to output a region to be rendered predicted by the original image of the indoor scene in response to receiving the original image of the indoor scene;

[0095] An adjustment module 420 is configured to, in response to receiving an editing operation on the area to be rendered, call an image segmentation model to determine a target area of ​​the original image of the indoor scene;

[0096] A preview module 430 is used to output a preview image of the target area in response to detecting the target material selected for the target area, wherein the target area is paved with the target material in the preview image;

[0097] The second generating module 440 is configured to output, in response to detecting an image rendering instruction, an indoor scene rendering image generated based on scene data corresponding to the target area and its preview image.

[0098] In some embodiments, the first generation module 410 includes: an initialization submodule, used to initialize the image output component according to the original image of the indoor scene; an area preselection submodule, used to determine the wall area and ground area included in the original image of the indoor scene through an image segmentation model based on the wall bounding box and ground bounding box obtained during the initialization process, and preselect any area from the wall area and the ground area as the area to be rendered.

[0099] In some embodiments, the adjustment module 420 includes: a bounding box determination submodule, used to determine the target bounding box when the location of the editing operation corresponds to a wall area or a ground area; a first target area determination submodule, used to input the target bounding box and related intermediate files into the image segmentation model, and the image segmentation model performs inference based on the target bounding box and the intermediate files to segment and determine the target area in the original image of the indoor scene.

[0100] In some embodiments, the adjustment module 420 also includes: a coordinate determination submodule, which is used to determine the coordinate information corresponding to the editing operation when the location of the editing operation does not correspond to the wall area and the ground area; a second target area determination submodule, which is used to input the coordinate information and the related intermediate files into the image segmentation model, and the image segmentation model performs reasoning based on the coordinate information and the intermediate files to segment and determine the target area in the original image of the indoor scene.

[0101] In some embodiments, the bounding box determination module also includes: a bounding box selection submodule, which is used to select one of the wall bounding boxes and ground bounding boxes of the original image of the indoor scene that have been acquired during the initialization process of the output component as the target bounding box based on the position corresponding to the editing operation.

[0102] In some embodiments, the image rendering device for indoor scenes further includes: a material paving module ( Figure 4 ), which is used to combine the current segmentation result of the image segmentation model and the image depth information, vanishing point prediction result and full-image segmentation result generated by the output component during the initialization process to perform material paving on the target area so that the target area is paved with the target material.

[0103] In some embodiments, the material paving module includes: a paving enhancement submodule, which is used to pave the material according to the wall orientation, wall size and material properties of each wall if there are multiple walls in the target area, and adjust the width, color and texture of the seams according to the connection relationship between adjacent walls to maintain the continuity and consistency of the material seams between different walls.

[0104] In some embodiments, the second generation module 440 includes: an information push submodule, which is used to send the area mask map, paving results and material information corresponding to each target area to the graphical reasoning engine; an image rendering submodule, which is used to control the graphical reasoning engine to use a pre-configured output process to perform image rendering to obtain a rendered image of the indoor scene.

[0105] Those skilled in the art should understand that the functions of each processing module in the image rendering device for indoor scenes of the embodiments of the present disclosure can be understood with reference to the relevant description of the image rendering method for indoor scenes mentioned above, and each processing module in the image rendering device for indoor scenes of the embodiments of the present disclosure can be implemented by an analog circuit that implements the functions of the embodiments of the present disclosure, or can be implemented by running software that executes the functions of the embodiments of the present disclosure on an electronic device.

[0106] The image rendering of the indoor scene in the embodiment of the present disclosure performs rendering of the indoor scene image based on the target area and scene data, which can improve the rendering effect of the indoor scene image.

[0107] The present disclosure provides a scene schematic diagram of image rendering of an indoor scene, such as Figure 5 As mentioned above, the image rendering of the indoor scene provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0108] Specifically, the electronic device can perform the following operations:

[0109] In response to receiving the original image of the indoor scene, outputting a region to be rendered predicted by the original image of the indoor scene;

[0110] In response to receiving an editing operation on the area to be rendered, calling an image segmentation model to determine a target area of ​​the original image of the indoor scene;

[0111] In response to detecting the target material selected for the target area, outputting a preview image of the target area, wherein in the preview image, the target area is paved with the target material;

[0112] In response to detecting the image rendering instruction, an indoor scene rendering image generated based on the scene data corresponding to the target area and its preview image is output.

[0113] The original image and target material of the indoor scene can be obtained from a data source, which can be various forms of data storage devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. In addition, the data source and the user terminal can be the same device.

[0114] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0115] In the technical solution disclosed herein, the acquisition, storage and application of the target object's personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0116] Figure 6 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 As shown, the electronic device includes: a memory 610 and a processor 620, and the memory 610 stores a computer program that can be run on the processor 620. The number of the memory 610 and the processor 620 can be one or more. The memory 610 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 630, which is used to communicate with external devices and perform data exchange transmission.

[0117] If the memory 610, the processor 620 and the communication interface 630 are implemented independently, the memory 610, the processor 620 and the communication interface 630 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0118] Optionally, in a specific implementation, if the memory 610, the processor 620 and the communication interface 630 are integrated on a chip, the memory 610, the processor 620 and the communication interface 630 can communicate with each other through an internal interface.

[0119] It should be understood that the above processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0120] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0121] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by the computer, or a data storage device such as a server or data center that includes one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0122] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0123] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0124] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0125] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0126] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for rendering an image of an indoor scene, characterized in that: The method comprises: In response to receiving an original image of an indoor scene, outputting a region to be rendered predicted by the original image of the indoor scene; In response to receiving an editing operation on the area to be rendered, calling an image segmentation model to determine a target area of ​​the original image of the indoor scene; In response to detecting the target material selected for the target area, outputting a preview image of the target area, wherein in the preview image, the target area is paved with the target material; In response to detecting an image rendering instruction, an indoor scene rendering image generated based on the scene data corresponding to the target area and the preview image thereof is output.

2. The method according to claim 1, characterized in that In response to receiving the original image of the indoor scene, outputting the area to be rendered predicted by the original image of the indoor scene includes: Initializing according to the original image of the indoor scene; According to the wall bounding box and the ground bounding box obtained during the initialization process, the wall area and the ground area included in the original image of the indoor scene are determined by the image segmentation model, and any area is pre-selected from the wall area and the ground area as the area to be rendered.

3. The method according to claim 1, characterized in that In response to receiving the editing operation on the area to be rendered, calling the image segmentation model to determine the target area of ​​the original image of the indoor scene includes: In a case where the location of the editing operation corresponds to a wall area or a ground area, determining a target bounding box, wherein the target bounding box can enclose or identify the area pointed to by the editing operation; The target bounding box and the related intermediate files are input into the image segmentation model, and the image segmentation model performs reasoning based on the target bounding box and the intermediate files to segment and determine the target area in the original image of the indoor scene; wherein the intermediate files include files output by the encoder of the image segmentation model during the initialization process.

4. The method according to claim 3, characterized in that In response to receiving the editing operation on the area to be rendered, calling the image segmentation model to determine the target area of ​​the original image of the indoor scene also includes: In a case where the position of the editing operation does not correspond to the wall area and the ground area, determining coordinate information corresponding to the editing operation; The coordinate information and the related intermediate files are input into the image segmentation model, and the image segmentation model performs reasoning based on the coordinate information and the intermediate files to segment and determine the target area in the original image of the indoor scene: wherein the intermediate files include the files output by the encoder of the image segmentation model during the initialization process.

5. The method according to claim 3, characterized in that: When the location of the editing operation corresponds to a wall area or a ground area, determining a target bounding box includes: Based on the position corresponding to the editing operation, one of the wall bounding box and the ground bounding box of the original image of the indoor scene that have been acquired during the initialization process is selected as the target bounding box.

6. The method according to claim 1, characterized in that The method further comprises: In combination with the current segmentation result of the image segmentation model and the image depth information, vanishing point prediction result and full-image segmentation result generated during the initialization process, material paving is performed on the target area so that the target area is paved with the target material; wherein the full-image segmentation result is the segmentation result obtained after the image segmentation model samples the full image.

7. The method according to claim 6, characterized in that The material paving of the target area further includes: If there are multiple walls in the target area, the material is paved according to the wall orientation, wall size and material properties of each wall, and the width, color and texture of the seams are adjusted according to the connection relationship between adjacent walls to maintain the continuity and consistency of the material seams between different walls.

8. The method according to claim 1, characterized in that The step of outputting, in response to detecting an image rendering instruction, an indoor scene rendering image generated based on scene data corresponding to the target area and the preview image thereof comprises: Sending the area mask map, paving results and material information corresponding to each target area to the graphical reasoning engine; The graphical reasoning engine is controlled to perform image rendering using a pre-configured image output process to obtain the indoor scene rendering image.

9. An image rendering device for an indoor scene, characterized in that: The device comprises: A first generating module, configured to, in response to receiving an original image of an indoor scene, output a region to be rendered predicted by the original image of the indoor scene; An adjustment module, configured to, in response to receiving an editing operation on the area to be rendered, call an image segmentation model to determine a target area of ​​the original image of the indoor scene; A preview module, configured to output a preview image of the target area in response to detecting the target material selected for the target area, wherein in the preview image, the target area is paved with the target material; The second generating module is used for outputting an indoor scene rendering image generated based on the scene data corresponding to the target area and the preview image thereof in response to detecting an image rendering instruction.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images