Home style image generation method and device based on diffusion model, and storage medium

Abstract

The invention relates to a home style image generation method and device based on a diffusion model, and a storage medium. The method comprises the steps of obtaining a first home image, and setting feature information based on a style demand for the first home image; inputting the feature information as a control condition into a U-NET model of a stable diffusion model for optimization training, and generating one or more demand home images by using the stable diffusion model; and performing image processing based on the style demand on the one or more demand home images to obtain a target home image.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, device and storage medium for generating a home style image based on a diffusion model. Background Art

[0002] The Stable Diffusion Model (SDM) is a text-to-image diffusion model that can generate photo-realistic images given a text input. Compared with traditional diffusion models, SDM does not directly diffuse and denoise the image on the original image, but diffuses the image compressed by the variational encoder (VAE), and then restores the compressed code to the image through the decoder. In this way, training the diffusion model can achieve a certain balance between reducing complexity and retaining details, thereby improving visual fidelity. Furthermore, by introducing the cross-attention layer into the model architecture, the diffusion model is transformed into a powerful and flexible generator for general conditional inputs, and convolution makes high-resolution synthesis possible. As a result, SDM is used in image restoration and quasi-conditional image synthesis, significantly reducing the computational requirements. Summary of the invention

[0003] A brief overview of the disclosure is given below in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give some concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating a home style image based on a diffusion model is provided, comprising: acquiring a first home image, and setting feature information based on style requirements for the first home image; inputting the feature information as a control condition into a U-NET model of a stable diffusion model for optimization training, and using the stable diffusion model to generate one or more required home images; and obtaining a target home image by performing image processing based on style requirements on the one or more required home images.

[0005] In some embodiments, responding to the style requirement includes performing partial image replacement on the first home image, and the feature information includes masked image features, a black and white mask, and additional noise.

[0006] In some embodiments, the control conditions include: scaling the masked image features that have been four times downsampled and the black and white masks that have passed the VAE encoder to the same size and then splicing them; inputting the spliced ​​results into the stable diffusion model as an additional parallel branch; and encoding the parallel branches in sequence to obtain multi-stage output features that are fused into the U-NET model as control conditions.

[0007] In some embodiments, in the U-NET model, the parallel branch and the main branch features are element-wise added to achieve fusion.

[0008] In some embodiments, the output of the stable diffusion model is partially decoded by a VAE decoder to generate the one or more desired home images.

[0009] In some embodiments, responding to the style requirement includes performing home style migration on the first home image, and the feature information includes text features associated with the style requirement input by the user.

[0010] In some embodiments, the control condition includes: generating at least one of a corresponding depth map and a semantic map based on the first home image; and controlling the home layout information using the corresponding at least one as a basic component.

[0011] In some embodiments, the control conditions also include: extracting additional image coding information based on home layout information and extracting text coding information based on text features; and fusing the additional image coding information and the text coding information into the middle layer of the U-NET model through a cross-attention module calculation.

[0012] According to a second aspect of an embodiment of the present disclosure, a method for generating a home style image based on a diffusion model is provided, comprising: acquiring a first home image, and in response to style requirements for the first home image including local image replacement, setting feature information to include masked image features, black and white masks, and additional noise; inputting the feature information as a control condition into a U-NET model of a stable diffusion model for optimization training, and partially decoding the output of the stable diffusion model through a VAE decoder to generate one or more required home images; and obtaining a target home image by performing image processing based on style requirements on the one or more required home images.

[0013] According to a third aspect of an embodiment of the present disclosure, a method for generating a home style image based on a diffusion model is provided, comprising: acquiring a first home image, and in response to style requirements for the first home image including home style migration, setting feature information as text features associated with the style requirements input by a user; extracting additional image encoding information and text encoding information based on the feature information, and inputting them as control conditions into a U-NET model of a stable diffusion model for optimization training, and generating one or more required home images using the stable diffusion model; and obtaining a target home image by performing image processing based on the style requirements on the one or more required home images.

[0014] According to a fourth aspect of an embodiment of the present disclosure, a home style image generation device based on a diffusion model is provided, comprising: a processor; and a memory storing computer executable instructions, which, when executed by the processor, enables the processor to execute the aforementioned home style image generation method based on the diffusion model.

[0015] According to a fifth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer-executable instructions is provided, which, when executed by a processor, enables the processor to implement the aforementioned diffusion model-based home style image generation method.

[0016] The advantage of the embodiments disclosed herein is that a specified image associated with a desired style is used as an additional control condition to be incorporated into a diffusion model for generating home style images. By understanding text features or region-related features contained in the specified image, the generation effect of the diffusion model is optimized and the stability of generating home images of a specified style and / or scene is improved.

[0017] It should be appreciated that the above advantages do not need to be all realized in one or some specific embodiments, but may be partially dispersed in different embodiments according to the present disclosure. The embodiments according to the present disclosure may have one or some of the above advantages, or may have other advantages alternatively or additionally. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The foregoing and other features and advantages of the present disclosure will become apparent from the following description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are incorporated herein and form a part of the specification, and are further used to explain the principles of the present disclosure and enable those skilled in the art to make and use the present disclosure. Among them:

[0019] Figure 1 A flowchart of a method for generating a home style image based on a diffusion model according to an embodiment of the present disclosure is shown;

[0020] Figure 2A schematic diagram of a method for generating a home style image based on a diffusion model according to some embodiments of the present disclosure is shown;

[0021] Figure 3 A flowchart of a method for generating a home style image based on a diffusion model according to some embodiments of the present disclosure is shown;

[0022] Figure 4 A schematic diagram showing a method for generating a home style image based on a diffusion model according to other embodiments of the present disclosure is shown;

[0023] Figure 5 A flowchart of a method for generating a home style image based on a diffusion model according to other embodiments of the present disclosure is shown;

[0024] Figure 6 A schematic block diagram of a home style image generation device based on a diffusion model according to an embodiment of the present disclosure is shown;

[0025] Figure 7 An exemplary configuration of a computing device that can implement an embodiment according to the present disclosure is shown.

[0026] Note that in the embodiments described below, sometimes the same reference numerals are used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In some cases, similar numbers and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0027] For ease of understanding, the position, size, range, etc. of each structure shown in the drawings and the like may not represent the actual position, size, range, etc. Therefore, the present disclosure is not limited to the position, size, range, etc. disclosed in the drawings and the like. DETAILED DESCRIPTION

[0028] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0029] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0030] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0031] The present disclosure provides a method for generating a home style image, which specifically combines a diffusion model represented by SDM to generate a target home image that meets the style requirements based on an existing home image combined with feature information. The home style image generation method according to the present disclosure will be described in detail below in conjunction with the accompanying drawings. It should be understood that the actual method may also include other additional steps, but in order to avoid blurring the main points of the present disclosure, these other additional steps are not discussed herein and are not shown in the accompanying drawings.

[0032] Figure 1 FIG. 1 is a flow chart of a method 1000 for generating a home style image based on a diffusion model (hereinafter referred to as method 1000 ) according to an embodiment of the present disclosure. Figure 1 As shown, the method 1000 mainly includes:

[0033] In step S101, a first home image is acquired, and feature information is set based on style requirements for the first home image;

[0034] In step S102, the feature information is input as a control condition into a U-NET (U-type network) model of a stable diffusion model for optimization training, and one or more required home images are generated using the stable diffusion model;

[0035] In step S103, the one or more required home images and the first home image are subjected to image processing based on style requirements to obtain a target home image.

[0036] Specifically, in the field of home decoration, there is often a need to design the same home scene in different styles, including replacing the pattern of a specified area in the original image according to the style requirements, or extracting style features based on the original image to perform style migration in combination with text features associated with the style requirements input by the user. In a non-limiting embodiment, a home scene may include home objects such as sofas and curtains. The positional relationship of these objects compared to the spatial structure of the scene itself (such as doors, windows, walls, etc.), the style and material of each object, and the color matching in the scene constitute the home layout information. Accordingly, the home image can be partially replaced according to the style requirements, such as replacing the color or style of the sofa, replacing the material and shape of the curtains, etc., to generate a target home image after a specific partial style change. Alternatively, the style migration of the home image can be performed according to the style requirements, such as obtaining a color card from an existing home image, combining the text feature "a modern living room with a large window on the left and wooden floors" as a control condition, and generating a home image with the style of the color card and conforming to the text feature. As a non-limiting example, the following can be provided respectively. Figures 2 to 5 The home style image generation method shown.

[0037] Next reference Figure 2 , Figure 2 A schematic diagram of a method for generating home style images based on a diffusion model according to some embodiments of the present disclosure is shown. In general, partial image replacement is one of the main applications of image editing, and the purpose is to effectively fill in the existing SDM under the premise of maintaining semantic consistency, so that the generated part matches the original scene more closely and the overall image is more natural. Specifically, compared with the traditional image generation model, as a conditional input, each module (such as from X t To X t+1 The module 200 of the model 200 is integrated with an additional parallel branch 201 to structurally adjust the diffusion model. First, the weights of the neural network block of the model 20 are copied into a "frozen" part (Frozen Original U-net) and a "trainable" part (Trainable CopyExtra U-net), where the "trainable" copy inputs the masked image and mask features after the upper module processing (including downsampling and VAE encoding), and the additional parallel branch features and the main branch features of each layer of U-NET are element-wise added to achieve the fusion effect.

[0038] Figure 2, the process of generating the first home image of a home scene with a sofa is shown, wherein the black and white mask feature 210 of the first home image is firstly downsampled four times (step S2101), and the masked image feature 220 of the first home image is input into the VAE encoder for VAE encoding (step S2201). The four-fold downsampling is mainly used to keep consistent with the sampling multiple of the VAE encoder, that is, to scale the processed image to the same size, so that the two can be spliced ​​together and input into the control branch of U-NET. Unlike black and white images, the image information granularity of the masked image is richer, and the VAE encoder can better extract pixel-level features and is more suitable for U-NET distribution. It can be understood that by using the VAE encoder, image features can be extracted and adapted more effectively, which helps to more accurately capture the structural information of the image during the image restoration process, thereby improving the effect of conditional generation. A hierarchical approach is used to gradually integrate additional parallel branches into the pre-trained U-NET. This hierarchical fusion method can make the model more controllable in details and improve the accuracy and consistency of the generation process; at the same time, it ensures that the generated images have finer details at different scales, improving the quality of image restoration.

[0039] In a non-limiting embodiment, the size of the input image is 1024×1024 after scaling (resize), then the size of the masked image feature 220 after VAE encoding is 256×256×4; the size of the black and white mask 210 after downsampling is 256×256×1; and the additional noise features (256×256×4) of the two inputs are spliced ​​to obtain 256×256×(4+4+1), which are input to the additional parallel branch 201, and the multi-stage output features of the additional parallel branch 201 are obtained by encoding in turn, and fused into the U-NET model 20. Further, the generated result is partially decoded by the VAE decoder (step S2102) to generate one or more required home images 230 that meet the style requirements, wherein the sofa part is image generated according to the style requirements, and the parts other than the sofa part can maintain coherence and realism with the sofa part. Targeted screening or further processing can be performed from the generated one or more required home images 230 to obtain a target home image (not shown). Therefore, this solution can achieve excellent text-to-image filling effects in home decoration scenarios, bringing new possibilities to the field of image editing.

[0040] Please refer to Figure 3 , Figure 3A flowchart of a method 3000 for generating a home style image based on a diffusion model according to some embodiments of the present disclosure is shown. In step S301, a first home image is acquired, and in response to the style requirements for the first home image including partial image replacement, feature information is set to include masked image features, black and white masks, and additional noise. In step S302, the feature information is input as a control condition into a U-NET model of a stable diffusion model for optimization training, and the output of the stable diffusion model is partially decoded by a VAE decoder to generate one or more required home images. In step S303, the one or more required home images are subjected to image processing based on style requirements with the first home image to obtain a target home image.

[0041] Alternatively, Figure 4 The schematic diagram of the method for generating home style images based on the diffusion model according to other embodiments of the present disclosure is shown. Generally speaking, overall style control belongs to a type of style transfer. The traditional text image model can control the generation of images based on text conditions, but the current algorithm is limited to rough style transfer (such as anime style, abstractionism, etc.) and cannot achieve detailed style adjustment. For this reason, Figure 4 A process of controlling style generation based on user input is shown, wherein the user inputs a first home image 410, which can be understood as including a window element on the right side of the image, a lamp element in the middle and upper part, a sofa element on the left side and lower part, etc., or can be understood as including a specific color scheme.

[0042] In a non-limiting embodiment, one or more image processing algorithms are used to obtain corresponding layout information corresponding to the current home scene from the first home image 410, such as obtaining a semantic graph of the first home image 410 by performing semantic segmentation of the image through Coco Oneformer, obtaining a depth graph of the first home image 410 by performing depth estimation of the image through Depth Anything (V2), and obtaining a layout graph of the first home image 410 through Layout Oneformer. At least one of the semantic graph, depth graph, and / or layout graph obtained above is used as a basic component control condition to control the home layout information, that is, to control the generation of the room layout in the image. On this basis, additional image encoding information is extracted based on the home layout information (step S4201), and text encoding information is extracted based on the text features input by the user; further, the additional image encoding information and the text encoding information are calculated and fused to the intermediate layer of the U-NET model (such as from X) through the cross attention module 401. t To X t+1The cross-attention module 401 adjusts the features of the intermediate layer, including refining the overall style transfer from a global level by combining text (such as home style and furniture description, etc.) with input image information (such as target color cards and ideal renovation pictures, etc.), which can better understand the user's decoration style requirements and achieve more accurate style control.

[0043] In some embodiments, the additional image encoding information mainly comes from the first home image 410 input via the image encoder, and together with the original separate text features, affects the generation of the image. Specifically, the pre-trained U-NET is a text graph model. When training its branches in parallel, it is necessary to first use an image description (Caption) model (such as blipv2-caption) to summarize the style map to obtain an image description (i.e., caption text) as a text feature, such as a non-restrictive style map that can be described as "a living room with a large window on the left. The living room has wooden floors and white walls". On this basis, a hardcover template image or a color card image of a specified color scheme can be used as a conditional input of the parallel branch of the U-NET module, where the input image is encoded by the image encoder (step S4201) to obtain image conditional features, which are calculated and fused into different network layers (stages) of the original U-NET model through the cross-attention module 401 together with the original text features. Additionally or alternatively, the image features encoded by the image encoder can be further adjusted in dimension through the linear module of step S4202, and the layer normalization (LN) module of step S4203 to help the neural network converge faster and more stably, thereby helping to improve the efficiency of model training.

[0044] Additionally, the image features are integrated into the pre-trained U-NET model via the decoupled criss-cross attention module 401. In the traditional SDM, the text features from the CLIP text encoder are connected to the U-NET model by inputting into the criss-cross attention module 401. Given the query feature Z and the text feature c t , the output Z′ of the cross-attention module 401 can be defined by the following formula:

[0045]

[0046] Where Q = ZW q , K=c t W k 、V=c t W v are the query, key, and value matrices in the attention operation, respectively, and W q , W k , W vis the weight matrix of the trainable linear projection layer.

[0047] A simple method to insert image features into the model is to concatenate the image features and text features and then input them into the cross-attention layer. Considering that the effect of this method is not ideal, this disclosure mainly applies a decoupled cross-attention mechanism, in which the cross-attention layers of text features and image features are separated. Specifically, a new cross-attention layer is added to each cross-attention layer in the original U-NET model to insert image features. Given an image feature c i , the new cross-attention output Z′′ is calculated as follows:

[0048]

[0049] Where Q = ZW q , K′=c i W′ k , V′=c i W′ v is the query, key, and value matrix from the image features, W′ k and W′ v is the corresponding weight matrix. Note that the same query is used for image cross-attention as for text cross-attention. Therefore, only two parameters W′ need to be added for each cross-attention layer. k and W′ v In order to speed up the convergence, W′ k and W′ v From W k and W v Initialization. Further, the output of the image cross-attention is added to the output of the text cross-attention. Therefore, the final formula of the decoupled cross-attention is defined as follows:

[0050]

[0051] Where Q = ZW q , K=c t W k 、V=c t W v , K′=c i W′ k , V′=c i W′ v .

[0052] Therefore, the cross-attention module 401 realizes encoding the image as conditional input and the original text prompt features to control the diffusion model together, which helps the trained U-NET model to generate home scenes of a specified style more stably.

[0053] Please refer to the following Figure 5 , Figure 5 A flowchart of a method 5000 for generating a home style image based on a diffusion model according to some other embodiments of the present disclosure is shown. In step S501, a first home image is acquired, and in response to the style requirement for the first home image including home style migration, the feature information is set to a text feature associated with the style requirement input by the user. In step S502, additional image encoding information and text encoding information are extracted based on the feature information, and are input as control conditions into a U-NET model of a stable diffusion model for optimization training, and one or more required home images are generated using the stable diffusion model. In step S503, the one or more required home images are subjected to image processing based on the style requirement with the first home image to obtain a target home image.

[0054] It should be understood that the division of steps in the aforementioned flowcharts is only for illustration and not for limitation of the embodiments of the present application. These steps can be completed by each illustrated subject individually, two or more of them are completed together, or by an external system not shown in cooperation with the illustrated subject, and are not limited to the illustrated distribution method, and can be completed in sequence according to the illustrated order or in parallel in accordance with the data flow direction and system operation rules, and can be completed step by step or in combination.

[0055] The present disclosure also provides a home style image generation device 6000 based on a diffusion model, which may include a processor 6010 and a memory 6020 storing computer executable instructions, which, when executed by the processor 6010, causes the processor 6010 to perform the home style image generation method based on the diffusion model according to any of the aforementioned embodiments of the present disclosure. The processor 6010 may be a central processing unit (CPU) of an image generation device, which may be any type of general-purpose processor, or may be a processor specially designed for the home style image generation method based on the diffusion model, such as an application-specific integrated circuit ("ASIC"). The memory 6020 may include various computer-readable media accessible by the processor 6010.

[0056] The present disclosure also provides a non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, enables the processor to implement the method for generating a home style image based on a diffusion model according to any of the aforementioned embodiments of the present disclosure.

[0057] The present disclosure also provides a computer program product, comprising computer executable instructions, which, when executed by a processor, enable the processor to implement the method for generating a home style image based on a diffusion model according to any of the aforementioned embodiments of the present disclosure.

[0058] Figure 7An exemplary configuration of a computing device that can implement an embodiment of the present disclosure is shown. The computing device includes one or more processors 701, an input / output interface 705 connected to the processor 701 via a bus 704, and memories 702 and 703 connected to the bus 704. In some embodiments, the memory 702 may be a read-only memory (ROM), and the memory 703 may be a random access memory (RAM).

[0059] Processor 701 may be any kind of processor, and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). Memories 702 and 703 may be any non-transitory storage device that can implement data storage, and may include, but is not limited to, disk drives, optical storage devices, solid-state memories, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, compressed disks or any other optical media, cache memories and / or any other storage chips or modules, and / or any other media from which a computer can read data, instructions and / or codes.

[0060] The bus 704 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0061] In some embodiments, the input / output interface 705 is connected to an input unit 706 configured by an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 707 that outputs a processing operation screen and an image of a processing result to a display device, a storage unit 708 including a hard disk drive and the like for storing programs and various data, and a communication unit 709 including a local area network (LAN) adapter and the like and performing communication processing via a network represented by the Internet. In addition, a drive 710 that reads and writes data from and on a removable storage medium 711 is connected.

[0062] Various aspects, embodiments, implementations or features of the foregoing embodiments may be used alone or in any combination. Various aspects of the foregoing embodiments may be implemented by software, hardware or a combination of hardware and software.

[0063] For example, the foregoing embodiments may be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can thereafter be read by a computer system. Examples of computer readable media include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard drive, solid state drive, and optical data storage device. The computer readable medium may also be distributed in a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0064] For example, the aforementioned embodiments may be in the form of hardware circuits. The hardware circuits may include any combination of combinational logic circuits, clock storage devices (such as floppy disks, flip-flops, latches, etc.), finite state machines, memories such as static random access memories or embedded dynamic random access memories, custom designed circuits, programmable logic arrays, etc.

[0065] In one embodiment, hardware circuits according to the present disclosure may be implemented by coding and designing one or more integrated circuits using a hardware description language (HDL) such as Verilog or VHDL, or in combination using discrete circuits.

[0066] The terms used herein are only for describing specific exemplary embodiments and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular forms of "one" and "the" used herein are intended to include plural forms as well. It should also be understood that when the word "including" is used in this article, it indicates the existence of the indicated features, wholes, steps, operations, units and / or components, but does not exclude the existence or increase of one or more other features, wholes, steps, operations, units and / or components and / or their combinations. In addition, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or order. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0067] References to "embodiments" or similar expressions in this specification refer to specific features, structures, or characteristics described in conjunction with the embodiment as being included in at least one specific embodiment of the present disclosure. Therefore, the appearance of "in an embodiment of the present disclosure" and similar expressions in this specification does not necessarily refer to the same embodiment.

[0068] Those skilled in the art will appreciate that the present disclosure may be implemented in various forms, such as a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microprogram code, etc.), or a combination of software and hardware, which will be referred to below as a "circuit," "module," "unit," or "system." In addition, the present disclosure may also be implemented in any tangible media form as a computer program product having a computer-usable program code stored thereon.

[0069] The relevant description of the present disclosure is described with reference to the flowchart and / or block diagram of the system, device, method and computer program product according to the specific embodiment of the present disclosure. It can be understood that each block in each flowchart and / or block diagram, and any combination of blocks in the flowchart and / or block diagram, can be implemented using computer program instructions. These computer program instructions can be executed by a machine composed of a processor of a general-purpose computer or a special computer or other programmable data processing device, and the instructions are processed by a computer or other programmable data processing device to implement the functions or operations described in the flowchart and / or block diagram.

[0070] The accompanying drawings show the flow charts and block diagrams of the architecture, functions and operations that can be implemented by the systems, devices, methods and computer program products according to various embodiments of the present disclosure. Therefore, each block in the flow chart or block diagram can represent a module, segment, or part of the program code, which includes one or more executable instructions to implement the specified logical function. It should also be noted that in some other embodiments, the functions described in the blocks may not be performed in the order shown in the figure. For example, two blocks connected to each other in the figure can actually be executed simultaneously, or in some cases according to the functions involved, they can also be executed in the opposite order of the icons. It should also be noted that each block diagram and / or flow chart block, as well as the combination of blocks in the block diagram and / or flow chart, can be implemented by a system based on special hardware, or by a combination of special hardware and computer instructions to perform specific functions or operations.

[0071] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to market technologies, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a home style image based on a diffusion model, comprising: Acquire a first home image, and set feature information based on style requirements for the first home image; Input the feature information as a control condition into a U-NET model of a stable diffusion model for optimization training, and use the stable diffusion model to generate one or more required home images; as well as The target home image is obtained by performing image processing based on the style requirement on the one or more required home images.

2. The method for generating a home style image according to claim 1, wherein: In response to the style requirement, a partial image replacement is performed on the first home image, and the feature information includes masked image features, a black and white mask, and additional noise.

3. The method for generating a home style image according to claim 2, wherein the control conditions include: The masked image features that have been downsampled four times and the black and white mask that has passed the VAE encoder are scaled to the same size and then concatenated; inputting the spliced ​​results into the stable diffusion model as an additional parallel branch; and The parallel branches are encoded in sequence to obtain multi-stage output features which are used as the control conditions and integrated into the U-NET model.

4. The method for generating a home style image according to claim 3, wherein: In the U-NET model, the parallel branch and the main branch features are added element by element layer by layer to achieve fusion.

5. The method for generating a home style image according to claim 3, wherein: The output of the stable diffusion model is partially decoded by a VAE decoder to generate the one or more required home images.

6. The method for generating a home style image according to claim 1, wherein: In response to the style requirement including performing home style migration on the first home image, the feature information includes text features associated with the style requirement input by the user.

7. The method for generating a home style image according to claim 6, wherein the control condition comprises: generating at least one of a corresponding depth map and a semantic map based on the first home image; as well as At least one of the corresponding depth map and semantic map is used as a basic component to control home layout information.

8. The method for generating a home style image according to claim 7, wherein the control condition further comprises: extracting additional image encoding information based on the home layout information and extracting text encoding information based on the text features; as well as The additional image encoding information and the text encoding information are fused into the middle layer of the U-NET model through the cross-attention module calculation.

9. A method for generating a home style image based on a diffusion model, comprising: Acquire a first home image, and in response to style requirements for the first home image including partial image replacement, set feature information to include masked image features, black and white masks, and additional noise; Inputting the feature information as a control condition into a U-NET model of a stable diffusion model for optimization training, and partially decoding the output of the stable diffusion model through a VAE decoder to generate one or more required home images; and The target home image is obtained by performing image processing based on the style requirement on the one or more required home images.

10. A method for generating a home style image based on a diffusion model, comprising: Acquire a first home image, and in response to a style requirement for the first home image including home style migration, set feature information to a text feature associated with the style requirement input by a user; Extracting additional image encoding information and text encoding information based on feature information, inputting them into a U-NET model of a stable diffusion model as control conditions for optimization training, and using the stable diffusion model to generate one or more required home images; as well as The target home image is obtained by performing image processing based on the style requirement on the one or more required home images.

11. A home style image generation device based on a diffusion model, comprising: processor; as well as A memory storing computer executable instructions, wherein when the computer executable instructions are executed by the processor, the processor executes the method for generating a home style image based on a diffusion model according to any one of claims 1 to 10. 12 . A non-transitory computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor implements the method for generating a home style image based on a diffusion model according to claim 1 . 13 . A computer program product, comprising computer executable instructions, which, when executed by a processor, enable the processor to implement the method for generating a home style image based on a diffusion model according to claim 1 .

Images