Exemplar-guided Image Generation System and Method Using Bridge Diffusion Models

Abstract

The present invention relates to an image generation system and method based on an example image utilizing a bridge diffusion model, and more specifically, to a system for generating an image based on an example image, which is implemented by instructions executed by a processor and includes an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) that generates a result image from an example image and a structure control input input from a user terminal, wherein the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) includes an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal.

Description

Technology Field

[0001] The present invention relates to an image generation system and method based on an example image utilizing a bridge diffusion model, and more specifically, to a system for generating an image based on an example image, which is implemented by instructions executed by a processor and includes an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) that generates a result image from an example image and a structure control input input from a user terminal, wherein the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) includes an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal. Background Technology

[0002] AI-powered image synthesis and transformation technology is simultaneously changing image production processes for both the general public and the industry. Beyond simply generating images using text prompts, there is a growing demand for more precise control over the style and texture intended by the user.

[0003] Amidst this trend, Diffusion Model Based Image Synthesis (DMI) technology is gaining attention as a machine learning technique that generates high-quality images through an iterative denoising process. A Diffusion Model is a deep learning model that generates a result image by gradually removing noise from samples of random noise or original images that have been denoised. Compared to existing Generative Adversarial Networks (GANs), this approach has the advantages of capturing a wider data distribution, a stable training process, and excellent scalability. Based on these strengths, Text-to-Image (T2I) technology conditioned on text prompts has been extensively researched and has been proven to possess excellent performance in generating complex scenes. To meet the demands of users seeking more sophisticated control, various technologies have been derived, such as prompt engineering and fine-tuning (e.g., DreamBooth, LoRA), or adapter networks that add structural control (e.g., ControlNet, T2I-Adapter).

[0004] However, these technologies have limitations in that text prompts alone cannot perfectly convey all the visual details intended by the user, particularly fine textures or complex patterns. Furthermore, fine-tuning the model for a desired style is costly and is limited to specific domains. In addition, existing models that must simultaneously apply multiple conditions—such as structure, style, and text prompts—are highly sensitive to hyperparameters like guidance scales that control the intensity of each condition, inevitably leading to highly unstable training and inference processes.

[0005] In particular, in example-based transformations where the user provides example images, text-associative embeddings such as CLIP encoders often capture the overall meaning of the image well but fail to adequately reflect fine textures or unique visual features.

[0006] Furthermore, while image-to-image conversion technology can utilize structural control or reference images as conditions, it still fails to overcome the limitations of text-based representation and has the problem of not adequately reflecting the identity or detailed textures of the reference image.

[0007] In other words, conventional technology suffered from problems such as limitations in fine-grained control due to reliance on text prompts, high computational costs and loss of local details in matching-based frameworks, and instability that occurs when combining multiple conditions.

[0008] Therefore, a new model design is required that can effectively combine style and structure control of example images while simultaneously ensuring stability and computational efficiency in the learning and inference processes.

[0009] The technology forming the background of the present invention is disclosed in Korean Registered Patent No. 10-2490060. The problem to be solved

[0010] The present invention has been devised to solve the above-mentioned problems,

[0011] The objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model that generates an image based on an example image, which is implemented by instructions executed by a processor and generates a result image from a structure control input and an example image received from a user terminal, wherein the example-guided Brownian-Bridge Diffusion Model (EBDM) comprises an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal, thereby providing an example image-based image generation system that sufficiently reflects the overall style and fine texture of the example image while increasing structure preservation and computational efficiency.

[0012] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model, which can provide a system architecture capable of combining computational efficiency and fidelity to an example image, wherein the potential generation unit comprises a global encoder that extracts global style features from the example image, an example network that extracts local texture features from the example image, and a denoising network that sets the structure control input as the starting state of the diffusion process, sets the potential representation as the target state based on the example image, and performs deterministic inverse sampling based on an Ordinary Differential Equation (ODE) from the starting state to the target state.

[0013] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model that can precisely transplant local details into a resulting image while maintaining consistency in the overall atmosphere of the example image without distortion of the content, by including an example attention module in the denoising network that receives the global style feature through a cross attention module during the deterministic backsampling process and selectively fuses the local texture feature with the denoising feature of the denoising network based on attention.

[0014] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model, wherein the example attention module is configured to increase the attention weight of a specific area designated by the user to enhance texture reflection for that area, thereby enabling the system to actively respond to the user's local and detailed control requirements and provide customized results.

[0015] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model, wherein the example-based brown bridge diffusion model (EBDM) is trained by a two-stage training method in which a global style feature injection path is learned through the denoising network and global encoder in a first training stage, and a local texture feature fusion path is learned by combining the example network and the example attention module in a second training stage, thereby ensuring content consistency in global style first and then learning the transplantation of local textures stably and enabling optimized performance.

[0016] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model that can be applied to a wide range of applications by accommodating various forms of structural guides input by a user, wherein the structure control input includes at least one of a segmentation mask, an edge map, or a pose keypoint.

[0017] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model, wherein when a plurality of example images are acquired, the potential generation unit is configured to dynamically route and combine local texture features extracted from the plurality of example images based on patch similarity by image region, thereby allowing the system to combine the advantages of the plurality of example images to fuse the example texture most suitable for each region of the image, and thus respond to the complex and creative demands of the user with high satisfaction.

[0018] Another objective of the present invention is to provide an example image-based image generation system utilizing a bridge diffusion model that can optimize the efficiency of system resource usage, wherein the system comprises a plurality of mutually communicating computation nodes, the processor of the first node is configured to perform low-latency preview computation, and the processor of the second node is configured to perform high-resolution final generation and / or model training, such that some inference is performed at the terminal, and the generation or training of the high-resolution final image is performed at the server.

[0020] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model, which sufficiently reflects the overall style and fine texture of the example image while increasing structure preservation and computational efficiency, by comprising the steps of: obtaining a structure control input and an example image from a user in an image generation method implemented by instructions executed by a processor; generating a latent representation from the structure control input and the example image based on a Brownian-Bridge Diffusion Model (BBDM); and decoding the generated latent representation to provide a result image.

[0021] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model, wherein the generation of the latent representation is performed through an example-guided Brownian-Bridge Diffusion Model (EBDM), and the example-guided Brownian-Bridge Diffusion Model (EBDM) includes a global encoder that encodes global style features, an example network that encodes local texture features, and an example attention module that fuses the features into a denoising process, thereby enabling the generation of a high-quality image in which the overall style and detailed texture of the example are precisely processed without relying on text embeddings.

[0022] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model that enables control to a style desired by a user, wherein the step of generating the latent representation further includes the step of extracting global style features from the example image through the global encoder and the step of extracting local texture features from the example image through the example network, thereby separating and capturing style information inherent in the example image into global style from a macroscopic perspective and local texture features from a microscopic perspective.

[0023] Another objective of the present invention is to provide an example image-based image generation method and system utilizing a bridge diffusion model that can solve the problem of learning instability caused by multiple conditions and maintain structural consistency, by including the step of generating the latent representation based on a Brownian-Bridge diffusion model, the step of setting the structure control input as the starting state of the diffusion process and setting the target state, which is the latent representation of the final result image, using the example image as a condition.

[0024] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model, wherein the step of generating the latent expression generates the latent expression using the global style feature and the local texture feature through a denoising network, and the latent expression is generated by deterministic inverse sampling based on an Ordinary Differential Equation (ODE) moving from a set starting state to a target state according to the condition, thereby ensuring the stability of the inference process, having high reproducibility of the generated image, and improving the conversion speed by generating a high-quality image with a small number of steps.

[0025] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model that can precisely transplant local details into a resulting image while maintaining consistency in the overall atmosphere of the example image without distortion of the content, by including the step of injecting the global style features into a denoising network through a cross-attention module and the step of selectively fusing the local texture features with the denoising features of the denoising network through an example attention module based on attention.

[0026] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model, wherein the example attention module can increase the attention weight of a specific area designated by the user to enhance texture reflection for that area, thereby locally reflecting detailed intentions such as texture reflection of the area of ​​interest intended by the user, and thus increasing satisfaction with the result.

[0027] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model that can increase the stability of learning and systematically learn global styles and local details to maximize the performance of the model by training the model using a two-stage learning method comprising: a first learning stage in which a global style feature injection path is learned through the denoising network and a global encoder; and a second learning stage in which, after the first learning stage, a local texture feature fusion path is learned by combining the example network and an example attention module.

[0028] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model that can be applied to a wide range of applications by accommodating various forms of structural guides input by a user, wherein the structure control input includes at least one of a segmentation mask, an edge map, or a pose keypoint.

[0029] Another objective of the present invention is to provide an example image-based image generation method utilizing a bridge diffusion model capable of fusing the most suitable example texture for each region of an image by combining the advantages of multiple example images, by further including the step of dynamically routing and combining local texture features extracted from multiple example images based on patch similarity by image region when multiple example images are obtained.

[0030] Another objective of the present invention is to provide a non-transient computer-readable recording medium that records instructions for a processor to perform operations when executed by a processor, wherein the instructions are configured such that the processor obtains a structure control input and an example image from a user, extracts global style features and local texture features from the example image through an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM), sets the structure control input to a starting state for the diffusion process, generates a potential representation by performing deterministic inverse sampling based on an Ordinary Differential Equation (ODE) toward a target state which is a potential representation of a final result image using the example image as a condition, and decodes the generated potential representation to display the result image through a display, thereby enabling the example image-based image generation method utilizing the bridge diffusion model of the present invention to be easily distributed, installed, and executed on various devices.

[0031] However, the technical problems that the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0032] According to one embodiment of the present invention, a system for generating an image based on an example image is implemented by instructions executed by a processor and includes an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) that generates a result image from a structure control input and an example image received from a user terminal. The Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) may include an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal.

[0033] According to another embodiment of the present invention, the potential generation unit may include a global encoder that extracts global style features from the example image, an example network that extracts local texture features from the example image, and a denoising network that sets the structure control input to a starting state of the diffusion process, sets the potential representation to a target state based on the example image, and performs deterministic reverse sampling based on an Ordinary Differential Equation (ODE) from the starting state to the target state.

[0034] According to another embodiment of the present invention, the denoising network may include an example attention module that receives the global style feature through a cross attention module during the deterministic backsampling process and selectively fuses the local texture feature with the denoising feature of the denoising network based on attention.

[0035] According to another embodiment of the present invention, the example attention module may be configured to increase the attention weight of a specific area designated by a user to enhance texture reflection for that area.

[0036] According to another embodiment of the present invention, the example-based brown bridge diffusion model (EBDM) may be configured to be trained by a two-step training method in which a global style feature injection path is learned through the denoising network and the global encoder by a first training step, and a local texture feature fusion path is learned by combining the example network and the example attention module by a second training step.

[0037] According to another embodiment of the present invention, when a plurality of example images are obtained, the potential generation unit may be configured to dynamically route and combine local texture features extracted from the plurality of example images based on patch similarity by image region.

[0038] According to another embodiment of the present invention, the system may include a plurality of mutually communicating computation nodes, and the processor of the first node may be configured to perform a low-latency preview operation, and the processor of the second node may be configured to perform high-resolution final generation and / or model training.

[0039] According to one embodiment of the present invention, an image generation method implemented by instructions executed by a processor may include the steps of obtaining a structure control input and an example image from a user, generating a latent representation from the structure control input and the example image based on a Brownian-Bridge Diffusion Model (BBDM), and decoding the generated latent representation to provide a result image.

[0040] According to another embodiment of the present invention, the generation of the latent representation is performed through an exemplar-guided Brownian-Bridge Diffusion Model (EBDM), and the exemplar-guided Brownian-Bridge Diffusion Model (EBDM) may include a global encoder that encodes global style features, an exemplar network that encodes local texture features, and an exemplar attention module that fuses the features into a denoising process.

[0041] According to another embodiment of the present invention, the step of generating the latent representation may further include the step of extracting global style features from the example image through the global encoder and the step of extracting local texture features from the example image through the example network.

[0042] According to another embodiment of the present invention, the step of generating the latent representation may include, based on a Brownian-Bridge diffusion model, setting the structure control input as the starting state of the diffusion process and setting the target state, which is the latent representation of the final result image, as a condition of the example image.

[0043] According to another embodiment of the present invention, the step of generating the latent expression comprises generating the latent expression using the global style feature and the local texture feature through a denoising network, and the latent expression may be generated by deterministic inverse sampling based on an Ordinary Differential Equation (ODE) moving from a set starting state to a target state according to the condition.

[0044] According to another embodiment of the present invention, the deterministic backsampling may include the step of injecting the global style feature into a denoising network through a cross-attention module, and the step of selectively fusing the local texture feature with the denoising feature of the denoising network based on attention through an example attention module.

[0045] According to another embodiment of the present invention, the example attention module of the present invention can increase the attention weight of a specific area designated by a user, thereby enabling the enhancement of texture reflection for that area.

[0046] According to another embodiment of the present invention, the present invention can train the model by a two-step training method comprising: a first training step of training a global style feature injection path through the denoising network and a global encoder; and a second training step of training a local texture feature fusion path by combining the example network and an example attention module after the first training step.

[0047] According to another embodiment of the present invention, the structure control input may include at least one of a segmentation mask, an edge map, or a pose keypoint.

[0048] According to another embodiment of the present invention, when a plurality of example images are obtained, the present invention may further include the step of dynamically routing and combining local texture features extracted from a plurality of example images based on patch similarity by image region.

[0049] According to another embodiment of the present invention, the present invention comprises a non-transient computer-readable recording medium having recorded instructions that cause the processor to perform an operation when executed by the processor, wherein the instructions may include a non-transient computer-readable recording medium configured to cause the processor to perform an example image-based image generation method utilizing the bridge diffusion model.

[0050] The above-described means for solving the problem are merely exemplary and should not be interpreted as intended to limit the invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention. Effects of the invention

[0051] The present invention can achieve the following effects through the combination and usage relationship of the embodiments described above and the configuration described below.

[0052] The present invention is a system for generating an image based on an example image, which is implemented by instructions executed by a processor and includes an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) that generates a result image from a structure control input and an example image received from a user terminal. The Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) includes an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal. This has the effect of sufficiently reflecting the overall style and fine texture of the example image while increasing structure preservation and computational efficiency.

[0053] The present invention has the effect of providing a system architecture that combines computational efficiency and fidelity to an example image by including, in the latent generation unit a global encoder that extracts global style features from the example image, an example network that extracts local texture features from the example image, and a denoising network that sets the structure control input as the starting state of the diffusion process, sets the latent representation as the target state based on the example image, and performs deterministic inverse sampling based on an ODE (Ordinary Differential Equation) from the starting state to the target state.

[0054] The present invention has the effect of precisely implanting local details into a resulting image while maintaining consistency in the overall atmosphere of the example image without distortion of the content, by including an example attention module in the denoising network that receives the global style feature through a cross attention module during the deterministic reverse sampling process and selectively fuses the local texture feature with the denoising feature of the denoising network based on attention.

[0055] The present invention has the effect of enabling the system to actively respond to the user's local and detailed control requirements and provide customized results, as the example attention module is configured to increase the attention weight of a specific area designated by the user to enhance texture reflection for that area.

[0056] The present invention has the effect of enabling stable learning and optimized performance by first securing content consistency in global styles and then learning the transplantation of local textures, wherein the example-based brown bridge diffusion model (EBDM) is trained by a two-stage learning method in which a global style feature injection path is learned through the denoising network and global encoder in the first learning stage, and a local texture feature fusion path is learned by combining the example network and the example attention module in the second learning stage.

[0057] The present invention has the effect of being applicable to a wide range of applications by accommodating various types of structural guides input by a user, by including at least one of a segmentation mask, an edge map, or a pose keypoint in the structure control input.

[0058] The present invention has the effect of enabling the system to respond to the complex and creative demands of users with high satisfaction by combining the advantages of multiple example images and fusing the most suitable example texture for each area of ​​the image, wherein, when the example attention module is acquired in multiple example images, the potential generation unit is configured to dynamically route and combine local texture features extracted from the multiple example images based on patch similarity by image region.

[0059] The present invention has the effect of optimizing the efficiency of system resource usage, such that some inference is performed at the terminal, and the generation of high-resolution final images or learning is performed on the server, by configuring the system to include a plurality of mutually communicating computation nodes, wherein the processor of the first node performs low-latency preview computation and the processor of the second node performs high-resolution final generation and / or model learning.

[0061] The present invention relates to an image generation method implemented by instructions executed by a processor, comprising the steps of obtaining a structure control input and an example image from a user, generating a latent representation from the structure control input and the example image based on a Brownian-Bridge Diffusion Model (BBDM), and decoding the generated latent representation to provide a result image, thereby having the effect of preserving structure and increasing computational efficiency while sufficiently reflecting the overall style and fine texture of the example image.

[0062] The present invention has the effect of generating a high-quality image in which the overall style and detailed texture of the example are precisely processed without relying on text embeddings, by performing the generation of the latent representation through an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM), wherein the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) includes a global encoder that encodes global style features, an example network that encodes local texture features, and an example attention module that fuses the features into a denoising process.

[0063] The present invention has the effect of enabling control over the style desired by the user by separating and capturing style information inherent in the example image into a global style from a macroscopic perspective and local texture features from a microscopic perspective, wherein the step of generating the latent expression further includes the step of extracting global style features from the example image through the global encoder and the step of extracting local texture features from the example image through the example network.

[0064] The present invention has the effect of resolving the problem of learning instability caused by multiple conditions and maintaining structural consistency by including the step of generating the latent representation, based on a Brownian-Bridge diffusion model, setting the structure control input as the starting state of the diffusion process and setting the target state, which is the latent representation of the final result image, as a condition.

[0065] The present invention provides that the step of generating the latent expression generates the latent expression using the global style feature and the local texture feature through a denoising network, and the latent expression is generated by deterministic inverse sampling based on an Ordinary Differential Equation (ODE) moving from a set starting state to a target state according to the condition, thereby ensuring the stability of the inference process, having high reproducibility of the generated image, and having the effect of improving the conversion speed by generating a high-quality image with a small number of steps.

[0066] The present invention has the effect of precisely transplanting local details into a resulting image while maintaining consistency in the overall atmosphere of the example image without distortion of the content, by including the step of injecting the global style feature into a denoising network through a cross-attention module and the step of selectively fusing the local texture feature with the denoising feature of the denoising network through an example attention module based on attention.

[0067] The present invention has the effect of increasing satisfaction with the result by locally reflecting detailed intentions, such as the texture reflection of the area of ​​interest intended by the user, by increasing the attention weight of the example attention module of the user-specified specific area to enhance texture reflection for that area.

[0068] The present invention has the effect of maximizing the performance of a model by increasing the stability of learning and systematically learning global styles and local details by training the model using a two-stage learning method comprising: a first learning stage in which a global style feature injection path is learned through the denoising network and a global encoder; and a second learning stage in which, after the first learning stage, a local texture feature fusion path is learned by combining the example network and an example attention module.

[0069] The present invention has the effect of being applicable to a wide range of applications by accommodating various types of structural guides input by a user, wherein the structure control input includes at least one of a segmentation mask, an edge map, or a pose keypoint.

[0070] The present invention further includes the step of dynamically routing and combining local texture features extracted from a plurality of example images based on patch similarity by image region when a plurality of example images are obtained, thereby having the effect of fusing the most suitable example texture for each region of an image by combining the advantages of a plurality of example images.

[0071] The present invention provides a non-transient computer-readable recording medium that records instructions for causing a processor to perform an operation when executed by a processor, wherein the instructions are configured to cause the processor to perform an example image-based image generation method utilizing the bridge diffusion model, thereby providing a non-transient computer-readable recording medium that enables the example image-based image generation method utilizing the bridge diffusion model of the present invention to be easily distributed, installed, and executed on various devices.

[0072] However, the effects obtainable from the present invention are not limited to those described above, and other effects may exist. Brief explanation of the drawing

[0073] FIG. 1 is a conceptual diagram of an example image-based image generation system utilizing a bridge diffusion model according to an embodiment of the present invention. FIG. 2 is a block diagram of a user terminal according to an embodiment of the present invention. FIG. 3 is a block diagram of an example-based brown bridge diffusion model according to an embodiment of the present invention. FIG. 4 is a block flowchart illustrating an example-based brown bridge diffusion model according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating an image generation method according to an embodiment of the present invention. FIG. 6 is a diagram comparing an example-based brown bridge diffusion model according to one embodiment of the present invention with an image generated by another diffusion model. FIG. 7 is a drawing showing the image generation result based on an example-based brown bridge diffusion model according to an embodiment of the present invention. Specific details for implementing the invention

[0074] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0075] Throughout the specification of the present invention, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0076] Throughout the entire specification of the present invention, when a component is described as being "connected" or "joined" with another component, this includes not only cases where the component is directly connected or joined to the other component, but also cases where it may be connected or joined through a new component.

[0077] Throughout this specification, singular expressions may include the meaning of the plural form unless otherwise stated, and expressions such as "first," "second," etc., are used to distinguish multiple components from one another and do not limit the order or importance of said components.

[0078] Throughout the entire specification of the invention, terms such as “~part” refer to a unit that processes at least one function or operation. The “~part” may refer to software or hardware components such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits), but is not limited to hardware and software. The “part” may be configured to reside in an addressable storage medium or may be configured to run one or more processors. Accordingly, as an example, the “part” may include components such as software components, object-oriented software components, class components, and task components, as well as processors, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and the “part” may be combined into a smaller number of components and “parts” or further separated into additional components and “parts.”

[0079] As used throughout this specification, "command" or "instruction" refers to a unit of information that causes a processor or computing system to perform a specific operation or a series of steps, and refers to all types of code and expressions defining the execution thereof that are executed by a processor to implement the functions, steps, operations, etc. of the present invention. Such "instruction" is not limited to a single instruction and may refer to a set of logically combined functions, procedures, subroutines, libraries, or part or all of program code constituting an application to perform a specific function. "Instruction" may be stored on a non-transient computer-readable recording medium (e.g., RAM, flash memory, SSD / HDD, optical recording medium, magnetic recording medium, embedded memory, etc.) and may be transmitted via a network for transmission and distribution.

[0080] Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the attached drawings.

[0082] The present invention relates to an image generation system and method based on example images utilizing a bridge diffusion model, and more specifically, to an image generation method and system utilizing an exemplar-guided Brownian-Bridge Diffusion Model (EBDM). The EBDM model of the present invention can solve the problem of learning instability inherent in existing multi-condition models and stably guarantee the structural consistency of the generated image by introducing a Brownian-Bridge Diffusion model that sets a Structural Control input as a fixed starting point of the diffusion process. Furthermore, the EBDM model extracts global style and local texture features from example images and selectively fuses these features into a denoising network, thereby efficiently performing high-quality image transformation that faithfully reflects both the macroscopic atmosphere and local details of the example images while reducing dependence on text prompts.

[0084] FIG. 1 is a conceptual diagram of an example image-based image generation system (1) utilizing a bridge diffusion model according to one embodiment of the present invention.

[0085] Referring to FIG. 1, the example image-based image generation system (1) utilizing the bridge diffusion model of the present invention can provide an example image-based image conversion service using an example-guided Brownian-Bridge Diffusion Model (EBDM).

[0086] The example image-based image generation system (1) utilizing the bridge diffusion model of the present invention is implemented by instructions executed by a processor and may include an example-guided Brownian-Bridge Diffusion Model (EBDM) (1000) that generates a result image from an example image and a structure control input input from a user terminal.

[0087] At this time, the processor is included in the user terminal (100) and / or server (200) and executes instructions stored in memory. The user terminal (100) and / or server (200) may be connected via the network (300).

[0088] At this time, the network (300) may refer to a connection structure capable of exchanging information between nodes such as a user terminal (100) and / or a server (200). The network (300) may include, but is not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a satellite broadcasting network and / or an analog broadcasting network.

[0089] Hereinafter, an example image-based image generation system (1) utilizing the bridge diffusion model of the present invention will be described in detail with reference to the attached drawings.

[0091] - User terminal (100)

[0092] A user terminal (100) according to one embodiment of the present invention may be understood as a wired / wireless communication-based device having an image generation application installed that can provide an image generation service based on an example image utilizing a bridge diffusion model. At this time, the user terminal (100) may be at least one of a smartphone, a tablet computer, a PC (Personal Computer), a mobile phone, a PDA (Personal Digital Assistant), an audio player, and a wearable device, and may further include a predetermined server computing device that can provide an image generation service environment based on an example image utilizing a bridge diffusion model.

[0094] FIG. 2 is a block diagram of the user terminal (100) according to one embodiment of the present invention.

[0095] Referring to FIG. 2, the user terminal (100) may include a memory (110), a processor (120), a communication unit (130), an interface unit (140), and a display unit (150).

[0096] The memory (110) above is configured to store an operating system for the use of the user terminal (100) or data generated according to the use of the user terminal (100), and can be understood as a configuration in which various applications, data, and commands (hereinafter referred to as 'applications') are stored to provide an example image-based image generation service utilizing the bridge diffusion model of the present invention.

[0097] For example, the memory (110) may include at least one non-transient computer-readable storage medium and a transient computer-readable storage medium, and may be various storage devices such as a flash drive, ROM, hard drive, solid-state drive (SSD), etc., and may include web storage that performs functions on the Internet.

[0098] The processor (120) can be understood as a configuration that executes commands stored in the memory (110) to control at least one component connected to the processor (120) and performs various operations such as computation, processing, data generation, and processing to provide an example image-based image generation method utilizing the bridge diffusion model of the present invention. The processor (120) may refer to a set of at least one or more processors capable of executing commands of an application stored in the memory (110).

[0099] For example, the processor (120) may be implemented by including at least one of ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, and at least some of the components inside and outside the user terminal (100) may be connected to each other via a bus, GPIO (General Purpose Input / Output), SPI (Serial Peripheral Interface), or MIPI (Mobile Industry Processor Interface), etc., so as to be able to exchange data and / or signals.

[0100] The above communication unit (130) is configured for the user terminal (100) to communicate with the server (200) and / or external device described later, and can be understood as a configuration that enables communication between devices to implement an example-based image generation service environment utilizing the bridge diffusion model of the present invention.

[0101] For example, the communication unit (130) can communicate with various user input components, such as a controller that receives input from a user, and can be configured to transmit and receive various data generated in relation to an image generation service utilizing a bridge diffusion model to other terminals and / or servers (200).

[0102] In addition, the communication unit (130) may be configured to transmit and receive data to and from other devices or servers via wired / wireless communication methods. For example, the communication unit (130) may be capable of performing wireless communication according to methods such as eMBB (enhanced Mobile Broadband), URLLC (Ultra Reliable Low-Latency Communications), MMTC (Massive Machine Type Communications), LTE (Long-Term Evolution), LTE-A (LTE Advance), NR (New Radio), UMTS (Universal Mobile Telecommunications System), GSM (Global System for Mobile communications), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), WiBro (Wireless Broadband), WiFi (Wireless Fidelity), Bluetooth, NFC (Near Field Communication), GPS (Global Positioning System), or GNSS (Global Navigation Satellite System), and may be configured to perform wired communication according to methods such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), RS-232 (Recommended Standard-232), or POTS (Plain Old Telephone Service).

[0103] The interface unit (140) can be understood as a configuration for connecting the user terminal (100) to communicate with one or more other devices. Specifically, the interface unit (140) may include a wired / wireless communication device compatible with one or more different communication protocols, and through this interface unit (140), the user terminal (100) may be connected to various input / output devices.

[0104] For example, the interface unit (140) may be configured to be connected to an input device such as a keyboard and / or mouse to obtain input from a user, and connected to an audio interface such as a speaker or headset to output audio.

[0105] To this end, the interface unit (140) may be configured to include at least one of a wired / wireless audio output port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0106] The above display unit (150) can be understood as a display device for displaying various user interfaces, images, and / or text, etc., for providing an image generation service utilizing the bridge diffusion model of the present invention. Meanwhile, the above display unit (150) may be implemented as a touchscreen including a touch sensor that detects user touch input in addition to the display device for images and / or text.

[0107] Meanwhile, the above user terminal (100) can perform deep learning related to an image generation service utilizing the bridge diffusion model of the present invention based on a predetermined deep-learning neural network.

[0108] For example, the deep learning neural networks include Custom Diffusion Deep-learning Neural Networks, Stable Diffusion Deep-learning Neural Networks, CLIP (Contrastive Language-Image Pre-training Encoder), VGG-19, GPT-3, GPT-3.5, EfficientNet, ResNet, ARIMA (Autoregressive Integrated Moving Average), VAR (Vector Auto Regression), and RNN (Recurrent Neural Networks). Networks), LSTM (Long Short-Term Memory models), GRU (Gated Recurrent Unit), GAN (Generative Adversarial Networks), DualStyleGAN, StyleGAN, Graph Convolution Network (GCN), CNN (Convolution Neural Network, CNN), DPSNet (Deep Plane Sweep Network, DPSNet), AGN (Attention Guided Network, AGN), R-CNN (Regions with CNN features), Fast May include R-CNN, Faster RCNN, Mask R-CNN and / or U-Net network, etc. There are, and are not limited to, those used in the industry.

[0109] Through such a configuration, the user terminal (100) can perform a series of processes to provide an image generation service utilizing the bridge diffusion model of the present invention, and, depending on the embodiment, may also perform at least some of the functional operations performed by the server (200) described later.

[0111] - Server(200)

[0112] A server (200) according to one embodiment of the present invention can be understood as being configured to provide an environment in which an application for providing an example image-based image generation service utilizing the bridge diffusion model of the present invention can operate on an external device by exchanging data with the external device for providing an example image-based image generation service utilizing the bridge diffusion model of the present invention on an external device such as the user terminal (100).

[0113] To this end, the server (200) may include an application program, data and / or commands, etc., for the application to operate, and may transmit and receive various data based thereon with an external device. For example, the server (200) may receive user input data input from the user terminal (100) and perform a series of processes to provide an image generation service utilizing the bridge diffusion model of the present invention.

[0114] To this end, the server (200) may include a memory in which data acquired, processed, or used by the server is stored, and a processor that executes commands stored in the memory to control various components mounted on the server (200). The description of the memory and processor included in the server is the same as that in the case of the user terminal (100) described above.

[0115] Meanwhile, the above server (200) may additionally include a communication unit that enables communication between internal and / or external devices of the server (200), a communication unit that handles secure connections and tokens, etc., and the components of the server (200) of the present invention are not limited to the above-mentioned contents.

[0116] Through such a configuration, the server (200) can perform a series of processes to provide an image generation service utilizing the bridge diffusion model of the present invention, and according to one embodiment of the present invention to be described later, it may also perform at least a part of the functional operation performed at the user terminal (100).

[0117] Meanwhile, an example image-based image generation system utilizing a bridge diffusion model according to one embodiment of the present invention may be implemented as a distributed system comprising a plurality of mutually communicating computation nodes. Specifically, the processor of a first node constituting the system may be configured to perform low-latency preview operations, and the processor of a second node may be configured to perform high-resolution final generation and / or model training.

[0118] For example, the processor of the first node, the user terminal (100), can perform low-latency preview operations, and the processor of the second node, the server (200), can perform high-resolution final generation and model training. When a user edits an image on the user terminal (100), the processor (120) of the user terminal or the server (200) can quickly generate a result image in a small number of steps and provide it as a 'preview', and when the user requests the generation of a final image, the server (200) can generate and deliver a high-resolution result in a full number of steps. This configuration can simultaneously optimize user experience and resource usage efficiency.

[0120] - Exemplar-guided Brownian-Bridge Diffusion Model (EBDM)(1000)

[0121] The present invention relates to a technique for generating a new image by applying the style of an exemplar image to a structure control input, and in particular, utilizes an exemplar-based brown bridge diffusion model (EBDM).

[0122] The example-based Brownian-Bridge Diffusion Model (EBDM) of the present invention is an extension of the Brownian-Bridge Diffusion Model (BBDM) specialized for example image-based image-to-image transformation. While the BBDM provides a general principle for learning transformations between two images, the EBDM of the present invention can provide a unique style encoding and fusion architecture to faithfully and precisely reflect the style of the example image within this framework as a structure control input.

[0124] Before explaining this, in a Brown Bridge Diffusion Model (BBDM), the starting state and the target state learn a probabilistic process specified at both ends of time. In the forward process, the starting sample and the target sample are progressively mixed with noise over time, and in the backward process, they are restored to obtain an output that meets the target conditions. That is, unlike general diffusion models such as DDPM that restore images from random noise, BBDM directly learns and infers a diffusion path with the two states from input to output as boundary conditions at both ends. Therefore, compared to conventional diffusion models, it is advantageous to simultaneously achieve structure preservation and target fidelity in image transformation.

[0126] FIG. 3 is a block diagram showing the internal configuration of an example-based brown bridge diffusion model (EBDM) according to one embodiment of the present invention.

[0127] Referring to FIG. 3, the example-based brown bridge diffusion model (EBDM) (1000) may include an image acquisition unit (1100), a potential generation unit (1200), and a decoding unit (1300).

[0128] The image acquisition unit (1100) can be understood as being configured to acquire a structure control input and an example image input from the interface unit (140) of the user terminal (100) in accordance with the transmission of the communication unit (130).

[0129] The above-mentioned structural control input can be understood as any kind of visual guide that determines the framework, such as the shape, contour, and composition of the image to be generated, and may include a segmentation mask, an edge map, a pose keypoint, a label map, an inpainting mask, an image of another object, etc., and preferably may include at least one of a segmentation mask, an edge map, or a pose keypoint. The above-mentioned structural control input is not limited to the examples described above.

[0130] The potential generation unit (1200) can be understood as a part corresponding to a core pipeline that oversees the image generation of the present invention, and may include a Global Encoder (1210), an Exemplar Network (1230), and an Exemplar Attention Module (1250).

[0132] The above global encoder (1210) is an example image (Exemplar I y It can be understood as a composition that captures macroscopic style information, such as the overall color tone and atmosphere, and extracts it as a global style feature.

[0133] The above global style features can be understood as color distribution, lighting, contrast, the mood of the background / subject, and the average tendency of texture, and the generated global style features are injected into each stage of the denoising network through the cross-attention path of the denoising network to effectively guide the overall style of the image to be generated.

[0134] Specifically, the global encoder (1210) receives an example image as input, extracts [CLS]-based style tokens, and encodes them into global style features through a linear layer. The global encoder is trained using self-supervised learning, so it can capture a richer and wider range of visual semantic features than conventional encoders such as CLIP, which focuses on text-image alignment.

[0135] For example, the global encoder (1210) may include self-supervised learning-based models such as DINOv2, MoCo, MAE, CLIP, or supervised learning-based models such as ResNET, EfficientNet, and ViT, and may also include all vision models that can be easily selected and used by a person skilled in the art, and the scope of the invention is not limited by the above examples.

[0136] The above example network (1230) can be understood as a configuration that captures microscopic and detailed texture information, such as patterns or textures of clothing, which are difficult for the global encoder (1210) to capture, and extracts it as a local texture feature (Detailed Texture). While the global encoder (1210) processes the image at low resolution and may miss fine details, the above example network (1230) maintains high-resolution features, thereby improving the quality of the final result.

[0138] Referring to FIG. 4, which will be described later, the example attention module (1250) is one of the key technical elements of the present invention, and the input structure control input (Control I x ) and example image (Exemplar I yIt can solve the problem of spatially unaligned inputs. In conventional models, simply concatenating or adding features could cause image distortion or blurring when the inputs were not aligned.

[0139] The exemplary attention module (1250) of the present invention is a local texture feature (F) extracted by the exemplary network (1230). l 1) intelligently integrates into the denoising process. The mode of operation is as follows: at each scale, denoising features (F l 2) and example features (F l 1) is combined along the spatial dimension, and self-attention is applied to the combined feature map to calculate which part of the texture of the example image each location (Query) of the image currently being generated should pay attention to. That is, if there is a region of interest provided by the user or the system, the attention module (1250) operates to increase the attention weight for that region or attenuate the non-interest region, thereby reducing collisions and turbidity. After attention is applied, only the feature part corresponding to the denoising path is separated again and passed to the next layer, so that the fine texture of the example image can be selectively reflected without spatial constraints.

[0141] The decoding unit (1300) can be understood as having a configuration that decodes the generated potential representation, which is the final target state, restores it to an actual image, and transmits the generated final image to a user terminal (100). Such decoding can be performed by a VAE decoder, and the generated image can be provided to the user through a separate display means connected to the display unit (150) or interface unit (140) of the user terminal (100).

[0143] The example-based brown bridge diffusion model (EBDM) of the present invention adopts the framework of the aforementioned brown bridge diffusion model (BBDM), but introduces the following configuration and method to effectively inject the style of the example image.

[0144] FIG. 4 is a block flowchart for explaining an example-based brown bridge diffusion model according to an embodiment of the present invention, and the example-based brown bridge diffusion model of the present invention will be explained below with reference to FIG. 4.

[0145] Referring to FIG. 4, the EBDM of the present invention can be understood as a diffusion pipeline operating in latent space and is characterized by including an exemplar network, a global encoder, and an exemplar attention module.

[0146] Specifically, a structure control input (Control I) for controlling the structure of an image x ) and an example image (Exemplar I) for referencing the image style y ) is input into the system. Subsequently, the input structure control input and example image input are converted into latent representations through a pre-trained VAE encoder. At this time, the style information contained in the example image is precisely extracted into global style features and local texture features through two independent paths.

[0147] First, the example image above is encoded through a global encoder into a global style feature, which is a macroscopic style feature such as the overall color tone or mood of the image. The global style feature is injected into the cross-attention module (CA) in the diffusion path of Fig. 4 and plays a role in determining the overall tone or mood of the final image to be generated.

[0148] At the same time, the latent representation of the above example image is transmitted to the above example network to local texture features (Exemplar Feature F), which are microscopic and detailed features such as the pattern or texture of the clothing. l It is encoded as 1).

[0149] The core process of image generation in the EBDM of the present invention takes place in the Brown-bridge diffusion core (Latent Brownian Bridge Diffusion Model of Fig. 4). This model is a structural control input (I x The latent representation of ) is frozen as the starting point of the diffusion process, and an iterative noise removal process is performed through T steps. In this iterative process, the Exemplar-attention (EA) module receives local texture features (F) from the exemplar network. l 1) features of the image currently being generated (F l 2) In combination with the above structure control input (Control I x While maintaining the structure of ), the example image (Exemplar I y The detailed style of ) can be effectively applied. When all T iterations are finished, the above structure control input (Control I x The structure of ) and example image (Exemplar I y A final latent representation with the styles of ) perfectly fused is completed, which then passes through a VAE decoder to the final result image (Output I x→y It can be generated as ).

[0150] At this time, the above-mentioned Bridge-Brown diffusion core can be understood as an engine that performs the entire process of converting the structure control input and example image into a final image in the EBDM model of the present invention.

[0152] Meanwhile, the example-based brown bridge diffusion model (EBDM) of the present invention can be trained by a two-stage learning method for stable and efficient learning.

[0153] In the first training phase, training can be conducted primarily using the denoising network and the global encoder. The goal of this phase is to first enable the model to reliably inject global style features of example images via cross-attention while transforming images from structure control inputs. In this phase, the aforementioned example network and example attention module may not be involved in the training.

[0154] In the second learning stage, additional training can be performed by combining an example network and an example attention module with the model that has secured stable basic transformation performance through the first learning stage. In this stage, the weights of the denoising network and the global encoder that have already been trained are fixed, and training can be performed by focusing on the path that extracts and fuses local texture features. Through this two-stage learning method, the present invention has the effect of stably training a complex model and systematically learning both global style and local details to maximize the overall performance of the model.

[0155] Meanwhile, in one embodiment of the present invention, the invention enables fine control by a user. For example, a user may specify a specific area of ​​an image to be generated (e.g., 'hair' or 'top' area) through the interface or display of a user terminal. The system may artificially increase the attention weight for the specified area to induce the denoising network to refer more intensively to the local texture features of the example image when generating the texture of that area compared to other areas. Through this, the user can enhance the reflection of texture in areas they wish to particularly emphasize, thereby increasing satisfaction with the resulting output.

[0156] In addition, in one embodiment of the present invention, when a plurality of example images are input by a user, the processor can dynamically determine which example image texture to use for each region of the image to be generated. Specifically, the image region can be divided into patches, and routing can be performed to retrieve local texture features from the corresponding patch of the example image that shows the highest similarity to the structural features of each patch. For example, when generating a person image, it is possible to generate images with complex conditions, such as having the upper body follow the clothing texture of the first example image and the lower body follow the pants texture of the second example image.

[0158] - Image generation method using the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM)

[0159] FIG. 5 is a flowchart illustrating an image generation method according to an embodiment of the present invention.

[0160] Referring to FIG. 5, an example image-based image generation method (S) utilizing a bridge diffusion model according to one embodiment of the present invention may be performed by including the following steps.

[0162] S1: Step of obtaining structural control input and example images from the user

[0163] S2: Step to extract global style features from example images using a global encoder

[0164] S3: Step to extract local texture features from example images using a trained example network

[0165] S4: A step of setting the target state, which is a potential representation of the final result image, using the structural control input as the starting state of the diffusion process and the example image as a condition.

[0166] S5: A step of generating a latent representation by performing ODE-based deterministic backsampling.

[0167] S6: Step of decoding the generated latent representation to provide a result image

[0169] The above S1 step can be understood as a step in which a processor included in a user terminal (100) or a server (200) acquires a structure control input and an example image input from the user terminal. More specifically, the input structure control input and the example image may be acquired by the image acquisition unit (1200) of the example-based brown bridge diffusion model (EBDM) of the present invention by the processor.

[0170] The user can transmit a structural control input that forms the skeleton of an image and an example image containing the desired style and texture to the server (200) through an application of the user terminal (100). At this time, the user terminal (100) may be a smartphone, a portable terminal, a PC, etc., and the structural control input may be in various forms such as a segmentation mask, an edge map, or a pose keypoint, and the processor may be a processor of the user terminal (100) and / or the server (200).

[0171] The above steps S2 and S3 can be understood as steps of extracting global style features from an example image through a global encoder and extracting local texture features from an example image through a trained example network. These steps may be performed in a processor or in a potential generation unit (1200) of the above EBDM model (1000).

[0172] At this time, the extraction of the global style features and local texture features is the same as that described in the Brown Bridge diffusion model based on the previous example. By extracting the style information of an image by separating it into global and local characteristics in this way, the present invention enables more sophisticated image control in subsequent steps.

[0173] The above S4 step can be understood as a step of setting a target state, which is a potential representation of the final result image, using the structure control input as the starting state of the diffusion process and the example image as a condition. This step may be performed in a processor or in the potential generation unit (1200) of the EBDM model (1000).

[0174] In this step, boundary conditions for the diffusion path are set by applying the principles of the Brown-bridge diffusion model. First, the VAE encoder projects the structure control input obtained in step S1 onto the latent space and fixes it as the starting state of the diffusion process. Simultaneously, the latent representation of the image to be finally generated can be set as the target state based on the global style and local texture features extracted in steps S2 and S3. Through this process, the present invention enforces the image generation process to always follow the structure provided by the user, thereby improving the structural stability of the generated image.

[0175] The above S5 step can be understood as a step of generating a latent representation by performing ODE-based deterministic reverse sampling. This step may be performed in a processor or in the latent generation unit (1200) of the above EBDM model (1000).

[0176] This step is the stage where the final image is generated, and it can be performed by a denoising network. The denoising network performs an iterative noise removal process over several steps, moving from the starting state set in step S4 toward the target state.

[0177] At this time, within the sampling step, the global style features and local texture features are fused into the denoising network. Referring again to FIG. 4, first, the global style features are injected into the denoising network through a cross-attention module to form the overall tone and atmosphere of the image being generated. At the same time, the local texture features are fused with the features of the image currently being generated and attention-based by an example attention module.

[0178] Through this process, the present invention can generate a stable and high-quality latent representation in which the detailed style of an example image is precisely applied with a small number of steps while maintaining the structural characteristics of the structure control input.

[0179] The above S6 step can be understood as a step of decoding the generated latent representation to provide a result image. This step may be performed in a processor or in the decoding unit (1300) of the EBDM model (1000).

[0180] Once the above reverse sampling is completed, a clean latent representation with all noise removed can be finally produced. This latent representation is converted into a pixel-space image perceptible to humans through a VAE decoder. The final resulting image thus generated can be transmitted to a user terminal (100) via a network (300) and provided to the user.

[0182] Referring to FIGS. 6 and FIGS. 7, it can be seen that the image generated through the image generation method of the present invention described so far is superior to other models and can maintain high quality even under various conditions.

[0183] FIG. 6 is a diagram comparing an example-based brown bridge diffusion model according to one embodiment of the present invention with an image generated by another diffusion model.

[0184] Each column of FIG. 6 represents, in order from left to right, an example image (Exemplar) providing style, an edge map which is a structure control input (Control) providing structure, a result from a conventional state-of-the-art technology (ControlNet + IP-Adapter), and a result of an EBDM model according to the present invention (Ours).

[0185] As shown in Figure 6 above, the conventional method combining ControlNet and IP-Adapter generates realistic images that follow the structure of the structure control input to some extent, but tends not to perfectly reflect the unique style of the example image, particularly the identity or fine features of the person. Specifically, the facial features or unique atmosphere of the person in the example image may be expressed in a somewhat awkward or generic form in the generated result.

[0186] On the other hand, according to the results of the EBDM model (Ours) of the present invention, it can be confirmed that the facial features, hair color and style, and skin tone of the person in the example image are reproduced much more accurately and faithfully while maintaining the structural consistency of the structure control input. This demonstrates that the present invention achieves more stable and superior style transfer performance than hyperparameter-sensitive multi-condition models by effectively separating structure and style conditions through a brown-bridge diffusion model and precisely injecting style information through a global encoder and an example network.

[0188] FIG. 7 is a diagram showing the result of image generation based on an example-based brown bridge diffusion model for a combination of various example images and structural control inputs according to an embodiment of the present invention.

[0189] The top row of Fig. 7 represents structural control inputs (Control), i.e., segmentation masks, for faces with various shapes and expressions. The first column on the left represents example images (Exemplar) with different races, genders, ages, lighting environments, hairstyles, etc.

[0190] The images located in the central area within the Control row and Exemplar column are results generated by fusing styles (e.g., hair color, skin tone, makeup, etc.) from the example images in the corresponding row and structures (e.g., facial contours, poses, mouth shapes, etc.) from the structure control inputs in the corresponding column, respectively.

[0191] Specifically, referring to the images in the 3rd and 7th rows, if the example image of a blonde woman is used, a blonde-haired figure is consistently generated for all control masks. Referring to the images in the 2nd and 6th rows, if the example image of dim lighting is used, the corresponding lighting environment is consistently reflected in all generated images. Additionally, the structure of the image appearing in each control mask reflects the key features of the segmentation mask corresponding to the control structure input (Control) in the corresponding column.

[0192] As shown in Figure 7 above, it can be confirmed that the EBDM model of the present invention can be applied very faithfully to the structure defined by the control input, with the unique style and texture of the example image.

[0194] Meanwhile, the embodiments according to the present invention described above may be implemented in the form of application instructions that can be executed through various computer components and recorded on a non-transient computer-readable recording medium.

[0195] The above-mentioned non-transient computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The instructions may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. The recording medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute instructions, such as ROM, RAM, and flash memory.

[0197] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0198] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0199] 1: Example image-based image generation system using a bridge diffusion model 100: User terminal 110: Memory 120: Processor 130: Communications Department 140: Interface section 150: Display section 200: Server S: Example image-based image generation method using a bridge diffusion model S1: A step in which the processor obtains structure control input and example images from the user. S2: Step to extract global style features from example images using a global encoder S3: Step to extract local texture features from example images using a trained example network S4: A step of setting the target state, which is a potential representation of the final result image, using the structural control input as the starting state of the diffusion process and the example image as a condition. S5: A step of generating a latent representation by performing ODE-based deterministic backsampling. S6: Step of decoding the generated latent representation to provide a result image

Claims

Claim 1 A system for generating an image based on an example image, implemented by instructions executed by a processor, and comprising an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) that generates a result image from a structure control input input from a user terminal and an example image, wherein the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) comprises an image acquisition unit that acquires the structure control input and the example image, a latent generation unit that generates a latent representation of the result image based on a Brownian-Bridge Diffusion Model (BBDM) from the structure control input and the example image, and a decoding unit that decodes the latent representation to generate a result image and transmits it to the user terminal, wherein the latent generation unit comprises a global encoder that extracts global style features from the example image, an example network that extracts local texture features from the example image, and a deterministic ODE (Ordinary Differential Equation) based deterministic that sets the structure control input as the starting state of the diffusion process, sets the latent representation as the target state using the example image as a condition, and moves from the starting state to the target state. An example image-based image generation system utilizing a bridge diffusion model, including a denoising network that performs reverse sampling. Claim 2 delete Claim 3 An example image-based image generation system according to claim 1, wherein the denoising network includes an example attention module that receives the global style feature through a cross attention module during the deterministic reverse sampling process and selectively fuses the local texture feature with the denoising feature of the denoising network based on attention. Claim 4 An example image-based image generation system according to claim 3, wherein the example attention module is configured to increase the attention weight of a specific area designated by a user to enhance texture reflection for that area. Claim 5 An example image-based image generation system utilizing a bridge diffusion model according to claim 1, wherein the example-based brown bridge diffusion model (EBDM) is trained by a two-step training method in which a global style feature injection path is learned through the denoising network and a global encoder in a first training step, and a local texture feature fusion path is learned by combining the example network and an example attention module in a second training step. Claim 6 An example image-based image generation system utilizing a bridge diffusion model according to claim 1, wherein the structure control input comprises at least one of a segmentation mask, an edge map, or a pose keypoint. Claim 7 An example image-based image generation system utilizing a bridge diffusion model according to claim 1, wherein, when a plurality of example images are obtained, the potential generation unit is configured to dynamically route and combine local texture features extracted from the plurality of example images based on patch similarity by image region. Claim 8 An example image-based image generation system utilizing a bridge diffusion model, wherein, in any one of claims 1 and claims 3 to 7, the system comprises a plurality of mutually communicating computation nodes, wherein a processor of a first node performs a low-latency preview operation and a processor of a second node is configured to perform high-resolution final generation and / or model learning. Claim 9 An image generation method implemented by instructions executed by a processor, comprising the steps of obtaining a structure control input and an example image from a user, generating a latent representation from the structure control input and the example image based on a Brownian-Bridge Diffusion Model (BBDM), and decoding the generated latent representation to provide a result image, wherein the generation of the latent representation is performed through an Exemplar-guided Brownian-Bridge Diffusion Model (EBDM), and the Exemplar-guided Brownian-Bridge Diffusion Model (EBDM) comprises a global encoder that encodes global style features, an exemplar network that encodes local texture features, and an exemplar attention module that fuses the features into a denoising process. Claim 10 delete Claim 11 A method for generating an example image based on a bridge diffusion model according to claim 9, wherein the step of generating the latent representation further comprises the step of extracting global style features from the example image through the global encoder and the step of extracting local texture features from the example image through the example network. Claim 12 A method for generating an example image based on a bridge diffusion model according to claim 11, wherein the step of generating the potential representation comprises, based on a Brownian-Bridge diffusion model, setting the structure control input to a starting state of the diffusion process and setting a target state, which is a potential representation of the final result image, using the example image as a condition. Claim 13 In claim 12, the step of generating the latent expression generates the latent expression using the global style feature and the local texture feature through a denoising network, and the latent expression is generated by deterministic inverse sampling based on an ODE (Ordinary Differential Equation) from a set starting state toward a target state according to the condition, an example image-based image generation method utilizing a bridge diffusion model. Claim 14 A method for generating an example image based on a bridge diffusion model according to claim 13, wherein the deterministic reverse sampling comprises the steps of injecting the global style feature into a denoising network through a cross-attention module and selectively fusing the local texture feature with the denoising feature of the denoising network based on attention through an example attention module. Claim 15 In claim 14, the example attention module is an example image-based image generation method utilizing a bridge diffusion model capable of enhancing texture reflection for a specific area designated by a user by increasing the attention weight of that area. Claim 16 A method for generating an image based on an example image using a bridge diffusion model according to claim 14, wherein the example-based brown bridge diffusion model (EBDM) is trained by a two-step training method comprising: a first training step of training a global style feature injection path through the denoising network and a global encoder; and a second training step of training a local texture feature fusion path by combining the example network and an example attention module after the first training step. Claim 17 A method for generating an example image based on a bridge diffusion model, wherein the structure control input of claim 9 comprises at least one of a segmentation mask, an edge map, or a pose keypoint. Claim 18 A method for generating an example image based on a bridge diffusion model according to claim 12, further comprising the step of dynamically routing and combining local texture features extracted from a plurality of example images based on patch similarity by image region when a plurality of example images are obtained. Claim 19 A non-transient computer-readable recording medium having recorded instructions that cause the processor to perform an operation when executed by the processor, wherein the instructions are configured to cause the processor to perform an example image-based image generation method utilizing a bridge diffusion model according to any one of claims 9 and 11 to 18.

Images