Scrabble-to-vector image generation

Abstract

Embodiments of the present disclosure relate to scribble-to-vector image generation. Embodiments of the present disclosure relate to a method, apparatus, non-transitory computer readable medium, apparatus, and system for image generation, including acquiring sketch input depicting an object, processing the sketch input to acquire sketch guidance, and generating a composite image based on the sketch guidance using an image generation model, wherein the composite image depicts an object from the sketch input.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Application Serial No. 63 / 588,375, filed on October 6, 2023, entitled “SCRIBBLE2VECTOR” and U.S. Patent Application No. 18 / 625,432, filed on April 3, 2024, entitled “SCRIBBLE-TO-VECTORIMAGE GENERATION”. The entire contents of the above applications are incorporated herein by reference for all purposes. Background Art

[0003] The following relates generally to image processing, and more specifically to image generation using scribbles. Scribbles cover a variety of marks made by hand or digital tools. Scribbles can range from basic scribbles that capture the essence of an idea to more elaborate sketches that outline preliminary design concepts. In the field of image processing, scribbles can serve as the initial structure for manually formulating digital graphics and designs, and provide a direct and engaging way to capture and refine creative visions, thereby promoting innovation and efficiency in the design workflow.

[0004] In the field of machine learning, image generation models have been used to create images from inputs by learning from a dataset of existing images. However, conventional image generation models do not generate synthetic images based on informal inputs, such as scribblings that capture the intent of the input. Summary of the invention

[0005] The present disclosure relates to a machine learning model trained to generate a synthetic image based on a sketch input. The sketch input is processed by a sketch encoder. In some examples, the sketch encoder is initialized based on a copy of a pre-trained image generation model and is trained using training data including the training sketch input.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for image generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include obtaining a sketch input depicting an object; processing the sketch input using a sketch encoder to obtain sketch guidance; and generating a composite image based on the sketch guidance using an image generation model, wherein the composite image depicts the object from the sketch input.

[0007] A method, apparatus, non-transitory computer-readable medium, and system for image generation are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include creating a training set including sketch inputs, initializing a sketch encoder based on parameters of an image generation model, and training the sketch encoder to generate sketch guidance for the image generation model based on the sketch inputs.

[0008] An apparatus, system, and method for image generation are described. One or more aspects of the apparatus, system, and method include at least one processor, at least one memory storing instructions executable by the at least one processor, a sketch encoder including parameters stored in the at least one memory and trained to generate sketch guidance based on a sketch input, and an image generation model including instructions stored in the at least one memory and trained to generate a synthetic image based on the sketch guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of an image processing system according to aspects of the present disclosure is shown;

[0010] Figure 2 An example of an image processing application according to aspects of the present disclosure is shown;

[0011] Figure 3 An example of a method of generating a composite image according to aspects of the present disclosure is shown;

[0012] Figure 4 shows an example of a composite image generated according to aspects of the present disclosure;

[0013] Figure 5 An example of a method for image processing according to aspects of the present disclosure is shown;

[0014] Figure 6 An example of an image processing apparatus according to aspects of the present disclosure is shown;

[0015] Figure 7 An example of a U-Net according to aspects of the present disclosure is shown;

[0016] Figure 8 An example of a sketch encoder according to aspects of the present disclosure is shown;

[0017] Fig. 9 An example of a sketch encoder with a fit factor mechanism according to aspects of the present disclosure is shown;

[0018] Fig.10 An example of a data enhancement pipeline according to aspects of the present disclosure is shown;

[0019] Fig.11 shows example database creation for training an image generation model in accordance with aspects of the present disclosure;

[0020] Fig.12 An example of a method for image processing according to aspects of the present disclosure is shown; and

[0021] Fig.13 An example of a system according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0022] In the personal field of digital art and design, the transition from basic scribbles or sketches to precise digital output is a complex process involving complex technical solutions. A scribble or sketch can be the first step in this process, serving as an initial expression for artists and designers. In the following disclosure, the terms "sketch" and "scrawl" are used synonymously to refer to informal drawings that show a rough outline or approximate features of a desired image. These initial marks, combined with image processing techniques such as diffusion models, form the basis for converting informal ideas into structured images. Diffusion models refine the input to an image by learning from patterns in the data. Diffusion models can be used to generate images based on inputs including scribbles.

[0023] Some approaches involve analyzing and interpreting the inherent characteristics of scribbles, such as line quality, direction, and implicit shape, to reconstruct scribbles into graphics. However, these approaches have difficulty accurately discerning the intention behind the user's scribbles and may not provide the flexibility to accommodate the huge diversity and complexity of scribble styles, resulting in overly standardized or generic vector outputs that fail to faithfully represent the uniqueness of the original sketch.

[0024] Embodiments of the present disclosure include an image generation model that improves a conventional image generation model by generating a more accurate synthetic image that can better capture user intent via informal visual inputs such as scribbles. In some cases, the synthetic image is generated in a manner that enables the synthetic image to be converted into vector graphics. This is achieved using a sketch encoder that generates guidance for the image generation model. The present invention also provides a method for generating scribble training data to improve the efficiency of training an image generation model to obtain sketch inputs. In some examples, the sketch encoder is initialized using parameters from the image generation model and trained using the scribble training data.

[0025] Embodiments of the present disclosure improve the accuracy and fidelity of generating images from scribbles or sketches. By adapting diffusion models to convert scribbles into vectors, and training these models to maintain the capabilities of the decoder while customizing the models for specific scribble-based tasks, and by employing a novel data augmentation pipeline to generate training data that closely mimics scribbles, the system effectively adapts to a variety of scribble styles and complexities, thereby improving the generalization ability of the model and providing artists and designers with a versatile tool to convert their ideas into precise, scalable digital formats.

[0026] Image processing methods

[0027] Therefore, the present disclosure includes the following aspects. A method for image generation is described. One or more aspects of the method include obtaining a sketch input; processing the sketch input using a sketch encoder to obtain a sketch guide; and generating a synthetic image based on the sketch guide using an image generation model.

[0028] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include a machine learning model. Some examples also include encoding the text prompt to obtain a text encoding, wherein the synthetic image is generated based on the text encoding.

[0029] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include encoding the sketch input includes performing zero convolution on the sketch input. Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include generating the synthesized image includes providing the sketch guidance as an input to a decoder layer of the image generation model.

[0030] Some examples of the methods, apparatus, non-transitory computer-readable media, and systems also include generating the composite image comprising performing a back diffusion process. Some examples of the methods, apparatus, non-transitory computer-readable media, and systems also include encoding the sketch input comprising obtaining a noisy input image for an image generation model, wherein the composite image is based on the noisy input image. In some aspects, the composite image depicts elements of the sketch input with additional detail.

[0031] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include encoding the sketch input includes obtaining a class label indicating a class of the synthesized image, wherein the sketch guidance is generated based on the class label. In some aspects, the class is selected from a class set including a contour class and a color image class.

[0032] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include generating the composite image including obtaining a fit factor indicating a degree of adherence of the composite image to the sketch input, wherein the composite image is generated based on the fit factor. In some aspects, the fit factor is selected from a continuous range.

[0033] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include generating the composite image including applying a fit factor to the sketch guidance before providing the sketch guidance to the image generation model. In some aspects, the sketch encoder is trained using a trainable copy of the image generation model.

[0034] Figure 1 An example of an image processing system according to aspects of the present disclosure is shown. The example shown includes a user 100 , a user device 105 , an image processing apparatus 110 , a cloud 115 , and a database 120 .

[0035] exist Figure 1 In the example shown, user 100 provides a sketch input of a cute dog to image processing apparatus 110 via user device 105, where the interaction is facilitated by cloud 115. Image processing apparatus 110 equipped with a sketch encoder processes the sketch input to capture essential qualities and attributes of the cute dog depicted in the sketch.

[0036] The sketch encoder in the image processing device 110 is used to analyze and encode the sketch input, focusing on the dog's shape, features, and any other unique elements that contribute to the perception of "cuteness". For example, the image processing device 100 converts these visual elements into a structured format suitable for image synthesis. After this encoding process, the image processing device 110 uses a generator that takes the encoded data and generates a feature embedding. This embedding is a comprehensive representation of the sketch input prepared for the final stage of image synthesis.

[0037] The decoder component of the image processing device 110 then uses the feature embedding to create a composite image that embodies the characteristics of the cute dog originally drawn by the user 100. The resulting composite image is a detailed and visually appealing representation of the concept conveyed in the sketch input. The image is then transmitted back to the user 100 via the cloud 115 and the user device 105, demonstrating the device's ability to transform the user's sketch input into a high-quality, accurate visual output. This process effectively demonstrates the system's proficiency in interpreting and materializing the user-generated sketch into a finished image that meets the user's creative intent.

[0038] User device 105 may be a personal computer, laptop computer, mainframe computer, palmtop computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes software that incorporates image processing applications (e.g., query answering, image editing, relationship detection). In some examples, the image editing application on user device 105 may include the functionality of image processing device 110.

[0039] The user interface may enable user 100 to interact with user device 105. In some embodiments, the user interface may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the user interface may be represented by code that is sent to user device 105 and rendered locally by a browser. Figure 2 The process of using the image processing device 110 is further described.

[0040] The image processing device 110 includes a computer-implemented network including an image encoder, a text encoder, a multi-mode encoder, and a decoder. The image processing device 110 may also include a processor unit, a memory unit, an I / O module, and a training component. The training component is used to train the machine learning model (or image processing network). In addition, the image processing device 110 may communicate with the database 120 via the cloud 115. In some cases, the architecture of the image processing network is also referred to as a network, a machine learning model, or a network model. Reference Figure 5-Figure 6 Further details are provided regarding the architecture of the image processing device 110. Figure 5-Figure 6 Further details regarding the operation of the image processing device 110 are provided.

[0041] In some cases, the image processing device 110 is implemented on a server. The server provides one or more functions to users linked through one or more of the various networks. In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses a microprocessor and a protocol to exchange data with other devices / users on one or more networks via the Hypertext Transfer Protocol (HTTP) and the Simple Mail Transfer Protocol (SMTP), but other protocols such as the File Transfer Protocol (FTP) and the Simple Network Management Protocol (SNMP) may also be used. In some cases, the server is configured to send and receive files in the Hypertext Markup Language (HTML) format (e.g., for displaying web pages). In various embodiments, the server includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0042] Cloud 115 is a computer network that is configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, cloud 115 provides resources without active management by users. The term "cloud" is sometimes used to describe data centers available to many users on the Internet. The functions of some large cloud networks are distributed in multiple locations from a central server. If a server has a direct or close connection with a user, the server is designated as an edge server. In some cases, cloud 115 is limited to a single organization. In other examples, cloud 115 can be used for many organizations. In one example, cloud 115 includes a multi-layer communication network that includes multiple edge routers and core routers. In another example, cloud 115 is based on a local collection of switches in a single physical location.

[0043] Database 120 is an organized collection of data. For example, database 120 stores data in a specified format called a schema. Database 120 can be structured as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the storage and processing of data in database 120. In some cases, a user interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0044] Figure 2 An example of an image processing application 200 according to aspects of the present disclosure is shown. The image processing application 200 is a reference Figure 3 , Figure 4 , Figure 6-Figure 11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0045] At operation 205, the user provides a sketch input. For example, the user begins the image generation process by inputting a sketch depicting a cute dog. In this way, the user provides visual input that will guide subsequent image synthesis. The sketch input captures the basic shape, features, and possible expected expressions or poses of the cute dog envisioned by the user.

[0046] At operation 210, the system generates a sketch guide. For example, the system processes the sketch input to generate a sketch guide. This operation involves analyzing the sketch input, breaking it down into its constituent elements, and encoding it into a format that the image generation model can understand and use. The sketch guide serves as a detailed blueprint for the user input, preserving the key attributes and nuances of the cute dog depicted in the sketch, while converting it into a structured form ready for synthesis.

[0047] At operation 215, the system generates a composite image based on the sketch guide. For example, the system employs an image generation model to create a composite image that visually represents the cute dog drawn by the user. Using the sketch guide as a guide, the image generation model synthesizes an image that reflects the user's artistic vision by paying close attention to the form, features, and overall aesthetics of the cute dog. The job of the generator is to ensure that the composite image is visually logical and faithful to the original sketch input, thereby producing a high-quality and representative depiction of the cute dog.

[0048] At operation 220, the system provides the composite image to the user. For example, the system presents the generated composite image to the user. This final step completes the transformation cycle from sketch to composite image, providing the user with a visual representation of their initial sketch input. The user can review the composite image to assess its quality, accuracy, and overall consistency with the creative intent. The generated composite image demonstrates the system's ability to interpret and render the sketch into a detailed and engaging composite image, allowing the user to see their creative concepts realized in a vivid visual form.

[0049] Figure 3 An example of a method 300 for generating a composite image according to aspects of the present disclosure is shown. The method 300 is a reference Figure 2 , Figure 4 , Figure 6-Figure 11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0050] refer to Figure 3 , describing an example method of generating a composite image. At the beginning of the process, sketch input 305 represents the initial visual guidance provided by the user. This can be a hand-drawn sketch or a digital drawing that serves as the main structural and conceptual guidance for the composite image. In some examples, sketch input 305 captures the basic shapes, lines, and intended design elements that the user wants to be reflected in the final image.

[0051] In some cases, textual hints 310 are obtained. Textual hints 310 provide an additional layer of semantic guidance to the image generation process. Textual hints 310 may include descriptions, instructions, or any other textual information that provides context or specific details about the desired outcome of the synthesized image. Textual hints 310 supplement the visual information provided by the sketch input, providing a more nuanced understanding of the user's intent.

[0052] The sketch input 305 is then processed by a sketch encoder 315, which converts the visual information into a structured format that can be effectively utilized by the image generation model. The sketch encoder 315 analyzes the lines, shapes, and overall structure of the sketch to encode these elements into a form that captures the essence of the original input while making it interpretable by the image generation system.

[0053] The textual prompt 310 is processed by the text encoder 320. The text encoder 320 encodes the textual information into a feature-rich representation that summarizes the semantic nuances of the text. The text encoder 320 ensures that the details, context, and specificity contained in the textual prompt are effectively conveyed to the image generation model, thereby affecting the properties and characteristics of the synthesized image.

[0054] The outputs of the sketch encoder 315 and the text encoder 320 are then input into the image generation model 325. The image generation model 325 integrates the encoded sketch and text input to generate a composite image consistent with the guidance provided by the user. The image generation model 325 utilizes advanced algorithms and models to interpret the combined guidance to synthesize an image that reflects both the structural and stylistic elements conveyed by the sketch and text input.

[0055] Finally, the image generation model 325 generates a composite image 330 as an output image. The composite image 330 represents a visual creation that embodies the user's artistic vision, interpreted and rendered through the complex architecture of the image generation system.

[0056] Figure 4 An example of a composite image 400 according to aspects of the present disclosure is shown. The composite image is referenced Figure 2 , Figure 3 , Figure 6-Figure 11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0057] refer to Figure 4, depicts a series of images showing the results of an image generation process that incorporates user input, including rough scribbles, optional text prompts, and specified output types and enhancement levels. The images shown are the results after vectorization using a number of real-world image editing applications, demonstrating the system's ability to generate both outline and color images with varying levels of detail and complexity based on user-defined preferences and input.

[0058] Sketch input 405 represents an initial scribble provided by a user that serves as a basic guide for image generation. For example, sketch input 405 is a rough drawing provided by a user that outlines the basic shapes, structures, or concepts they would like to see in the final image. Sketch input 405 lays the foundation for the subsequent image generation and enhancement process, providing basic visual clues and guidelines that will be interpreted and elaborated by the system.

[0059] exist Figure 4 In the first row of , a series of outline images 410, 420, 430, and 440 are shown, each of which represents a different level of enhancement ranging from 0 to 3. These outline images are the results of applying increasing levels of enhancement to the original sketch input, demonstrating the system's ability to refine and diversify the output based on the specified enhancement level. The outline image 410 with enhancement level 0 represents the most basic and least altered version of the original sketch. For the outline images 410, 420, 430, and 440, each subsequent image exhibits a higher level of enhancement, introducing more variation and complexity, reflecting a greater deviation from the original sketch input. These outline images 410, 420, 430, and 440 demonstrate the system's ability to produce a range of results from a single input to allow users to select the level of detail and abstraction that best suits their needs or preferences.

[0060] exist Figure 4 In the second row of FIG. , a series of color images 415, 425, 435, and 445 are shown, which also have increasing enhancement levels from 0 to 3, respectively. Similar to the first row, these color images demonstrate the system's ability to apply different levels of enhancement to produce a range of results. Color image 415, with an enhancement level of 0, shows the most direct and least enhanced interpretation of the original sketch input, while color images 425, 435, and 445 show increasingly complex and detailed results, reflecting higher levels of enhancement. These color images vary not only in the level of detail and stylization, but also include colorization, providing a richer and more visually appealing representation of the original sketch input. The variations in color and shading add depth and dimension to the image, enhancing the overall aesthetic appeal, and providing the user with a diverse set of options for the final output.

[0061] Figure 5An example of a method 500 for image processing according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in combination with other operations.

[0062] At operation 505, the system obtains sketch input. In some cases, the operation of this step involves referring to Figure 6 The machine learning model may be executed thereby.

[0063] For example, at operation 505, the system obtains sketch input. In some examples, obtaining sketch input may involve direct digital sketching on a user interface or uploading a pre-drawn sketch. The operation of this step provides initial data input for the image generation process.

[0064] In some examples, obtaining sketch input involves receiving a visual representation drawn or provided by a user, which is a form of guidance for a composite image. In some examples, obtaining sketch input involves direct digital sketching on a user interface, where a user can create a sketch on a digital canvas using various tools. In some cases, a user can upload a pre-drawn sketch or image, which is then interpreted by the system as a sketch input. These inputs range from simple line drawings to more complex compositions, depending on the user's preferences and the capabilities of the system. The sketch input is used as an initial data input to lay the foundation for subsequent operations that will collectively transform the initial sketch into a refined composite image. The sketch input captures the user's intent and creative vision, which will be reflected and beautifully designed in the final output.

[0065] At operation 510, the system processes the sketch input using a sketch encoder to obtain sketch guidance. In some cases, the operation of this step involves referring to Figure 6 The sketch encoder may be executed thereby.

[0066] For example, at operation 510, the system processes the sketch input using a sketch encoder to obtain sketch guidance. For example, encoding the sketch input involves performing zero convolution on the sketch input, or obtaining a noisy input image for an image generation model. Zero convolution involves applying a convolution filter with all weights set to zero, which effectively serves as a mechanism to initialize or reset the encoding process. This technique can be particularly useful in preparing the sketch input for further, more detailed encoding processes by standardizing or normalizing the input data.

[0067] In some examples, the system obtains noisy input images for the image generation model. This can involve intentionally introducing noise into the sketch input to simulate the variability and imperfections inherent in human-drawn sketches. This noise can be added uniformly or according to a specific distribution to reflect the randomness of hand-drawing. Noisy images provide more challenging and realistic training or processing inputs for the image generation model, encouraging the development of robust and adaptive image generation capabilities.

[0068] In some examples, the sketch encoder is trained using a trainable copy of the image generation model to adjust its encoding strategy based on the unique characteristics of the input. For example, by utilizing a version of the image generation model specifically tailored for training, the sketch encoder can learn from the image generation patterns and results to continuously improve its ability to convert sketch inputs into effective sketch guidance. In some cases, the sketch guidance is generated based on various factors, including a class label indicating the category of the synthesized image. In some examples, the category is selected from a set including a contour category and a color image category.

[0069] At operation 515, the system generates a composite image based on the sketch guidance using the image generation model. In some cases, the operation of this step involves referring to Figure 6 The image generation model may be executed thereby.

[0070] For example, at operation 515, the system generates a composite image based on the sketching instructions using an image generation model. This operation converts the detailed and structured sketching instructions into a visual output that reflects the user's initial sketch input. According to some embodiments, the image generation model employs algorithms and models to interpret the sketching instructions to synthesize an image that is consistent with the style and structural characteristics conveyed in the original sketch.

[0071] In some examples, generating a synthetic image involves providing sketch guidance as input to a decoder layer of an image generation model. The decoder layer is responsible for reconstructing and refining the image based on the encoded guidance. The decoder layer works by incrementally building up the image, adding details and refinement at each stage of the decoding process. By feeding the sketch guidance into the decoder layer, the system ensures that the final image retains the essential elements and style of the original sketch, while also benefiting from the enhanced ability of the image generation model to produce polished and refined output.

[0072] For example, the system performs a back-diffusion process to generate a synthetic image. In the back-diffusion process, the image generation model starts with a deliberately noisy or distorted version of the sketch guide, and then gradually operates to reverse this noise, iteratively refining and clarifying the image. The back-diffusion process generates highly detailed images that are consistent with the sketch guide by allowing controlled and progressive accumulation of visual information.

[0073] In some embodiments, a composite image is generated based on a fit factor that indicates how closely the composite image adheres to the sketch input. This fit factor can be selected from a continuous range to provide a range of fit levels from strict adherence to more creative freedom. The fit factor is applied to the sketch guidance before providing it to the image generation model, allowing for fine control over the fidelity of the resulting image to the original sketch. In some examples, the composite image depicts elements of the sketch input in additional detail, enhancing the visual complexity and depth of the final output.

[0074] In some embodiments, a machine learning model such as image generator 325 is trained to generate a vectorizable image. A vectorizable image may refer to an image that can be efficiently vectorized. For example, a vectorizable image may have flat colors and relatively few strokes. Thus, in some examples, the image generator may generate a pixel-based image and then transform the pixel-based image into a vector image (i.e., "vectorize"). Alternatively, in some examples, the image generator may directly generate a vector image.

[0075] Image generating device

[0076] An apparatus for image generation is described. One or more aspects of the apparatus include at least one processor; at least one memory storing instructions executable by the at least one processor; a sketch encoder including parameters stored in the at least one memory and trained to generate sketch guidance based on a sketch input; and an image generation model including instructions stored in the at least one memory and trained to generate a synthetic image based on the sketch guidance.

[0077] In some aspects, the image generation model includes a diffusion model. In some aspects, the sketch encoder includes zero convolutional layers and layers of encoder layers based on the image generation model.

[0078] Figure 6 An example of an image generating device 600 according to aspects of the present disclosure is shown. The image generating device 600 is a reference Figure 2-Figure 4 , Figure 7-Figure 11 and Fig.13 Examples of corresponding elements described or include aspects thereof. In one aspect, the image generation device 600 includes a processor unit 605, an I / O module 610, a training component 615, a memory unit 620, a machine learning model 625 including an image generation model 630, a text encoder 635, and a sketch encoder 640.

[0079] The processor unit 605 includes one or more processors. A processor is an intelligent hardware device, such as a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof.

[0080] In some cases, processor unit 605 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into processor unit 605. In some cases, processor unit 605 is configured to execute computer-readable instructions stored in memory unit 620 to perform various functions. In some aspects, processor unit 605 includes dedicated components for modem processing, baseband processing, digital signal processing, or transport processing. According to some aspects, processor unit 605 includes reference Fig.13 Describes one or more processors.

[0081] The memory unit 620 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software, including instructions that, when executed, cause at least one processor of the processor unit 605 to perform various functions described herein.

[0082] In some cases, memory unit 620 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, memory unit 620 includes a memory controller that operates the memory cells of memory unit 620. For example, the memory controller may include a row address decoder, a column address decoder, or both. In some cases, the memory cells within memory unit 620 store information in the form of logical states. According to some aspects, memory unit 620 includes a reference Fig.13 Describes the memory subsystem.

[0083] According to some aspects, image generation device 600 uses one or more processors of processor unit 605 to execute instructions stored in memory unit 620 to perform the functions described herein. For example, in some cases, image generation device 600 obtains a prompt. In some cases, the prompt includes a text prompt. For example, in some cases, image generation device 600 obtains a sketch input.

[0084] Machine learning parameters (also called model parameters or weights) are variables that provide the behavior and characteristics of a machine learning model. Machine learning parameters can be learned or estimated from training data and used to make predictions or perform tasks based on learned patterns and relationships in the data.

[0085] Machine learning parameters are typically adjusted during the training process to minimize a loss function or maximize a performance metric. The goal of the training process is to find the best values ​​for the parameters so that the machine learning model can make accurate predictions or perform well on a given task.

[0086] For example, during training, the algorithm adjusts machine learning parameters based on optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms to minimize the error or loss between the predicted output and the actual target. Once the machine learning parameters are learned from the training data, the machine learning parameters are used to make predictions on new, unseen data.

[0087] Artificial neural networks (ANNs) have many parameters, including weights and biases associated with each neuron in the network, which control the degree of connectivity between neurons and affect the neural network's ability to capture complex patterns in data.

[0088] An ANN is a hardware or software component that includes multiple connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another (just like a physical synapse in the brain). When a node receives a signal, it processes the signal and then transmits the processed signal to other connected nodes.

[0089] In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. In some examples, the nodes may use other mathematical algorithms to determine their outputs, such as selecting the maximum value from the inputs as the output, or using any other suitable algorithm for activating nodes. Each node and edge is associated with one or more node weights that determine how the signal is processed and transmitted.

[0090] In an ANN, a hidden (or intermediate) layer includes hidden nodes and is located between the input layer and the output layer. The hidden layer performs nonlinear transformations on the inputs fed into the network. Each hidden layer is trained to produce a defined output that contributes to the joint output of the output layer of the ANN. The hidden representation is a machine-readable data representation of the input learned from the hidden layers of the ANN and produced by the output layer. As the ANN's understanding of the input improves as the ANN is trained, the hidden representation gradually differs from earlier iterations.

[0091] During the training process of an ANN, node weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss that corresponds in some way to the difference between the current result and the target result). The weights of the edges increase or decrease the strength of the signal transmitted between the nodes. In some cases, nodes have a threshold below which no signal is transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0092] According to some aspects, an image generation model 630 is included in the image generation device 600. As part of the machine learning model 625, the image generation model 630 generates a composite image based on input received from various sources (such as text prompts or sketches). In some examples, the image generation model 630 works in conjunction with other components of the device to utilize algorithms and data processed by the processor unit 605 to create visually logical and contextually relevant images.

[0093] According to some aspects, a text encoder 635 is included in the image generation device 600. The text encoder 635 interprets and encodes the text data into a format that can be understood and used by the machine learning model 625, in particular the image generation model 630. In some examples, the text encoder 635 obtains a text prompt, which can contain descriptive or instructive information about the desired image, and converts it into a feature-rich representation. The representation is used as a guide for the image generation model 630 to influence the properties and characteristics of the generated image to ensure that they are consistent with the user's intention expressed in the text prompt.

[0094] According to some aspects, a sketch encoder 640 is included in the image generation device 600 to process and encode the sketch input. The sketch encoder 640 transforms the sketch drawn by the user into a structured format that the image generation model 630 can understand and use in the image generation process. For example, the sketch encoder 640 is used to capture the style and structural elements of the user's sketch to ensure that these elements are accurately reflected in the final image output. By encoding these sketches, the sketch encoder 640 allows the image generation device 600 to incorporate user-specific details and nuances, thereby producing images that are not only high quality but also personalized and closely aligned with the user's creative vision.

[0095] Figure 7 An example of a U-Net 700 according to aspects of the present disclosure is shown. U-Net 700 is a reference Figure 2-Figure 4 , Figure 6 , Figure 8-Figure 11 and Fig.13Examples of corresponding elements described herein or include aspects thereof. According to some aspects, the diffusion model includes an ANN architecture known as U-Net. In some cases, U-Net 700 implements a reverse diffusion process.

[0096] According to some aspects, U-Net 700 receives input features 705. The input features 705 include an initial resolution and an initial number of channels, and the input features 705 are processed using an initial neural network layer 710 (e.g., a convolutional neural network layer) to produce intermediate features 715.

[0097] In some cases, the intermediate features 715 are then downsampled using a downsampling layer 720 so that the resolution of the downsampled features 725 is less than the initial resolution and the number of channels is greater than the initial number of channels. In some cases, the process is repeated multiple times and then the process is reversed. For example, the downsampled features 725 are upsampled using an upsampling process 730 to obtain upsampled features 735. In some cases, the upsampled features 735 are combined with the intermediate features 715 having the same resolution and number of channels via a skip connection 740. In some cases, the combination of the intermediate features 715 and the upsampled features 735 is processed using a final neural network layer 745 to produce an output feature 750. In some cases, the output feature 750 has the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0098] According to some aspects, U-Net 700 receives additional input features to produce conditionally generated outputs. In some cases, the additional input features include a vector representation of an input prompt. In some cases, the additional input features are combined with intermediate features 715 within U-Net 700 at one or more layers. For example, in some cases, a cross-attention module is used to combine the additional input features and the intermediate features 715.

[0099] Training an Image Generation Model

[0100] A method for training an image generation model is described. One or more aspects of the method include creating a training set including sketch inputs; initializing a sketch encoder based on parameters of the image generation model; and training the sketch encoder to generate sketch guidance for the image generation model based on the sketch inputs. In some cases, creating the training set may include obtaining a set of pre-existing training data to train a machine learning model.

[0101] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include that training the sketch encoder includes generating a synthetic image based on the output of the sketch encoder using the image generation model; and calculating a loss function based on the synthetic image and the ground truth image. Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include that creating a training set includes generating a sketch input based on the ground truth image by applying white filling, black strokes, path discarding, stroke width enhancement, control point perturbation, path translation, rotation, or any combination thereof. Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include that initializing the sketch encoder includes copying parameters of the encoder of the image generation model.

[0102] Figure 8 An example of a sketch encoder 800 according to aspects of the present disclosure is shown. The sketch encoder 800 is a reference Figure 2-Figure 4 , Figure 6 , Figure 7 , Figure 9-11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0103] According to some embodiments, the sketch encoder uses the ControlNet architecture as a guidance network architecture to facilitate image synthesis based on user-specified image generation guidance. For example, ControlNet is used as an encoder to encode a scribble input to generate a scribble guidance. However, embodiments of the present disclosure are not limited to this. An architecture different from the ControlNet architecture can be used as a scribble encoder. For example, a visual transformer adapter can be used as a scribble encoder.

[0104] According to some embodiments, the sketch encoder guides the creation of both color and black and white images using scribble input and optional text prompts to effectively utilize vector-based representations and outlines. In some examples, the sketch encoder, which is fine-tuned for specific image characteristics, orchestrates the synthesis process on top of a pre-trained diffusion model framework to ensure that the generated image adheres to the desired style and structural parameters. By using the sketch encoder, the image generation model generates synthesized images that are closely aligned with the user's intent and aesthetic preferences.

[0105] ControlNet is a neural network architecture that allows image generation to be guided via a diffusion model by a user-specified control signal. It creates a trainable copy of a U-Net encoder and fine-tunes only that copy, freezing the original U-Net. This encoder copy is connected to the U-Net decoder via a trainable 1×1 zeroed convolutional block. Since the original U-Net is unaffected and the new encoder copy learns to follow the control signal, ControlNet allows efficient fine-tuning of such task-specific large diffusion models without losing the generative power of the base model. Thus, sketch encoders such as ControlNet provide scribbles as control signals for guiding image generation with optional textual cues.

[0106] The guidance network is used to provide a control signal for guiding image generation. In some embodiments, the guidance network is a scribble encoder that encodes a scribble input to generate a scribble guide. In some examples, the scribble encoder encodes a contour image to generate a scribble guide. In some embodiments, the guidance network can be ControlNet. However, the embodiments of the present disclosure are not limited thereto. For example, the guidance network can have an architecture other than the ControlNet architecture, such as a visual transformer adapter architecture. The sketch encoder can be a neural network architecture that enables image generation of the image generation model to be guided by a user-specified control signal. In some examples, the guidance network creates a trainable copy of the encoder and only fine-tunes the copy to freeze the original decoder. The encoder copy can be connected to the decoder through a trainable 1×1 zero convolution block. Since the original encoder is unaffected and the new encoder copy learns to follow the control signal, large models for specific tasks can be efficiently fine-tuned without losing the basic model generation capability. The scribble can be used as a control signal to guide image generation. In addition, optional text prompts can be used, and the image generation model generates images based on the scribbles and text prompts.

[0107] For example, suppose Let C be the guidance network, θ and θ train As their locked and trainable parameters, respectively, C is the class label. The generated RGB image Obtained as follows:

[0108] y=C(x,h c ,c;θ,θ train ) (1)

[0109] In one example, h=w=128.

[0110] In some embodiments, a pre-trained diffusion model is instantiated and its weights are locked. The trainable copy of the encoder can then also be initialized with the same weights as the base encoder. The weights of the trainable copy are unlocked. The base model receives the noise map as before, and the trainable copy concatenates the noise map with the scribble hint. The output of each layer of the trainable encoder can be multiplied by a trainable 1×1 zeroed convolutional layer and added to the corresponding output of the base encoder layer. Class labels are also passed to the trainable encoder to control the type of image generated and the fidelity of the input scribble. In some embodiments, the image generation model employed can be different from the diffusion model and can include alternatives such as GANs or autoencoders.

[0111] refer to Figure 8 , the image generation process is guided by a guidance signal including scribble input 820 and text hint 805. In some cases, text hint 805 may be optional, and the guidance signal may include scribble input 820 without text hint 805. For example, scribble input 820 provides structural or stylistic guidance as the primary control signal, and text hint 805 provides an additional semantic context or guidance layer for the image generation process to enhance the specificity and relevance of the generated image to the user's intent.

[0112] In some embodiments, time 810 determines the image generation stage in the diffusion process to affect how the noise map 815 and other inputs are processed over time. Time 810, when encoded and input into the sketch encoder, provides a temporal component for coordinating the sequential nature of the diffusion process and managing the transformation of the inputs to the final logical output image.

[0113] In some embodiments, the noise map 815 is an input to the diffusion model and helps generate high quality images through controlled introduction and subsequent noise inversion. In some embodiments, the noise map 815 is manipulated over time 810. This manipulation promotes a gradual evolution of the output image 830. The evolution is influenced by guidance from the sketch encoder, which utilizes the scribble input 820 and the text prompt 805.

[0114] In some embodiments, the scribble input 820 acts as the primary control signal for guiding image synthesis and is encoded and interpreted by the sketch encoder to ensure that the generated image conforms to the structural guidelines specified by the user. In some examples, the scribble input 820 is a user-generated input and allows for personalized and detailed control over the image generation process.

[0115] In some embodiments, class labels 825 are used as classification controls in the image generation process to provide a mechanism for refining and directing image synthesis according to predefined parameters. Class labels 825 are components of the diffusion model for class adjustment, which enhances the adaptability and accuracy of the generated output. In some embodiments, class labels 825 enable modulation of the fidelity in the generated image relative to the scribbling hint. For example, class labels 825 allow the closeness of the generated image to the structural and stylistic details of the original scribbling to be adjusted. The system combines several fidelity levels, each associated with a specific class label, to provide users with a range of options from high fidelity to more abstract representations. In some embodiments, class labels 825 are used to determine the style results of the image generation process. It facilitates the selection between generating contour images and color images. Therefore, class labels 825 enable users to influence whether the model produces a contour-based representation or generates a full-color and detailed illustration, thereby further enhancing the creative control available in the image generation process.

[0116] In some embodiments, output image 830 is a composite image generated by an image generation model including a sketch encoder that reflects the combined effects of text prompt 805, scribble input 820, class label 825, and noise map 815 that varies over time 810. Output image 830 conforms to the user-defined input and has both visual integrity and contextual consistency.

[0117] Fig. 9 An example of a sketch encoder 900 with a fit factor mechanism according to aspects of the present disclosure is shown. The sketch encoder 900 with the fit factor mechanism 900 is a reference Figure 2-Figure 4 , Figure 6-Figure 8 , Fig.10 , Fig.11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0118] In some embodiments, a fit factor in the image generation model enables the user to exercise fine-grained control over image generation based on user-specific instructions or prompts. For example, the incorporation of a fit factor addresses the challenges posed by scribble-to-image transformation by providing control at the inference stage without requiring class conditioning at the training stage. In some examples, the model achieves an admirable balance between respecting the scribble and improving image quality when the fit factor is calibrated in the range of 0.3 to 0.6. Rather than strictly replicating every aspect of the scribble, the model retains key attributes such as pose and orientation, but gives itself the freedom to emphasize and refine other elements. This produces images that are consistent with the user's intent while maintaining exemplary visual quality. This modulation capability provides users with fine-grained control, bridging the split between realistic representation and superior image synthesis.

[0119] In some embodiments, the sketch encoder with a fit factor mechanism exhibits adaptability when given a complex scribble. In addition to the base representation, the image generation model generates images from complex scribbles that summarize a wide range of visual narratives, spanning scenes, backgrounds, or composite visuals. The ability to interpret and replicate beautiful scribbles, especially at optimally tuned fit factors, highlights the potential of this model in countless graphics and design projects.

[0120] Therefore, the fit factor improves the neural model's ability to handle the transition from scribbles to images. By effectively balancing input precision and output quality, the model demonstrates the ability to refine basic sketches into detailed visuals. This advancement has proven valuable in the design context, facilitating faster iteration and visualization from basic inputs. As the field of digital design advances, this approach has the potential to serve as a key tool to optimize rapid prototyping and support design exploration.

[0121] For example, suppose Let C be the guidance network, θ and θ tr As the locking and trainable parameters, respectively, f is the fitting factor. The generated image Obtained as follows:

[0122] y=C(x,h e ,f;θ,0 tr ) (2)

[0123] Here, h=w=128, and the generated image is RGB image.

[0124] refer to Fig. 9 In some embodiments, the image generation process using the modified ControlNet architecture is guided by guidance signals including sketch input 920 and textual prompts 905. The textual prompts 905 provide an additional semantic context or guidance layer to the image generation process to enhance the specificity and relevance of the generated image with respect to the user's intent. The sketch input 920 provides structural or style guidance as the primary control signal.

[0125] In some embodiments, time 910 determines the image generation stage in the diffusion process to affect how the noise map 915 and other inputs are processed over time. Time 910, when encoded and input into the sketch encoder, provides a temporal component for coordinating the sequential nature of the diffusion process and managing the transformation of the inputs to the final logical output image.

[0126] In some embodiments, the noise map 915 is an input to the diffusion model and helps generate high quality images through controlled introduction and subsequent noise inversion. The noise map 915 is manipulated over time 910. This manipulation promotes a gradual evolution of the output image 930. The evolution is influenced by guidance from the sketch encoder, which utilizes the sketch input 920 and the text prompt 905.

[0127] In some embodiments, a fit factor 925 is introduced as a parameter in the image generation model to adjust the model's compliance with a user-provided sketch or scribbling. The fit factor 925 takes a value between 0 and 1, where a value of 1 represents strict conformance to the sketch input 920 and a value of 0 represents complete independence from the sketch input 920. Intermediate values ​​allow varying degrees of conformance to provide a continuous possibility between strict conformance and complete freedom. In some examples, during training, the fit factor 925 can be set to its maximum value to ensure complete consistency with the sketch input 920. In some examples, during reasoning, the value of the fit factor 925 can be dynamically adjusted, allowing for customized responses to user-provided sketches and enabling a flexible and adaptable image generation process.

[0128] In some embodiments, the sketch input 920 acts as the primary control signal for guiding the synthesis of the image encoded and interpreted by the sketch encoder to ensure that the generated image conforms to the structural guidelines specified by the user. This user-generated input allows for personalized and detailed control over the image generation process.

[0129] In some embodiments, output image 930 is a result image generated by the model that reflects the combined effects of text prompt 905, sketch input 920, fit factor 925, and noise map 915 that varies over time 910. Output image 930 conforms to the user-defined input and has both visual integrity and contextual consistency. Incorporating fit factor 925 allows fine-tuning of the level of detail and stylistic consistency of the output image with the initial user input to ensure a high-quality and relevant final product.

[0130] Fig.10 An example of a data enhancement pipeline according to aspects of the present disclosure is shown. The data enhancement pipeline is referenced Figure 2-Figure 4 , Figure 6-Figure 9 , Fig.11 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0131] According to some embodiments, class adjustment is used with the image generation model. Class adjustment can be used for a variety of purposes, including guiding image generation with different fidelity levels for the scribbling hints provided to the model. Some embodiments use four discrete fidelity levels. Class adjustment can be used to choose between generating contour images and color images. Some embodiments use four classes (1, 2, 3, 4) as the fidelity level for contours, and these four classes as the fidelity level for color images.

[0132] In some embodiments, the base model is trained using paired image and text data. Vector data is used for training. Various enhancements can be used to enhance the training data. In some cases, vector data is used, such as SVG format. Various attributes of the vector image can be modified, including line, stroke, and fill attributes. In some images, the stroke is changed to black, and the color fill attribute is changed to white. In some cases, additional enhancements are applied to these modified black and white vector images. In some cases, the vector image is rasterized to obtain the training data.

[0133] In some embodiments, a real or synthetic hand-drawn scribble is provided to the guidance network as a control cue. For example, a ground truth vector image can be used for augmentation, and randomness is introduced in one or more steps to generalize the training data. Once augmented, the vector image is rasterized to generate a control cue scribble image. In some embodiments, augmentation includes various techniques.

[0134] For example, path discarding is discarding a portion of a path from a vector image to create a vector with less details. Stroke width augmentation is increasing or decreasing the stroke width of some paths of a vector image. Control point perturbation is perturbing the control points of some paths of a vector image to create a distorted effect. This augmentation is useful for the model to generalize to user scribbles with distorted shapes. Path translation and rotation is translating and rotating some paths of a vector image to mimic user scribbles with overlapping strokes that are not necessarily aligned with the intended resulting rotation.

[0135] Class conditioning refers to applying these enhancements with increasing strength based on multiple levels of class conditioning. Class information is passed to the model at training and inference time to allow the user to control the fidelity of the generated scribbles to the input. Image-level enhancement refers to performing enhancements in the image domain after the vector image has been rasterized. For example, an edge map is first extracted (e.g., via HED or Canny edge detector). Next, random binary thresholding, random pixel dropping, random edge dilation / erosion, random warping, etc. are performed. These enhancements, while good for photorealistic images, may not generalize well to vector-like color images and contours.

[0136] After obtaining vector-like images from the image generation model, these images can be converted into vectors using vectorization techniques. By creating vectorizable images, the ability to generate meaningful and logical vectors from different types of user scribbles can be enhanced. The model can also generate outline and / or color results with different fidelity levels to the input scribbles based on class conditioning at inference time.

[0137] In some embodiments, after obtaining vector-like images from the sketch encoder model, a vectorization method is provided to convert these vector-like images into vectors. Therefore, embodiments of the present disclosure provide real-world applications with the ability to generate meaningful and logical vectors from different types of user scribbles. Embodiments of the present disclosure can also generate contour and / or color results with different fidelity levels for the input scribbles based on class adjustment during reasoning.

[0138] refer to Fig.10 , in some embodiments, depicts a data augmentation pipeline showing various stages and techniques for refining and enhancing vector images for the purpose of training an image generation model. The diagram illustrates a series of transformation processes applied to a vector image, such as vector image 1005, each of which is intended to simulate the variability and characteristics of a human-drawn sketch and thereby enrich the training data for the model.

[0139] In some embodiments, the first stage involves extracting contours from the original vector image 1005 to obtain an image in which all strokes are changed to black and color fill attributes are changed to white to create extracted contours 1010 by applying white fill and black stroke. This is a foundational step for subsequent enhancements to strip the image down to its basic structure.

[0140] SVG is an XML-based format for creating two-dimensional vector graphics. SVG is composed of geometric shapes to allow SVG to scale without losing quality. This makes SVG an option for logos, icons, and illustrations that require clarity at various sizes. Additionally, SVG supports animation and interactivity, making it versatile for web use.

[0141] In some embodiments, the pipeline includes a random path discarding technique, as shown by the outline extracted by random path discarding 1015. This technique involves discarding a portion of a path from a vector image to create a vector with less detail. It is intended to simulate the incomplete or rough nature of some human drawings to prepare the model to interpret and complete such inputs.

[0142] In some embodiments, the contours extracted by random stroke width enhancement 1020 represent the application of stroke width enhancement. This involves increasing or decreasing the stroke width of some paths of the vector image. This variation in stroke width helps to mimic the natural inconsistencies found in manual drawing and makes the model more robust to variations in user input.

[0143] In some embodiments, the perturbation of the control points of some paths in the outlines extracted by random control point perturbation 1025 in the vector image creates a distortion effect. This enhancement helps the model generalize to user scribbles that may have distorted or unconventional shapes to enhance the model's ability to understand and interpret a variety of artistic styles.

[0144] In some embodiments, the contours extracted by random path translation and rotation 1030 reflect the technique of translating and rotating some paths of a vector image. This step is intended to simulate the dynamic and often overlapping nature of user sketches, where strokes are not always perfectly aligned or isolated.

[0145] In some embodiments, Fig.10 The bottom row of presents outlines with different levels of enhancement controlled by class adjustments, demonstrating the different degrees to which the original vector image can be altered. These levels (labeled Enhancement = 0, 1, 2, 3, respectively) represent the degree of incremental modification from the original version, providing a range of fidelity and complexity. This aspect of the figure illustrates the system's ability to cater to different levels of detail and abstraction to allow the user to control the fidelity of the generated scribble to the input during training and inference, thereby providing a customized general image generation method. Each level reflects a deeper degree of enhancement, demonstrating how class adjustments can fine-tune the output to meet specific style or structural preferences. Therefore, embodiments of the present disclosure provide flexibility in controlling the fidelity of the result of the input scribble, which allows the user to generate variations and provide choices for specific use cases.

[0146] Fig.11 An example of a database creation method for training an image generation model according to various aspects of the present disclosure is shown. The method for training an image generation model is referenced Figure 2-Figure 4 , Figure 6-Figure 10 and Fig.13 Examples of corresponding elements are described or include aspects thereof.

[0147] According to some embodiments, the training of machine learning models with the ability to interpret user sketches depends on obtaining extensive, high-quality training datasets. Directly obtaining real user sketches presents inherent challenges, primarily due to the large amount of manual input involved and potential deficiencies in capturing the diverse incorporation required for robust training of the model. Given that real sketches always embody human intent, spontaneity, and variability, a derived training dataset designed to reflect these characteristics must thoroughly and faithfully represent these inherent characteristics. To address this challenge, embodiments of the present disclosure generate new training data that simulates user sketches.

[0148] In some embodiments, a holistic nested edge detection (HED) algorithm is used. HED generates multi-resolution side outputs and can therefore expertly resolve edges at multiple scales. In the image generation model, each layer is tailored to detect edge information at different granularities, from capturing extended forms to complex details. Initial layers emphasize fine details and sharp edges, while deeper layers are suited to identifying overall structures and patterns. Notably, HED combines the side outputs and synthesizes them into a single detailed and comprehensive edge map.

[0149] According to some embodiments, a randomization mechanism is implemented when processing images via HED. Implementing a randomization mechanism can enhance the integrity of the dataset compared to deterministic methods that only use specific layer outputs or avoid combining outputs. By randomly selecting from side outputs or merged outputs, it is ensured that the dataset summarizes a large number of edge definitions. This approach enhances the richness of the dataset and increases its resistance to potential biases, making the training method consistent with the multifaceted nature of user sketches.

[0150] In some embodiments, a novel pipeline is provided to generate data samples for training, and a method is provided to control the fidelity of the generated images to the input scribbles. The method includes randomly selecting from the side outputs of holistic nested edge detection (HED) applied to the input image to generate data with different levels of detail to enhance the generalization ability of the model.

[0151] In some embodiments, a method is provided to control the fidelity of results generated at inference time to an input scrawl. The provided method for controlling adapter strength is not limited to ControlNet, but is applicable to adapter-based configurations. Thus, embodiments of the present disclosure provide a general method for combining fine-grained control with user-provided hints.

[0152] After applying the HED algorithm, the dataset is refined via various augmentation techniques. A random pixel omission technique is invoked to periodically remove pixels to simulate sporadic discontinuities inherent in sketches. To foster spatial invariance in the model, random spatial translations and rotations across different angles are incorporated. To generalize subtle differences in line softness and resolution, Gaussian blur is integrated. Overall, these methods enhance the resilience of the training data, ensuring an inclusive and representative training environment for the model. The output will be conditioned to the predefined specifications of the model. For example, the output is conditioned to a resolution of 128×128.

[0153] refer to Fig.11 , a database creation method is shown to enhance the ability of machine learning models to interpret user sketches by generating a comprehensive and diverse training dataset. Fig.11 The use of a holistic nested edge detection (HED) algorithm to produce multi-resolution side outputs is shown to capture the complexity and variability inherent in real user sketches.

[0154] In some embodiments, the original image 1105 is processed using a HED algorithm to generate multiple side outputs, each representing a different level of detail and abstraction. These side outputs include a first side output 1110, a second side output 1115, a third side output 1120, a fourth side output 1125, and a fifth side output 1130. Each layer in the HED algorithm is tailored to detect edge information of a specific granularity, where the initial layers focus on fine details and sharp edges, and the deeper layers are suitable for recognizing broader structures and patterns.

[0155] In some embodiments, a randomization mechanism is implemented when processing images through HED. This approach ensures that a wide range of edge definitions are captured, enhancing the richness and robustness of the dataset. By randomly selecting from the side outputs or the combined fusion outputs, the dataset is diversified to make it more representative of the multifaceted nature of the user's sketches.

[0156] In some embodiments, the fused output 1135 represents the merging of multiple side outputs into a single comprehensive edge map. This fusion synthesizes different levels of detail into a detailed and comprehensive representation that is closely aligned with the spontaneity and diversity of human-drawn sketches.

[0157] In some embodiments, after using the HED algorithm, the dataset is further refined via various enhancement techniques designed to simulate discontinuities, spatial variability, and nuances commonly found in hand-drawn sketches, making the model ready to handle a variety of user inputs and artistic styles.

[0158] In some embodiments, outputs from the HED algorithm, including side outputs and fused outputs, are randomly selected and resized to a predefined specification of the model, such as 128×128 resolution. This selection process combined with the subsequent resizing ensures that the training data is not only diverse and representative, but also compatible with the requirements of the model, thereby facilitating an effective and efficient learning process. Therefore, Fig.11 The database creation method in represents a complex and subtle approach for generating training data that closely matches the inherent properties of real sketches to enable the image generation model to produce results that are faithful to the user input and creatively realized.

[0159] Fig.12 An example of a method 1200 for image processing according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in combination with other operations.

[0160] At operation 1205, the system creates a training set including sketch inputs. In some cases, the operation of this step involves referencing Figure 6 The training component may be executed thereby.

[0161] For example, at operation 1205, the system creates a training set including sketch input. In some cases, this step involves compiling or generating a variety of sketches to train the sketch encoder. The sketches in the training set can be obtained from direct user input, a sketch database, or synthetically generated to represent a variety of styles, details, and complexities. According to some embodiments, this diversity ensures that the sketch encoder learns to interpret and process a wide range of artistic inputs to enhance its generalization ability and effectiveness.

[0162] At operation 1210, the system initializes the sketch encoder based on the parameters of the image generation model. In some cases, the operation of this step involves referring to Figure 6 The training component may be executed thereby.

[0163] For example, at operation 1210, the system initializes the sketch encoder based on the parameters of the image generation model. In some cases, initializing the sketch encoder involves copying the parameters of the encoder of the image generation model to ensure a level of consistency and synergy between the two components. This initialization step is critical because it sets a basic set of parameters for the sketch encoder that will guide its learning and adaptation process during training.

[0164] At operation 1215, the system trains the sketch encoder to generate sketch guidance for the image generation model based on the sketch input. In some cases, the operation of this step involves reference Figure 6 The training component may be executed thereby.

[0165] For example, at operation 1215, the system trains a sketch encoder to generate sketch guidance for an image generation model based on sketch inputs. The training involves exposing the sketch encoder to various sketch inputs in a training set to enable it to learn and improve strategies for encoding these inputs into effective sketch guidance. For example, training the sketch encoder may involve generating a synthetic image based on the output of the sketch encoder, and then calculating a loss function based on the synthetic image and the ground truth image. This process enables the system to measure the accuracy and effectiveness of the sketch encoder to guide its continued adaptation and improvement.

[0166] In some embodiments, creating the training set includes generating sketch inputs based on ground truth images by applying a series of augmentations such as white filling, black strokes, path discarding, stroke width augmentation, control point perturbations, path translation, rotation, or any combination thereof. These augmentations introduce a range of variations and challenges to the sketches to mimic the diversity in real user inputs and prepare the sketch encoder to handle a variety of artistic styles and complexities.

[0167] Fig.13 An example of a computing device 1300 according to aspects of the present disclosure is shown. The computing device 1300 includes (multiple) processors 1305, a memory subsystem 1310, a communication interface 1315, an I / O interface 1320, (multiple) user interface components 1325, and a channel 1330. In some embodiments, the computing device 1300 is a reference Figure 2-Figure 4 and Figure 6-Figure 11 Examples of image generating apparatus are described or include aspects thereof.

[0168] In some embodiments, computing device 1300 includes one or more processors 1305 that can execute instructions stored in memory subsystem 1310 to generate a composite image including a first attribute and a second attribute by providing first attribute tokens to a first set of layers of an image generation model during a first set of time steps and providing second attribute tokens to a second set of layers of the image generation model during a second set of time steps.

[0169] According to some aspects, computing device 1300 includes one or more processors 1305. Processor(s) 1305 are as described with reference to Figure 5Examples of the processor unit described herein or aspects thereof. In some cases, the processor is an intelligent hardware device, such as a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof.

[0170] In some cases, the processor is configured to operate the memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0171] According to some aspects, the memory subsystem 1310 includes one or more memory devices. The memory subsystem 1310 is as shown in FIG. Figure 5 Examples of the memory unit described herein may include aspects thereof. Examples of memory devices include random access memory (RAM), read-only memory (ROM) or hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software, including instructions that cause the processor to perform various functions described herein when executed. In some cases, among other aspects, the memory also includes a basic input / output system (BIOS), which controls basic hardware or software operations, such as interactions with peripheral components or devices. In some cases, a memory controller operates the memory unit. For example, the memory controller may include a row address decoder, a column address decoder or both. In some cases, the storage unit in the memory stores information in the form of a logical state.

[0172] According to some aspects, the communication interface 1315 operates at the boundary between the communication entities (such as the computing device 1300, one or more user devices, the cloud, and one or more databases) and the channel 1330, and can record and process the communication. In some cases, the communication interface 1315 is provided to enable the processing system to couple to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna.

[0173] According to some aspects, I / O interface 1320 is controlled by an I / O controller to manage input and output signals of computing device 1300. In some cases, I / O interface 1320 manages peripheral devices that are not integrated into computing device 1300. In some cases, I / O interface 1320 represents a physical connection or port to an external peripheral device. In some cases, an I / O controller uses an operating system, such as or other known operating systems. In some cases, the I / O controller represents or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, the I / O controller is implemented as a component of the processor. In some cases, a user interacts with the device via the I / O interface 1320 or via a hardware component controlled by the I / O controller.

[0174] According to some aspects, user interface component 1325 enables a user to interact with computing device 1300. In some cases, user interface component 1325 includes an audio device (such as an external speaker system), an external display device (such as a display screen), an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller), or a combination thereof. In some cases, user interface component 1325 includes a GUI.

[0175] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, the operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in the form of block diagrams to indicate the relationships between components and to avoid obscuring the concepts described. Similar components or features may have the same name but may have different reference numerals corresponding to different drawings.

[0176] Some modifications to the present disclosure may be clear to those skilled in the art, and the principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should be given the widest scope consistent with the principles and novel features disclosed herein.

[0177] The above method can be implemented or executed by a device including a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination thereof. The general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller or a state machine. The processor can also be implemented as a combination of computing devices (for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be performed by a processor, firmware or any combination thereof. If implemented in software executed by a processor, the functions can be stored in the form of instructions or codes on a computer-readable medium.

[0178] Computer-readable media include both non-transient computer storage media and communication media, including any media that facilitates code or data transmission. Non-transient storage media can be any available media that a computer can access. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc (CD) or other optical disc storage device, magnetic disk storage device, or any other non-transient medium for carrying or storing data or code.

[0179] In addition, the connecting component may be properly referred to as a computer-readable medium. For example, if the code or data is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio or microwave signals), the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is also included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0180] In this disclosure and the following claims, the word "or" means an inclusive list, for example, a list of X, Y or Z means X or Y or Z or XY or XZ or YZ or XYZ. In addition, the word "based on" is not used to indicate a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as "based at least in part on". In addition, the word "a" or "an" means "at least one".

Claims

1. A method for image generation, comprising: Get the sketch input that depicts the object; Processing the sketch input using a sketch encoder to obtain sketch guidance; as well as A composite image is generated based on the sketch guidance using an image generation model, wherein the composite image depicts the object from the sketch input.

2. The method according to claim 1, further comprising: Get text prompts; as well as The text cue is encoded to obtain a text encoding, wherein the composite image is generated based on the text encoding.

3. The method of claim 1 , wherein generating the composite image comprises: The sketch guidance is provided as input to a decoder layer of the image generation model.

4. The method according to claim 1, further comprising: A vector image is generated based on the composite image.

5. The method of claim 1 , wherein generating the composite image comprises: Perform the reverse diffusion process.

6. The method of claim 1 , wherein encoding the sketch input comprises: A noisy input image is obtained for an image generation model, wherein the synthesized image is based on the noisy input image.

7. The method of claim 1 , wherein encoding the sketch input comprises: A class label indicating a class of the composite image is obtained, wherein the sketch guidance is generated based on the class label.

8. The method according to claim 7, wherein: The category is selected from a set of categories including a contour category and a color image category.

9. The method of claim 1 , wherein generating the composite image comprises: A fit factor is obtained that indicates how closely the composite image fits the sketch input, wherein the composite image is generated based on the fit factor.

10. The method of claim 9, wherein generating the composite image comprises: The fitting factor is applied to the sketch guidance before providing the sketch guidance to the image generation model.

11. The method according to claim 1, wherein: The sketch encoder and the image generation model are diffusion models.

12. The method of claim 1, wherein: The sketch encoder is initialized using a trainable copy of the image generation model.

13. The method of claim 1, wherein: The sketch encoder is trained using training data comprising training sketch inputs.

14. A method for training a machine learning model, comprising: Get a training set including sketch inputs; Initialize the sketch encoder based on the parameters of the image generation model; as well as The sketch encoder is trained to generate sketch guidance for the image generation model based on the sketch input.

15. The method of claim 14, wherein training the sketch encoder comprises: generating a synthetic image based on an output of the sketch encoder using the image generation model; as well as A loss function is calculated based on the synthetic image and the ground truth image.

16. The method of claim 14, wherein creating the training set comprises: The sketch input is generated based on a ground truth image by applying white filling, black stroke, path discarding, stroke width enhancement, control point perturbation, path translation, rotation, or any combination thereof.

17. The method of claim 14, wherein initializing the sketch encoder comprises: The parameters of the encoder of the image generation model are copied.

18. An apparatus comprising: at least one processor; at least one memory storing instructions executable by the at least one processor; a sketch encoder comprising parameters stored in the at least one memory and trained to generate sketch guidance based on a sketch input; as well as An image generation model includes instructions stored in the at least one memory and is trained to generate a synthetic image based on the sketch guidance.

19. The device according to claim 18, wherein: The image generation model includes a diffusion model.

20. The apparatus of claim 18, wherein: The sketch encoder includes zero convolutional layers and layers of encoder layers based on the image generation model.

Images