A hand-drawn sketch-based complex scene image generation system and method

Abstract

The application belongs to the technical field of image processing, and discloses a complex scene image generation system and method based on hand-drawn sketches, which comprises the following steps: obtaining a sketch image, performing feature extraction on the sketch image, and obtaining sketch features; meanwhile, analyzing the sketch based on a large language model to obtain a sketch semantic text description, performing feature extraction on the text to obtain text features; then, aligning the sketch features and the text features to obtain object and background layout information contained in the sketch; finally, based on the sketch layout and shape contour information, performing image generation through a preset generation model to obtain a target complex scene image. The application can quickly and intelligently obtain a high-quality scene image based on a simple human hand-drawn sketch, without complex prompts, effectively improving the scene image generation efficiency and the scene image generation quality.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a system and method for generating complex scene images based on hand-drawn sketches. Background Technology

[0002] The continuous development of artificial intelligence technology has provided new opportunities for computer vision technology, with its ability to automatically generate images becoming increasingly powerful. Text-based image generation technology has provided many content creators with entirely new tools, making the creation of image content, which previously required professionals and expensive equipment, easier and more cost-effective. However, existing text-based image generation technologies face significant challenges in generating sketches of complex scenes, namely, that text cannot accurately express and control the model to generate the scene diagrams that users expect.

[0003] With the widespread use of touchpad devices such as smartphones and tablets, hand-drawn sketches have gradually become a common tool in multimedia applications and industrial design. Hand-drawn sketches can bridge language barriers, helping users better realize their ideas and create innovative designs. Compared to text, hand-drawn sketches can provide information such as object relationships, shapes, and scene layouts, explicitly guiding scene image generation. Therefore, there is an urgent need for a method to generate complex scenes based on hand-drawn sketches, which can balance the efficiency and quality of target scene image generation. Summary of the Invention

[0004] The purpose of this invention is to provide a system and method for generating complex scene images based on hand-drawn sketches, so as to solve the above-mentioned technical problems.

[0005] To address the aforementioned technical problems, the specific technical solution of the complex scene image generation system and method based on hand-drawn sketches of the present invention is as follows:

[0006] A complex scene image generation system based on hand-drawn sketches includes a user terminal, a model inference device on the server side, and an image generation device on the server side.

[0007] The server-side model inference device performs real-time inference on the image generation model online.

[0008] The user terminal interacts with the image generation device on the server via a network.

[0009] The server-side image generation device provides a service interface to the user terminal via a network. The user terminal displays this service interface to the user, who then draws a sketch. The user terminal sends the sketch to the server-side image generation device via the network. The image generation device, using the pre-obtained image generation model and guided by the input hand-drawn sketch, generates a result image and sends it to the user terminal via the network for display. This invention also discloses a method for generating complex scene images using a complex scene image generation system based on hand-drawn sketches. This method utilizes the SketchLLM framework for complex scene image generation to perform sketch recognition, and includes the following steps:

[0010] Step 1: Obtain the hand-drawn sketch image;

[0011] Step 2: Analyze the sketch based on the large language model to obtain the text features of the overall scene and local objects;

[0012] Step 3: Align the sketch with the text features to obtain scene layout and object shape information;

[0013] Step 4: Input the scene layout and object shape information into the generation model to obtain the target complex scene image.

[0014] Furthermore, the hand-drawn sketch in step 1 is a simple sketch by the user describing the image they want to generate, thereby helping the image generation device generate the image content described by the prompt text.

[0015] Furthermore, step 1 includes the server-side image generation device providing a service interface to the user terminal, the user terminal displaying the service interface to the user, and the user inputting a sketch through the service interface, the service interface including components for inputting the sketch.

[0016] Furthermore, step 2, based on a large language model, includes a sketch parsing module, an alignment module, and a generative model. Step 2 is executed using the sketch parsing module and includes the following steps:

[0017] The hand-drawn sketch is input into a multimodal large language model. A preset template is used to query the large language model to analyze the sketch, obtaining information about the objects and scene contained in the sketch, as well as the positional relationships between the objects. This yields information about the local objects drawn in the sketch, including simple noun descriptions of multiple objects and the background. Then, the content is expanded by adding more complex descriptions of the objects and background based on the prior knowledge of the large model. Finally, scene synthesis is performed, combining the object and background descriptions into a unified scene prompt text, which serves as the input to the generation model.

[0018] Furthermore, the preset template includes a task introduction, detailed description, and contextual learning.

[0019] Furthermore, step 3 is performed using the alignment module. The prompt text and sketch image features generated by the sketch parsing module are input into the CLIP model to calculate the attention distribution map, obtain the salient response region of the sketch corresponding to the text, and obtain the scene object layout information. The calculation formula is as follows:

[0020]

[0021] Where, f(I) i ) represents the embedding representation of an image. τ represents the embedded representation of the text, and τ represents the learnable temperature coefficient.

[0022] Furthermore, the generation model in step 4 includes a denoising network, a control network, and a decoding network. The scene layout and object shape information obtained by the alignment module in step 3 are input into the denoising network and the control network respectively to initialize the noisy image embedding representation. The denoising network predicts noise and controls the scene layout. The control network regulates the object shape and controls the edge contour shape of the generated image to obtain the latent features in the final latent space. The decoding network decodes the latent features to obtain the final result image.

[0023] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the complex scene image generation method.

[0024] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the complex scene image generation method.

[0025] The present invention provides a complex scene image generation system and method based on hand-drawn sketches, which has the following advantages: The system and method fully utilize the image parsing and information augmentation capabilities of a multimodal large language model. Through a sketch parsing module, sparse and complex hand-drawn scene sketches are parsed to obtain object and scene-level information. An alignment module aligns the sketch image and text information to obtain the scene layout and object shape information contained in the scene sketch. This information is then input into an image generation model to achieve complex scene image generation with sketch semantic awareness, better assisting designers. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram applicable to the embodiments of the present invention;

[0027] Figure 2This is a flowchart of a method for generating complex scene images based on hand-drawn sketches provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the model architecture provided in the embodiments of the present invention;

[0029] Figure 4 This is a schematic diagram of the composition of the sketch parsing module in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the information alignment module in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the image generation model in an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of the device in an embodiment of the present invention. Detailed Implementation

[0033] To better understand the purpose, structure, and function of this invention, the following detailed description of a complex scene image generation system and method based on hand-drawn sketches is provided in conjunction with the accompanying drawings.

[0034] Figure 1 An exemplary system architecture to which embodiments of the present invention can be applied is shown, such as Figure 1 As shown, the system architecture may include: a user terminal, and a model inference device and an image generation device located on the server.

[0035] The server-side model inference device can use the method provided in this embodiment of the invention to perform real-time inference on the image generation model online.

[0036] User terminals can interact with the image generation device on the server via the network.

[0037] One possible approach is for the server-side image generation device to provide a service interface to the user terminal via a network, which the user terminal then displays to the user. The user can draw a sketch through this interface. The user terminal then sends the sketch to the server-side image generation device via the network. The image generation device, using the pre-existing image generation model and guided by the input sketch, generates the resulting image and sends it to the user terminal via the network for display.

[0038] Figure 1 The number of user terminals, image generation devices, and image generation models shown is merely illustrative. Depending on implementation needs, any number of user terminals, image generation devices, and image generation models can be included.

[0039] Figure 2 This application provides a method for generating complex scene images based on hand-drawn sketches. The method utilizes the SketchLLM framework for generating complex scene images based on hand-drawn sketches to perform sketch recognition. Specifically, it includes the following steps:

[0040] Step 1: Obtain the hand-drawn sketch image

[0041] First, step 1, "obtaining the hand-drawn sketch image," will be described in detail with reference to the embodiments.

[0042] Hand-drawn sketches are created by users using simple strokes to describe the image they want to generate, thus helping the image generation device to generate the image content described by the prompt text.

[0043] One possible approach is for the server-side image generation device to provide a service interface to the user terminal, which then displays this interface to the user. The user can input sketches through this service interface. The service interface includes components for inputting sketches. For example, the service interface can be in the form of a canvas or a specific page. The user can draw and input sketches on the canvas area of ​​the service interface, or browse sketches uploaded as files.

[0044] Step 2: Analyze the sketch based on a large language model to obtain the text features of the overall scene and local objects.

[0045] The following section provides a detailed description of step 2 above, namely, "analyzing the sketch based on a large language model to obtain the text features of the overall scene and local objects," using examples.

[0046] To facilitate understanding, the model architecture of this application will first be described, such as... Figure 3 As shown, the model architecture mainly includes a sketch parsing module, an alignment module, and a model generation module. Step 2 is performed using the sketch parsing module.

[0047] like Figure 4 As shown, a hand-drawn sketch is input into a multimodal large language model. Using a preset template (which includes task introduction, detailed descriptions, and contextual learning), the large language model is queried to analyze the sketch, obtaining information about the objects and scene contained in the sketch, as well as the positional relationships between objects. This yields information about the local objects drawn in the sketch, including simple noun descriptions of multiple objects and the background. Then, content expansion is performed, adding more complex descriptions of objects and the background based on the model's prior knowledge. Finally, scene synthesis is performed, combining the object and background descriptions into a unified scene prompt text, which serves as input to the generation model.

[0048] Step 3: Align the sketch with the text features to obtain scene layout and object shape information.

[0049] The following section provides a detailed description of step 3 above, namely, "aligning the sketch with text features to obtain scene layout and object shape information," using examples.

[0050] like Figure 3 As shown, step 3 is performed using the alignment module.

[0051] like Figure 5 As shown, the prompt text and sketch image features generated by the sketch parsing module in step 3 above are input into the CLIP model to calculate the attention distribution map, obtain the salient response region of the sketch corresponding to the text, and obtain the scene object layout information. The calculation formula is as follows:

[0052]

[0053] Where, f(I) i ) represents the embedding representation of an image. τ represents the embedded representation of the text, and τ represents the learnable temperature coefficient.

[0054] Step 4: Input the scene layout and object shape information into the generation model to obtain the target complex scene image.

[0055] The following section provides a detailed description of step 4 above, namely, "inputting scene layout and object shape information into the generation model to obtain the target complex scene," using examples.

[0056] like Figure 6 As shown, the generation model includes a denoising network, a control network, and a decoding network. The scene layout and object shape information obtained from the alignment module in step 3 are input into the denoising network and the control network, respectively. The noisy image embedding representation is initialized, noise is predicted by the denoising network, the scene layout is controlled, the object shape is adjusted by the control network, and the edge contour shape of the generated image is controlled to obtain the latent features in the final latent space. The decoding network is then used to decode the latent features to obtain the final result image.

[0057] Corresponding to the aforementioned embodiments of the method for generating complex scene images based on hand-drawn sketches, the present invention also provides embodiments of a device and storage medium for generating complex scene images based on hand-drawn sketches.

[0058] See Figure 7 The present invention provides a method, apparatus and storage medium for generating complex scene images based on hand-drawn sketches, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for generating complex scene images based on hand-drawn sketches in the above embodiments.

[0059] The embodiments of the present invention, including the method, apparatus, and storage medium for generating complex scene images based on hand-drawn sketches, can be applied to any device with data processing capabilities, such as a computer. The apparatus embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logically defined device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the method, apparatus, and storage medium for generating complex scene images based on hand-drawn sketches according to the present invention. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0060] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0061] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0062] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for generating complex scene images based on hand-drawn sketches. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data-processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0063] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for generating complex scene images based on a hand-drawn sketch-based complex scene image generation system, comprising a user terminal, a model inference device on the server side, and an image generation device on the server side. The server-side model inference device performs real-time inference on the image generation model online. The user terminal interacts with the image generation device on the server via a network. The image generation device on the server provides a service interface to the user terminal via the network. The user terminal displays the service interface to the user, who draws a sketch through the service interface. The user terminal sends the sketch drawn by the user to the image generation device on the server via the network. The image generation device uses a pre-obtained image generation model and, guided by the input hand-drawn sketch, generates the result image and sends the generated result image to the user terminal via the network for the user terminal to display. Its features are, The method for generating complex scene images includes the following steps: Step 1: Obtain the hand-drawn sketch image; Step 2: Analyze the sketch based on the large language model to obtain the text features of the overall scene and local objects; Step 2, based on a large language model, includes a sketch parsing module, an alignment module, and a generative model. Step 2 is executed using the sketch parsing module and includes the following steps: The hand-drawn sketch is input into a multimodal large language model. A preset template is used to query the large language model to analyze the sketch, obtaining information about the objects and scene contained in the sketch, as well as the positional relationships between the objects. This yields information about the local objects drawn in the sketch, including simple noun descriptions of multiple objects and the background. Then, the content is expanded by adding more complex descriptions of the objects and background based on the prior knowledge of the large model. Finally, scene synthesis is performed, combining the object and background descriptions into a unified scene prompt text, which serves as the input to the generation model. Step 3: Align the sketch with the text features to obtain scene layout and object shape information; Step 3 is performed using the alignment module. The prompt text and sketch image features generated by the sketch parsing module are input into the CLIP model to calculate the attention distribution map, obtain the salient response region of the sketch corresponding to the text, and obtain the scene object layout information. The calculation formula is as follows: ， in, Representing the embedding representation of an image, This represents the embedded representation of text. This represents the learnable temperature coefficient; Step 4: Input the scene layout and object shape information into the generation model to obtain the target complex scene image; The generation model in step 4 includes a denoising network, a control network, and a decoding network. The scene layout and object shape information obtained by the alignment module in step 3 are input into the denoising network and the control network respectively to initialize the noisy image embedding representation. The denoising network predicts noise and controls the scene layout. The control network regulates the object shape and controls the edge contour shape of the generated image to obtain the latent features in the final latent space. The decoding network decodes the latent features to obtain the final result image.

2. The method for generating complex scene images according to claim 1, characterized in that, The hand-drawn sketch in step 1 is a simple sketch by the user to describe the image they want to generate, thereby helping the image generation device to generate the image content described by the prompt text.

3. The method for generating complex scene images according to claim 1, characterized in that, Step 1 includes the server-side image generation device providing a service interface to the user terminal, the user terminal displaying the service interface to the user, and the user inputting a sketch through the service interface, which includes components for inputting the sketch.

4. The method for generating complex scene images according to claim 1, characterized in that, The preset template includes a task introduction, detailed description, and contextual learning.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the complex scene image generation method as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the complex scene image generation method as described in any one of claims 1 to 4.

Images