Complex scene image generation system and method based on freehand sketch

Abstract

The invention belongs to the technical field of image processing, and discloses a freehand sketch-based complex scene image generation system and method, and the method comprises the steps: obtaining a sketch image, and carrying out the feature extraction of the sketch image, and obtaining the sketch features; meanwhile, analyzing the sketch based on a large language model to obtain sketch semantic text description, and performing feature extraction on the text to obtain text features; then, the sketch features are aligned with the text features, and object and background layout information contained in the sketch is obtained; and finally, based on the sketch layout and the shape contour information, performing image generation through a preset generation model to obtain a target complex scene image. According to the method, the high-quality scene image can be quickly and intelligently obtained based on the simple human hand-drawn sketch, complex prompts are not needed, and the scene image generation efficiency and the scene image generation quality are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a system and method for generating complex scene images based on hand-drawn sketches. Background Art

[0002] The continuous development of artificial intelligence technology has provided new opportunities for computer vision technology, among which the ability to automatically generate images is becoming increasingly powerful. Text-based image generation technology provides many content creators with new tools, making the creation of image content that originally required professionals and expensive equipment easier and more cost-effective. However, existing text-based image generation technology faces great challenges in generating complex scene sketches, that is, text cannot accurately express and control the model to generate the scene graph expected by the user.

[0003] With the widespread use of touchpad devices such as smartphones and tablets, hand-drawn sketches have gradually become a common tool for multimedia applications and industrial design. Hand-drawn sketches can break the language gap and help users better realize their ideas and conduct innovative designs. Compared with text, hand-drawn sketches can provide information such as object relationships, shapes, and scene layouts, and can explicitly guide scene image generation. Therefore, there is an urgent need for a method to generate complex scenes based on hand-drawn sketches that can take into account both the efficiency and quality of target scene image generation. Summary of the invention

[0004] The purpose of the present invention is to provide a complex scene image generation system and method based on hand-drawn sketches to solve the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the specific technical solutions of a complex scene image generation system and method based on hand-drawn sketches of the present invention are as follows:

[0006] A complex scene image generation system based on hand-drawn sketches includes a user terminal, a model reasoning device located at a server end, and an image generation device at the server end.

[0007] The model inference device at the server end performs real-time inference on the image generation model online;

[0008] The user terminal exchanges information with the image generating device at the server end through the network;

[0009] The image generation device at the server end provides a service interface to the user terminal through the network, and the user terminal displays the service interface to the user. The user draws a sketch through the service interface, and the user terminal sends the sketch drawn by the user to the image generation device at the server end through the network. The image generation device uses the above-mentioned pre-obtained image generation model, under the guidance of the input hand-drawn sketch, to generate a result image, and sends the generated result image to the user terminal through the network for display by the user terminal. The present invention also discloses a complex scene image generation method using a complex scene image generation system based on hand-drawn sketches, and performs sketch recognition through SketchLLM, a complex scene image generation framework based on hand-drawn sketches, including the following steps:

[0010] Step 1: Get a hand-drawn sketch image;

[0011] Step 2: Analyze the sketch based on the large language model to obtain the text features of the overall scene and local objects;

[0012] Step 3: Align the sketch with the text features to obtain the scene layout and object morphology information;

[0013] Step 4: Input the scene layout and object morphology information into the generative model to obtain the target complex scene image.

[0014] Furthermore, the hand-drawn sketch in step 1 is a user's description of the image to be generated using simple strokes, thereby helping the image generation device to generate the image content described by the prompt text.

[0015] Furthermore, step 1 includes the image generation device at the server end providing a service interface to the user terminal, the user terminal displays the service interface to the user, and the user inputs the sketch through the service interface, wherein the service interface includes a component for inputting the sketch.

[0016] Furthermore, the large language model based on step 2 includes a sketch parsing module, an alignment module and a generation model, and step 2 is performed using the sketch parsing module, including the following steps:

[0017] The hand-drawn sketch is input into the multimodal large language model, and the preset template is used to query the large language model to analyze the sketch to obtain the objects, scene information and positional relationships contained in the sketch, and the local object information drawn in the sketch is obtained, including simple noun descriptions of multiple objects and backgrounds; then, content expansion is performed to expand the complex description of objects and backgrounds based on the prior knowledge of the large model; finally, scene synthesis is performed, and the object and background descriptions are collectively referred to as the overall scene prompt text as the input of the generation model.

[0018] Furthermore, the preset template includes task introduction, detailed description and context learning.

[0019] Furthermore, the step 3 is performed using the alignment module, the prompt text and sketch image features generated by the sketch parsing module are input into the CLIP model, the attention distribution map is calculated, the sketch significant response area corresponding to the text is obtained, and the scene object layout information is obtained. The calculation formula is as follows:

[0020]

[0021] Among them, f(I i ) represents the embedded representation of the image, represents the embedding representation of the text, and τ represents the learnable temperature coefficient.

[0022] Furthermore, the generation model of step 4 includes a denoising network, a control network and a decoding network. The scene layout and object morphology information obtained by the alignment module in step 3 are respectively input into the denoising network and the control network, the noisy image embedding representation is initialized, the noise is predicted by the denoising network, the scene layout is controlled, the object morphology is regulated by the control network, the edge contour morphology of the generated image is controlled, the final latent features in the latent space are obtained, and the final result image is obtained by decoding through the decoding network.

[0023] The present invention also discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the complex scene image generation method when executing the computer program.

[0024] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the complex scene image generation method are implemented.

[0025] The complex scene image generation system and method based on hand-drawn sketches of the present invention have the following advantages: The complex scene image generation system and method based on hand-drawn sketches of the present invention fully utilize the image parsing and information expansion capabilities of the multimodal large language model, and parse the sparse complex hand-drawn scene sketches through the sketch parsing module to obtain object and scene level information. The sketch image and text information are aligned through the alignment module to obtain the scene layout and object morphology information contained in the scene sketch. The above information is input into the image generation model to realize the complex scene image generation with sketch semantic perception, which better assists designers. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a system architecture diagram applicable to the embodiment of the present invention;

[0027] Figure 2is a flow chart of a method for generating a complex scene image based on a hand-drawn sketch provided by an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of the composition of the model architecture provided by an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of the composition of a sketch parsing module in an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of the structure of an information alignment module in an embodiment of the present invention;

[0031] Figure 6 is a structural schematic diagram of an image generation model in an embodiment of the present invention;

[0032] Figure 7 Schematic diagram of the structure of the device in the embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a complex scene image generation system and method based on hand-drawn sketches of the present invention in conjunction with the accompanying drawings.

[0034] Figure 1 An exemplary system architecture to which the present invention can be applied is shown. Figure 1 As shown in , the system architecture may include: a user terminal, and a model reasoning device and an image generating device located at the server end.

[0035] Among them, the model inference device on the server side can adopt the method provided by the embodiment of the present invention to perform real-time inference on the image generation model online.

[0036] The user terminal can exchange information with the image generating device at the server end through the network.

[0037] As one of the achievable methods, the image generation device at the server end can provide a service interface to the user terminal through the network, and the user terminal displays the service interface to the user. The user can draw a sketch through the service interface. The user terminal sends the sketch drawn by the user to the image generation device at the server end through the network. The image generation device can use the above-mentioned pre-obtained image generation model, under the guidance of the input hand-drawn sketch, to generate a result image, and send the generated result image to the user terminal through the network for display by the user terminal.

[0038] Figure 1 The number of user terminals, image generation devices, and image generation models in the embodiment is only for illustration. Any number of user terminals, image generation devices, and image generation models may be provided according to implementation requirements.

[0039] Figure 2 A complex scene image generation method based on hand-drawn sketches provided in an embodiment of the present application performs sketch recognition through a complex scene image generation framework SketchLLM based on hand-drawn sketches, specifically comprising the following steps:

[0040] Step 1: Get a hand-drawn sketch image

[0041] First, step 1, namely, “obtaining a hand-drawn sketch image”, is described in detail in conjunction with the embodiment.

[0042] The hand-drawn sketch is a user's description of the image he wants to generate using simple strokes, thereby helping the image generation device to generate the image content described by the prompt text.

[0043] As one of the achievable methods, the image generation device at the server end may provide a service interface to the user terminal, and the user terminal displays the service interface to the user. The user may input a sketch through the service interface. The service interface includes a component for inputting a sketch. For example, the service interface may be in the form of a canvas or a specific page. The user may draw and input a sketch in the canvas area for drawing sketches on the service interface, or may browse a file to upload a drawn sketch.

[0044] Step 2: Analyze the sketch based on the large language model to obtain the text features of the overall scene and local objects

[0045] The following is a detailed description of step 2 above, i.e., "analyzing the sketch based on a large language model to obtain text features of the overall scene and local objects" with reference to an example.

[0046] For ease of understanding, the model architecture of this application is first described. Figure 3 As shown in FIG, the model architecture mainly includes a sketch parsing module, an alignment module and a generation model. This step 2 is performed using the sketch parsing module.

[0047] like Figure 4 As shown in the figure, the hand-drawn sketch is input into the multimodal large language model, and a preset template is used (the preset template includes task introduction, detailed description and context learning). The large language model is asked to analyze the sketch to obtain the objects, scene information and positional relationship between objects contained in the sketch, and the local object information drawn in the sketch is obtained, including simple noun descriptions of multiple objects and backgrounds; then, content expansion is performed, and the complex description of objects and backgrounds is expanded according to the prior knowledge of the large model; finally, scene synthesis is performed, and the description of objects and background is collectively referred to as the overall scene prompt text as the input of the generation model.

[0048] Step 3: Align the sketch with the text features to obtain scene layout and object morphology information

[0049] The following describes in detail the above step 3, i.e., "aligning the sketch with the text features to obtain the scene layout and object morphology information", with reference to an example.

[0050] like Figure 3 As shown, step 3 is performed using the alignment module.

[0051] like Figure 5 As shown in the figure, the prompt text and sketch image features generated by the sketch parsing module in step 3 above are input into the CLIP model, and the attention distribution map is calculated to obtain the sketch salient response area corresponding to the text and the scene object layout information. The calculation formula is as follows:

[0052]

[0053] Among them, f(I i ) represents the embedded representation of the image, represents the embedding representation of the text, and τ represents the learnable temperature coefficient.

[0054] Step 4: Input the scene layout and object morphology information into the generative model to obtain the target complex scene image

[0055] The above step 4, i.e., “inputting the scene layout and object morphology information into the generative model to obtain the target complex scene”, is described in detail below with reference to an example.

[0056] like Figure 6 As shown in the figure, the generation model includes a denoising network, a control network and a decoding network. The scene layout and object morphology information obtained by the alignment module in step 3 are input into the denoising network and the control network respectively. The noisy image is initialized to embed the representation, the noise is predicted by the denoising network, the scene layout is controlled, the object morphology is regulated by the control network, the edge contour morphology of the generated image is controlled, and the final latent features in the latent space are obtained. The final result image is obtained by decoding through the decoding network.

[0057] Corresponding to the aforementioned embodiment of the complex scene image generation method based on hand-drawn sketches, the present invention also provides embodiments of a complex scene image generation device and storage medium based on hand-drawn sketches.

[0058] See also Figure 7 The complex scene image generation method, device and storage medium based on hand-drawn sketches provided in an embodiment of the present invention include a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the complex scene image generation method based on hand-drawn sketches in the above embodiment.

[0059] The embodiments of the complex scene image generation method, device and storage medium based on hand-drawn sketches of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where the complex scene image generation method, device and storage medium based on hand-drawn sketches of the present invention are located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0060] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0061] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0062] The embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, a complex scene image generation method based on a hand-drawn sketch is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (FlashCard), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0063] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A complex scene image generation system based on hand-drawn sketches, comprising a user terminal, a model inference device located at a server, and an image generation device at the server, characterized in that: The model inference device at the server end performs real-time inference on the image generation model online; The user terminal exchanges information with the image generating device at the server end through the network; The image generating device at the server end provides a service interface to the user terminal via the network, and the user terminal displays the service interface to the user. The user draws a sketch via the service interface, and the user terminal sends the sketch drawn by the user to the image generating device at the server end via the network. The image generating device uses the above-mentioned pre-obtained image generation model to generate a result image under the guidance of the input hand-drawn sketch, and sends the generated result image to the user terminal via the network for display by the user terminal.

2. A complex scene image generation method using the complex scene image generation system based on hand-drawn sketches according to claim 1, performing sketch recognition through a complex scene image generation framework based on hand-drawn sketches SketchLLM, characterized in that: The steps include: Step 1: Get a hand-drawn sketch image; Step 2: Analyze the sketch based on the large language model to obtain the text features of the overall scene and local objects; Step 3: Align the sketch with the text features to obtain the scene layout and object morphology information; Step 4: Input the scene layout and object morphology information into the generative model to obtain the target complex scene image.

3. The complex scene image generation method according to claim 2, characterized in that: The hand-drawn sketch in step 1 is a user's description of the image he wants to generate using simple strokes, thereby helping the image generation device to generate the image content described by the prompt text.

4. The complex scene image generation method according to claim 2, characterized in that: Step 1 includes an image generating device at the server end providing a service interface to a user terminal, the user terminal presenting the service interface to a user, and the user inputting a sketch through the service interface, wherein the service interface includes a component for inputting a sketch.

5. The complex scene image generation method according to claim 2, characterized in that: The step 2 based on the large language model includes a sketch parsing module, an alignment module and a generation model. The step 2 is performed using the sketch parsing module and includes the following steps: The hand-drawn sketch is input into the multimodal large language model, and the preset template is used to query the large language model to analyze the sketch to obtain the objects, scene information and positional relationships contained in the sketch, and the local object information drawn in the sketch is obtained, including simple noun descriptions of multiple objects and backgrounds; then, content expansion is performed to expand the complex description of objects and backgrounds based on the prior knowledge of the large model; finally, scene synthesis is performed, and the object and background descriptions are collectively referred to as the overall scene prompt text as the input of the generation model.

6. The complex scene image generation method according to claim 5, characterized in that: The preset template includes task introduction, detailed description and context learning.

7. The complex scene image generation method according to claim 2, characterized in that: The step 3 is performed by the alignment module, and the prompt text and sketch image features generated by the sketch parsing module are input into the CLIP model to calculate the attention distribution map, obtain the sketch significant response area corresponding to the text, and obtain the scene object layout information. The calculation formula is as follows: map(i,j)=τ·f(I i )·g(T j ) T , Among them, f(I i ) represents the embedded representation of the image, g(T j ) T represents the embedding representation of the text, and τ represents the learnable temperature coefficient.

8. The complex scene image generation method according to claim 2, characterized in that: The generation model of step 4 includes a denoising network, a control network and a decoding network. The scene layout and object morphology information obtained by the alignment module in step 3 are respectively input into the denoising network and the control network, the noisy image embedding representation is initialized, the noise is predicted by the denoising network, the scene layout is controlled, the object morphology is regulated by the control network, the edge contour morphology of the generated image is controlled, the final latent features in the latent space are obtained, and the final result image is obtained by decoding through the decoding network.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the complex scene image generation method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the complex scene image generation method according to any one of claims 1 to 8 are implemented.

Images