Image cartoonization generation method and device, and electronic equipment
By introducing two linear layers into the U-Net network of the stable diffusion model and combining it with a specific training strategy, the problems of high training cost and model complexity in traditional methods in cartoon style conversion are solved, and an efficient and flexible portrait cartoon style is achieved. Transformation that captures finer cartoon style features.
Patent Information
- Application Number
- CN202411364650.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-24
AI Technical Summary
Traditional image style conversion methods require a large amount of training data and complex model structures in the conversion of specific styles (such as cartoon style), resulting in high training costs, many model parameters, and difficulty in deploying.
The K and V part of the intersection layer of the stable diffusion model of the stable diffusion model introduced two layers of linear layers in the K and V part of the force layer, and combined with specific training strategies to obtain integrated stable diffusion models to achieve high-efficiency and flexible portrait cartoon style conversion.
By reducing the number of new parameters, it can capture more fine cartoon style characteristics, reduce training costs and deployment difficulty, and make the cartoon style conversion model more practical.
Smart Images
Figure CN119359529A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of deep learning technology, and more specifically, to a method, device, and electronic device for generating cartoon images. Background Art
[0002] With the development of deep learning technology, image style transfer has become one of the research hotspots in the field of computer vision. As an advanced image generation model, the Stable Diffusion model has shown great potential in image generation, editing, and style transfer due to its excellent generation quality and flexibility. However, in the conversion of specific styles (such as cartoon style), traditional methods often require a large amount of training data and complex model structures, resulting in high training costs, many model parameters, and difficulty in deployment. Summary of the invention
[0003] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide an image cartoonization generation method, device, and electronic device, which achieves efficient and flexible portrait cartoon style conversion by introducing two linear layers in the K (key) and V (value) parts of the cross-attention layer in the U-Net network of the stable diffusion model, and combining with a specific training strategy.
[0004] In a first aspect, an embodiment of the present application provides a method for generating a cartoon image, comprising the following steps:
[0005] Enter the image to be converted and the text description;
[0006] Add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model; and
[0007] The image to be converted is cartoonized by using the integrated stable diffusion model and combining the text description.
[0008] Further, after two linear layers are added to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain the integrated stable diffusion model, the method further includes:
[0009] Get a training dataset containing cartoon-style images; and
[0010] The integrated stable diffusion model is trained using the training data set.
[0011] Furthermore, before using the training data set to train the integrated stable diffusion model, the method further includes:
[0012] All parameters except the two linear layers in the integrated stable diffusion model are frozen.
[0013] Further, after the integrated stable diffusion model is trained using the training data set, the method further includes:
[0014] The integrated stable diffusion model that has completed training is tested.
[0015] Furthermore, after obtaining the training data set containing cartoon-style pictures, the method further includes:
[0016] The training data set is processed using data enhancement technology.
[0017] Furthermore, after obtaining the training data set containing cartoon-style pictures, the method further includes:
[0018] The training data set is processed using a regularization technique.
[0019] Furthermore, the number of the cartoon-style pictures is less than 1,000.
[0020] In a second aspect, the embodiment of the present application further provides an image cartoon generation device, comprising:
[0021] A video acquisition module, used to acquire a video to be processed containing a virtual digital human;
[0022] A driving result generating module, used for adjusting the mouth area of the virtual digital human in the video to be processed by strengthening the face driving model of lip shape supervision, and generating a lip shape driving result; and
[0023] The video generation module is used to render the video to be processed according to the lip-driven result to generate a broadcast video.
[0024] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the image cartoonization generation method according to the first aspect when executing the program.
[0025] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the image cartoonization generation method according to the first aspect above.
[0026] The embodiments of the present application bring the following beneficial effects:
[0027] In the image cartoon generation method provided in the embodiment of the present application, after the image to be converted and the text description are input, two linear layers are respectively added to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model, and the image to be converted is processed into an image cartoon by combining the integrated stable diffusion model with the text description. Therefore, the image cartoon generation method provided in the embodiment of the present application can reduce the number of newly added parameters in the process of generating cartoon images from text descriptions, and can capture more refined cartoon style features. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.
[0029] Figure 1 A schematic diagram of a process for generating an image cartoon provided in an embodiment of the present application;
[0030] Figure 2 A structural block diagram of an image cartoonization generation device provided in an embodiment of the present application;
[0031] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0032] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments described in the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0034] In the specification and claims of this application and the above-mentioned drawings, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "multiple" means two or more. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.
[0035] Figure 1 FIG. 1 is a flow chart of a method for generating an image cartoon according to an embodiment of the present application. Figure 1 As shown, the image cartoon generation method of the embodiment of the present application includes the following steps:
[0036] S101: Input the image to be converted and the text description;
[0037] Specifically, the image to be converted is usually a real shot image, including portraits and landscapes, etc. The text description contains specific requirements for the cartoon style of the image, such as color, line thickness, degree of exaggeration of expression, etc.
[0038] S102: adding two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model, respectively, to obtain an integrated stable diffusion model; and
[0039] Specifically, the U-Net network of the stable diffusion model consists of two parts: an encoder and a decoder, which extract and restore image features through downsampling and upsampling processes. In the U-Net network, the CrossAttention Layer is one of the key components to improve the performance of the model. Here, K and V represent Key and Value, respectively.
[0040] In the attention mechanism, Key is the benchmark for comparison, which contains information used to calculate the similarity between Query and Key. In the cross-attention layer, Key (K) extracts the key features of the input data, which are used to measure the similarity between Query and Key, thereby deciding which information to pay more attention to. Value is the actual information carrier in the attention mechanism, which contains the information content that the model needs to pay attention to. In the cross-attention layer, Value (V) provides rich feature information, which is assigned different weights according to the similarity between Query and Key, so as to achieve effective extraction and utilization of information.
[0041] In the U-Net network of the Stable Diffusion model, the K and V parts of the cross attention layer are expanded to add a small two-layer linear layer respectively. The trained two-layer linear layer is added to the original K and V inputs in a similar way to residual connections to enhance the expressiveness of the model in cartoon style conversion. These linear layers are used to capture more detailed cartoon style features while keeping the overall lightweight of the model to reduce the number of new parameters and avoid excessive impact on the overall performance of the model.
[0042] S103: Cartoonizing the image to be converted by using the integrated stable diffusion model and combining it with the text description.
[0043] As mentioned above, the integrated stable diffusion model expands the K and V parts of the cross attention layer in the U-Net network of the Stable Diffusion model by adding a small two-layer linear layer respectively. Therefore, in the process of generating cartoon images from text descriptions, the number of new parameters can be reduced while capturing more refined cartoon style features.
[0044] Therefore, in the image cartoon generation method provided in the embodiment of the present application, after the image to be converted and the text description are input, two linear layers are respectively added to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model, and the image to be converted is cartoonized by combining the integrated stable diffusion model with the text description. Therefore, the image cartoon generation method provided in the embodiment of the present application can reduce the number of newly added parameters in the process of generating cartoon images from text descriptions, and can capture more refined cartoon style features.
[0045] Further, in some embodiments of the present application, after two linear layers are respectively added to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain the integrated stable diffusion model, the method further includes:
[0046] Get a training dataset containing cartoon-style images; and
[0047] The integrated stable diffusion model is trained using the training data set.
[0048] Specifically, in order to improve the application effect of the integrated stable diffusion model, it needs to be trained, that is, firstly a training data set containing cartoon-style pictures is obtained, and then the integrated stable diffusion model is trained using the training data set, thereby improving the accuracy and generation effect of the integrated stable diffusion model.
[0049] Furthermore, in some embodiments of the present application, before using the training data set to train the integrated stable diffusion model, the method further includes:
[0050] All parameters except the two linear layers in the integrated stable diffusion model are frozen.
[0051] Specifically, all parameters except the two newly added linear layers in the U-Net network are frozen, and only the newly added linear layers are trained. This strategy can speed up the training process and reduce the demand for computing resources. The trained two linear layers are added to the original K and V inputs, and a similar residual connection method is used to enhance the expressiveness of the model in cartoon style conversion.
[0052] Furthermore, in some embodiments of the present application, after the integrated stable diffusion model is trained using the training data set, the method further includes:
[0053] The integrated stable diffusion model that has completed training is tested.
[0054] That is, after the integrated stable diffusion model is trained, the trained model is applied to unseen cartoon-style pictures to verify its effect and generalization ability in cartoon-style conversion, thereby further improving the application effect of the integrated stable diffusion model.
[0055] Furthermore, in some embodiments of the present application, after obtaining the training data set containing cartoon-style pictures, the method further includes:
[0056] The training data set is processed using data enhancement technology.
[0057] Specifically, data augmentation technology is a method of generating new training samples by transforming and processing existing data, aiming to increase the diversity and quantity of data sets. The image cartoon generation method provided in the embodiment of the present application can increase data diversity and reduce the risk of overfitting through data augmentation technology (such as random cropping, rotation, color transformation, etc.).
[0058] Furthermore, in some embodiments of the present application, after obtaining the training data set containing cartoon-style pictures, the method further includes:
[0059] The training data set is processed using a regularization technique.
[0060] Specifically, regularization technology is a general term for a class of methods that introduce additional information into the original model at this time in order to prevent overfitting and improve the generalization performance of the model. Therefore, the image cartoon generation method provided in the embodiment of the present application can increase data diversity, reduce the risk of overfitting, and improve the generalization ability of the model through regularization technology.
[0061] Furthermore, in some embodiments of the present application, the number of the cartoon-style pictures is less than 1,000.
[0062] That is, the image cartoon generation method provided in the embodiment of the present application uses a small number (e.g., less than 1,000) but representative cartoon-style pictures as a training data set to complete the training of the model. Since only a small number of new parameters are trained, the image cartoon generation method provided in the embodiment of the present application can significantly reduce the training cost and deployment difficulty, making the cartoon style conversion model more practical.
[0063] Figure 2 2 is a structural block diagram of an image cartoon generation device 200 provided in an embodiment of the present application. Figure 2 As shown, the image cartoon generation device 200 of the embodiment of the present application includes: a video acquisition module 210, a drive result generation module 220 and a video generation module 230, wherein:
[0064] The video acquisition module 210 is used to acquire the video to be processed containing the virtual digital human;
[0065] A driving result generating module 220 is used to adjust the mouth area of the virtual digital human in the video to be processed by strengthening the face driving model of lip shape supervision to generate a lip shape driving result; and
[0066] The video generation module 230 is used to render the video to be processed according to the lip-activated result to generate a broadcast video.
[0067] In the image cartoon generation device provided in the embodiment of the present application, after the image to be converted and the text description are input, two linear layers are respectively added to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model, and the image to be converted is processed into an image cartoon by combining the integrated stable diffusion model with the text description. Therefore, the image cartoon generation method provided in the embodiment of the present application can reduce the number of newly added parameters in the process of generating cartoon images from text descriptions, and can capture more refined cartoon style features.
[0068] It should be noted that the specific implementation of the image cartoonization generation device of the embodiment of the present application is similar to the specific implementation of the image cartoonization generation method of the embodiment of the present application. Please refer to the description of the method part for details, and no further details will be given here.
[0069] Figure 3 Schematic diagram of the structure of an electronic device 300 according to an embodiment of the present application.
[0070] like Figure 3As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage part 302 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0071] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.
[0072] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, the above-mentioned functions defined in the electronic device of the present application are executed.
[0073] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0074] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction-executing electronic device, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction-executing electronic device, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0075] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based electronic device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0076] The units or modules involved in the embodiments described in this application may be implemented by software or hardware. The units or modules described may also be set in a processor, and the processor is used to implement the image cartoon generation method when executing the program:
[0077] Enter the image to be converted and the text description;
[0078] Add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model; and
[0079] The image to be converted is cartoonized by using the integrated stable diffusion model and combining the text description.
[0080] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the image cartoon generation method described in the present application:
[0081] Enter the image to be converted and the text description;
[0082] Add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model; and
[0083] The image to be converted is cartoonized by using the integrated stable diffusion model and combining the text description.
[0084] As another aspect, the present application further provides a computer program product, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the image cartoon generation method described in the present application:
[0085] Enter the image to be converted and the text description;
[0086] Add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model; and
[0087] The image to be converted is cartoonized by using the integrated stable diffusion model and combining the text description.
[0088] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structural changes made by using the contents of the present application specification and drawings under the application concept of the present application, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A method for generating cartoon images, characterized in that: The following steps are involved: Enter the image to be converted and the text description; Add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain an integrated stable diffusion model; and The image to be converted is cartoonized by using the integrated stable diffusion model and combining the text description.
2. The image cartoon generation method according to claim 1, characterized in that: After adding two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model to obtain the integrated stable diffusion model, the method further includes: Get a training dataset containing cartoon-style images; and The integrated stable diffusion model is trained using the training data set.
3. The image cartoon generation method according to claim 2, characterized in that: Before using the training data set to train the integrated stable diffusion model, the method further includes: All parameters except the two linear layers in the integrated stable diffusion model are frozen.
4. The image cartoon generation method according to claim 2 or 3, characterized in that: After the integrated stable diffusion model is trained using the training data set, the method further includes: The integrated stable diffusion model that has completed training is tested.
5. The method for generating an image cartoon according to claim 2, characterized in that: After obtaining the training data set containing cartoon-style pictures, the method further includes: The training data set is processed using data enhancement technology.
6. The method for generating an image cartoon according to claim 2, characterized in that: After obtaining the training data set containing cartoon-style pictures, the method further includes: The training data set is processed using a regularization technique.
7. The image cartoon generation method according to claim 2, characterized in that: The number of the cartoon-style pictures is less than 1,000.
8. An image cartoon generation device, characterized in that: include: The initial input module is used to input the image to be converted and the text description; A model acquisition module, used to add two linear layers to the K and V parts of the cross attention layer of the U-Net network of the stable diffusion model, respectively, to obtain an integrated stable diffusion model; and The cartoon generation module is used to cartoonize the image to be converted by using the integrated stable diffusion model in combination with the text description.
9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the image cartoon generation method according to any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the image cartoon generation method according to any one of claims 1 to 7.