Picture-to-pencil-drawing style image conversion method and device, equipment and medium

Abstract

The invention provides a picture-to-pencil drawing style image conversion method and device, equipment and a medium, and the method comprises the steps: selecting a pencil sketch as a training sample, training a LoRA model through the training sample, and obtaining the parameters of the trained LoRA model; establishing a standard pencil color card, establishing a blank image with the same size as the to-be-drawn picture, converting the to-be-drawn picture into a grey-scale image, mapping the grey-scale image to the blank image according to the standard pencil color card, and finally obtaining a mapping image; the method comprises the following steps: extracting a to-be-converted picture through a line extraction model and a Canny edge detection algorithm to obtain Lineart information and Canny information; the LoRA model parameters, the mapping picture, the combined cue word, the Lineart information and the Canny information are input into the ControlNet to guide the Flux model to generate the required transfer drawing, so that the generation speed is increased, and the generation quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for converting a picture into an image in pencil drawing style. Background Art

[0002] In the field of image style conversion, converting various images into pencil drawing styles has broad application prospects, such as artistic creation, personalized decoration, and cultural and creative product design.

[0003] Current methods for converting images into pencil drawings have significant drawbacks. Firstly, manual adjustment of image styles requires specialized drawing skills and a significant amount of time, resulting in complex and inefficient operations and a significant difficulty in meeting the demands of batch processing. Secondly, existing automated image processing technologies are inaccurate in reproducing the delicate texture of pencil lines, crosshatching, and paper texture. The resulting images often lack the texture and artistic expression of a true pencil drawing, resulting in a low degree of stylistic fidelity.

[0004] In addition, traditional methods cannot be flexibly adjusted according to different image contents and users' personalized needs for pencil drawing styles, and are difficult to adapt to diverse style conversion scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for converting a picture into an image in the style of a pencil drawing, which can quickly generate a pencil-style image, greatly improving the generation speed, and the generated image is of high quality, which can be used directly by users without any modification.

[0006] In a first aspect, the present invention provides a method for converting a picture into a pencil drawing-style image, comprising the following steps:

[0007] Step 1: Select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters;

[0008] Step 2: Create a standard pencil color chart and a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color chart to obtain a mapped image.

[0009] Step 3: Extract the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information

[0010] Step 4: Input the LoRA model parameters, mapping image, combination prompt words, Lineart information, and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

[0011] In a second aspect, the present invention provides a device for converting a picture into a pencil drawing-style image, comprising:

[0012] Train the LoRA model module, select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters;

[0013] Establish a pencil mapping module, create a standard pencil color card, and create a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color card to finally obtain a mapped image.

[0014] The image information extraction module extracts the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information

[0015] Generate a pencil-style image module, input the LoRA model parameters, mapping image, combination prompt word, Lineart information and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed by a processor.

[0018] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0019] 1. High efficiency:

[0020] Compared with traditional manual style conversion, the present invention realizes the rapid conversion of images into pencil drawing style images. Traditional manual drawing of a pencil drawing may take several hours or even longer, while the time for processing a single image by the algorithm of the present invention can be shortened to within 30 seconds, which greatly improves the processing efficiency and can meet the needs of batch image style conversion. It is suitable for scenarios such as e-commerce product image processing, personalized image customization, etc. that require the generation of a large number of pencil drawing style images.

[0021] 2. Accuracy:

[0022] The Flux LoRA model is trained with a rich variety of pencil sketch training samples, enabling the model to accurately learn the various stylistic features of pencil drawings, including the natural texture of graphite lines, realistic cross-hatching effects, and unique paper textures. At the same time, ControlNet uses the extracted Lineart and Canny information to accurately lock the image lines, ensuring that the generated pencil drawing style images are consistent with the original in terms of object shape and structure. Figure 1 Through the coordinated collaboration and precise processing of various technologies, the generated pencil drawing-style images excel in restoring stylistic features, with natural and smooth lines and realistic textures. They can highly restore the artistic effect of pencil drawings and significantly improve image quality and professionalism.

[0023] 3. Flexibility:

[0024] Users do not need to possess professional drawing knowledge or image processing skills; simply provide any image and the method described in this invention will convert it into a pencil drawing-style image. Furthermore, users can flexibly customize pencil drawing effects of different styles by modifying the samples used to train the Flux LoRA model. For example, users can choose prompt word combinations for delicate realistic or bold abstract styles based on different application scenarios, generating pencil drawing-style images with different characteristics, providing rich creative space and diverse options for artistic creation and image design.

[0025] 4. Compatibility:

[0026] The proposed method can be seamlessly integrated with existing image processing software and systems. For example, it can be integrated as a plug-in into common image editing software (such as Photoshop and Illustrator), allowing users to utilize pencil drawing-style transfer within a familiar software environment without changing their existing workflow. Furthermore, this method can be combined with other image generation and processing technologies to form a more powerful image processing solution, lowering the barrier to entry and cost of technology application, and facilitating its promotion and application in multiple fields.

[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] Figure 1 This is a flowchart of the method in Example 1 of the present invention;

[0030] Figure 2This is a schematic diagram of the structure of the device in Example 2 of the present invention. DETAILED DESCRIPTION

[0031] The embodiments of the present application provide a method, apparatus, device, and medium for converting images into pencil drawing styles. By training a specific Flux LoRA model and combining Flux Kontext's rough style conversion and ControlNet's line locking technology, any image can be quickly and high-quality converted into a pencil drawing style image, accurately restoring the line texture, cross-hatching effect, and paper texture of the pencil drawing, meeting users' demand for diverse pencil drawing styles and significantly improving the efficiency and quality of image style conversion.

[0032] The technical solutions in the embodiments of this application have the following general ideas:

[0033] Flux: A deep learning-based generative model with powerful image generation and style transfer capabilities. It can generate images of corresponding styles based on input prompt words and related parameters.

[0034] LoRA (Low-Rank Adaptation): Model fine-tuning technology, by adding a low-rank adaptation layer to the original model, to achieve adaptation to a specific style or task with fewer parameter adjustments. This invention is used to train a specific model for pencil drawing style.

[0035] Florence Inference Model: A deep learning model for inferring cues from images, which can analyze image content and generate text cues describing its content.

[0036] ControlNet: A deep learning architecture that enables conditional control of neural networks. It uses input-specific information (such as Lineart and Canny information) to constrain the image generation process and achieve precise image control.

[0037] Lineart: Image line contour information extracted through image processing technology, reflecting the edges and shapes of objects in the image.

[0038] Canny: The image edge information extracted by the Canny algorithm can accurately detect edge details in the image.

[0039] The main implementation is as follows:

[0040] (1) LoRA model training stage

[0041] Training Sample Collection: Carefully select 30-40 high-quality, diverse pencil sketches as training samples. These samples must cover a variety of subject matter, including figures, landscapes, and still lifes, and include different painting styles, ranging from delicate realism to bold abstraction. They must also include different pencil line expressions (such as light and heavy lines, sparse and dense line arrangement), as well as paper texture effects, to ensure comprehensive coverage of the various stylistic characteristics of pencil sketches.

[0042] Model training: Collected pencil sketch training samples are fed into the Flux model and trained using LoRA technology. During training, we focus on the stylistic characteristics of pencil sketches, such as variations in line thickness, the pattern of crosshatching, and the influence of paper texture on lines. By adjusting the parameters of LoRA layers 2 and 7, the model learns the unique style of pencil sketches, forming a LoRA model specifically designed for pencil drawing style generation.

[0043] (2) Color card mapping

[0044] Create 8 pencil strokes of varying hardness and softness, map pencil colors according to their softness and hardness, and include pure white to create a standard pencil color card;

[0045] The 8-color brushstrokes can be the colors corresponding to 8B pencil, 6B pencil, 4B pencil, 2B pencil, B pencil, HB pencil, 2HB pencil and 3HB pencil. You can color with a pencil and then take a photo to obtain the corresponding grayscale value; you can also obtain the corresponding grayscale value according to an existing color card;

[0046] Create a mapping image of the same size as the image to be transferred, and grayscale the image to be transferred first. The grayscale formula is gray = (R+G+B) / 3. For example, the white color of 255,255,255 will be mapped to 255. Then map the values ​​between 0-255 to the 8-color standard pencil drawing color card. The specific method is to use the values ​​between 0-255 in the grayscale image, subtract the color values ​​in the color card, calculate the minimum difference to get the color card value of the pencil color, and then assign it to the mapping image.

[0047] (3) Style fine-tuning stage

[0048] 1. Combination of prompt words: Input the real photo to be transferred into the Florence reverse model. The model analyzes the objects, scenes, colors and other information in the photo, and then combines the reversed prompt words with the style prompt words: sketech_style, a detailedbalck and whitepencilpaitaing to obtain the final prompt words to be input into the model.

[0049] 2. Information Extraction: Extract Lineart and Canny information from the original image. Leveraging existing image processing algorithms, such as deep learning-based line extraction models and Canny edge detection algorithms, accurately capture the line outlines and edge details of objects in the image.

[0050] 3. Image fine-tuning: The extracted Lineart and Canny information is fed into ControlNet, and the pencil drawing style LoRA model trained in the first step is used as an auxiliary. ControlNet locks and constrains the lines of the image based on the Lineart and Canny information, ensuring that the line outlines and edge details of the objects in the generated pencil drawing style image are consistent with the original. Figure 1 The LoRA model provides stylistic features of pencil drawings for image generation, guiding adjustments to the image's color, line texture, and paper texture. During the generation process, the model continuously optimizes the image using a deep learning algorithm, further refining the rough transfer image and ultimately outputting a high-quality, precisely styled pencil drawing.

[0051] Example 1

[0052] like Figure 1 As shown, this embodiment provides a method for converting a picture into a pencil drawing-style image, including the following steps:

[0053] Step 1: Select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters;

[0054] Step 2: Create a standard pencil color chart and a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color chart to obtain a mapped image.

[0055] Step 3: Extract the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information

[0056] Step 4: Input the LoRA model parameters, mapping image, combination prompt words, Lineart information, and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

[0057] In this embodiment, preferably, step 1 is specifically: selecting pencil sketches as training samples, wherein the training samples include pencil sketches of different themes, different color painting styles, and different pencil lines; training the LoRA model through the training samples to obtain the trained LoRA model parameters; setting the block weight of LoRA, setting the 2nd and 7th layers to 1, and setting the other layers to 0.

[0058] In this embodiment, preferably, step 2 specifically includes: creating a standard pencil color chart and a blank image of the same size as the image to be transferred, converting the image to be transferred into a grayscale image, traversing each pixel of the grayscale image, calculating the difference between the grayscale value of each pixel and the grayscale value of each color in the standard pencil color chart, taking the grayscale value of the color in the standard pencil color chart corresponding to the minimum difference, and assigning it to the corresponding pixel of the blank image, thereby finally obtaining a mapped image.

[0059] In this embodiment, preferably, the combined prompt words are: inputting the picture to be transferred into the Florence inverse inference model, inversely inferring to obtain content prompt words, and combining the content prompt words with the set prompt words to obtain the combined prompt words.

[0060] Based on the same inventive concept, this application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0061] Example 2

[0062] like Figure 2 As shown, in this embodiment, a device for converting a picture into a pencil drawing-style image is provided, comprising:

[0063] Train the LoRA model module, select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters;

[0064] Establish a pencil mapping module, create a standard pencil color card, and create a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color card to finally obtain a mapped image.

[0065] The image information extraction module extracts the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information

[0066] Generate a pencil-style image module, input the LoRA model parameters, mapping image, combination prompt word, Lineart information and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

[0067] In this embodiment, preferably, the training LoRA model module is specifically as follows: pencil sketches are selected as training samples, and the training samples include pencil sketches of different themes, different color painting styles, and different pencil lines; the LoRA model is trained through the training samples to obtain the trained LoRA model parameters; the block weight of LoRA is set, and the 2nd and 7th layers are set to 1, and the other layers are set to 0.

[0068] In this embodiment, preferably, the pencil mapping module specifically includes: creating a standard pencil color chart and a blank image of the same size as the image to be transferred; converting the image to be transferred into a grayscale image; traversing each pixel of the grayscale image; calculating the difference between the grayscale value of each pixel and the grayscale value of each color in the standard pencil color chart; taking the grayscale value of the color in the standard pencil color chart corresponding to the minimum difference, and assigning it to the corresponding pixel of the blank image, thereby finally obtaining a mapped image.

[0069] In this embodiment, preferably, the combined prompt words are: inputting the picture to be transferred into the Florence inverse inference model, inversely inferring to obtain content prompt words, and combining the content prompt words with the set prompt words to obtain the combined prompt words.

[0070] Since the device described in the second embodiment of the present invention is used to implement the method of the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the device based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All devices used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0071] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0072] Example 3

[0073] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0074] Since the electronic device described in this embodiment is the device used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection to be provided by this application.

[0075] Based on the same inventive concept, this application provides a storage medium corresponding to Example 1, see Example 4 for details.

[0076] Example 4

[0077] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0078] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for converting a picture into a pencil drawing style image, characterized by: The steps include: Step 1: Select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters; Step 2: Create a standard pencil color chart and a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color chart to obtain a mapped image. Step 3: Extract the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information Step 4: Input the LoRA model parameters, mapping image, combination prompt words, Lineart information, and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

2. The method for converting a picture into a pencil drawing style according to claim 1, characterized in that: The step 1 specifically includes: selecting pencil sketches as training samples, wherein the training samples include pencil sketches of different themes, different color painting styles, and different pencil lines; training the LoRA model through the training samples to obtain the trained LoRA model parameters; setting the blockweight of LoRA, setting the second and seventh layers to 1, and setting the other layers to 0.

3. The method for converting a picture into a pencil drawing style according to claim 1, characterized in that: Step 2 specifically includes: creating a standard pencil color chart and a blank image of the same size as the image to be transferred, converting the image to be transferred into a grayscale image, traversing each pixel of the grayscale image, calculating the difference between the grayscale value of each pixel and the grayscale value of each color in the standard pencil color chart, taking the grayscale value of the color in the standard pencil color chart corresponding to the minimum difference, and assigning it to the corresponding pixel of the blank image, thereby finally obtaining a mapped image.

4. The method for converting a picture into a pencil drawing style according to claim 1, characterized in that: The combination prompt words are: inputting the picture to be transferred into the Florence reverse deduction model, reverse deducing to obtain the content prompt words, and combining the content prompt words with the set prompt words to obtain the combination prompt words.

5. A device for converting a picture into a pencil drawing-style image, characterized by: include: Train the LoRA model module, select pencil sketches as training samples, train the LoRA model through the training samples, and obtain the trained LoRA model parameters; Establish a pencil mapping module, create a standard pencil color card, and create a blank image of the same size as the image to be transferred. Convert the image to be transferred into a grayscale image and map it to the blank image according to the standard pencil color card to finally obtain a mapped image. The image information extraction module extracts the image to be transferred through the line extraction model and Canny edge detection algorithm to obtain Lineart information and Canny information Generate a pencil-style image module, input the LoRA model parameters, mapping image, combination prompt word, Lineart information and Canny information into ControlNet to guide the Flux model to generate the required transfer image.

6. The device for converting a picture into a pencil drawing style according to claim 5, characterized in that: The training LoRA model module specifically includes: selecting pencil sketches as training samples, wherein the training samples include pencil sketches of different themes, different color painting styles, and different pencil lines; training the LoRA model through the training samples to obtain the trained LoRA model parameters; setting the block weight of LoRA, setting the 2nd and 7th layers to 1, and setting the other layers to 0.

7. The device for converting a picture into a pencil drawing style according to claim 5, characterized in that: The pencil mapping module specifically creates a standard pencil color chart and a blank image of the same size as the image to be transferred, converts the image to be transferred into a grayscale image, iterates over each pixel of the grayscale image, calculates the difference between the grayscale value of each pixel and the grayscale value of each color in the standard pencil color chart, takes the grayscale value of the color in the standard pencil color chart corresponding to the minimum difference, and assigns it to the corresponding pixel of the blank image, thereby finally obtaining a mapped image.

8. The device for converting a picture into a pencil drawing style according to claim 5, characterized in that: The combination prompt words are: input the picture to be transferred into the Florence reverse inference model, reverse inference to obtain the content prompt words, and combine the content prompt words with the set prompt words to obtain the combination prompt words.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images