A black and white line drawing coloring method and device based on transfer learning

Abstract

A method and device for coloring black and white line drawings based on transfer learning, which relates to the field of image processing, solves the problems that the existing GAN models designed for a single task are difficult to process multiple image tasks at the same time, and these models are often difficult to maintain the authenticity of image content and consistency of style at the same time. The present invention provides the following scheme: the method includes: preparing data and preprocessing the data; constructing a multi-task generative adversarial network model for coloring black and white line drawings, the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D; performing transfer learning and multi-task training on the generated color image to obtain an ultra-high resolution image; optimizing the loss function of the multi-task training in the ultra-high resolution image through at least one iterative training, and using an alternating training method to update the parameters in the generator G and the discriminator D respectively; completing the coloring of black and white line drawings. It is also suitable for the field of high-quality image generation in different industries.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing. Background Art

[0002] In the field of digital image processing, especially in the application of automatic image generation, editing and enhancement, the generative adversarial network (GAN) technology has become a key force in promoting the development of this field in recent years. GAN consists of two main parts: the generator and the discriminator. The main task of the generator is to create images that are extremely similar to real images, while the discriminator strives to identify whether the image is created by the generator in order to distinguish between machine-generated images and real images. This competitive mechanism enables both parties to make progress in constant confrontation, promoting the continuous improvement of the quality of generated images.

[0003] Although generative adversarial networks have made remarkable achievements in the field of image generation, they face many challenges in practical applications. The main problem is that most GAN models are designed for a single task, such as limited to image generation or style transfer, and it is difficult to handle multiple image tasks at the same time, which greatly limits their application scope in real-world scenarios with diverse requirements. In addition, when faced with complex image conversion tasks, these models often find it difficult to simultaneously maintain the authenticity of image content and the consistency of style. In addition, GANs may also encounter problems such as mode collapse or unstable training during training, which not only affect image quality, but also may make the training process difficult to proceed smoothly. Finally, the generation of high-quality images usually requires a lot of computing resources and data support, which places higher requirements on hardware equipment and increases costs accordingly. Summary of the invention

[0004] The present invention aims at the problem that the GAN models in the prior art are designed for a single task and are difficult to handle multiple image tasks at the same time. When performing complex image conversion tasks, these models often find it difficult to simultaneously maintain the authenticity of the image content and the consistency of the style. In addition, the GAN may also encounter mode collapse or training instability during the training process. To solve the above technical problems, the present invention is implemented through the following technical solutions:

[0005] Solution 1: The present invention proposes a black and white line drawing coloring method based on transfer learning, and the method comprises the following steps:

[0006] S1. Prepare data, the data including color images And the corresponding black and white line drawing , and preprocessing the data;

[0007] S2. Construct black and white line drawing A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D;

[0008] S3, the color image generated in S2 Perform transfer learning and multi-task training to obtain ultra-high-resolution images;

[0009] S4, optimizing the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and using an alternating training method to update the parameters in the generator G and the discriminator D respectively;

[0010] S5. In each iteration, the generator G is used to generate the color image according to the discriminator. The style and clarity of the real color image are generated, and the discriminator D is used to generate the color image according to the generated Optimize the color images and complete the coloring of the black and white line drawings.

[0011] Further, a preferred embodiment is provided, wherein the data preprocessing in S1 comprises the following steps:

[0012] S1.1. The color image Generate the corresponding black and white line drawing through the edge detection algorithm ;

[0013] S1.2, and the color image Adjust to consistent size.

[0014] Further, a preferred embodiment is provided, wherein the color image in S1.1 generates a corresponding black and white line drawing through an edge detection algorithm, comprising the following steps:

[0015] S1.11. Color image Convert to grayscale image ;

[0016] S1.12. Grayscale images Use edge detection algorithm to get black and white line drawing ;

[0017] S1.13. Based on the obtained black and white line drawing The image quality is processed by corrosion or expansion method to generate line drawings The thickness of black and white lines;

[0018] S1.14. Collect the processed black and white line drawings Set with color images storage.

[0019] Furthermore, a preferred implementation is provided, wherein S1.2 also includes a step of data enhancement.

[0020] Further, a preferred embodiment is provided, wherein the generator G in S2 includes a shared feature extraction module and a multi-task processing module;

[0021] The shared feature extraction module is used to extract black and white line drawings Features in the image and output a color image ;

[0022] The multi-task processing module is used to enhance the color image size.

[0023] Furthermore, a preferred implementation is provided, in which transfer learning and multi-task training in S3 include three steps: task weight adjustment, loss function design, and multi-task training strategy, wherein the loss function design includes adversarial loss , content loss , style loss and super-resolution loss .

[0024] Furthermore, a preferred implementation is provided, in which the loss function in the multi-task training is:

[0025] + +

[0026] in, , , , They are the weight coefficients of adversarial loss, content loss, style loss and super-resolution loss respectively.

[0027] Solution 2: A black and white line drawing coloring device based on transfer learning, the device comprising:

[0028] Data preparation module for the data including color images And the corresponding black and white line drawing , preprocessing the data;

[0029] Model building module for building black and white line drawings A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D;

[0030] A training module for the color image generated in the model building module Perform transfer learning and multi-task training to obtain ultra-high-resolution images;

[0031] An optimization module, used to optimize the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and to update the parameters in the generator G and the discriminator D respectively by an alternating training method;

[0032] Update module, used in each iteration, the generator G, used to generate the color image according to the discriminator The style and clarity of the real color image are generated, and the discriminator D is used to generate the color image according to the generated Optimize the color images and complete the coloring of the black and white line drawings.

[0033] Solution three: A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution one.

[0034] Solution 4: A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of Solution 1.

[0035] The present invention is beneficial in that:

[0036] The present invention aims at the fact that the existing GAN models in the prior art are all designed for a single task, such as limited to image generation or style conversion, and it is difficult to handle multiple image task problems at the same time. The present invention introduces a new multi-task loss function and adopts a dynamic training strategy, which enables the network to automatically adjust the focus during the training process to balance and optimize the needs of different image generation tasks. This method not only solves the shortcomings of traditional GAN ​​in multi-task processing, but also significantly improves the efficiency and stability of training.

[0037] The multi-task GAN design of the present invention can be widely used in a variety of industries and scenarios, including digital media, online advertising, video game development, virtual reality, and visual simulation in autonomous driving systems. In these applications, high-quality and style-diverse image generation is particularly critical, and the solution provided by the present invention can meet the high standards of image processing technology in these industries. Therefore, this new type of GAN not only promotes the development of image processing technology, but also brings substantial innovation and value to related industries.

[0038] In order to solve the problem in the prior art that these models often find it difficult to simultaneously maintain the authenticity of image content and consistency of style when faced with complex image conversion tasks, the present invention provides a comprehensive image processing solution, namely, an innovative multi-task learning generative adversarial network. This improved network design can not only generate high-quality color images from basic black and white line drawings, but also perform style transfer and super-resolution processing at the same time, significantly improving the practical value and aesthetics of the image. It also supports everything from basic image line drawing coloring to complex style transfer and resolution enhancement.

[0039] By combining adversarial loss, content loss, style loss and super-resolution loss, the present invention can comprehensively optimize multiple aspects of the generated image and improve the color authenticity, style consistency and clarity of the image. This makes the conversion from black and white line drawings to color images not only natural but also visually attractive.

[0040] The present invention uses a dynamic training strategy, and the model can automatically adjust the learning rate and loss weight according to real-time training feedback. This adaptive adjustment method increases the model's responsiveness to different training conditions and changes, and improves the model's stability and effect in a changing application environment.

[0041] The present invention is widely used in various industries, including but not limited to digital media production, advertising design, artistic creation and education and training, meeting the needs of different industries for high-quality image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a black and white line drawing coloring method based on transfer learning described in Implementation Method 1.

[0043] Figure 2 This is a rendering of a black and white line drawing coloring method based on transfer learning described in Implementation Method 1.

[0044] Figure 2 In the figure, (a) is the line drawing image, (b) is the original image, (c) is the generated image, and (d) is the super-high resolution image. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the implementation methods of the present application clearer, the technical solutions in the implementation methods of the present application will be clearly and completely described below in conjunction with the drawings in the implementation methods of the present application. Obviously, the described implementation methods are only part of the implementation methods of the present application, not all of the implementation methods.

[0046] Embodiment 1: This embodiment provides a black and white line drawing coloring method based on transfer learning, and the method comprises the following steps:

[0047] S1. Prepare data, the data including color images And the corresponding black and white line drawing , and preprocessing the data;

[0048] S2. Construct black and white line drawing A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D;

[0049] S3, the color image generated in S2 Perform transfer learning and multi-task training to obtain ultra-high-resolution images;

[0050] S4, optimizing the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and using an alternating training method to update the parameters in the generator G and the discriminator D respectively;

[0051] S5. In each iteration, the generator G is used to generate the color image according to the discriminator. The style and clarity of the real color image are generated, and the discriminator D is used to generate the color image according to the generated Optimize the color images and complete the coloring of the black and white line drawings.

[0052] Implementation method 2: This implementation method further limits the black and white line drawing coloring method based on transfer learning described in implementation method 1. The data preprocessing in S1 includes the following steps:

[0053] S1.1. The color image Generate the corresponding black and white line drawing through the edge detection algorithm ;

[0054] S1.2, and the color image Adjust to consistent size.

[0055] Implementation method 3: This implementation method further limits the black and white line drawing coloring method based on transfer learning described in implementation method 1. The color image in S1.1 generates a corresponding black and white line drawing through an edge detection algorithm, including the following steps:

[0056] S1.11. Color image Convert to grayscale image ;

[0057] S1.12. Grayscale images Use edge detection algorithm to get black and white line drawing ;

[0058] S1.13. Based on the obtained black and white line drawing The image quality is processed by corrosion or expansion method to generate line drawings The thickness of black and white lines;

[0059] S1.14. Collect the processed black and white line drawings Set with color images storage.

[0060] Implementation method 4: This implementation method further limits the black and white line drawing coloring method based on transfer learning described in implementation method 3, and S1.2 also includes a step of data enhancement.

[0061] Embodiment 5: This embodiment further limits the black and white line drawing coloring method based on transfer learning described in Embodiment 1. In S2, the generator G includes a shared feature extraction module and a multi-task processing module;

[0062] The shared feature extraction module is used to extract black and white line drawings Features in the image and output a color image ;

[0063] The multi-task processing module is used to enhance the color image size.

[0064] Implementation 6. This implementation is a further limitation of the black and white line drawing coloring method based on transfer learning described in Implementation 1. In S3, transfer learning and multi-task training include three steps: task weight adjustment, loss function design, and multi-task training strategy. The loss function design includes adversarial loss. , content loss , style loss and super-resolution loss .

[0065] Embodiment 7: This embodiment further limits the black and white line drawing coloring method based on transfer learning described in Embodiment 6. The loss function in the multi-task training is:

[0066] + +

[0067] in, , , , They are the weight coefficients of adversarial loss, content loss, style loss and super-resolution loss respectively.

[0068] Embodiment 8: This embodiment proposes a black and white line drawing coloring method and device based on transfer learning, and the device includes:

[0069] Data preparation module for the data including color images And the corresponding black and white line drawing , preprocessing the data;

[0070] Model building module for building black and white line drawings A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D;

[0071] A training module for the color image generated in the model building module Perform transfer learning and multi-task training to obtain ultra-high-resolution images;

[0072] An optimization module, used to optimize the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and to update the parameters in the generator G and the discriminator D respectively by an alternating training method;

[0073] Update module, used in each iteration, the generator G, used to generate the color image according to the discriminator The style and clarity of the real color image are generated, and the discriminator D is used to generate the color image according to the generated Optimize the color images and complete the coloring of the black and white line drawings.

[0074] Embodiment 9: A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in any one of embodiments 1 to 7.

[0075] Embodiment 10: A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of embodiments 1 to 7 are implemented.

[0076] Implementation eleven: This implementation provides an example, which is used to explain the above implementations one to ten. The specific example is as follows:

[0077] See also Figure 1 and Figure 2 To illustrate the present embodiment, this embodiment will describe in detail an application example of a multi-task generative adversarial network (GAN) architecture, particularly in a scenario of automatically converting black and white line drawings into color images while implementing style transfer and super-resolution processing.

[0078] See also Figure 1 This embodiment is described, wherein Figure 1This is a flow chart of a black and white line drawing coloring method based on transfer learning. This embodiment provides a black and white line drawing coloring method based on transfer learning, including the following steps:

[0079] S1. Prepare data. In addition to the original color images and the corresponding black and white line drawings, if the line drawings are not available, use edge detection algorithms (such as Canny, etc.) to generate line drawings from color images. Prepare or generate corresponding training data, such as high-resolution images, images with a specific style, etc. Perform appropriate preprocessing operations on all types of data to ensure that the data format and size match the model input;

[0080] S2. Select a suitable pre-trained deep learning model as the basis of the feature extraction layer to improve the generation quality. Design a multi-task generative adversarial network architecture for line drawing coloring. The network contains a generator and a discriminator with multiple tasks. Among them, the generator is responsible for generating color images from black and white line drawings and performing auxiliary tasks such as edge enhancement, style transfer and super-resolution, while the discriminator is designed to evaluate the authenticity, style consistency and quality of the generated images. It is necessary not only to distinguish the generated images from the real images, but also to evaluate the style and clarity of the generated images;

[0081] S3, transfer learning and multi-task training process, including task weight adjustment, loss function design, and specific training strategy. The loss function consists of four parts: adversarial loss, content loss, style loss, and super-resolution loss. Adversarial loss ensures that the generated color image is visually indistinguishable from the real color image; content loss ensures the accuracy of the generated image content; style loss makes the style of the generated image close to the target style; super-resolution loss improves the resolution of the generated image. Through multiple iterative experiments, the best loss weight configuration is found to achieve optimization of each task.

[0082] S4. When implementing the training strategy, a large number of iterative trainings are used to optimize the multi-task loss function. The process uses an alternating training method to update the parameters of the generator and the discriminator separately. In each iteration, the generator first tries to generate a color image that is as realistic as possible based on the current discriminator model, and then the discriminator optimizes its ability to judge authenticity based on these generated images and real images. In addition, the weights and learning rates of each task loss are dynamically adjusted based on the performance of the model on the validation set.

[0083] S5. Based on user feedback and application scenario requirements, continue to optimize and iterate the model to improve the quality and diversity of coloring.

[0084] In this embodiment, in S1, the sample data of the line drawing dataset before training is constructed according to the following sub-steps: data preparation is performed. In addition to the original color image and the corresponding black and white line drawing, if the black and white line drawing is not available, the black and white line drawing is generated from the color image using an edge detection algorithm (such as Canny, etc.). Prepare or generate corresponding training data, such as high-resolution images, images with a specific style, etc.; perform appropriate preprocessing operations on all types of data to ensure that the data format and size match the model input;

[0085] S1.1. Prepare and collect data, and select a large color image dataset as the basis. These images should cover a wide range of topics and styles so that the model can learn rich color distribution and scene composition. Commonly used datasets include ImageNet, COCO, or custom-made datasets. For each color image, an edge detection algorithm (such as the Canny operator, etc.) can be used to generate the corresponding black and white line drawing. This step is to create the input data for the model. Collect additional data based on specified tasks (such as super-resolution or style transfer). For example, for style transfer, you can collect a set of images with a specific artistic style;

[0086] S1.2. According to the input requirements of the multi-task generative adversarial network model, resize all images to a uniform size. At the same time, standardize the pixel values ​​so that they fall within a standard range (e.g., 0 to 1 or -1 to 1) to improve the stability of model training. Apply data augmentation techniques (such as rotation, scaling, cropping, color adjustment) to increase data diversity and the generalization ability of the model;

[0087] In one embodiment, S1.1 includes:

[0088] S1.11. The original color image Convert to grayscale image ;

[0089] S1.12, Apply edge detection algorithm (such as Canny operator) to Get the line drawing ;

[0090] S113. Based on the obtained line drawing To improve the image quality, you can choose to further process the line thickness of the line drawing by corrosion or expansion.

[0091] S114. Collect the processed line drawings With original color image set Store together, ready for subsequent training steps;

[0092] In this embodiment, the data enhancement technology in S1.2 above may be selected from the following methods:

[0093] ;

[0094] in is the original data, is the transformed data, represents the data augmentation operation applied, The operations can be scaling, rotating, denoising, cropping, etc.

[0095] In this embodiment, S2 includes: selecting a suitable pre-trained deep learning model as the basis of the feature extraction layer to improve the generation quality. Design a multi-task generative adversarial network architecture for black and white line drawing coloring, the network includes a generator and a discriminator with multiple tasks. Among them, the generator is responsible for generating color images from black and white line drawings, and performing auxiliary tasks such as edge enhancement, style transfer and super-resolution, while the discriminator is designed to evaluate the authenticity, style consistency and quality of the generated images. It is necessary not only to distinguish the generated images from the real images, but also to evaluate the style and clarity of the generated images;

[0096] In the embodiment, S2 completes the design of a multi-task generative adversarial network architecture for black and white line drawing coloring according to the following sub-steps:

[0097] S2.1. Design the basic structure of multi-task GAN for line drawing coloring task, and refine the structure of generator (G) and discriminator (D);

[0098] S2.2. Select a deep learning model that has been pre-trained on a large image dataset (such as ImageNet) as the basis for the feature extractor. Commonly used models include VGG-16, ResNet-50, etc.; modify the pre-trained model appropriately according to the task requirements to adapt to the new output format (such as colored images) and the needs of auxiliary tasks;

[0099] S2.3. Design a generator according to the task requirements, the input of which is a black and white line drawing image, including a shared feature extraction module and a multi-task processing module. The main goal of the generator is to receive the black and white line drawing as input and output the corresponding color image. In addition, it also contains several sub-networks, each responsible for different auxiliary tasks.

[0100] The shared feature extraction module of the main network is composed of a deep neural network to extract the features of the input image. The feature map extracted by this module will be used for subsequent different tasks. In the embodiment, the shared feature extraction module can be composed of a deep network consisting of a convolutional layer, a normalization layer, and a ReLU activation layer.

[0101] The multi-task processing module is composed of multiple task branches composed of upsampling layers (such as deconvolution, bilinear interpolation, etc.), which are used to increase the size of the feature map, refine the features, and gradually increase the coloring details. In the embodiment, the multi-task branch includes sub-networks such as edge enhancement, style transfer, and super-resolution restoration.

[0102] Depending on the task requirements, the output layer can directly generate the final color image, or an image after style transfer, edge enhancement and resolution improvement.

[0103] S2.4. Design a discriminator based on the task requirements, which needs to be able to evaluate the rationality, style consistency, and resolution quality of the generated images. This may mean implementing multiple outputs in the discriminator, each for a specific evaluation task, or designing independent discriminator networks to handle different task evaluations.

[0104] In this embodiment, the above step S3 is the network transfer learning and multi-task training process, including task weight adjustment, loss function design, and specific training strategy. The loss function consists of four parts: adversarial loss, content loss, style loss, and super-resolution loss. Step S3 includes:

[0105] S3.1. Define the components of the loss function, including adversarial loss , content loss , style loss and super-resolution loss ,in:

[0106] ① Fighting losses It is used to calculate the degree of distinction between the generated image c and the artist's drawing in the prior art on the discriminator, and encourage the generator to produce images that are difficult to distinguish by the discriminator. The adversarial loss in the embodiment is as follows:

[0107] Generator adversarial loss: ;

[0108] Discriminator adversarial loss:

[0109] ② Content loss By comparison Figure 2 The feature representation of the generated image c and the target image at a specific depth layer is ensured Figure 2 The generated image c is consistent with the target image in content. The content loss in the embodiment is as follows:

[0110]

[0111] in From the reference image x Extracted from i The characteristics of the layer, is to generate an image, is the corresponding feature extracted from the generated image c;

[0112] ③ Style Loss By matching the style features of the generated image c with the target style image (usually represented by a Gram matrix), the generated image is made to mimic the target style. The style loss in the embodiment is as follows:

[0113]

[0114] in, is an image x In the j The Gram matrix of the layer, y is a style reference image;

[0115] ④ Super-resolution loss It is used to calculate the difference between the generated high-resolution image and the real high-resolution image, where the generated high-resolution image and the real high-resolution image are super-resolution images d, so as to improve the clarity of the generated image. The super-resolution loss in the embodiment is as follows:

[0116]

[0117] in, is the high-resolution target image, is a high-resolution image generated by the model, using the L1 norm to measure the difference.

[0118] S3.2. Different weight parameters are assigned to different tasks according to the contribution of each loss in the total loss function. The weight selection is adjusted according to the importance of the task and the performance of the model in the pre-training stage;

[0119] S3.3. Combine all the above losses into a total loss function,

[0120] + +

[0121] in , , , are the weight coefficients of adversarial loss, content loss, style loss and super-resolution loss respectively;

[0122] In this embodiment, the above step S4 includes optimizing the multi-task loss function through a large number of iterative training when implementing the training strategy. The process adopts an alternating training method to update the parameters of the generator G and the discriminator D respectively:

[0123] S4.1. Set the initial parameters of the generator G and the discriminator D.

[0124] S4.2. In order to improve the convergence speed and stability of the model, a suitable optimizer is selected for GAN training; the optimizer in the embodiment may be an Adam optimizer, and the formula is as follows:

[0125]

[0126] in is a parameter, is the learning rate, and are the first and second moment estimates, is a decimal number that prevents division by zero;

[0127] S4.3, using an alternating training method to update the parameters of the generator and discriminator respectively. In each iteration, the generator first tries to generate color images that are as realistic as possible based on the current discriminator model, and then the discriminator optimizes its ability to judge authenticity based on these generated images and real images.

[0128] In an embodiment, the discriminator D is trained to maximize the discriminator loss:

[0129]

[0130] In an embodiment, the generator G is trained to minimize the generator loss:

[0131]

[0132] In this embodiment, the above step S5 includes: continuously optimizing and iterating the model according to user feedback and requirements of application scenarios to improve the quality and diversity of coloring.

[0133] Figure 1 Any process or method description in the flowcharts described in the flowcharts or otherwise described herein can be understood as representing a module, segment or portion of code including one or more executable instructions for implementing the steps of a custom logic function or process, and the scope of the preferred embodiments of the present invention includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong. The logic and / or steps represented in the flowcharts or otherwise described herein, Figure 1The system architecture, function and operation of the apparatus and method according to various embodiments of the present disclosure are shown. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram or flow chart, and the combination of the square boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. For example, a sequenced list of executable instructions for implementing logical functions may be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0134] For the purposes of the present specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or apparatus or in conjunction with such instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) with one or N wirings, a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, because the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or processing in other suitable ways as necessary, and then stored in a computer memory. It should be understood that various parts of the present invention may be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0136] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A black and white line drawing coloring method based on transfer learning, characterized in that: The method comprises the following steps: S1. Prepare data, the data including color images And the corresponding black and white line drawing , and preprocessing the data; S2. Construct black and white line drawing A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D; S3, the color image generated in S2 Perform transfer learning and multi-task training to obtain ultra-high-resolution images; S4, optimizing the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and using an alternating training method to update the parameters in the generator G and the discriminator D respectively; S5. In each iteration, the generator G is used to generate a color image that is as realistic as possible based on the current discriminator model, and the discriminator D is used to generate a color image based on the generated color image. Optimize the real image and complete the coloring of the black and white line drawing; The generator G in S2 includes a shared feature extraction module and a multi-task processing module; The shared feature extraction module is used to extract black and white line drawings Features in the image and output a color image ; The multi-task processing module is used to enhance the color image Size; The shared feature extraction module is composed of a deep neural network to extract features of the input image, and the feature map extracted by the feature extraction module is used for subsequent different tasks; The multi-task processing module is composed of multiple task branches composed of upsampling layers, which are used to increase the size of feature maps, refine features, and gradually increase coloring details; the multi-task branches include edge enhancement, style transfer, and super-resolution restoration sub-networks; a discriminator is designed according to task requirements, and the discriminator is used to evaluate the rationality, style consistency, and resolution quality of the generated image; so as to realize black and white line drawing coloring based on transfer learning of the multi-task generative adversarial network model; Transfer learning and multi-task training in S3 include three steps: task weight adjustment, loss function design, and multi-task training strategy. The loss function design includes adversarial loss. , content loss , style loss and super-resolution loss ; The loss function in the multi-task training is: + + in, , , , They are the weight coefficients of adversarial loss, content loss, style loss and super-resolution loss respectively.

2. The black and white line drawing coloring method based on transfer learning according to claim 1, characterized in that: The data preprocessing described in S1 includes the following steps: S1.

1. The color image Generate the corresponding black and white line drawing through the edge detection algorithm ; S1.2, and the color image Adjust to consistent size.

3. The black and white line drawing coloring method based on transfer learning according to claim 1, characterized in that: The color image in S1.1 generates the corresponding black and white line drawing through the edge detection algorithm, including the following steps: S1.

11. Color image Convert to grayscale image ; S1.

12. Grayscale images Use edge detection algorithm; S1.

13. Based on the obtained black and white line drawing The image quality is processed by corrosion or expansion method to generate line drawings The thickness of black and white lines; S1.

14. Collect the processed black and white line drawings Set with color images storage.

4. The line drawing coloring method based on transfer learning according to claim 1, characterized in that: S1.2 also includes steps for data enhancement.

5. A black and white line drawing coloring device based on transfer learning, characterized in that: The device comprises: Data preparation module for the data including color images And the corresponding black and white line drawing , preprocessing the data; Model building module for building black and white line drawings A colored multi-task generative adversarial network model, wherein the multi-task generative adversarial network model includes a multi-task generator G and a discriminator D; A training module for the color image generated in the model building module Perform transfer learning and multi-task training to obtain ultra-high-resolution images; An optimization module, used to optimize the loss function of multi-task training in ultra-high resolution images through at least one iterative training, and to update the parameters in the generator G and the discriminator D respectively by an alternating training method; The update module is used in each iteration to generate a color image that is as realistic as possible based on the current discriminator model and to generate a color image based on the generated color image. Optimize the real image and complete the coloring of the black and white line drawing. The generator G in the model building module includes a shared feature extraction module and a multi-task processing module; The shared feature extraction module is used to extract black and white line drawings Features in the image and output a color image ; The multi-task processing module is used to enhance the color image Size; The shared feature extraction module is composed of a deep neural network to extract features of the input image, and the feature map extracted by the feature extraction module is used for subsequent different tasks; The multi-task processing module is composed of multiple task branches composed of upsampling layers, which are used to increase the size of feature maps, refine features, and gradually increase coloring details; the multi-task branches include edge enhancement, style transfer, and super-resolution restoration sub-networks; a discriminator is designed according to task requirements, and the discriminator is used to evaluate the rationality, style consistency, and resolution quality of the generated image; so as to realize black and white line drawing coloring based on transfer learning of the multi-task generative adversarial network model; Transfer learning and multi-task training in the training module include three steps: task weight adjustment, loss function design, and multi-task training strategy. The loss function design includes adversarial loss. , content loss , style loss and super-resolution loss ; The loss function in the multi-task training is: + + in, , , , They are the weight coefficients of adversarial loss, content loss, style loss and super-resolution loss respectively.

6. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images