A method, system, device and storage medium for coloring a cartoon image
By combining a self-attention Transformer network and an adversarial generative network, the problem of insufficient computational resources in existing technologies is solved, achieving efficient comic image coloring and generating high-quality color comics.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2022-11-21
- Publication Date
- 2026-06-05
Smart Images
Figure CN115760577B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, system, apparatus and storage medium for coloring cartoon images. Background Technology
[0002] Color comics offer more information and are closer to reality than black and white comics, thus better pleasing readers. Coloring is a crucial step in comic creation, but it's extremely time-consuming and labor-intensive for creators. Therefore, computer and image processing technologies can be used to automate comic image coloring, significantly improving creators' efficiency. Furthermore, since a large number of comics are currently in black and white, automatic coloring technology can greatly enhance the reader's experience.
[0003] Image colorization is a challenging task, and existing methods can be broadly categorized into three approaches: user-input based, reference image based, and automatic colorization. User-input based methods require users to provide color cues, such as drawing on a black and white image. Reference image based methods require finding a similar color image and then transferring the colors there. Automatic colorization, on the other hand, only requires a black and white image to obtain one or more color images.
[0004] Automatic coloring of comics can be viewed as an image generation task in image processing, predicting unknown color information based on certain input conditions. However, this is an ill-posed problem; for the same image, there can be multiple plausible coloring results. Image generation can be achieved using machine learning techniques such as autoencoders (AEs) or variational autoencoders (VAEs). Since the advent of Generative Adversarial Networks (GANs), these have become the mainstream solution for such tasks. The coloring model for comics can draw inspiration from the idea of GAN models for colorizing black and white natural images. It uses an encoder-decoder structure, where the encoder performs convolutional downsampling, the decoder performs deconvolutional upsampling, and the model is trained using GAN Loss and L1 Loss provided by the discriminator. The Transformer model, based on a self-attention model, was initially proposed for natural language processing tasks, but it has also shown advantages in other tasks, thus it can also be applied to image generation. However, a drawback of applying the Transformer model to image processing tasks is the high computational resource requirement. It necessitates reducing image resolution and color depth for processing and adding a boosting network model to reconstruct the resolution and color depth. Comic book images differ from natural images. Comic book images lack the rich details of natural images, while natural images lack the distinct outlines and diverse colors of comic book images. Transformer models, which are shaded at low resolutions, are more suitable for comic book images with fewer textures. Summary of the Invention
[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, system, apparatus and storage medium for coloring comic images.
[0006] The technical solution adopted in this invention is:
[0007] A method for coloring comic images includes the following steps:
[0008] A comic image of size H×W is obtained, and the comic image is downsampled to obtain a first resolution image; wherein, the first resolution image is a grayscale image;
[0009] The first resolution image is input into a Transformer network based on a self-attention mechanism for feature extraction, and the color distribution of each pixel in the first resolution image is predicted to obtain a coarsely colored image with color depth.
[0010] The coarse-colored image and the cartoon image are input into an adversarial generative enhancement network to reconstruct the resolution and color depth, thereby obtaining a second-resolution image.
[0011] The second resolution image is upsampled to obtain a finely colored image with a size of H×W;
[0012] The resolution of the second resolution image is greater than that of the first resolution image, and the number of bits of color depth of the second resolution image is greater than that of the coarsely colored image.
[0013] Furthermore, the Transformer network based on the self-attention mechanism performs the following processing on the input grayscale image:
[0014] The grayscale image x g The input is processed by the encoder of the Transformer network; wherein the encoder consists of an embedding layer, a row self-attention module, and a column self-attention module;
[0015] The output of the encoder is passed sequentially through a linear layer and a softmax layer to obtain...
[0016] The output of the encoder is input into the decoder of the Transformer network for processing; wherein the decoder comprises an embedding layer, a conditional row self-attention module, and a conditional column self-attention module;
[0017] The output of the encoder is passed sequentially through a linear layer and a softmax layer to obtain p. c (x|x g The colored image is obtained by randomly sampling on the distribution or calculating the expected value.
[0018] Furthermore, during training, the Transformer network calculates the model weight gradient using a loss function, performs gradient descent, and iterates multiple times to obtain the optimized model weights.
[0019] The expression for the loss function L during the training process is as follows:
[0020]
[0021]
[0022]
[0023] In the formula, M and N represent the row and column dimensions of the downsampled matrix, and x ij x represents the color value of the pixel in row i and column j. g x represents the input grayscale image matrix.<i x represents all pixels less than row i. i,<j Let λ represent the pixels in the i-th row that are less than the pixels in the j-th column, and let λ represent the hyperparameters of a trained model.
[0024] Furthermore, the adversarial generation-based boosting network is obtained in the following way:
[0025] The generator part of the boosting network adopts a U-net structure, which includes a corresponding number of convolutional layers and upsampling layers. The feature maps output by the convolutional layers are short-linked to the upsampling layers at the same level.
[0026] The discriminator of the improved network is the PatchGAN discriminator, whose output is a matrix, where each point represents the true or false value of the corresponding small patch.
[0027] Generative Adversarial Networks (GANs) require alternating training of the generator G and the discriminator D: during the training phase of the generator G, the false values generated by G are input into the discriminator D to calculate the GAN Loss, then the L1 Loss is calculated, and the network weights are updated; during the training phase of the discriminator D, the generator G generates false values, and the original input conditions are used as true values, which are then input into the discriminator D to calculate the Loss and update the network weights.
[0028] Furthermore, the objective function G of the adversarial generative network during the training process... * The expression is as follows:
[0029] L L1 (G)=E x,y [||yG(x)||1]
[0030]
[0031] In the formula, L L1 (G) represents the L1 Loss function of the generator G, E x,y Let ||yG(x)||1 represent the mean, and let L represent the absolute value of the difference between the target y and the generated result G(x). cGAN (G,D) represents the GAN Loss provided by the discriminator, and λ represents an adjustable hyperparameter.
[0032] Further, acquiring a comic image with a size of H×W includes:
[0033] If the comic image is a color image, then the comic image is converted to grayscale.
[0034] Furthermore, the comic image coloring method also includes an evaluation step:
[0035] The coloring effect of the obtained fine coloring image is evaluated using a preset comprehensive objective index; wherein the objective index includes at least one of the following: PSNR index, SSIM index, LPIPS index, or FID index.
[0036] Another technical solution adopted in this invention is:
[0037] A comic image coloring system, comprising:
[0038] An image sampling module is used to acquire a comic image of size H×W, and downsample the comic image to obtain a first resolution image; wherein, the first resolution image is a grayscale image;
[0039] The image coloring module is used to input the first resolution image into a Transformer network based on a self-attention mechanism, extract features, and predict the color distribution of each pixel in the first resolution image to obtain a coarsely colored image with color depth.
[0040] The coloring optimization module is used to input the coarsely colored image and the comic image into an adversarial generation-based enhancement network to reconstruct the resolution and color depth, thereby obtaining a second resolution image.
[0041] The upsampling module is used to upsample the second resolution image to obtain a finely colored image with a size of H×W;
[0042] The resolution of the second resolution image is greater than that of the first resolution image, and the number of bit depths of the second resolution image is greater than that of the coarsely colored image.
[0043] Another technical solution adopted in this invention is:
[0044] A comic image coloring device, comprising:
[0045] At least one processor;
[0046] At least one memory for storing at least one program;
[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.
[0048] Another technical solution adopted in this invention is:
[0049] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.
[0050] The beneficial effects of this invention are: This invention uses the Transformer model to perform coarse coloring with low resolution and low color depth, and then uses an adversarial generative model to reconstruct the resolution and color depth, which can effectively improve the coloring effect of comic images and has obvious advantages over ordinary image coloring techniques. Attached Figure Description
[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a flowchart of the steps of a comic image coloring method according to an embodiment of the present invention;
[0053] Figure 2 This is an overall structural diagram of the self-attention Transformer network and the adversarial generative network in an embodiment of the present invention. Detailed Implementation
[0054] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0055] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0056] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0057] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0058] For comic images with diverse color variations, a comic image coloring method based on a combination of self-attention and adversarial generative models can be implemented. The method generates diverse coloring results in two stages using two models, which has significant advantages over ordinary image coloring techniques.
[0059] like Figure 1 This embodiment provides a cartoon image coloring method that utilizes a Transformer model for low-resolution, low-color-depth coarse coloring, and then employs an adversarial generative model to reconstruct the resolution and color depth. This effectively improves the coloring effect of cartoon images and can produce a variety of different coloring results. The method includes the following steps:
[0060] S1. Obtain a comic image with size H×W, downsample the comic image to obtain a first resolution image; wherein, the first resolution image is a grayscale image.
[0061] In this embodiment, a comic image of size H×W is read in, the corresponding image format is decoded, and an H×W matrix is obtained. If the input image is a color image, it also needs to be grayscale processed and resampled to 64×64 based on the pixel area relationship.
[0062] S2. Input the first resolution image into a Transformer network based on a self-attention mechanism to extract features and predict the color distribution of each pixel in the first resolution image to obtain a coarsely colored image with color depth.
[0063] A 64×64 grayscale image is input into a Transformer network based on a self-attention mechanism to extract features and predict the color distribution of each pixel. The expected value of the predicted distribution on each pixel can be calculated or random sampling can yield one or more coloring results, resulting in a coarsely colored image of 64×64×3 with a color depth of 3-bit RGB.
[0064] S3. Input the coarsely colored image and the cartoon image into an adversarial generative enhancement network to reconstruct the resolution and color depth, and obtain a second resolution image.
[0065] S4. Upsample the second resolution image to obtain a finely colored image with a size of H×W.
[0066] The coarse-colored image and the original-size input image are fed into an adversarial generative enhancement network to reconstruct the resolution and color depth. The coarse-colored result is enhanced to an image of size 256×256 and color depth of 8 bits. Then, the image is restored to H×W through interpolation upsampling.
[0067] As an optional implementation, step S4 further includes step S5: evaluating the coloring effect of the obtained fine coloring image using a preset comprehensive objective index; wherein the objective index includes at least one of the following: PSNR index, SSIM index, LPIPS index, or FID index.
[0068] As an optional implementation method, the coloring effect is evaluated by combining objective and subjective indicators. Objective indicators include PSNR, SSIM, LPIPS, FID, etc., while subjective indicators include the observer's subjective feelings and the comparison between the original image and the generated image.
[0069] The following section provides a detailed explanation of the Transformer network based on the self-attention mechanism and the boosting network based on adversarial generation, with reference to the accompanying diagrams.
[0070] like Figure 2 As shown, this embodiment provides a colorization method based on a Transformer network with a self-attention mechanism, which predicts the pixel-level color distribution of a grayscale comic image through a self-attention mechanism module. The input two-dimensional image is converted into a tensor through an embedding layer. If a self-attention module is used, the qkv weights would be calculated between each pixel, requiring a large amount of computation. Therefore, by using an axial self-attention module, only the self-attention between rows and columns needs to be calculated, which can greatly reduce the amount of computation.
[0071] The Transformer network has an encoder and decoder structure. The encoder's output serves not only as input to the decoder but also as part of the loss function calculation to aid training. The decoder's output is not an image but a matrix, where each point represents the color distribution of the corresponding image pixel. Due to computational resource constraints, the output can only represent 3-bit RGB, a total of 512 colors; that is, each point in the output matrix is a 512-dimensional vector.
[0072] This model requires a large amount of data for training. A large number of color cartoon images are collected and converted to grayscale to serve as training data pairs. The grayscale images are input into the network, and the prediction results are compared with the original images using a loss function. The model weight gradient is then calculated using the loss function, and gradient descent is performed. This process is repeated multiple times to obtain the optimized model weights. The specific loss function is as follows:
[0073]
[0074]
[0075]
[0076] By inputting an image into a trained network, a tensor of size 64×64×512 can be obtained. Random sampling on 512 color distributions represented by a 512-dimensional vector can yield a 64×64 coloring result. Furthermore, depending on the sampling method, one or more coloring results can be obtained.
[0077] like Figure 2 As shown, this embodiment also provides an enhancement network based on adversarial generative networks to reconstruct resolution and color depth for low-resolution, low-color-depth colorization results. The generator part of the network is a U-net structure, consisting of a corresponding number of convolutional layers and upsampling layers, with short links between the feature maps output from the convolutional layers and the upsampling layers at the same level. Furthermore, a PatchGAN discriminator is used to calculate the GAN Loss for the images generated by the generator.
[0078] This model is also trained on a large amount of image data. Cartoon images are resampled to a low resolution of 64×64 based on pixel area relationships, and the color depth is downsampled from 8 bits to 3 bits, resulting in low-resolution, low-color-depth image-original image training data pairs. The low-resolution, low-color-depth image serves as the input image, and the original image serves as the label image. The model training process alternates between two training phases: First, the generator phase, where the input image is input to the generator to obtain the generated image. The loss function is calculated between the generator and the corresponding label image, and the generator weights are updated to minimize the loss function, enabling the generator to generate images closer to the labels. Second, the discriminator phase, where the input image is input to the generator to obtain the generated image. The generated image and the label image are then input to the discriminator, where the loss function is calculated, and the discriminator weights are updated to maximize the loss function value, allowing the discriminator to better distinguish the generated images. The generator and discriminator compete against each other during training to improve generation performance. In addition to the GAN Loss provided by the discriminator, L1Loss is introduced to enhance the details of the generated images. The objective function can be expressed as:
[0079] L L1 (G)=E x,y [||yG(x)||1]
[0080]
[0081] By inputting the output image of the Transformer network based on the self-attention mechanism into the trained boosting network, the resolution can be reconstructed to 256×256 and the color depth to 8 bits. Then, by interpolation and upsampling, the image can be restored to the original input size, and finally, one or more comic image coloring can be completed.
[0082] This embodiment also provides a comic image coloring system, including:
[0083] An image sampling module is used to acquire a comic image of size H×W, and downsample the comic image to obtain a first resolution image; wherein, the first resolution image is a grayscale image;
[0084] The image coloring module is used to input the first resolution image into a Transformer network based on a self-attention mechanism, extract features, and predict the color distribution of each pixel in the first resolution image to obtain a coarsely colored image with color depth.
[0085] The coloring optimization module is used to input the coarsely colored image and the comic image into an adversarial generation-based enhancement network to reconstruct the resolution and color depth, thereby obtaining a second resolution image.
[0086] The upsampling module is used to upsample the second resolution image to obtain a finely colored image with a size of H×W;
[0087] The resolution of the second resolution image is greater than that of the first resolution image, and the number of bit depths of the second resolution image is greater than that of the coarsely colored image.
[0088] This embodiment of the comic image coloring system can execute the comic image coloring method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0089] This embodiment also provides a comic image coloring device, including:
[0090] At least one processor;
[0091] At least one memory for storing at least one program;
[0092] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.
[0093] This embodiment of the comic image coloring apparatus can execute a comic image coloring method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0094] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.
[0095] This embodiment also provides a storage medium storing instructions or programs that can execute the comic image coloring method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.
[0096] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.
[0097] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0099] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0100] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0101] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0102] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0103] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0104] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
[0105] This patent was funded by the National Natural Science Foundation of China (NSFC) No. 62071183.
Claims
1. A method for coloring comic images, characterized in that, Includes the following steps: A comic image of size H×W is obtained, and the comic image is downsampled to obtain a first resolution image; wherein, the first resolution image is a grayscale image; The first resolution image is input into a Transformer network based on a self-attention mechanism for feature extraction, and the color distribution of each pixel in the first resolution image is predicted to obtain a coarsely colored image with color depth. The coarse-colored image and the cartoon image are input into an adversarial generative enhancement network to reconstruct the resolution and color depth, thereby obtaining a second-resolution image. The second resolution image is upsampled to obtain a finely colored image with a size of H×W; Wherein, the resolution of the second resolution image is greater than that of the first resolution image, and the number of bits of color depth of the second resolution image is greater than that of the coarsely colored image; The Transformer network based on the self-attention mechanism performs the following processing on the input grayscale image: grayscale image The input is processed by the encoder of the Transformer network; wherein the encoder consists of an embedding layer, a row self-attention module, and a column self-attention module; The output of the encoder is passed sequentially through a linear layer and a softmax layer to obtain... ; The output of the encoder is input into the decoder of the Transformer network for processing; wherein the decoder comprises an embedding layer, a conditional row self-attention module, and a conditional column self-attention module; The output of the decoder is passed sequentially through a linear layer and a softmax layer to obtain... The colored image is obtained by randomly sampling on the distribution or calculating the expected value; The adversarial generation-based boosting network is obtained through the following methods: The generator part of the boosting network adopts a U-net structure, which includes a corresponding number of convolutional layers and upsampling layers. The feature maps output by the convolutional layers are short-linked to the upsampling layers at the same level. The discriminator of the improved network is the PatchGAN discriminator, whose output is a matrix, where each point represents the true or false value of the corresponding small patch. Generative Adversarial Networks (GANs) require alternating training of the generator G and the discriminator D: during the training phase of the generator G, the false values generated by G are input into the discriminator D to calculate the GAN Loss, then the L1 Loss is calculated, and the network weights are updated; during the training phase of the discriminator D, the generator G generates false values, and the original input conditions are used as true values, which are then input into the discriminator D to calculate the Loss and update the network weights.
2. The method for coloring comic images according to claim 1, characterized in that, During training, the Transformer network calculates the model weight gradient using a loss function, performs gradient descent, and iterates multiple times to obtain the optimized model weights. The expression for the loss function L during the training process is as follows: In the formula, M and N represent the row and column dimensions of the downsampled matrix. This represents the color value of the pixel in row i and column j. This represents the input grayscale image matrix. This represents all pixels smaller than row i. This represents the pixel in row i that is less than column j. This represents the hyperparameters of a trained model.
3. The method for coloring comic images according to claim 1, characterized in that, The objective function of the adversarial generative network during training. The expression is as follows: In the formula, This represents the L1 Loss function of the generator G. This represents the mean. This represents the absolute value of the difference between the target y and the generated result G(x). This represents the GAN Loss provided by the discriminator. This represents an adjustable hyperparameter.
4. The method for coloring comic images according to claim 1, characterized in that, The process of acquiring a comic image with dimensions H×W includes: If the comic image is a color image, then the comic image is converted to grayscale.
5. The method for coloring comic images according to claim 1, characterized in that, The comic image coloring method further includes an evaluation step: The coloring effect of the obtained fine coloring image is evaluated using a preset comprehensive objective index; wherein the objective index includes at least one of the following: PSNR index, SSIM index, LPIPS index, or FID index.
6. A comic image coloring system for implementing the method according to any one of claims 1-5, characterized in that, include: An image sampling module is used to acquire a comic image of size H×W, and downsample the comic image to obtain a first resolution image; wherein, the first resolution image is a grayscale image; The image coloring module is used to input the first resolution image into a Transformer network based on a self-attention mechanism, extract features, and predict the color distribution of each pixel in the first resolution image to obtain a coarsely colored image with color depth. The coloring optimization module is used to input the coarsely colored image and the comic image into an adversarial generation-based enhancement network to reconstruct the resolution and color depth, thereby obtaining a second resolution image. The upsampling module is used to upsample the second resolution image to obtain a finely colored image with a size of H×W; The resolution of the second resolution image is greater than that of the first resolution image, and the number of bits of color depth of the second resolution image is greater than that of the coarsely colored image.
7. A comic image coloring device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-5.
8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-5.