An image coloring method, device and medium based on iterative optimization

Abstract

The application discloses an image coloring method and device based on iterative optimization and a medium, relates to the technical field of image processing, and comprises the following steps: acquiring a grayscale image; performing initial coloring on the grayscale image by using a trained conditional coloring model to obtain an initial coloring result; performing color evaluation on the initial coloring result by using a trained evaluation model, identifying a region in the initial coloring result that does not meet a set standard of color confidence, and obtaining an unsatisfactory region; performing re-coloring on the unsatisfactory region by using the trained conditional coloring model to obtain an updated coloring result; replacing the initial coloring result with the updated coloring result, repeating the re-coloring and evaluation processes, obtaining a final coloring result, and decoding the final coloring result to obtain a final color image. The application realizes higher reality and fidelity in image coloring.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image coloring method, device and medium based on iterative optimization. Background Technology

[0002] Many black-and-white photographs from bygone eras still exist today, often requiring the skillful hands of artists to add color and bring them to life. However, this manual coloring process is not only time-consuming but also labor-intensive. Employing a one-shot coloring paradigm, numerous methods have been proposed to reproduce the prior knowledge and intuition of human experts. These methods have made significant progress, providing high-quality results. Although some of these techniques incorporate additional conditions, they often still lack the iterative optimization typically performed by human experts, resulting in lower realism and fidelity in the image coloring. Summary of the Invention

[0003] The purpose of this application is to provide an image coloring method, device, and medium based on iterative optimization, which can achieve higher realism and fidelity in image coloring.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] In a first aspect, this application provides an image coloring method based on iterative optimization, comprising the following steps.

[0006] Obtain a grayscale image.

[0007] The grayscale image is initially colored using a trained conditional coloring model to obtain the initial coloring result.

[0008] The trained evaluation model is used to evaluate the initial coloring results, and the regions in the initial coloring results that do not meet the set criteria for color confidence are identified, thus obtaining unsatisfactory regions.

[0009] The unsatisfactory regions are recolored using the trained conditional coloring model to obtain updated coloring results.

[0010] The updated coloring result is used to replace the initial coloring result, and the recoloring and evaluation process is repeated to obtain the final coloring result. The final coloring result is then decoded to obtain the final color image.

[0011] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described iterative optimization-based image coloring method.

[0012] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described image coloring method based on iterative optimization.

[0013] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0014] This application provides an image colorization method, device, and medium based on iterative optimization. The method uses a trained evaluation model to evaluate the initial colorization result, identifying regions in the initial colorization result that do not meet a set color confidence standard, thus obtaining unsatisfactory regions. A trained conditional colorization model is then used to recolor these unsatisfactory regions. In the colorization stage, the colorization model generates initial color predictions. In the evaluation stage, the evaluation model is used to identify regions with low color confidence. In the recoloring stage, the unsatisfactory regions (i.e., regions with low color confidence) are recolored, allowing for targeted adjustments to regions requiring improvement. The adaptive feedback loop iteratively improves color quality and consistency, achieving higher realism and fidelity in colorization and enhancing image colorization quality. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an application environment diagram of an image coloring method based on iterative optimization in one embodiment of this application.

[0017] Figure 2 This is a flowchart illustrating an image coloring method based on iterative optimization, provided as an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of an iterative coloring-evaluation-recoloring process provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram illustrating the training and inference phases of image coloring according to an embodiment of this application.

[0020] Figure 5 This is a demonstration diagram of intermediate results of selective coloring provided in an embodiment of this application.

[0021] Figure 6 This is a schematic diagram illustrating the visual contrast results of unconditional coloring provided in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram illustrating the visual contrast results of conditional coloring provided in an embodiment of this application.

[0023] Figure 8 Ablation study diagrams for different loss settings used for training, provided as an embodiment of this application.

[0024] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Image colorization uses generative models to colorize grayscale images, and these methods are divided into conditional and unconditional methods. Generative models are further divided into single-step and multi-step methods, the latter being more suitable for the process of colorization, evaluation, and recoloring. This section will review these two types of models.

[0027] Unconditional coloring aims to automatically add color to grayscale images. Early methods treated it as a regression or classification task, focusing on learning the relationship between semantics and color from large datasets. To enhance color vibrancy, additional semantic information, such as class labels, semantic segmentation maps, or instance bounding boxes, is often incorporated. Recent advances have leveraged Generative Adversarial Networks (GANs), Transformers, and diffusion models to enhance coloring by incorporating prior knowledge, such as DGP, MGANPrior, GCP, BigColor, ColTran, and CT. 2 UniColor, DISCO, and DDColor are examples of color recognition technologies. However, their knowledge of color remains limited by the range of training data, restricting their ability to generate vibrant and realistic colors.

[0028] Conditional coloring has traditionally relied on graffiti guidance. Example-based methods simplify the process by copying colors from a reference image. Some techniques use AdaIN to convey a global color distribution, while others convey colors through pixel-level semantic alignment. Despite performance improvements, finding ideal reference images remains time-consuming and inefficient. Generative coloring offers an alternative by extending unconditional generative models to include grayscale image conditions. Methods include cINN using conditional normalized flow, SCC-DC using conditional VAEs, UniColor using Transformers, Diffusing Colors leveraging diffusion models, and GAN-based methods such as ColorFormer, BigColor, and DDColor.

[0029] Single-shot generation models, such as GANs and diffusion models, generate the final output in a single step. GANs have made significant progress in learning low-dimensional latent representations of natural color images, enabling their application in inpainting, super-resolution, and colorization. Diffusion models, driven by DDPM and DDIM, have led to the emergence of LDMs such as Stable Diffusion, setting benchmarks in text-to-image generation. ControlNet leverages LDMs to conditionally control pre-trained diffusion models through task-specific conditions, supporting multimodal inputs and broadening application scenarios. However, directly using grayscale images as conditional training for ControlNet results in significant texture and detail inconsistencies, ruining the goal of colorization.

[0030] Multi-step generative models, such as autoregressive models and masked visual token models, iteratively generate outputs by repeatedly feeding the input and partially completed outputs back into the model until the final result is obtained. Early autoregressive image models focused on pixel sequences, while VQGAN transforms these sequences into potential tokens for predicting the next token, similar to BERT. Regression prediction of color tokens has shown excellent results. Traditional methods treat images as a 1D token sequence in a raster scan order (from left to right, row by row), which is not ideal in terms of efficiency and representation. In contrast, masked generation models predict new tokens in all directions or generate multiple tokens simultaneously in a random order, while maintaining the autoregressive principle of predicting subsequent tokens based on known tokens.

[0031] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The image colorization method based on iterative optimization provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the grayscale image to be processed to server 104. After receiving the grayscale image, server 104 uses a trained conditional coloring model to perform initial coloring on the grayscale image, obtaining an initial coloring result. It then uses a trained evaluation model to evaluate the color of the initial coloring result, identifying areas in the initial coloring result that do not meet the set color confidence standard, thus obtaining unsatisfactory areas. The trained conditional coloring model is then used to recolor the unsatisfactory areas, obtaining an updated coloring result. This updated coloring result replaces the initial coloring result, and the recoloring and evaluation process is repeated to obtain the final coloring result. The final coloring result is then decoded to obtain the final color image. Server 104 can then feed back the obtained final color image for the grayscale image to terminal 102. In addition, in some embodiments, the image coloring method based on iterative optimization can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform image coloring processing on the grayscale image to be processed, or the server 104 can obtain the grayscale image to be processed from the data storage system and perform image coloring processing on the grayscale image to be processed.

[0033] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0034] In one exemplary embodiment, such as Figure 2 As shown, an image colorization method based on iterative optimization is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 205.

[0035] Step 201: Obtain a grayscale image. This grayscale image can be an uncolored image of any region.

[0036] Step 202: Use the trained conditional coloring model to perform initial coloring on the grayscale image to obtain the initial coloring result.

[0037] Step 203: Use the trained evaluation model to perform color evaluation on the initial coloring result, identify the regions in the initial coloring result that do not meet the set standard of color confidence, and obtain the unsatisfactory regions.

[0038] Step 204: Use the trained conditional coloring model to recolor the unsatisfactory region to obtain the updated coloring result.

[0039] Step 205: Replace the initial coloring result with the updated coloring result, repeat the recoloring and evaluation process to obtain the final coloring result, and decode the final coloring result to obtain the final color image.

[0040] Implementing steps 201 to 205 above provides a CAR method, an iterative coloring method involving coloring, evaluation, and recoloring. It resembles the iterative workflow of a professional colorist. This method combines a conditional coloring model and an evaluation model to reproduce the cycle of coloring, evaluation, and recoloring, similar to the color correction stage in film post-production. Unlike traditional single-step coloring methods, CAR employs a progressive multi-step approach. In the coloring stage, the coloring model generates initial color predictions. The evaluation stage uses the evaluation model to identify areas with low color confidence, while in the recoloring stage, well-colored areas are masked so that areas needing improvement can be targeted and adjusted using the coloring model. This adaptive feedback loop allows CAR to iteratively improve color quality and consistency, approaching the fine-tuning of a professional colorist. Experimental results show that CAR outperforms traditional single-step methods, achieving higher realism and fidelity in coloring.

[0041] Colorists begin with a basic sketch and progressively enhance it by adding or adjusting details. They evaluate the initial coloring, identify unsatisfactory areas, and recolor those areas. Knowing that perfection in coloring cannot be achieved overnight, they achieve a high degree of visual consistency and accuracy through multiple coloring-evaluation-recoloring cycles. This human-involved cyclical approach differs from the single-step coloring used in earlier methods.

[0042] This application introduces an iterative coloring method involving coloring, evaluation, and recoloring, such as... Figure 3As shown, this is a novel approach that simulates the colorization-evaluation-recoloring process, transitioning from single-step colorization to multi-step colorization. The left section outlines the iterative colorization process, the top image shows the results of the conditional colorization model during the colorization and recoloring steps, and the bottom image highlights the unsatisfactory areas identified by the evaluation model. The right section expands on the colorization iterations described in the left image. Notably, a close-up view of the recoloring results is appended at the top to highlight the quality improvements achieved through recoloring. CAR consists of two main models: an evaluation model that simulates the colorist's evaluation steps; and a conditional colorization model that incorporates expert knowledge and supports iterative color optimization. Specifically, the evaluation model identifies unsatisfactory latent codes in the colorization results, while the conditional colorization model combines a masked visual token model and a conditional diffusion model to perform initial colorization based on the grayscale image and recolor unsatisfactory latent codes from the mask.

[0043] Key advancements in CAR include: (1) using an evaluation model to identify unsatisfactory latent codes requiring optimization; (2) retaining satisfactory latent codes, along with brushstrokes and grayscale images, as input to a masked autoregressive model; (3) enhancing results through a conditional diffusion model instead of directly using an autoregressive model to predict the final image; and (4) achieving high-quality and consistent output through iterative optimization of coloring via a multi-step process. This shifts the coloring paradigm from single-step coloring to multi-stage coloring to simulate a colorist's color correction process. This application demonstrates that the masked autoregressive model provides controllability and user interactivity, while the diffusion model enables diverse coloring results. Compared to previous automatic coloring methods, the iterative coloring framework of coloring, evaluation, and recoloring provided in this application achieves state-of-the-art performance and generalization capabilities.

[0044] In another exemplary embodiment of this application, step 202 may include steps 301 to 304.

[0045] Step 301: Encode the grayscale image and the color cue image respectively to obtain a gray token sequence and a color cue token sequence; the color cue image is an image obtained by color-marking a portion of the grayscale image; the gray token sequence includes several gray tokens; the color cue token sequence includes several color cue tokens.

[0046] Step 302: Generate an intermediate token sequence based on the gray token sequence and the color-coded token sequence. Specifically, step 302 includes: concatenating the gray token sequence and the color-coded token sequence to obtain a concatenated token sequence; and processing the concatenated token sequence using an MLP to obtain the intermediate token sequence.

[0047] Step 303: Concatenate the intermediate token sequence and the masking sequence to obtain a concatenated masking sequence; the masking sequence is a token sequence of the same size as the intermediate token sequence that is completely masked.

[0048] Step 304: Input the spliced ​​mask sequence into the trained conditional coloring model for initial coloring to obtain the initial coloring result.

[0049] Step 203 above specifically includes: inputting the initial coloring result and the gray token sequence into the trained evaluation model, evaluating the color confidence in the initial coloring result, identifying the regions whose color confidence in the initial coloring result does not meet the set standard as unsatisfactory regions, and masking the unsatisfactory regions to obtain the updated masking mask sequence.

[0050] The coloring framework used in this application consists of a tokenizer, a conditional coloring model, and an evaluation model.

[0051] (1) The Tokenizer is used in step 301 to encode the grayscale image and the color cue image respectively, obtaining a grayscale token sequence and a color cue token sequence. The Tokenizer encodes the image. The Tokenizer is a KL-16Tokenizer, which is based on the VQ-16Tokenizer and originates from VQ-GAN. Unlike VQ-GAN, the KL-16Tokenizer uses Kullback-Leibler (KL) divergence for regularization instead of vector quantization. "16" indicates the stride of the Tokenizer.

[0052] The main network components of the Tokenizer include a Convolutional Neural Network (CNN), an embedding layer, and a Transformer encoder. The CNN's main function is to extract local image features. The embedding layer maps the image information extracted by the CNN to a vector space. Finally, the transformer enables the model to learn the relationships between different image regions. The Tokenizer's function is to segment the image into blocks, then extract image features through its network structure to form a sequence of image tokens. Specifically, step 301 involves segmenting the grayscale image and the color-coded image to obtain several first and second segmentation blocks. Each first or second segmentation block is input into the Tokenizer to obtain a gray token corresponding to each first segmentation block or a color-coded token corresponding to each second segmentation block. All gray tokens corresponding to the first segmentation blocks form a gray token sequence, and all color-coded tokens corresponding to the second segmentation blocks form a color-coded token sequence.

[0053] (2) Conditional coloring model (i.e.) Figure 4 The colorization model in this paper comprises a visual mask token model and a conditional diffusion model. The visual mask token model processes the concatenated mask sequence to obtain a conditional token sequence. The conditional diffusion model iteratively adds noise to the conditional token sequence in multiple steps to generate an intermediate colorization sequence. The conditional diffusion model includes several residual blocks; each residual block consists of a LayerNorm, a linear transformation, and a SiLU activation function layer. The conditional diffusion model uses a compact MLP containing several residual blocks. Each residual block includes a LayerNorm, a linear transformation, a SiLU activation, and residual connections. We use three blocks containing 1024 channels each. The conditional diffusion model is conditioned on a vector z (conditional token sequence) generated by the masked visual token model and concatenated with a noise scheduler at time step t. This concatenated vector is conditionalized through the LayerNorm layer in the MLP using AdaLN, thus achieving effective refinement during the colorization process.

[0054] The conditional colorization model comprises a masked visual token model and a conditional diffusion model. The colorization framework accepts three inputs: a grayscale image of the target color content, user-provided color strokes for local cues (i.e., color cue images), and the colorization result from the previous iteration. These inputs are converted from pixel-level latent representations using a tokenizer. The evaluation model then identifies unsatisfactory latent encodings in the colorized image and replaces them with [MASK]Tokens. The masked visual token model combines the unmasked latent features with stroke information from the grayscale and color cue images to generate conditional (conditional token sequence) inputs for the conditional diffusion model, which iteratively refines the colorized image.

[0055] The 256×16 tokens (token sequence) extracted by the tokenizer are concatenated along the channel dimension with the latent codes from grayscale and color cue images (the latent code for grayscale images is a gray token sequence, and the latent code for color cue images is a color cue token sequence) to form a 256×32 code, which is the concatenated token sequence. This concatenated token sequence is processed by an MLP with an output size of 256×768 to obtain the intermediate token sequence. Unsatisfactory tokens identified by the evaluation model are masked and replaced with [MASK] tokens, which are also processed by an MLP. The two processed token sets are concatenated along the width dimension and enhanced by positional embedding, i.e., the intermediate token sequence and the masking sequence are concatenated to obtain the concatenated mask sequence. The generated sequence is input into a Transformer inspired by the original Transformer, i.e., a masked visual token model, and adapted in ViT. We adopt a bidirectional attention mechanism similar to that in masked autoencoders, allowing known tokens to provide information to unknown tokens, thereby promoting effective communication between tokens.

[0056] (3) Evaluation model: This model is used to evaluate the quality of the colorization result generated by the conditional diffusion model relative to the original grayscale image. It uses a classifier to process tokens from the grayscale image and the colorization result, generating a score through softmax output. The higher the score, the higher the confidence level that the colorization result is unsatisfactory. This evaluation model is based on an MLP-Mixer architecture, relies entirely on MLP, and iterates on tokens and feature channels.

[0057] The evaluation model (primarily MLP-Mixer) consists of: fully connected layers, and N Mixer layers (N being the number of patches into which the image is segmented, i.e., the number of image blocks). The Mixer layers are still composed of MLPs. In other words, the evaluation model comprises a first fully connected layer, a Mixer module, and a second fully connected layer connected sequentially; the Mixer module contains several Mixer layers. Its function is to evaluate which colorings in the generated coloring token sequence are good and which are bad.

[0058] The visual mask token model (mainly the MAE architecture) consists of an encoder-decoder structure. The entire processing flow is as follows: the user inputs a grayscale image... g You can choose to add an extra color cue point I. s Alternatively, no tokens can be added. After encoder processing, the grayscale image becomes a 256*16 sequence containing gray tokens, and the color cue image becomes a 256*16 sequence of color cue tokens containing color information on some of the tokens. The two are concatenated to form a 256*32 concatenated token sequence. After MLP processing, it becomes a 256*768 intermediate token sequence.

[0059] During the initial coloring process, the intermediate token sequence and the masking sequence (the entire token sequence is masked) are concatenated. Inside the visual mask token model, the masking sequence learns the preceding 256 token sequences through an internal transformer layer, then discards the preceding 256 token sequences, and the remaining token sequence is used as a condition in the conditional diffusion model. After t-step denoising, a token sequence with the coloring result is generated, serving as the initial coloring result.

[0060] The initial coloring result (the token sequence generated by the conditional diffusion model) and the initially acquired gray token sequence are repeatedly processed on the tokens and feature channels under the different effects of multiple MLP layers within the evaluation model. Feature information is extracted and compared. That is, the image feature information in the tokens of the generated color image is compared with the image feature information of the tokens of the original input gray image. Finally, a score for each token is output. Unsatisfactory areas with scores exceeding a threshold are re-masked, and then the next iteration is performed.

[0061] In subsequent iterations, the unsatisfactory parts of the evaluation model that align with the evaluation results are masked, while the normal output token sequence is used for the satisfactory parts, resulting in an updated occlusion mask sequence. The token sequence fused from grayscale and color cues is no longer concatenated with the full occlusion mask sequence. Instead, it is concatenated with the unsatisfactory parts of the evaluation model that align with the evaluation results, and the normal output token sequence for the satisfactory parts. In other words, the intermediate token sequence is concatenated with the updated occlusion mask sequence to obtain an updated concatenated mask sequence. This updated concatenated mask sequence passes through the transformer structure of the visual mask token model. After the mask part learns information from the previously concatenated token sequence and the token blocks already containing color information, all tokens except those masked are discarded. It is then fed into the conditional diffusion model. The subsequent process is largely the same as the first iteration, except that the conditional diffusion model no longer generates all tokens, but rather the colored tokens corresponding to the masked parts. These are then combined with the previously unmasked tokens containing colored information to form a complete colored token sequence, which is then fed into the evaluation model for evaluation. After repeated iterations, the color image is finally decoded by the decoder.

[0062] Step 204 above specifically includes: replacing the masking mask sequence with the updated masking mask sequence, returning to the step of "concatenating the intermediate token sequence and the masking mask sequence" in step 303 to obtain the updated concatenated mask sequence; inputting the updated concatenated mask sequence into the trained conditional coloring model to obtain the updated coloring result.

[0063] The evaluation model and the conditional coloring model work together to perform the coloring process, such as Figure 4 As shown on the left. Starting with a zero image as a priori result, the conditional coloring model applies user-provided color cues to colorize the grayscale image. In subsequent iterations, the evaluation model identifies unsatisfactory regions exceeding a threshold and masks them for optimization. The input is encoded, and the unsatisfactory regions identified by the evaluation model are replaced with [MASK]Tokens, which are processed by a masked visual token model to generate a conditional input conditionalized diffusion model.

[0064] In each iteration, the conditional diffusion model predicts all tokens in parallel, retaining only those whose evaluation model score is above a threshold. Tokens with lower scores are re-predicted in the next iteration. The threshold is gradually lowered, allowing token generation to be completed in several optimization loops. The conditional diffusion model uses conditional probabilities p(z|y) for prediction, where y includes the unmasked token, the grayscale image, and the user's color cue. From z... TStarting at ~N(0,I), the backdiffusion process generates z0 aligned with p(z|y). This iteration automates the colorist's coloring-evaluation-recoloring process. It becomes interactive by replacing the evaluation model with human input: the user marks incorrect areas, providing additional color cues, and in the next iteration, the conditional coloring model optimizes the results based on coloring priors and user feedback.

[0065] In the colorization-evaluation-recoloring process flowchart, during inference, the evaluation model checks the colorization quality of previous iterations on the grayscale image, masks any poor markers, and fills in missing markers with [MASK] markers to create a partial result. If it's the first iteration, previous results are treated as zero. Then, the grayscale image and user-provided color cues are encoded to form the colorization condition, which, combined with the partial colorization, is fed into the visual mask token model. This output instructs the conditional diffusion model to convert Gaussian noise into a color image in the reverse process. During training, the conditional colorization model (including the visual mask token model and the conditional diffusion model) and the evaluation model are independently tuned. To simulate the function of the evaluation model during inference, color markers on the ground are randomly masked, and the evaluation model is tuned into a classifier.

[0066] The training process of the evaluation model and the conditional coloring model will be introduced below.

[0067] The evaluation model measures the quality of the coloring, while the conditional coloring model uses feedback from the evaluation model to optimize the results. Although they collaborate to generate the final output during inference, they are trained independently. This separation of training makes the optimization of the coloring process more efficient.

[0068] Evaluation Model Training: The evaluation model is compatible with the output of the conditional diffusion model, operating on a token basis. The input consists of two tokens: one from the grayscale image and the other from the colorized image. It outputs a score representing the quality of the colorized token relative to the grayscale token; a lower value indicates a better result. The evaluation model is trained using cross-entropy because logistic values ​​effectively reflect the quality of the colorized result, similar to the work of a colorist in the evaluation step.

[0069] Conditional coloring model training: This work uses a conditional diffusion model to model the conditional probability p(z|y), where z represents the real token and y is generated by the masked visual token model. To simulate the colorist's evaluation, a random mask is applied to remove tokens from the previous coloring result. For the initial iteration, all uncolored tokens are masked. During training, the random mask ratio is sampled between [0.7, 1.0].

[0070] Let ∈R dLet N(0,I) be the noise sampled from N(0,I), and t be the noise scheduling time step. Define noise scheduling. The loss function is formulated as the denoising objective.

[0071] L diff =E ε,t [||ε-ε θ (z t |y,t)|| 2 (1);

[0072] In the formula, L diff It is the diffusion loss function of the conditional diffusion model, used to measure the difference between the noise predicted by the model and the actual noise; E ε,t ε is the expectation of the noise and time step t; ε is a noise vector sampled from a standard Gaussian distribution N(0,I); ε θ This represents a network parameterized by θ, which is contaminated with a noise vector z. t The input is time step t, and the condition vector y is the condition; ||ε-ε θ (z t |y,t)|| 2 This represents the mean square error (MSE) between the predicted noise and the actual noise.

[0073] Where the noise pollution vector z t Given by formula (2):

[0074]

[0075] In the formula, z t The noise data is generated in step t, and z is the original input of the model. These are noise scheduling parameters, similar to the previous use of heavyweight architectures such as U-Net and DiT as noise estimators ε. θ Unlike other approaches, we employ a lightweight MLP network. The motivation for this shift is that conditional diffusion models focus on individual tokens rather than the entire image.

[0076] To enhance color generation performance, this application optimizes the color generation process by explicitly optimizing the cyclic consistency between the generated image and the ground truth image. Specifically, when the noise added in formula (2) is small (corresponding to a low time step t), it disrupts consistency, thus achieving effective reward fine-tuning. Conversely, when t is large, z t Approximate random noise z T directly from z t Predicting z'0 is prone to significant distortion. Therefore, this application improves the accuracy of the perturbation image z'0 at a low time step t. T By applying single-step sampling to predict the original image z'0, the image fidelity is maintained, and the color generation can be represented as shown in Equation (3).

[0077]

[0078] In the formula, z0 represents the final generated image token, i.e., the coloring result output by the conditional diffusion model, and z'0 represents the original image token predicted through single-step sampling; z t It is a noisy image at time step t, ε θ (z t |y,t) is a noise estimator, where the symbols have the same meaning as in equation (1); This represents the noise correction factor, used to adjust noise levels. It is a scaling factor used to adjust the step size.

[0079] Then, this application applies the denoised image z'0 for reward fine-tuning, as shown in Equation (4), where h(z) represents the image edge detection (Holistically-Nested Edge Detection, HED) operator, which calculates the HED image from the image.

[0080] L reward =E ε,t [||h(z0)-h(z'0)|| 2 (4);

[0081] In the formula, L reward It is a reward-fine-tuned loss function, the purpose of which is to make the HED map of the generated image z'0 closer to the HED map of the real image z0, E ε ,t is the expectation of noise and time step t, ||h(z0)-h(z'0)|| 2 It is the mean square error (MSE) between the real image and the generated HED image, and h(·) represents the HED operator.

[0082] Reward fine-tuning guides the conditional diffusion model in generating images, thereby restoring cycle consistency, improving adherence to conditions during generation, and prompting the masked visual token model to generate more accurate color conditions. The total loss combines the diffusion training loss and the reward fine-tuning loss; therefore, the total loss function of the conditional diffusion model is expressed as:

[0083]

[0084] Among them, L total T represents the total loss function; λ is a hyperparameter that adjusts the weights of the reward-adjusted loss; threshold The time-step threshold is a hyperparameter that adjusts whether the total loss function at the current time step includes the reward fine-tuning loss, used to determine whether the noisy image z should be used. t Make minor adjustments to the rewards.

[0085] Training Data Preparation: Training the conditional colorization model requires cue points to provide color information for grayscale images, while evaluating the model requires color-grayscale token pairs indicating good and bad coloring results. Since collecting large-scale annotated datasets is impractical, this application uses the SLIC algorithm for superpixel segmentation. The average color of each segment is assigned to its corresponding superpixel, creating a superpixel image. During training, these superpixels are randomly selected as cue points, generating different color cue images.

[0086] For the evaluation model, high-performing token pairs are obtained by encoding the color image and its grayscale version using a KL-16 tokenizer. Low-performing token pairs are generated by colorizing the grayscale input image using a trained conditional colorization model that processes the zero-stroke image. This application computes HED images of the ground truth and the colored output, identifying regions where colorization failed due to issues such as color overflow or omission. The tokens for these regions are classified as low-performing color-grayscale token pairs.

[0087] Intermediate results of CAR iterative coloring are shown below. Figure 5 As shown, given a grayscale input, CAR produces the final colorization result through multiple steps. In the first step, the conditional colorization model generates an initial colorization result based on the input. Then, the evaluation model identifies unsatisfactory latent codes with higher scores. In subsequent steps, the conditional colorization model uses the input and satisfactory latent codes to recolor the unsatisfactory regions. The final colorization result is obtained after several evaluation-recoloring iterations.

[0088] CAR is a novel coloring method that simulates the iterative coloring process of a colorist. Experiments are presented below to demonstrate the effectiveness of the proposed iterative optimization-based image coloring method.

[0089] The experimental setup is as follows:

[0090] (1) Dataset: The method in this application is trained on ImageNet and OpenImage. Evaluation is performed on (i) ImageNet images; (ii) COCO images; and (iii) 500 natural scene photographs. Evaluating the colorization performance on grayscale images from COCO and the Internet is crucial because the color distribution of these images is often different from that in ImageNet and OpenImage, which challenges the generalization ability of the model.

[0091] (2) Baseline: To verify the effectiveness of the method in this application, it is benchmarked against seven state-of-the-art unconditional coloring methods: DISCO, CT, etc. 2The application compares the following baselines: ColorFormer, BigColor, UniColor, DDColor, and CtrlColor. It uses the official code and pre-trained weights for all baselines. For UniColor, the application evaluates its automatic coloring capabilities without user-provided prompts. For conditional coloring, the application compares its method with UniColor and iColoriT, which uses brush strokes.

[0092] (3) Evaluation Metrics: To assess perceived realism, this application uses FID, which measures the distributional similarity between the generated image and the ground reality image. To assess color vibrancy, this application uses CF to quantify the vibrancy of the generated image. However, since a high CF score does not necessarily indicate higher visual quality, this application also incorporates PSNR assessment, following previous research, to provide additional insights. Furthermore, this application conducted user research to measure subjective preferences, ensuring a comprehensive evaluation that combines quantitative metrics and human perception assessment.

[0093] (4) Training Configuration: This application is implemented in PyTorch and trained on 10 3090 GPUs. The AdamW optimizer was used with β1 = 0.9 and β2 = 0.95 for 200 epochs. The learning rate was set to 10. -5 This application maintains the exponential moving average (EMA) of the model parameters with a momentum of 0.9999.

[0094] The visual contrast results of unconditional coloring are as follows Figure 6 As shown, qualitative visualization of image coloring is demonstrated using various automatic coloring methods, including DISCO, CT2, ColFormer, BigColor, UniColor, DDColor, and CtrlColor. Compared with related techniques, the method in this application can generate more natural and realistic colors.

[0095] CAR is the first method to implement an iterative coloring-evaluation-recoloring process, therefore direct comparison with intermediate results in the iteration process of related techniques is impractical. Therefore, this application... Figure 5 The intermediate results are shown to illustrate how the coloring results progressively improve as the evaluation model detects unsatisfactory potential codes and the conditional coloring model optimizes them.

[0096] This application categorizes the final coloring results into two types: unconditional coloring and conditional coloring. For each type, this application provides qualitative and quantitative comparisons.

[0097] 1) Qualitative comparison: This application visualizes the results of unconditional coloring and conditional coloring for quantitative comparison.

[0098] Unconditional coloring: This application compares its method with seven related technical methods for colorizing grayscale images without user prompting. The results are as follows: Figure 6 As shown. It is worth noting that the ground-based image is omitted because the evaluation should not be based solely on color similarity, given the multimodal uncertainty inherent in the coloring task. The method in this application generates more natural and vibrant results while reducing color spillover.

[0099] Conditional Coloring: This application evaluates its method in comparison with four stroke-based coloring methods, where the user provides color strokes as conditions. For fairness, this application does not generate cue points from superpixel images, but instead follows UniColor and assigns the average color of the cells to the cue points. Figure 7 As shown, the method of this application smoothly and consistently propagates brushstroke color across the entire object. Furthermore, for areas without specified cue points, the method generates diverse and vibrant colors. Unlike unconditional coloring, the input includes both grayscale images and stroke-based color cues. This application provides a qualitative comparison with iColoriT, UniColor, DDColor, and CtrlColor, demonstrating that the method of this application generates more natural and realistic images.

[0100] 2) Quantitative Comparison: This application collected statistical data on unconditional and conditional colorization results, covering ImageNet, COCO, and natural scene images. This application compared the performance of its method with unconditional and conditional colorization methods on the test datasets, and the quantitative results are shown in Table 1. The upper part of the table corresponds to unconditional colorization, while the lower part focuses on conditional colorization. All methods were evaluated using their official code and weights. The method in this application achieved the lowest value on the FID (Fréchet Inception Distance) metric, demonstrating its ability to generate high-quality colorization results and showcasing its strong generalization ability. Although the color vibrancy score reflects the vividness of the image, and some methods outperformed this application's method on this metric, a higher score is not always associated with better visual quality. To further evaluate performance, this application calculated the PSNR (Peak Signal-to-Noise Ratio) to measure the deviation between the generated image and the real image. The method in this application achieved the highest PSNR on all datasets, confirming its ability to generate more natural and realistic colorization results.

[0101] To evaluate the realism of the output results on natural scene photographs, this application conducted a study on "preference and non-preference" perception. An experiment was conducted with 25 randomly selected participants from three age groups (old, middle-aged, and young). Participants were shown five grayscale images with corresponding brushstroke color cues, along with the color results generated by the application's model and compared models. They were asked to select their favorite color image. Data for each algorithm was collected from only 25 participants. The results of unconditional and conditional colorization are shown in Table 1.

[0102] This application provides a comprehensive quantitative evaluation of multiple objective benchmarks, including ImageNet, COCO, and wild images (i.e., unlabeled images not present in the aforementioned training sets), measuring FID, CF, and PSNR metrics. Furthermore, we conduct user research to evaluate subjective preferences. Our colorization framework achieves the highest FID score, demonstrating superior perceived realism, while the highest preference rate confirms its overall quality compared to existing methods.

[0103] Table 1 Quantitative Comparison

[0104] ImageNet COCO UserStudy FID↓ CF↑ PSNR↑ DISCO 10.29 40.95 20.72 CT2 5.51 38.48 23.50 ColFormer 4.91 38.00 23.10 BigColor 5.36 39.74 21.24 UniColor 9.46 39.01 22.17 DDColor 4.38 37.66 23.54 CtrlColor 8.88 47.17 24.67 Ours 4.35 39.13 28.36 iColoriT 5.01 34.33 28.86 UniColor 7.04 36.16 24.06 CtrlColor 4.67 32.36 26.14 Ours 4.23 36.59 28.99

[0105] In Table 1, Ours refers to the coloring framework in this application, including the Tokenizer, conditional coloring model, and evaluation model.

[0106] CAR uses two loss terms to train the coloring network: L diff This refers to the diffusion loss used to train the conditional diffusion model; L reward This is used to ensure that the HED (Holistically-Nested Edge Detection) edges of the colorized result are aligned with the original color image; it is the reward-based fine-tuning loss function. Relying solely on diffusion loss leads to a degraded colorization quality, while combining it with reward loss significantly improves performance. This improvement is evident in... Figure 8 The improvement is clearly demonstrated in both unconditional and conditional coloring scenarios. The performance enhancement stems from resolving the issues of missing colors (first row) and color overflow (second row), which distort the HED edges of the color output. By incorporating this constraint, the model is guided to prioritize color edges, a key factor in achieving higher visual quality.

[0107] Ablation studies using different loss settings for training, such as Figure 8As shown, using diffusion loss alone leads to color gaps and brilliance issues, reducing the final coloring result. However, combining it with reward loss can improve the coloring results for both unconditional (first row) and conditional (second row) coloring.

[0108] This application proposes a novel iterative image colorization framework that mimics the colorization process of a human colorist. The main contributions of this application are the conditional colorization model and the evaluation model. The evaluation model identifies regions with unsatisfactory colorization results, while the conditional colorization model specifically optimizes for these masked regions. This framework surpasses current state-of-the-art benchmark methods both qualitatively and quantitatively, providing controllable and editable colorization results. This application has the potential to advance the development of a wide range of computer vision tasks.

[0109] This application also provides an application scenario in which the above-mentioned image colorization method based on iterative optimization is applied. Specifically, the image colorization method based on iterative optimization provided in this embodiment can be applied to image coloring scenarios. Image coloring scenarios include an image acquisition stage and an image coloring stage; a grayscale image enters the image coloring stage from the image acquisition stage, and the corresponding final color image is obtained through human-computer collaboration. The image colorization method based on iterative optimization provided in this embodiment belongs to the image coloring stage. Specifically, in the image coloring stage for grayscale images, a trained conditional coloring model can be used to initially colorize the grayscale image to obtain an initial coloring result. A trained evaluation model is then used to evaluate the color of the initial coloring result, identifying regions in the initial coloring result that do not meet the set standard of color confidence, thus obtaining unsatisfactory regions. The trained conditional coloring model is then used to recolor the unsatisfactory regions to obtain an updated coloring result. The updated coloring result replaces the initial coloring result, and the recoloring and evaluation process is repeated to obtain the final coloring result. Finally, the final coloring result is decoded to obtain the final color image.

[0110] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image colorization processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements an iterative optimization-based image colorization method.

[0111] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0112] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0113] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0116] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0117] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image colorization method based on iterative optimization, characterized in that, The image colorization method based on iterative optimization includes: Obtain a grayscale image; The initial colorization of the grayscale image is performed using a trained conditional colorization model to obtain an initial colorization result. Specifically, this includes: encoding the grayscale image and the color cue image respectively to obtain a gray token sequence and a color cue token sequence; the color cue image is an image obtained by color-marking a portion of the grayscale image; the gray token sequence includes several gray tokens; the color cue token sequence includes several color cue tokens; generating an intermediate token sequence based on the gray token sequence and the color cue token sequence; concatenating the intermediate token sequence with a masking sequence to obtain a concatenated mask sequence; the masking mask sequence is a token sequence of the same size as the intermediate token sequence that is completely masked; inputting the concatenated mask sequence into the trained conditional colorization model for initial colorization to obtain the initial colorization result; the conditional colorization model includes a visual mask token model and a conditional diffusion model; the visual mask token model is used to process the concatenated mask sequence to obtain a conditional token sequence; the conditional diffusion model is used to perform multi-step noise-adding iterations on the conditional token sequence to generate an intermediate colorization sequence. The initial coloring result is evaluated using a trained evaluation model to identify regions in the initial coloring result that do not meet the set standard for color confidence, thus obtaining unsatisfactory regions. Specifically, this includes: inputting the initial coloring result and the gray token sequence into the trained evaluation model; evaluating the color confidence in the initial coloring result; identifying regions in the initial coloring result whose color confidence does not meet the set standard as unsatisfactory regions; and masking the unsatisfactory regions to obtain an updated masking sequence. The unsatisfactory regions are recolored using the trained conditional coloring model to obtain updated coloring results; The updated coloring result is used to replace the initial coloring result, and the recoloring and evaluation process is repeated to obtain the final coloring result. The final coloring result is then decoded to obtain the final color image.

2. The image colorization method based on iterative optimization according to claim 1, characterized in that, Generating an intermediate token sequence based on the gray token sequence and the color-coded token sequence specifically includes: The gray token sequence and the color-coded token sequence are concatenated to obtain a concatenated token sequence; The concatenated token sequence is processed using MLP to obtain the intermediate token sequence.

3. The image colorization method based on iterative optimization according to claim 1, characterized in that, The conditional diffusion model includes several residual blocks; each residual block includes a LayerNorm, a linear transformation, and a SiLU activation function layer.

4. The image colorization method based on iterative optimization according to claim 1, characterized in that, The trained conditional coloring model is used to recolor the unsatisfactory regions to obtain updated coloring results, specifically including: Replace the masking mask sequence with the updated masking mask sequence, and return to the step of "concatenating the intermediate token sequence and the masking mask sequence" to obtain the updated concatenated mask sequence; The updated spliced ​​mask sequence is input into the trained conditional coloring model to obtain the updated coloring result.

5. The image colorization method based on iterative optimization according to claim 1, characterized in that, The evaluation model includes a first fully connected layer, a Mixer module, and a second fully connected layer connected in sequence; the Mixer module includes several Mixer layers.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the iterative optimization-based image coloring method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the image coloring method based on iterative optimization as described in any one of claims 1-5.

Images