Controllable image coloring method based on text coding optimization

Abstract

The embodiment of the invention provides a controllable image coloring method based on text coding optimization, and belongs to the technical field of image processing, and the method specifically comprises the steps: coding a to-be-colored grayscale image, and coding a target text cue word; constructing a negative color sample cue word set and encoding; optimizing the to-be-optimized code by using the target loss function; inputting the optimal code and the grayscale image code into an SDM model to obtain an initial coloring image; inputting the initial coloring image and the to-be-colored grayscale image into a space alignment module to obtain a repaired coloring image; freezing the optimal code and finely adjusting the SDM model; performing semantic interpolation to obtain a final text code; inputting the final text code and a coloring image code corresponding to the restored coloring image into the fine-tuned SDM model to obtain a target coloring image; and inputting the target colored image and the to-be-colored grayscale image into a space alignment module to obtain a final colored image. Through the scheme disclosed by the invention, the coloring efficiency, controllability and accuracy are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of image processing technology, and in particular to a controllable image colorization method based on text encoding optimization. Background Art

[0002] At present, the automatic image colorization method refers to inferring the missing color information only from the input grayscale image through deep learning or other methods. It is essentially an underdetermined problem. The text-guided automatic image colorization method can additionally accept text prompt words as conditions, so that the colorization result matches the text semantics, thereby realizing a controllable automatic image colorization method. This method can be widely used in old photo restoration, advertising design and other fields.

[0003] Traditional methods for automatic image colorization include user interactive labeling, reference image colorization, etc. These methods usually require a lot of manual intervention and are not ideal when dealing with objects with complex details or uncertain colors. Although the reference image-based colorization method can provide reasonable color selection to a certain extent, it usually requires complex image alignment or image segmentation operations. With the rise of deep learning, neural networks are trained through large-scale image datasets to achieve image colorization in an end-to-end manner. For example, generative adversarial networks are used to produce realistic color distributions by training generators and discriminators. However, this method performs poorly when the fabric color control is poor and the color interaction between objects is blurred. This type of deep learning method has high training cost requirements and cannot achieve controllable coloring results.

[0004] It can be seen that there is an urgent need for a controllable image colorization method based on text encoding optimization with high colorization efficiency, controllability and accuracy. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a controllable image colorization method based on text encoding optimization, which at least partially solves the problems of poor colorization efficiency, controllability and accuracy in the prior art.

[0006] The present disclosure provides a controllable image coloring method based on text encoding optimization, including:

[0007] Step 1, encoding the grayscale image to be colored to obtain grayscale image encoding, and encoding the target text prompt word input by the user to obtain target text encoding;

[0008] Step 2, construct a negative color sample prompt word set and encode it to obtain a negative color text encoding set;

[0009] Step 3, initializing the code to be optimized with a random vector and optimizing the code to be optimized using the target loss function to obtain the optimal code;

[0010] Step 4, input the optimal encoding and grayscale image encoding into the SDM model to obtain an initial colored image based on diffusion prior;

[0011] Step 5, inputting the initial colored image and the grayscale image to be colored into a spatial alignment module to obtain a repaired colored image;

[0012] Step 6, freeze the optimal encoding and fine-tune the SDM model;

[0013] Step 7, based on the target text encoding and the optimal encoding, semantic interpolation is performed using a preset formula to obtain the final text encoding;

[0014] Step 8, input the final text code and the color image code corresponding to the repaired color image into the fine-tuned SDM model to obtain the target color image corresponding to the target text prompt word;

[0015] Step 9: Input the target colored image and the grayscale image to be colored into a spatial alignment module to obtain the final colored image.

[0016] According to a specific implementation of the embodiment of the present disclosure, before step 5, the method further includes:

[0017] Performing a preprocessing operation on the sample image data set to obtain a triplet data pair, wherein the triplet data pair includes a grayscale image, a texture-damaged color image, and an original color image;

[0018] Based on the triplet data pairs, the spatial alignment module of the CoCosNet network structure is trained using the Adam optimizer and the cosine decay learning rate strategy with Warmup.

[0019] According to a specific implementation method of the embodiment of the present disclosure, the target loss function is:

[0020] L Total =L DM +λL Color

[0021] Among them, L DM represents the noise prediction loss used in the default diffusion model training, L Color represents the contrast loss based on CLIP guidance, and λ is the contrast loss weight based on CLI guidance.

[0022] According to a specific implementation of the embodiment of the present disclosure, the noise prediction loss L DM The expression is

[0023]

[0024] Where t is a randomly selected time step, ∈t is the noise added in the forward process time step t, is the grayscale image x gray The noisy image encoding at time step t, ∈ θ (·) is the neural network for predicting noise;

[0025] The contrast loss L Color The expression is

[0026]

[0027] Where L is the number of negative color samples, e T is the CLIP image encoding corresponding to the colored image, e + Encode the text. is L negative color text encodings, and exp(·) means calculating the inner product of two vectors and then performing a power operation with a natural number as the base.

[0028] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:

[0029] Step 3.1, randomly select time step t and encode the grayscale image According to the preset noise adding formula, the noise adding process is performed to obtain the grayscale image coding after the noise adding Among them, the preset noise addition formula is:

[0030]

[0031] in, is a predefined hyperparameter in SDM, which indicates the true data retention rate during the noise addition process;

[0032] Step 3.2: Initialize the optimized code e opt And grayscale image coding after noise Input the SDM model to get the predicted noise And based on this, the predicted initial state is obtained

[0033]

[0034] Step 3.3, predict the initial state The SDM model is decoded into pixel space by the VAE decoder, and the corresponding CLIP image encoding is obtained by the CLIP image encoder. T ;

[0035] Step 3.4: Based on the target loss function, update the code to be optimized through back propagation opt , until the iteration is completed and the optimal encoding is obtained.

[0036] According to a specific implementation of the embodiment of the present disclosure, step 6 specifically includes:

[0037] Step 6.1, randomly select time step t and encode the inpainted image corresponding to the inpainted image The noise is added according to the preset noise adding formula to obtain the restored color image code after noise addition.

[0038] Step 6.2: Encode the restored color image with the optimal encoding and the noise added As the input of the SDM model, the noise prediction loss L used in the default diffusion model training is used. DM As a loss function;

[0039] Step 6.3, update the SDM model weights and repeat the iterative process until the iteration is completed.

[0040] According to a specific implementation of the embodiment of the present disclosure, after step 9, the method further includes:

[0041] When a new text prompt word input by the user is received, the target text prompt word is replaced with the new text prompt word, and steps 7 to 9 are repeated to obtain a final colored image corresponding to the new text prompt word.

[0042] The controllable image coloring scheme based on text encoding optimization in the embodiment of the present disclosure includes: step 1, encoding the grayscale image to be colored to obtain the grayscale image encoding, and encoding the target text prompt word input by the user to obtain the target text encoding; step 2, constructing a negative color sample prompt word set and encoding it to obtain a negative color text encoding set; step 3, initializing the code to be optimized with a random vector and optimizing the code to be optimized using a target loss function to obtain an optimal code; step 4, inputting the optimal code and the grayscale image code into the SDM model to obtain an initial coloring based on the diffusion prior. color image; step 5, input the initial color image and the grayscale image to be colored into the spatial alignment module to obtain the repaired color image; step 6, freeze the optimal encoding and fine-tune the SDM model; step 7, based on the target text encoding and the optimal encoding, use the preset formula to perform semantic interpolation to obtain the final text encoding; step 8, input the final text encoding and the color image encoding corresponding to the repaired color image into the fine-tuned SDM model to obtain the target color image corresponding to the target text prompt word; step 9, input the target color image and the grayscale image to be colored into the spatial alignment module to obtain the final color image.

[0043] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, a text encoding optimization method is used to utilize the prior knowledge of the pre-trained diffusion model to achieve text-guided controllable image colorization, and the CLIP-guided contrast loss is innovatively proposed for the image inversion process of the diffusion model. With the help of this loss, the color information missing from the grayscale image is successfully mapped to the diffusion prior space, thereby achieving low-computational-cost text-guided image colorization with the help of the pre-trained diffusion model, and improving the colorization efficiency, controllability, and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a controllable image coloring method based on text encoding optimization provided in an embodiment of the present disclosure;

[0046] Figure 2 A schematic diagram of a process for processing image and text input provided by an embodiment of the present disclosure;

[0047] Figure 3 A schematic diagram of a CLIP-guided contrast loss image inversion process provided in an embodiment of the present disclosure;

[0048] Figure 4 A schematic diagram of a flow chart of a spatial alignment process provided by an embodiment of the present disclosure;

[0049] Figure 5 A schematic diagram of a process of fine-tuning an SDM model provided in an embodiment of the present disclosure;

[0050] Figure 6 A flowchart of a text-guided coloring process provided by an embodiment of the present disclosure;

[0051] Figure 7 A comparison chart of coloring effects between another method of the present invention and an existing solution provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0053] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0054] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0055] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0056] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0057] The disclosed embodiment provides a controllable image colorization method based on text encoding optimization, which can be applied in image processing.

[0058] See also Figure 1 , is a flow chart of a controllable image coloring method based on text encoding optimization provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:

[0059] Step 1, encoding the grayscale image to be colored to obtain grayscale image encoding, and encoding the target text prompt word input by the user to obtain target text encoding;

[0060] The present invention proposes a controllable image colorization method based on text encoding optimization, which can support users to customize the coloring scheme with text prompt words and can obtain diversified color images.

[0061] The overall colorization includes: a colorization stage based on diffuse prior knowledge and a colorization stage guided by text.

[0062] The present invention divides the text-guided image coloring process into two stages: a coloring stage based on prior knowledge and a text-guided coloring stage.

[0063] Coloring stage based on diffusion prior knowledge

[0064] Previous image colorization models needed to be trained on a large amount of synthetic image data. However, the Stable Diffusion Model (SDM) that has emerged in recent years can serve as a powerful prior knowledge, providing a more convenient and low-computational cost basis for the research of image colorization methods. Ordinary SDM-based image editing methods use the noise prediction loss used in the default diffusion model training during the optimization process, as shown in Formula 1:

[0065] L DM =MSE(∈ t ,∈ θ (z t ,t,e opt ))

[0066] Where t is a randomly selected time step, ∈ t is the random noise introduced in the forward process time step t, z t is the grayscale image x gray The noisy image encoding at time step t, ∈ θ (·) is the neural network for predicting noise.

[0067] With the help of SDM, the present invention transforms the image colorization task into text encoding. opt Optimization: Optimize text encoding opt , so that As the input of SDM, the user input grayscale image x can be obtained gray Colored version of .

[0068] Formula 1 lacks guidance on color. The present invention introduces CLIP-based contrast loss as guidance for color space during the coding optimization process, as shown in Formula 2:

[0069]

[0070] Where L is the number of negative color samples, e T is the CLIP image encoding corresponding to the colored image, e + Encode the text. is the L negative sample text encoding, exp(·) represents the calculation of the inner product of two vectors and then the power operation with natural numbers as the base. Through the design of contrast loss, the color image encoding e T Color guidance is achieved by moving away from negative text encodings and by using a logarithmic function to maintain numerical stability during the optimization phase.

[0071] By combining the two, it is possible to achieve the optimization process while retaining the content of the grayscale image input by the user and applying the color information from the prior knowledge of the diffusion model. This approach solves the color information limitation caused by the noise loss in the training process of the diffusion model alone, but due to the random nature of the diffusion model itself, it is impossible to guarantee the complete consistency of the texture information of the generated image with the grayscale image. In order to compensate for this shortcoming, the present invention adds a spatial alignment module based on CoCosNet to correct the detail distortion caused by the denoising process.

[0072] When implementing it, Figure 2 As shown, the VAE encoder provided by SDM can be used to treat the colored grayscale image x gray Encode the image to get Use CLIP text encoder to preprocess the target text prompt word c input by the user to obtain the corresponding text encoding e tar .

[0073] Step 2, construct a negative color sample prompt word set and encode it to obtain a negative color text encoding set;

[0074] In specific implementation, a set of L negative color sample prompt words can be constructed, such as: - = "A grayscale photograph", "A color bleeding photograph", ...}, and obtain the corresponding negative color text encoding set through the CLIP text encoder

[0075] Step 3, initializing the code to be optimized with a random vector and optimizing the code to be optimized using the target loss function to obtain the optimal code;

[0076] Furthermore, the objective loss function is

[0077] L Total =L DM +λLColor

[0078] Among them, L DM represents the noise prediction loss used in the default diffusion model training, L Color represents the contrastive loss based on CLIP guidance, and λ is the contrastive loss weight based on CLIP guidance.

[0079] Furthermore, the noise prediction loss L DM The expression is

[0080]

[0081] Where t is a randomly selected time step, ∈ t is the noise added in the forward process time step t, is the grayscale image x gray The noisy image encoding at time step t, ∈ θ (·) is the neural network for predicting noise;

[0082] The contrast loss L Color The expression is

[0083]

[0084] Where L is the number of negative color samples, e T is the CLIP image encoding corresponding to the colored image, e + Encode the text. is L negative color text encodings, and exp(·) means calculating the inner product of two vectors and then performing a power operation with a natural number as the base.

[0085] Furthermore, the step 3 specifically includes:

[0086] Step 3.1, randomly select time step t and encode the grayscale image According to the preset noise adding formula, the noise adding process is performed to obtain the grayscale image coding after the noise adding Among them, the preset noise addition formula is:

[0087]

[0088] in, is a predefined hyperparameter in SDM, which indicates the true data retention rate during the noise addition process.

[0089] Step 3.2: Initialize the optimized code e opt And grayscale image coding after noise Input the SDM model to get the predicted noise And based on this, the predicted initial state is obtained

[0090]

[0091] Step 3.3, predict the initial state The SDM model is decoded into pixel space by the VAE decoder, and the corresponding CLIP image encoding is obtained by the CLIP image encoder. T ;

[0092] Step 3.4: Based on the target loss function, update the code to be optimized through back propagation opt , until the iteration is completed and the optimal encoding is obtained.

[0093] In specific implementation, the code to be optimized can be initialized by random vectors opt Based on the loss function of formula 1 and formula 2, the combination of the two is used as the target loss function to optimize e opt , in this process, the parameters in the SDM model do not participate in the gradient update. The target loss function is shown in Formula 3:

[0094] Formula 3:

[0095] L Total =L DM +λL Color

[0096] where λ is the contrastive loss weight based on CLIP.

[0097] Initialize the code to be optimized with a random vector opt , use the Adam optimizer, initialize the learning rate to 0.0005, use the constant learning rate strategy, and freeze the pre-trained Stable Diffusion model weights. Execute M (take 1000) iterations as follows, Figure 3 As shown:

[0098] Randomly select time step t and According to formula 5, the noise is added to obtain

[0099] Formula 5:

[0100]

[0101] Will The predicted noise is obtained as the input of SDM The predicted initial state is obtained by the following formula

[0102]

[0103] Will get Decode it into pixel space through the SDM VAE decoder, and obtain the corresponding CLIP picture encoding through the CLIP picture encoder T , use Formula 3 as the objective loss function, where the hyperparameter λ of Formula 3 is 0.5.

[0104] Update the encoding e by backpropagation opt , repeat the iterative process to get the optimal encoding.

[0105] Step 4, input the optimal encoding and grayscale image encoding into the SDM model to obtain an initial colored image based on diffusion prior;

[0106] In specific implementation, the optimal encoding obtained by the previous step can be combined with the grayscale image encoding As the input of SDM, the initial colored image x is generated by SDM pri .

[0107] Step 5, inputting the initial colored image and the grayscale image to be colored into a spatial alignment module to obtain a repaired colored image;

[0108] Based on the above embodiment, before step 5, the method further includes:

[0109] Performing a preprocessing operation on the sample image data set to obtain a triplet data pair, wherein the triplet data pair includes a grayscale image, a texture-damaged color image, and an original color image;

[0110] Based on the triplet data pairs, the spatial alignment module of the CoCosNet network structure is trained using the Adam optimizer and the cosine decay learning rate strategy with Warmup.

[0111] In specific implementation, the sample image data set can be preprocessed, and image enhancement operations can be performed on the color images in the data set to obtain diverse data with damaged content through texture distortion, flipping, etc. At the same time, the corresponding grayscale image can be obtained by graying the color image, and the corresponding triple data pair of <grayscale image, texture-damaged color image, original color image> can be obtained.

[0112] Then, based on the data obtained above, the Adam optimizer is used, the learning rate is initialized to 0.0001, and the cosine decay learning rate strategy with Warmup is used to train the spatial alignment module of the CoCosNet network structure. The CoCosNet model is used as a spatial alignment module, and the input is a grayscale image and a color image with damaged texture to predict the original color image. The network structure of CoCosNet consists of three parts: feature extraction, correspondence learning, and generation network. First, the feature extraction network performs deep feature encoding on the input grayscale image and the reference color image with damaged texture respectively; the correspondence module aligns the information of the reference color image with damaged texture to the spatial layout of the input image by establishing cross-domain feature correspondences. The generation network performs image synthesis based on the alignment information, and fuses the features of the reference image with the structural information of the grayscale image layer by layer to generate the same image as the original color image. Figure 1 This method effectively solves the shortcoming of the lack of fidelity of image details generated based on the diffusion model, and improves the accuracy and consistency of the colored image and the grayscale image obtained by the method disclosed in the present invention.

[0113] After training the spatial alignment module, Figure 4 As shown, the initial colored image x pri and the grayscale image x to be colored gray Input the spatial alignment module to perform texture detail restoration on the result to obtain a high-fidelity restored color image

[0114] Step 6, freeze the optimal encoding and fine-tune the SDM model;

[0115] Based on the above embodiment, step 6 specifically includes:

[0116] Step 6.1, randomly select time step t and encode the inpainted image corresponding to the inpainted image The noise is added according to the preset noise adding formula to obtain the restored color image code after noise addition.

[0117] Step 6.2: Encode the restored color image with the optimal encoding and the noise added As the input of the SDM model, the noise prediction loss L used in the default diffusion model training is used. DM As a loss function;

[0118] Step 6.3, update the SDM model weights and repeat the iterative process until the iteration is completed.

[0119] When implementing it, Figure 5 As shown, in order to further consolidate the optimized coding opt With colored images The mapping relationship between them is shown in Figure 1. In this step, the SDM is fine-tuned using Formula 1 as the loss function to obtain a more robust editing capability. In the optimization process of this step, no opt to update.

[0120] The specific process is:

[0121] Freeze optimized encoding Perform N (take 500) fine-tuning iterations on the SDM as follows: Figure 5 As shown:

[0122] (1) Randomly select a time step t, According to formula 5, the noise is added to obtain in is the SDM model obtained in step 5 as well as The initial colored image x obtained as input pri After being repaired by the spatial alignment module, the obtained The corresponding picture code;

[0123] (2) In order to establish e opt and Establish a more stable mapping relationship to As the input of SDM, use Formula 1 as the loss function;

[0124] (3) Update the SDM model weights and repeat the iterative process.

[0125] Step 7, based on the target text encoding and the optimal encoding, semantic interpolation is performed using a preset formula to obtain the final text encoding;

[0126] In specific implementation, after steps 1 to 6 are completed, the text-guided coloring stage begins. This stage aims to further improve the fit between the colored image and the user text target through user text guidance. This stage implements the text-guided coloring method through the following steps:

[0127] Based on user target text encoding tar And the optimal encoding, according to formula 4, the semantic interpolation of the text encoding is performed to obtain the final text encoding Where η is 0.7. Formula 4 is as follows:

[0128]

[0129] Where η is the semantic interpolation coefficient, which is used to control the strength of user text guidance.

[0130] Step 8, input the final text code and the color image code corresponding to the repaired color image into the fine-tuned SDM model to obtain the target color image corresponding to the target text prompt word;

[0131] When implementing it, Figure 6 As shown, the final text encoding and the color image encoding corresponding to the repaired color image are As the input of SDM, after text guidance, the final colored image x matching the target text is obtained fin .

[0132] Step 9: Input the target colored image and the grayscale image to be colored into a spatial alignment module to obtain the final colored image.

[0133] Based on the above embodiment, after step 9, the method further includes:

[0134] When a new text prompt word input by the user is received, the target text prompt word is replaced with the new text prompt word, and steps 7 to 9 are repeated to obtain a final colored image corresponding to the new text prompt word.

[0135] In the specific implementation, in order to further optimize the obtained colored image x fin , to obtain a colored image with more realistic texture details. This step reuses the spatial alignment module to further gray Repair x fin The texture details in the image are taken, and the result of this step is used as the final colored image corresponding to the target text prompt word c input by the user.

[0136] At the same time, in order to obtain the user grayscale image x gray To create a diverse color image, we only need to modify the target text prompt word c (e.g., modify “a white dog” to “a red dog”), and follow step 1 to convert e tar Replace the text code corresponding to the modified target text prompt word, repeat steps 7 to 9, and you can get a variety of colored images without additional fine-tuning or optimization.

[0137] The controllable image colorization method based on text encoding optimization provided in this embodiment realizes text-guided controllable image colorization by using the prior knowledge of the pre-trained diffusion model through a text encoding optimization method, and innovatively proposes a CLIP-guided contrast loss for the image inversion process of the diffusion model. With the help of this loss, the color information missing from the grayscale image is successfully mapped to the diffusion prior space, thereby realizing low-computational-cost text-guided image colorization with the help of the pre-trained diffusion model, and improving the colorization efficiency, controllability and accuracy.

[0138] The method of the present disclosure will be further described below in conjunction with a specific embodiment. A similar method is as follows:

[0139] UniColor proposed by Zhitong Huang et al., a multimodal colorization method implemented through Transformer;

[0140] Shuchen Weng et al. proposed L-CoDe, a language-based coloring using color object decoupling conditions;

[0141] Lvmin Zhang et al. proposed ControlNet to provide a technique for adding conditional control to the text-to-image diffusion model;

[0142] The method disclosed in the present invention is compared with the above method, and the results are as follows Figure 7 As shown, the present invention can realize the controllable text coloring function for any grayscale image. For objects in grayscale images, after completing the coloring steps proposed by the present invention, a variety of coloring results can be achieved simply by replacing the color modifiers of the objects. Compared with other public methods, the present invention can significantly outperform other public methods in image indicators such as FID, CLIP Score and LPIPS, indicating that the coloring function of the present invention can present the characteristics of being more consistent with the details of the grayscale image, more matching the color attributes with the user's target text, and better visual effects.

[0143] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0144] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A controllable image coloring method based on text encoding optimization, characterized in that: include: Step 1, encoding the grayscale image to be colored to obtain grayscale image encoding, and encoding the target text prompt word input by the user to obtain target text encoding; Step 2, construct a negative color sample prompt word set and encode it to obtain a negative color text encoding set; Step 3, initializing the code to be optimized with a random vector and optimizing the code to be optimized using the target loss function to obtain the optimal code; Step 4, input the optimal encoding and grayscale image encoding into the SDM model to obtain an initial colored image based on diffusion prior; Step 5, inputting the initial colored image and the grayscale image to be colored into a spatial alignment module to obtain a repaired colored image; Step 6, freeze the optimal encoding and fine-tune the SDM model; Step 7, based on the target text encoding and the optimal encoding, semantic interpolation is performed using a preset formula to obtain the final text encoding; Step 8, input the final text code and the color image code corresponding to the repaired color image into the fine-tuned SDM model to obtain the target color image corresponding to the target text prompt word; Step 9: Input the target colored image and the grayscale image to be colored into a spatial alignment module to obtain the final colored image.

2. The method according to claim 1, characterized in that Before step 5, the method further includes: Performing a preprocessing operation on the sample image data set to obtain a triplet data pair, wherein the triplet data pair includes a grayscale image, a texture-damaged color image, and an original color image; Based on the triplet data pairs, the spatial alignment module of the CoCosNet network structure is trained using the Adam optimizer and the cosine decay learning rate strategy with Warmup.

3. The method according to claim 2, characterized in that The objective loss function is THE Total =L DM +λL Color Among them, L DM represents the noise prediction loss used in the default diffusion model training, L Color represents the contrastive loss based on CLIP guidance, and λ is the contrastive loss weight based on CLIP guidance.

4. The method according to claim 3, characterized in that The noise prediction loss L DM The expression is Where t is a randomly selected time step, ∈ t is the noise added in the forward process time step t, is the grayscale image x gray The noisy image encoding at time step t, ∈ θ (·) is the neural network for predicting noise; The contrast loss L Color The expression is Where L is the number of negative color samples, e T is the CLIP image encoding corresponding to the colored image, e + Encode the text. i=1,2,…,L are L negative color text codes, and exp(·) indicates that the inner product of two vectors is calculated and then a power operation with a natural number as the base is performed.

5. The method according to claim 4, characterized in that The step 3 specifically includes: Step 3.1, randomly select time step t and encode the grayscale image According to the preset noise adding formula, the noise adding process is performed to obtain the grayscale image coding after the noise adding Among them, the preset noise addition formula is: in, is a predefined hyperparameter in SDM, which indicates the true data retention rate during the noise addition process; Step 3.2: Initialize the optimized code e opt And grayscale image coding after noise Input the SDM model to get the predicted noise And based on this, the predicted initial state is obtained Step 3.3, predict the initial state The SDM model is decoded into pixel space by the VAE decoder, and the corresponding CLIP image encoding is obtained by the CLIP image encoder. T ; Step 3.4: Based on the target loss function, update the code to be optimized through back propagation opt , until the iteration is completed and the optimal encoding is obtained.

6. The method according to claim 5, characterized in that The step 6 specifically includes: Step 6.1, randomly select time step t and encode the inpainted image corresponding to the inpainted image The noise is added according to the preset noise adding formula to obtain the restored color image code after noise addition. Step 6.2: Encode the restored color image with the optimal encoding and the noise added As the input of the SDM model, the noise prediction loss L used in the default diffusion model training is used. DM As a loss function; Step 6.3, update the SDM model weights and repeat the iterative process until the iteration is completed.

7. The method according to claim 6, characterized in that After step 9, the method further includes: When a new text prompt word input by the user is received, the target text prompt word is replaced with the new text prompt word, and steps 7 to 9 are repeated to obtain a final colored image corresponding to the new text prompt word.

Images