Image coloring method and device, electronic equipment and readable storage medium
Through multi-modal fusion of text description information and image features of the image to be colored, comprehensive features are generated, which solves the problem of insufficient accuracy and naturalness of image coloring in the existing technology, and achieves high-quality color image generation.
Patent Information
- Application Number
- CN202411545813.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-01-21
AI Technical Summary
In the prior art, the structure of image coloring is not accurate, natural enough, and lacks generalization ability, mainly due to the lack of comprehensive semantic understanding, limitations on representation space, training error accumulation, and color leakage.
By receiving text description information corresponding to the coloring image, extract text features and image features, and combine these characteristics through the preset color decoder to generate comprehensive features in order to fill more accurately.
A more fine and accurate color filling is implemented, the content of the image content of the preset color decoder treats the color image is generated, the high -quality color image is generated, and the consistency of the text description information is maintained.
Smart Images
Figure CN119338941A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image coloring method, device, electronic device and readable storage medium. Background Art
[0002] Image colorization is a classic computer vision task with wide application potential in many fields, such as legacy photo restoration, video reconstruction, and artistic creation. In the existing technology, with the development of deep learning technology, automatic image colorization has emerged, which can predict the appropriate color from the complex semantics of the image.
[0003] However, the existing models used for image colorization lack comprehensive semantic understanding, are limited by representation space, have accumulated training errors, and have color leakage, resulting in inaccurate and unnatural structures for image colorization and lack of generalization capabilities. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide an image colorization method, device, electronic device and readable storage medium to solve the problem that the structure of image colorization in the prior art is not accurate and natural enough and lacks generalization ability.
[0005] A first aspect of an embodiment of the present disclosure provides an image coloring method, comprising:
[0006] receiving text description information corresponding to the image to be colored, extracting text features from the text description information, and extracting image features from the image to be colored, where the image to be colored is a grayscale image;
[0007] Initialize the color vector of the image to be colored to obtain an initialized color vector;
[0008] Through the preset color decoder, the text features, image features and initialized color vectors are fused into comprehensive features;
[0009] The comprehensive features and image features are fused to obtain the color filling result, and the image to be colored and the color filling result are synthesized to complete the coloring of the image to be colored.
[0010] According to a second aspect of the embodiments of the present disclosure, there is provided an image coloring device, comprising:
[0011] An extraction module is configured to receive text description information corresponding to the image to be colored, extract text features from the text description information, and extract image features from the image to be colored, where the image to be colored is a grayscale image;
[0012] An initialization module is configured to initialize the color vector of the image to be colored to obtain an initialized color vector;
[0013] A fusion module is configured to fuse text features, image features and an initialized color vector through a preset color decoder to obtain a comprehensive feature;
[0014] The filling module is configured to fuse the comprehensive features and the image features to obtain a color filling result, and synthesize the image to be colored and the color filling result to complete the coloring of the image to be colored.
[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0016] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0017] Compared with the prior art, the disclosed embodiment has the following beneficial effects: receiving text description information corresponding to an image to be colored, extracting text features from the text description information, and extracting image features from the image to be colored, wherein the image to be colored is a grayscale image, initializing the color vector of the image to be colored to obtain an initialized color vector, fusing the text features, image features and the initialized color vector through a preset color decoder to obtain a comprehensive feature, the comprehensive feature can enhance the consistency between the initialized color vector and the text description information corresponding to the image to be colored, fusing the comprehensive feature and the image feature to obtain a color filling result, and synthesizing the image to be colored and the color filling result, so that the image to be colored is completely colored, and the color filling result obtained based on the comprehensive feature and the image feature improves the preset color decoder's understanding of the image content of the image to be colored, realizes more refined and accurate color filling, and obtains a high-quality color image, and the comprehensive feature includes the text description information, which can retain more image details while maintaining consistency with the text description information. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 is a flowchart of an image coloring method provided by an embodiment of the present disclosure;
[0020] Figure 2is a flowchart of another image coloring method provided by an embodiment of the present disclosure;
[0021] Figure 3 is a schematic diagram of a preset color decoder provided by an embodiment of the present disclosure;
[0022] Figure 4 is a schematic diagram of an image coloring device provided by an embodiment of the present disclosure;
[0023] Figure 5 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0025] An image coloring method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0026] In the prior art, automatic image colorization methods have emerged, aiming to predict appropriate colors from the complex semantics of images. Common methods include image colorization based on convolutional neural networks (CNN), image colorization based on generative adversarial networks (GAN), and image colorization based on Transformer (a neural network model based on the "encoder-decoder" architecture).
[0027] However, the CNN-based approach lacks comprehensive semantic understanding and is prone to inaccurate or unsaturated colorization results. The GAN-based approach is limited by space and has difficulty processing complex images, resulting in unnatural coloring of complex images. The Transformer-based approach has problems such as training error accumulation and color leakage, and relies on hand-crafted prior results, which limits its generalization ability.
[0028] In summary, the present disclosure provides an image colorization method, which can colorize an image to be colored by multimodal fusion, aiming to achieve a semantically reasonable and visually vivid colorization effect, using a pixel decoder to restore the spatial resolution of the image, and combining a query-based preset color decoder to enrich the color query, and establish the correlation between color and multi-scale semantic representation between the two decoders through cross-attention, effectively avoiding color leakage, improving the accuracy and richness of color, and at the same time enhancing the generalization ability and robustness of the preset color decoder, achieving high-quality and natural color filling effects without relying on artificially designed prior knowledge.
[0029] Figure 1 is a flow chart of an image coloring method provided by an embodiment of the present disclosure. The preset color decoder in the image coloring method can be set in a terminal device or a server, and the terminal device or the server completes the coloring of the image to be colored. Figure 1 As shown, the image coloring method includes:
[0030] S101, receiving text description information corresponding to the image to be colored, extracting text features from the text description information, and extracting image features from the image to be colored;
[0031] Among them, the image to be colored is a grayscale image;
[0032] Specifically, text description information corresponding to the image to be colored is received, wherein the text description information may include object names, color attributes, scene atmosphere, etc. in the image to be colored, and is used to describe the image content of the image to be colored.
[0033] Text features can be extracted from the text description information by deeply analyzing the text description information through natural language processing technology, or by inputting the text description information into a text encoder to obtain text features. The text encoder for extracting text features can be a CLIP text encoder, which is a multimodal model including an image encoder and a text encoder.
[0034] In addition, image features are extracted from the image to be colored. Image features may include image features such as edges, textures, and shapes in the image, which can be used to understand the spatial structure and content details of the image. Image features can be extracted through a UNet network. The UNet network is a convolutional neural network for image segmentation. It consists of a set of encoders and a set of decoders, which can gradually extract the hierarchical features of the image, thereby extracting image features using global information and local information, and retaining finer image details.
[0035] The image to be colored is a grayscale image that does not include any color information.
[0036] S102, initializing the color vector of the image to be colored to obtain an initialized color vector;
[0037] Specifically, the color vector of the image to be colored is initialized to obtain an initialized color vector, so that the result obtained after the initialization color vector is subsequently processed by self-attention can better integrate the key vector and value vector required by the preset color decoder, thereby increasing the generalization ability and robustness of the preset color decoder.
[0038] The color vector may be a color query vector of the grayscale image, which is used to represent the grayscale value of each pixel in the grayscale image.
[0039] Among them, the color learnable vector is a feature vector used for image retrieval, which represents the spatial distribution information of color. It improves the accuracy of retrieval by quantifying, dividing connected areas, judging aggregation, etc. It is also an evolution of the color histogram. Although the amount of calculation is large, the calculation cost can be reduced through optimization methods. The color learnable vector can describe the spatial distribution information of color in the image and improve the accuracy of retrieval. Compared with the color histogram, the color learnable vector considers the spatial relationship of color instead of just counting the frequency of color occurrence, making the color learnable vector more effective in processing images with complex color distribution.
[0040] S103, fusing the text features, image features and the initialized color vector through a preset color decoder to obtain a comprehensive feature;
[0041] Specifically, the preset color decoder is used to understand and generate color information. The text features, image features and initialized color vectors are input into the preset color decoder. The above data are fused through mechanisms such as self-attention and cross-attention in the preset color decoder to obtain comprehensive features.
[0042] Among them, the fusion process can be to perform self-attention processing on the initialized color vector to obtain a self-attention output result, determine the self-attention output result as a query vector, determine the text feature as a key vector and a value vector, perform cross-attention processing, obtain a first cross result, and fuse the image feature and the first cross result according to Q-Former (a neural network architecture) to obtain a query cross result; at the same time, determine the image feature as a key vector and a value vector, determine the self-attention output result as a query vector, perform cross-attention processing, obtain a second cross result, use the query cross result as a key vector and a value vector, use the second cross result as a query vector, perform cross-attention processing, obtain an initial comprehensive feature, and input the initial comprehensive feature into the feedforward neural network to obtain a comprehensive feature.
[0043] It can combine the content in the text description information with the visual features of the image to be colored and the initialized color vector, making the comprehensive features more accurate and comprehensive.
[0044] S104, fusing the comprehensive features and the image features to obtain a color filling result, and synthesizing the image to be colored and the color filling result, so that the image to be colored is colored.
[0045] Specifically, the obtained comprehensive features are fused with the image features again to obtain a color filling result, so that the color filling result can conform to the text description and can also be naturally integrated into the overall visual effect of the image to be colored. The color filling result represents the color value assigned to each pixel or area in the image to be colored.
[0046] The color filling result and the image to be colored (ie, the original grayscale image) are synthesized, and the color information in the color filling result is superimposed on the image to be colored to complete the coloring process of the image to be colored, wherein the synthesis process may be additive synthesis.
[0047] According to the technical solution provided in the embodiments of the present disclosure, the color filling result can be determined based on the text features, image features and initialized color vector of the image to be colored, thereby improving the preset color decoder's understanding of the image content of the image to be colored, strengthening the consistency between the text description information and the color, and achieving fine and accurate color filling.
[0048] In some embodiments, the text features, image features and the initialization color vector are fused by a preset color decoder to obtain comprehensive features, including:
[0049] Perform self-attention processing on the initialized color vector through a preset color decoder to obtain a self-attention output result;
[0050] Perform cross-attention processing on the self-attention output result and the text feature to obtain a first cross-attention result, and fuse the image feature and the first cross-attention result based on the query mechanism to obtain a query cross-attention result;
[0051] Perform cross attention processing on the self-attention output result and the image feature to obtain a second cross result;
[0052] Cross-attention processing is performed on the query cross-result and the second cross-result to obtain comprehensive features.
[0053] Specifically, by performing self-attention processing on the initialized color vector through a preset color decoder, the dependencies between the internal elements of the color vector can be captured, providing basic feature representation for subsequent cross-attention.
[0054] Among them, the initialized color vector can be empty. The purpose of self-attention processing is to learn the filling method. The process of performing self-attention processing can be to use the initialized color vector as the query vector, key vector and value vector of self-attention to obtain the self-attention output result, so that the self-attention output result can better integrate the key vector and value vector of the cross attention when used in the cross attention, thereby improving the generalization ability and robustness of the preset color decoder.
[0055] In addition, cross-attention processing is performed on the self-attention output results and text features to obtain a first cross result. The first cross result realizes cross-modal information interaction between text features and the initialized color vector, enhances the correlation, and enhances the consistency between the initialized color vector and the text semantics included in the text description information.
[0056] In addition, the first cross-result and the image features are input into the query mechanism to obtain the query cross-result, and the query cross-result is used to characterize the association information between the text description information and the image to be colored, wherein the query mechanism can be a Q-Former structure, which is a neural network architecture used to improve information retrieval and representation learning through a query mechanism, and can convert image features into a form that can be fused with text features to achieve cross-modal interaction between images and texts.
[0057] In addition, another round of cross-attention processing is performed on the self-attention output results and image features, which can focus on the direct correlation between the initialized color vector and the image features and obtain the second cross result.
[0058] Among them, the process of cross-attention processing is to use the self-attention output result as the query vector of the cross-attention, and the image features as the key vector and value vector of the cross-attention, to obtain the second cross result, which can integrate the image content and initialized color information of the image to be colored.
[0059] In addition, in order to integrate all relevant information of the image to be colored, cross-attention processing is performed on the query cross result and the second cross result to obtain comprehensive features, which can sort out and fuse the text features, initialized color vectors and image features of the image to be colored. The comprehensive features contain multi-dimensional information, thereby retaining the important information in the text features and image features corresponding to the image to be colored, and deep interaction and complementarity between features can be achieved through multiple cross-attention processing.
[0060] According to the technical solution provided by the embodiment of the present disclosure, it is possible to obtain multimodal comprehensive features by fusing the initialized color vector, text features and image features through a preset color decoder, thereby effectively alleviating the color leakage effect and improving the accuracy and richness of coloring.
[0061] In some embodiments, cross-attention processing is performed on the self-attention output result and the text feature to obtain a first cross-attention result, including:
[0062] The text features are used as the key vector and value vector, and the self-attention output result is used as the query vector. Cross-attention processing is performed to obtain the first cross result.
[0063] Specifically, the cross attention includes the query vector (Query), the key vector (Key) and the value vector (Value). The attention output can be obtained by calculating the similarity score between the query vector and each key vector, and performing weighted summation on the value vector based on the score.
[0064] The text features are used as key vectors and value vectors, and the self-attention output results are used as query vectors to obtain the first cross result, wherein the query vector can reflect the relative importance and correlation between the elements in the initialized color vector.
[0065] According to the technical solution provided by the embodiment of the present disclosure, the first cross result obtained by cross-attention processing of text features and self-attention output results reflects the internal dependencies of the initialized color vector, and also incorporates the additional information provided by the text features, thereby enhancing the consistency between the text semantics corresponding to the initialized color vector and the text description information, thereby enabling the preset color decoder to understand each data more comprehensively and improve performance.
[0066] In some embodiments, cross-attention processing is performed on the query cross-result and the second cross-result to obtain comprehensive features, including:
[0067] Perform cross-attention processing on the query cross-result and the second cross-result to obtain the initial comprehensive features;
[0068] The initial comprehensive features are processed based on a feedforward neural network to obtain comprehensive features.
[0069] Specifically, the query cross result is determined as the query vector, and the second cross result is determined as the key vector and the value vector, and cross-attention processing is performed to obtain an initial comprehensive feature. The initial comprehensive feature includes information on the query cross result and information on the second cross result, and realizes information fusion between the two groups of vectors, that is, the initial color information, text features and image features of the image to be colored are fused, which is a multimodal information.
[0070] In addition, taking the initial comprehensive features as the input of the feedforward neural network to propagate layer by layer in the feedforward neural network can enable the feedforward neural network to perform linear transformation and nonlinear activation on the initial comprehensive features, thereby extracting higher-level feature representations, enhancing the expressive power of the features, and generating the final output of the color decoder, namely, the comprehensive features.
[0071] Among them, the feedforward neural network is a multi-layer perceptron, which consists of multiple fully connected layers, and each fully connected layer may contain nonlinear activation functions, such as ReLU functions.
[0072] Furthermore, the initial comprehensive features are propagated layer by layer in the feedforward neural network, and each layer performs linear transformation and nonlinear activation on the input features, thereby extracting higher-level feature representations. After multi-layer processing of the feedforward neural network, comprehensive features are obtained. Compared with the initial comprehensive features, the comprehensive features have stronger expressiveness and robustness.
[0073] According to the technical solution provided by the embodiment of the present disclosure, the initial comprehensive feature can be obtained by querying the cross result and the second cross result. The initial comprehensive feature can fuse the information in the cross result and the second cross result to make the feature richer and more comprehensive. The initial comprehensive feature is further refined and processed by the feedforward neural network to obtain the comprehensive feature, which can enhance the expressive power of the feature.
[0074] In some embodiments, the color filling result is obtained by fusing the comprehensive features and the image features, including:
[0075] Determine a comprehensive feature dimension corresponding to the comprehensive feature, and adjust the image feature dimension corresponding to the image feature according to the comprehensive feature dimension;
[0076] Performing a dot product operation on the comprehensive features and the image features after dimension adjustment to fuse the comprehensive features and the image features after dimension adjustment to obtain an initial color filling result;
[0077] The dimension corresponding to the initial color filling result is adjusted to the image feature dimension before the dimension adjustment to obtain the color filling result.
[0078] Specifically, a comprehensive feature dimension corresponding to the comprehensive feature is determined, where the comprehensive feature dimension represents the complexity and amount of information contained in the feature in data processing or representation, and the image feature dimension corresponding to the image feature is adjusted according to the determined comprehensive feature dimension, that is, the image feature dimension is matched with the comprehensive feature dimension corresponding to the comprehensive feature obtained by a preset color decoder.
[0079] The process of adjusting the image feature dimension may be a mathematical transformation such as scaling, cropping, and interpolation, so as to facilitate integration with the comprehensive feature dimension.
[0080] In addition, after completing the dimensionality adjustment, a dot product operation is performed on the comprehensive features and the image features after dimensionality adjustment. Through the dot product operation, the adjusted image features and the comprehensive features can be fused into a feature vector, and the feature vector is determined as the initial color filling result. The initial color filling result can reflect the joint influence of the comprehensive features and the image features after dimensionality adjustment in the image colorization task.
[0081] In addition, since the dimension of the initial color filling result may be inconsistent with the dimension of the image features of the image to be colored, it is also necessary to adjust the dimension corresponding to the initial color filling result to the image feature dimension before the dimension adjustment to obtain the color filling result, so as to ensure that the color filling result is consistent with the image to be colored in space and resolution. The dimension adjustment process can be upsampling, interpolation or other processing methods of the initial color filling result.
[0082] Furthermore, usually, the dimension of the initial color filling result is higher than the image feature dimension, and the initial color filling result can be processed by convolution operation to reduce the dimension so as to form a color filling result consistent with the image feature dimension of the image to be colored.
[0083] According to the technical solution provided by the embodiments of the present disclosure, the dimension of the image feature can be adjusted through the comprehensive feature dimension. After the adjustment, the comprehensive feature and the image feature are fused to obtain an initial color filling result, so that the initial color filling result is more accurate and more in line with expectations. Then, the dimension of the initial color filling result is adjusted to the image feature dimension before the dimension adjustment to obtain the color filling result, so that a fine and accurate color filling result can be obtained.
[0084] In some embodiments, adjusting the image feature dimension corresponding to the image feature according to the comprehensive feature dimension includes:
[0085] Based on a preset fusion module, a convolution operation is performed on the image feature dimension to adjust the image feature dimension according to the comprehensive feature dimension. The preset fusion module is used to obtain a color filling result based on the comprehensive feature and the image feature.
[0086] Specifically, the preset fusion module is used to obtain the color filling result according to the comprehensive features and the image features, and the preset fusion module can adjust and fuse the comprehensive features and the image features respectively.
[0087] The image features and comprehensive features are input into the preset fusion module. The convolution layer inside the preset fusion module can slide on the image features through the learnable convolution kernel, determine the local feature value of each position, and generate a new feature map.
[0088] The adjustment parameters of the image feature dimension can be determined according to the comprehensive feature dimension. The adjustment parameters include the expected size and the number of channels. By adjusting the size and number of convolution kernels, the image feature dimension can be adjusted to be the same as the comprehensive feature dimension.
[0089] Furthermore, usually, the image feature dimension is lower than the comprehensive feature dimension, that is, the image feature dimension is increased in dimension through the convolution operation.
[0090] Furthermore, the preset fusion module is flexible and scalable, and can dynamically adjust the number and configuration of convolutional layers according to different application scenarios and task requirements to achieve fine-tuning of image feature dimensions. For example, when it is necessary to process high-resolution images or complex scenes, the number of convolutional layers can be increased to improve the accuracy and depth of feature extraction, while when processing low-resolution images or simple scenes, the number of convolutional layers can be reduced to reduce computational complexity and increase processing speed.
[0091] According to the technical solution provided in the embodiment of the present disclosure, a convolution operation is performed on the image feature dimension according to a preset fusion module to obtain an image feature dimension that is the same as the comprehensive feature dimension. This can improve the fineness of image processing to retain more details, and enable the color filling result to better express the multi-dimensional feature content, and the coloring result is more natural.
[0092] In some embodiments, after the image to be colored is synthesized with the color filling result so that the image to be colored is colored, the method further includes:
[0093] Determine a loss value based on the colored image and the loss function;
[0094] Based on the loss value, the preset encoding parameters of the preset color decoder are adjusted, and the preset color decoding parameters include self-attention parameters corresponding to self-attention, cross-attention parameters corresponding to cross-attention, and neural network parameters corresponding to the feedforward neural network.
[0095] Specifically, the color filling result is synthesized and added with the image to be colored, so that the coloring of the image to be colored is completed, so that each area of the image to be colored can be properly colored to achieve visual harmony and realism.
[0096] In addition, the loss value of the image to be colored after the coloring is completed is determined through the loss function, wherein the loss function can comprehensively consider multiple factors, such as the rationality of color distribution, the clarity of edge preservation, the similarity with the ideal colored image, etc., and the loss value is determined by the deviation between the actual coloring result and the ideal state.
[0097] In addition, according to the loss value, the preset encoding parameters of the preset color decoder are adjusted to improve the coloring effect, wherein the preset encoding parameters may include self-attention parameters corresponding to self-attention, cross-attention parameters corresponding to cross-attention, and neural network parameters corresponding to the feedforward neural network.
[0098] Specifically, the self-attention parameters corresponding to the self-attention are used to control the preset color decoder to capture and utilize the color correlation and dependency between different regions in the image when processing the image to be colored. The adjusted self-attention parameters can enhance the consistency and coordination of the colors within the image.
[0099] The cross attention parameters corresponding to the cross attention are used to establish connections between different modalities, ensuring that the color filling results can accurately reflect the intention of the external information and can improve the accuracy of the color filling results;
[0100] The parameters of the feedforward neural network are used to determine the operation mode and efficiency of the feedforward neural network, which affects the shading speed and effect. Through optimization, the computational efficiency and robustness of the shading process can be improved without sacrificing quality.
[0101] According to the technical solution provided in the embodiment of the present disclosure, by synthesizing the image to be colored and the color filling result, and utilizing the parameter adjustment strategy guided by the loss function, the performance of the preset color decoder can be iteratively optimized to achieve high-quality image colorization effects.
[0102] Figure 2 is a flow chart of another image coloring method provided by an embodiment of the present disclosure, such as Figure 2 As shown, the text description information of the image to be colored is input into the CLIP text encoder to obtain text features, and the text features are input into the preset color decoder; at the same time, the image to be colored is input into the UNet network, the image features are obtained through the UNet network, and the image features are input into the preset color decoder and the preset fusion module. Among them, the initialization color vector is also input into the preset color decoder, so that the preset color decoder obtains comprehensive features according to the initialization color vector, text features and image features, and the comprehensive features and image features are convolved and then dot product operations are performed to obtain color filling results, which are then synthesized with the image to be colored to complete the coloring of the image to be colored.
[0103] According to the technical solution provided by the embodiments of the present disclosure, the spatial resolution of the image to be colored can be restored through the UNet decoder, and at the same time, the preset query-based color decoder can be combined to improve the color query with rich visual features, thereby avoiding reliance on artificially designed priors, and using the cross-attention and Q-Former mechanisms to establish the correlation between color and multi-scale semantic representation between the two decoders, effectively alleviating the color leakage effect, improving the accuracy and richness of the color, and enhancing the generalization ability and robustness of the model, achieving high-quality and natural color filling effects.
[0104] Figure 3 is a schematic diagram of a preset color decoder provided by an embodiment of the present disclosure. Figure 3 As shown, the initialized color vector is self-attention processed to obtain a self-attention output result, the self-attention output result is determined as a query vector, the text feature is determined as a key vector and a value vector, and a cross-attention process is performed to obtain a first cross result, and the image feature and the first cross result are fused according to Q-Former to obtain a query cross result; at the same time, the image feature is determined as a key vector and a value vector, and a cross-attention process is performed to obtain a second cross result, the query cross result is used as a key vector and a value vector, and the second cross result is used as a query vector, and a cross-attention process is performed to obtain an initial comprehensive feature, and the initial comprehensive feature is input into a feedforward neural network to obtain a comprehensive feature.
[0105] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.
[0106] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.
[0107] Figure 4 Schematic diagram of an image coloring device provided by an embodiment of the present disclosure. Figure 4 As shown, the image coloring device includes: an extraction module 401, an initialization module 402, a fusion module 403, and a filling module 404, wherein:
[0108] The extraction module 401 is configured to receive text description information corresponding to the image to be colored, extract text features from the text description information, and extract image features from the image to be colored, where the image to be colored is a grayscale image;
[0109] An initialization module 402 is configured to initialize the color vector of the image to be colored to obtain an initialized color vector;
[0110] The fusion module 403 is configured to fuse the text feature, the image feature and the initialization color vector through a preset color decoder to obtain a comprehensive feature;
[0111] The filling module 404 is configured to fuse the comprehensive features and the image features to obtain a color filling result, and synthesize the image to be colored and the color filling result to complete the coloring of the image to be colored.
[0112] In some embodiments, the fusion module 403 is configured to:
[0113] Perform self-attention processing on the initialized color vector through a preset color decoder to obtain a self-attention output result;
[0114] Perform cross-attention processing on the self-attention output result and the text feature to obtain a first cross-attention result, and fuse the image feature and the first cross-attention result based on the query mechanism to obtain a query cross-attention result;
[0115] Perform cross attention processing on the self-attention output result and the image feature to obtain a second cross result;
[0116] Cross-attention processing is performed on the query cross-result and the second cross-result to obtain comprehensive features.
[0117] In some embodiments, the fusion module 403 is configured to:
[0118] The text features are used as the key vector and value vector, and the self-attention output result is used as the query vector. Cross-attention processing is performed to obtain the first cross result.
[0119] In some embodiments, the fusion module 403 is configured to:
[0120] Perform cross-attention processing on the query cross-result and the second cross-result to obtain the initial comprehensive features;
[0121] The initial comprehensive features are processed based on a feedforward neural network to obtain comprehensive features.
[0122] In some embodiments, the fill module 404 is configured to:
[0123] Determine a comprehensive feature dimension corresponding to the comprehensive feature, and adjust the image feature dimension corresponding to the image feature according to the comprehensive feature dimension;
[0124] Performing a dot product operation on the comprehensive features and the image features after dimension adjustment to fuse the comprehensive features and the image features after dimension adjustment to obtain an initial color filling result;
[0125] The dimension corresponding to the initial color filling result is adjusted to the image feature dimension before the dimension adjustment to obtain the color filling result.
[0126] In some embodiments, the fill module 404 is configured to:
[0127] Based on a preset fusion module, a convolution operation is performed on the image feature dimension to adjust the image feature dimension according to the comprehensive feature dimension. The preset fusion module is used to obtain a color filling result based on the comprehensive feature and the image feature.
[0128] In some embodiments, the filling module 404 synthesizes the image to be colored and the color filling result so that after the image to be colored is colored, the filling module 404 is further configured to:
[0129] Determine a loss value based on the colored image and the loss function;
[0130] Based on the loss value, the preset encoding parameters of the preset color decoder are adjusted, and the preset color decoding parameters include self-attention parameters corresponding to self-attention, cross-attention parameters corresponding to cross-attention, and neural network parameters corresponding to the feedforward neural network.
[0131] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.
[0132] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0133] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.
[0134] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0135] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.
[0136] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0137] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0138] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.
Claims
1. An image coloring method, characterized in that: include: Receiving text description information corresponding to the image to be colored, extracting text features from the text description information, and extracting image features from the image to be colored, wherein the image to be colored is a grayscale image; Initializing the color vector of the image to be colored to obtain an initialized color vector; By using a preset color decoder, the text feature, the image feature and the initialized color vector are fused to obtain a comprehensive feature; The comprehensive features and the image features are integrated to obtain a color filling result, and the image to be colored and the color filling result are synthesized to complete the coloring of the image to be colored.
2. The method according to claim 1, characterized in that: By using a preset color decoder, the text feature, the image feature and the initialization color vector are fused to obtain a comprehensive feature, including: Performing self-attention processing on the initialized color vector by the preset color decoder to obtain a self-attention output result; Performing cross attention processing on the self-attention output result and the text feature to obtain a first cross result, and fusing the image feature and the first cross result based on a query mechanism to obtain a query cross result; Performing cross attention processing on the self-attention output result and the image feature to obtain a second cross result; Cross-attention processing is performed on the query cross result and the second cross result to obtain comprehensive features.
3. The method according to claim 2, characterized in that Performing cross attention processing on the self-attention output result and the text feature to obtain a first cross result, including: The text feature is used as a key vector and a value vector, and the self-attention output result is used as a query vector, and cross-attention processing is performed to obtain a first cross result.
4. The method according to claim 2, characterized in that: Cross-attention processing is performed on the query cross-result and the second cross-result to obtain comprehensive features, including: Performing cross-attention processing on the query cross-result and the second cross-result to obtain an initial comprehensive feature; The initial comprehensive features are processed based on a feedforward neural network to obtain comprehensive features.
5. The method according to claim 1, characterized in that The comprehensive feature and the image feature are integrated to obtain a color filling result, including: Determining a comprehensive feature dimension corresponding to the comprehensive feature, and adjusting an image feature dimension corresponding to the image feature according to the comprehensive feature dimension; Performing a dot product operation on the comprehensive feature and the dimensionally adjusted image feature to fuse the comprehensive feature and the dimensionally adjusted image feature to obtain an initial color filling result; The dimension corresponding to the initial color filling result is adjusted to the image feature dimension before the dimension adjustment to obtain the color filling result.
6. The method according to claim 5, characterized in that Adjusting the image feature dimension corresponding to the image feature according to the comprehensive feature dimension includes: Based on a preset fusion module, a convolution operation is performed on the image feature dimension to adjust the image feature dimension according to the comprehensive feature dimension, and the preset fusion module is used to obtain a color filling result based on the comprehensive feature and the image feature.
7. The method according to claim 1, characterized in that After synthesizing the image to be colored and the color filling result so that the image to be colored is colored, the method further includes: Determine a loss value based on the colored image and the loss function; Based on the loss value, the preset encoding parameters of the preset color decoder are adjusted, and the preset color decoding parameters include self-attention parameters corresponding to self-attention, cross-attention parameters corresponding to cross-attention, and neural network parameters corresponding to the feedforward neural network.
8. An image coloring device, characterized in that: include: An extraction module is configured to receive text description information corresponding to the image to be colored, extract text features from the text description information, and extract image features from the image to be colored, wherein the image to be colored is a grayscale image; An initialization module is configured to initialize the color vector of the image to be colored to obtain an initialized color vector; A fusion module is configured to fuse the text feature, the image feature and the initialized color vector through a preset color decoder to obtain a comprehensive feature; The filling module is configured to fuse the comprehensive features and the image features to obtain a color filling result, and synthesize the image to be colored and the color filling result to complete the coloring of the image to be colored.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.