Image coloring method and device, electronic equipment and storage medium

Abstract

The invention discloses an image coloring method and device, electronic equipment and a storage medium. The method comprises the following steps: performing feature extraction on a to-be-processed grayscale image to obtain image features of the grayscale image; performing pixel decoding on the image features to obtain a decoded grayscale image corresponding to the grayscale image; performing color decoding based on the decoded grayscale image and preset knowledge information to obtain color information of the grayscale image; the preset knowledge information is used for indicating recommended color information of at least one target entity in a plurality of common sense scenes; the grayscale image at least comprises one target entity; and coloring the grayscale image based on the color information to obtain a target image corresponding to the grayscale image. According to the method provided by the invention, the image coloring accuracy is relatively high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic information, and more particularly, to an image coloring method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of information technology, image automatic coloring, as an important research direction in the field of image processing, has attracted widespread attention. The goal of image coloring technology is to restore grayscale images to color images that conform to human visual perception, which has important application value in image restoration, old photo restoration, artistic style conversion, and infrared detection.

[0003] In related technologies, first, a coloring model architecture such as a convolutional neural network (CNN), a generative adversarial network (GAN), or an autoencoder is constructed, and the coloring model is trained on large-scale color image data, so that the coloring model learns the mapping relationship between grayscale images and color images. Then, the trained coloring model can be used to realize automatic coloring. However, the coloring effect of the grayscale image is poor using this method. SUMMARY

[0004] Therefore, the embodiments of the present application provide an image coloring method and device, an electronic device, and a storage medium.

[0005] In a first aspect, the embodiments of the present application provide an image coloring method, which includes: performing feature extraction on a grayscale image to be processed to obtain image features of the grayscale image; performing pixel decoding on the image features to obtain a decoded grayscale image corresponding to the grayscale image; performing color decoding based on the decoded grayscale image and preset knowledge information to obtain color information of the grayscale image; the preset knowledge information is used to indicate recommended color information of at least one target entity in a plurality of common sense scenes; the grayscale image includes at least one target entity; and performing coloring processing on the grayscale image based on the color information to obtain a target image corresponding to the grayscale image.

[0006] In a fourth aspect, the embodiments of the present application provide an image coloring device, which includes: a feature extraction module configured to perform feature extraction on a grayscale image to be processed to obtain image features of the grayscale image; a decoding module configured to perform pixel decoding on the image features to obtain a decoded grayscale image corresponding to the grayscale image; a color decoding module configured to perform color decoding based on the decoded grayscale image and preset knowledge information to obtain color information of the grayscale image; the preset knowledge information is used to indicate recommended color information of at least one target entity in a plurality of common sense scenes; the grayscale image includes at least one target entity; and a coloring module configured to perform coloring processing on the grayscale image based on the color information to obtain a target image corresponding to the grayscale image.

[0007] In a third aspect, an electronic device is provided, including a processor and a memory. The memory has computer readable instructions stored thereon. When the computer readable instructions are executed by the processor, the method described above is implemented.

[0008] In a fourth aspect, a computer readable storage medium is provided, having computer readable instructions stored thereon. When the computer readable instructions are executed by a processor, the method described above is implemented.

[0009] In a fifth aspect, a computer program product is provided, including computer instructions. When the computer instructions are executed by a processor, the method described above is implemented.

[0010] The training method of the object detection model, the image coloring method and device, the electronic device and the storage medium provided by the embodiments of the present application are provided. In the present application, first, the feature extraction and pixel decoding are performed on the gray image in sequence to obtain a decoded gray image. Then, color decoding is performed based on the decoded gray image and preset knowledge information to obtain color information of the gray image. Finally, color decoding is performed based on the decoded gray image and the preset knowledge information to obtain the color information of the gray image. In this way, the coloring process is guided by the preset knowledge information of the recommended color information of the target entity in multiple common sense scenes. The coloring process considers the recommended color information of the target entity in different common sense scenes in the gray image, so that the color information determined for the gray image is more consistent with the common sense scene. The situation that the coloring model is difficult to learn the hidden layer knowledge of the recommended color information of the target entity in different common sense scenes is avoided, the color information determined for the gray image does not conform to the common sense scene when the gray image is colored by the coloring model, and the accuracy of the color information of the gray image is poor. The accuracy and rationality of the color information determined for the gray image are improved, and the accuracy and rationality of the target image after coloring based on the color information of the gray image are improved, and the situation of distortion of the target image after coloring is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A schematic diagram suitable for the application scenario to which the embodiments of the present application are applied is shown; Figure 2 A flowchart of an image coloring method according to an embodiment of the present application is shown; Figure 3A flow chart of a gray image encoder, a pixel decoder, a coloring model and a color fusion model training process in an embodiment of the present application is shown. Figure 4 A schematic diagram of an image coloring process in an embodiment of the present application is shown. Figure 5 A block diagram of an image coloring device in an embodiment of the present application is shown. Figure 6 A structural block diagram of an electronic device for performing an image coloring method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Any other embodiments obtained by those of ordinary skill in the art without creative work based on the embodiments in the present application shall fall within the scope of the present application.

[0014] In the following description, the terms "first\second" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that the "first\second" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application. It should be noted that "multiple" as mentioned herein refers to two or more. The association relationship of "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0016] Reference is made to Figure 1 which shows a schematic diagram of an application scenario applicable to the embodiments of the present application. The application scenario includes a terminal 110 and a server 120.

[0017] The terminal 110 is, for example, a smartphone, a tablet computer, an e-book reader, a music player, a wearable device, a smart home device, a vehicle-mounted terminal, and the like. The terminal 110 is installed with an image coloring client capable of performing coloring processing on a to-be-processed grayscale image to obtain a colored target image, or the image coloring client is capable of sending an image coloring request so that the server 120 performs coloring processing on the to-be-processed grayscale image based on the image coloring request to obtain the colored target image.

[0018] The server 120 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0019] The terminal 110 can generate and send an image coloring request to the server 120 (through the image coloring client), and the server 120 performs coloring processing on the to-be-processed grayscale image based on the image coloring request to obtain the colored target image, and finally returns the target image to the terminal 110 so that the terminal 110 displays the target image.

[0020] In some embodiments, the server 120 itself can perform coloring processing on the to-be-processed grayscale image in response to a user's locally triggered image coloring operation to obtain the colored target image.

[0021] Alternatively, the terminal 110 itself can perform coloring processing on the to-be-processed grayscale image in response to a user's locally triggered image coloring operation to obtain the colored target image.

[0022] In addition, it is worth mentioning that in the embodiments of the present application, the image coloring process involves a grayscale image encoder, a pixel decoder, a coloring model, and a color fusion model, and the training process of the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model can be performed on the server 120. After the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model are trained, the server 120 can implement image coloring based on the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model.

[0023] Of course, the server 120 can also send the trained gray image encoder, pixel decoder, coloring model and color fusion model to the terminal 110, and the terminal 110 caches the gray image encoder, pixel decoder, coloring model and color fusion model, and realizes coloring locally in the terminal 110 based on the gray image encoder, pixel decoder, coloring model and color fusion model. Alternatively, the terminal 110 trains the gray image encoder, pixel decoder, coloring model and color fusion model itself, and stores the trained gray image encoder, pixel decoder, coloring model and color fusion model for local coloring in the terminal 110 The data for training the gray image encoder, pixel decoder, coloring model and color fusion model can be stored in a database (not shown in the figure), and when the server 120 or the terminal 110 needs to train the gray image encoder, pixel decoder, coloring model and color fusion model, the data for training the gray image encoder, pixel decoder, coloring model and color fusion model is obtained from the database. Figure 1 The data for training the gray image encoder, pixel decoder, coloring model and color fusion model can be stored in a database (not shown in the figure), and when the server 120 or the terminal 110 needs to train the gray image encoder, pixel decoder, coloring model and color fusion model, the data for training the gray image encoder, pixel decoder, coloring model and color fusion model is obtained from the database.

[0024] In order to more clearly explain the scheme of the present application, the following embodiments are explained with the electronic device as the execution subject of the image coloring method.

[0025] Please refer to Figure 2 , Figure 2 a flowchart of a training method of an object detection model according to an embodiment of the present application is shown, the method is applied to an electronic device, which can be the terminal 110 or the server 120 in Figure 1 , and the method can include: S110, feature extraction is performed on a gray image to be processed to obtain image features of the gray image.

[0026] The gray image to be processed refers to a gray image to be colored, which can be a gray image obtained by a camera capturing a gray image of a certain area, can also be a gray image obtained from the network, or can also be a gray image obtained by performing gray processing on a captured color image, which is not limited in the present application.

[0027] The gray image is an image with only one sample color for each pixel, which is usually displayed as a gray scale from the darkest black to the brightest white. The pixel value of each pixel point of the gray image is the gray value, the higher the gray value, the closer the pixel point to white, and the lower the gray value, the closer the pixel point to black, for example, when the gray value interval is 0-255, the pixel value is 0, the pixel point is black, and the pixel value is 255, the pixel point is white.

[0028] In some embodiments, the feature extraction can be performed on the grayscale image to be processed by a visual feature extractor, such as a BERT model, a visual encoder in a CLIP (Contrastive Language-Image Pre-training) model, a convolutional neural network, or a deep neural network, to obtain image features of the grayscale image.

[0029] In yet some embodiments, S110 comprises: performing convolution processing on the grayscale image to obtain an initial feature map of the grayscale image; and performing multi-scale feature extraction based on the initial feature map to obtain image features of the grayscale image.

[0030] In this embodiment, the grayscale image can be convolved by a convolution layer with a large kernel (e.g., a 7x7 kernel) to increase the channel dimension and enhance the expression ability of local edges and textures of the image, and the result of the convolution is taken as the initial feature map of the grayscale image. Then, multi-scale feature maps are extracted based on the initial feature map of the grayscale image, and the multi-scale feature maps are taken as the image features of the grayscale image.

[0031] Of course, different grayscale images can be different, and therefore, the input grayscale image can be standardized to process the grayscale image into a standardized standard grayscale image, and then the standard grayscale image is convolved to obtain the initial feature map of the grayscale image. The standardization processing can refer to a method of removing the mean value and dividing by the standard deviation to make the pixel values of the grayscale image conform to the standard normal distribution, i.e., the mean value is 0 and the standard deviation is 1. The purpose of this processing method is to eliminate the influence of brightness, contrast, and other factors in the grayscale image, so that the data between different grayscale images has the same scale and dimension, facilitating subsequent processing and analysis.

[0032] Of course, the size of different grayscale images can also be different, and therefore, the grayscale image can be converted into a grayscale image with a fixed size before the grayscale image is standardized, and then the grayscale image with a fixed size is standardized. The fixed size can be, for example, 256x256 or 512x512.

[0033] Optionally, the step of performing multi-scale feature extraction based on the initial feature map to obtain image features of the grayscale image can comprise: A1, encoding the input feature map of the Nth step by the encoding module of the Nth step to obtain the encoded feature map of the Nth step; wherein N is a natural number greater than zero, and the input feature map of the first step is the initial feature map; the input feature map of the Nth step and the encoded feature map of the Nth step are different in scale. A2, if N is less than M, obtaining the encoding feature map of the Nth step as the input feature map of the N+1th step, and accumulating N by 1; M is the total number of encoding modules; A3, returning to execute the step of encoding the input feature map of the Nth step through the encoding module of the Nth step to obtain the encoding feature map of the Nth step, until N is equal to M, obtaining M encoding feature maps; A4, obtaining other encoding feature maps except the encoding feature map of the first step as the image features of the grayscale image.

[0034] In the present application, a grayscale image encoder can be trained, the grayscale image encoder including M encoding modules, each encoding process being processed through an encoding module, and the sizes of the feature maps before and after each encoding process being inconsistent, so as to achieve the goal of determining multi-scale feature maps.

[0035] Firstly, the input feature map of the 1st step (i.e. the initial feature map of the grayscale image) is encoded through the 1st encoding module to obtain the encoded result as the encoding feature map of the 1st step, then the encoding feature map of the 1st step is taken as the input feature map of the 2nd step, the input feature map of the 2nd step is encoded through the 2nd encoding module to obtain the encoded result as the encoding feature map of the 2nd step, and so on, until the encoding feature map of the Mth step is obtained, at this time, M encoding feature maps are obtained in total, the 2nd to Mth encoding feature maps are obtained as the image features extracted for the grayscale image, and the image features extracted for the grayscale image include (M-1) scale feature maps.

[0036] Since the resolution of the 1st encoding feature map is too high and the data amount is large, in the present application, the 1st encoding feature map is not selected to improve the data processing efficiency.

[0037] In some embodiments, the grayscale image encoder can adopt a ConvNeXt network structure to combine the advantages of modern convolutional networks and realize efficient extraction of multi-level and multi-scale semantic features of images. That is, for the M encoding modules in the grayscale image encoder, each encoding module includes a plurality of basic modules, and each basic module includes: a deep convolution operation for extracting structural information in the image space; a feature normalization process for normalizing the feature distribution of different channels and enhancing the training stability; a channel mapping process for learning the nonlinear relationship between channels and enhancing the expression ability; and a residual connection process for preserving input information and preventing feature degradation.

[0038] Therefore, the aforementioned A1 can include: sequentially performing, by the Nth encoding module, a deep convolution operation, a feature normalization processing, a channel mapping processing and a residual connection processing on the input feature map of the Nth step to obtain the Nth encoding feature map. That is, by each basic module in the Nth encoding module, the input of the basic module is sequentially subjected to a deep convolution operation, a feature normalization processing, a channel mapping processing and a residual connection processing to obtain a processed result as the input of the next basic module in the Nth encoding module, until the output of the last basic module in the Nth encoding module is obtained as the Nth encoding feature map.

[0039] However, as the network layer deepens, the spatial resolution of the encoding feature map output by the encoding module of each step gradually decreases, and the number of channels gradually increases, realizing hierarchical modeling from local texture to overall semantics to obtain a multi-scale feature map.

[0040] For example, when the grayscale image encoder includes M encoding modules, the size of the input grayscale image is 256x256, and the grayscale image encoder finally outputs four scale feature maps: a 128x128 feature map (from the first encoding module), a 64x64 feature map (from the second encoding module), a 32x32 feature map (from the third encoding module), and a 16x16 feature map (from the fourth encoding module). At this time, the 64x64 feature map, the 32x32 feature map and the 16x16 feature map are obtained as the multi-scale features of the grayscale image (the feature map obtained by the first encoder module is not selected), wherein the first scale feature map (64x64 feature map) has high resolution and is suitable for retaining image edge details and local texture information; the second scale feature map (32x32 feature map) has medium resolution and is suitable for representing regional structure and color distribution; and the third scale feature map (16x16 feature map) has low resolution but the most abundant semantic information, and is suitable for reasoning object categories and context relationships. The channel dimensions of the three scale feature maps gradually increase according to the stages, such as 128, 256 and 512.

[0041] The processing process of the aforementioned S110 is actually feature extraction and semantic modeling on the input grayscale image, which is used to embed structural information, texture distribution and semantic objects in the visual content into a high-dimensional feature space to obtain image features of the grayscale image, thereby providing a basis for subsequent color generation and knowledge fusion.

[0042] S120, pixel decoding the image features to obtain a decoded grayscale image corresponding to the grayscale image.

[0043] The image features of the grayscale image can be pixel-decoded by a visual feature decoder to obtain a decoded grayscale image corresponding to the grayscale image. The visual feature decoder can be, for example, a decoder in a BERT model, a self-encoder, a U-net network, a convolutional neural network, or a deep neural network.

[0044] In some embodiments, when the grayscale image encoder includes M encoding modules, S120 can further include: B1, performing decoding processing based on the decoding input feature map of the Pth step to obtain a decoding feature map of the Pth step; P is a natural number greater than 0; the decoding input feature map of the 1st step is the encoding feature map of the Mth step; the decoding input feature map of the Pth step and the decoding feature map of the Pth step are different in size; B2, if P is less than M, obtaining the decoding feature map of the Pth step as the decoding input feature map of the P+1th step, and accumulating P by 1; B3, returning to perform the step of performing decoding processing based on the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step, until P is equal to M, and obtaining the decoding feature map of the Mth step as the decoded grayscale image.

[0045] The pixel decoder module accepts multi-scale feature maps from the grayscale image encoder, which contain various high-level semantic information (such as shape, object contour, edge information, etc.) extracted from the original grayscale image. After down-sampling and convolution operations, these feature maps have encoded the high-level semantic features of the image and the low-level local details. After specific processing, these encoded feature maps can gradually recover the pixel details of the image in the decoding process.

[0046] Firstly, the decoding processing can be performed on the decoding input feature map of the 1st step (i.e., the encoding feature map of the Mth step obtained in the foregoing S120) by the pixel decoder to obtain the decoding feature map of the 1st step, and the decoding feature map of the 1st step is obtained as the decoding input feature map of the 2nd step, and then the decoding processing is performed on the decoding input feature map of the 2nd step by the pixel decoder, and so on, until the decoding feature map of the Mth step is obtained, and the decoding feature map of the Mth step is obtained as the decoded grayscale image corresponding to the grayscale image. Each decoding process is an up-sampling process, therefore, the decoding input feature map of the Pth step and the decoding feature map of the Pth step are different in size, and the finally obtained decoded grayscale image is consistent with the size of the grayscale image input into the grayscale image encoder.

[0047] Optionally, B1 can include: B11, performing up-sampling processing on the decoding input feature map of the Pth step to obtain an up-sampled image of the Pth step; B12, fusing the up-sampling image of the Pth step and the decoding input feature map of the Pth step to obtain a decoding feature map of the Pth step.

[0048] First, the decoding input feature map of the Pth step is up-sampled by a pixel decoder to change the size of the decoding input feature map of the Pth step, so as to restore the resolution (that is, the decoding input feature map of the Pth step obtained is consistent with the size of the input feature map of the input of the encoding module in the feature extraction process in S110), to obtain an up-sampling image of the Pth step, and then the up-sampling image of the Pth step and the decoding input feature map of the Pth step are fused by splicing or weighting to obtain a fused decoding feature map of the Pth step.

[0049] However, the channels of the up-sampling image of the Pth step and the decoding input feature map of the Pth step are inconsistent, so before fusion, the number of channels of the up-sampling image of the Pth step and the decoding input feature map of the Pth step can be adjusted to be consistent by convolution operation (for example, adjusting the number of channels by a 1x1 convolution layer), and then the up-sampling image of the Pth step and the decoding input feature map of the Pth step are fused to obtain a decoding feature map of the Pth step.

[0050] And this step-by-step restoration strategy can better balance the overall structure and local details of the gray-scale image, avoiding distortion problems caused by relying on a single scale feature.

[0051] Further, B12 can include: B121, fusing the up-sampling image of the Pth step and the decoding input feature map of the Pth step to obtain an intermediate fusion feature map of the Pth step; B122, fusing the encoding feature map of the M-Pth step and the intermediate fusion feature map of the Pth step to obtain a decoding feature map of the Pth step.

[0052] That is, the up-sampling image of the Pth step and the decoding input feature map of the Pth step can be first convolved to make their channels consistent, and then the convolved up-sampling image of the Pth step and the convolved decoding input feature map of the Pth step are fused to obtain an intermediate fusion feature map of the Pth step.

[0053] However, the M-1 encoding feature maps obtained are multi-scale feature maps, and their channel numbers are inconsistent, making it difficult to fuse them, so the encoding feature map of the M-Pth step (that is, the input feature map of the input of the encoding module of the M-P+1th step) also needs to be convolved to adjust the number of channels of the encoding feature map of the M-Pth step to be consistent with the number of channels of the intermediate fusion feature map of the Pth step, and then the encoding feature map of the M-Pth step after adjusting the number of channels is fused with the intermediate fusion feature map of the Pth step to obtain a decoding feature map of the Pth step.

[0054] It is worth mentioning that in the case of P being M, M-P is 0, so the intermediate fusion feature map of the P-th step is directly obtained as the decoding feature map of the P-th step.

[0055] Through such a skip connection (the input of the encoding module of the M-P+1-th step in the gray image encoder is connected with the decoding process of the P-th step in the pixel decoder), low-level image detail information (such as edges and textures) can be directly transmitted to high-level feature maps in the decoding process, so as to ensure that fine details in the decoding process will not be lost, and these low-level details are particularly important for the texture and boundary of the image, which helps to ensure that the generated decoded gray image is structurally real and natural.

[0056] However, the pixel decoder does not output a gray image, so after obtaining the decoding feature map of the M-th step, the decoding feature map of the M-th step can also be restored to a high-resolution gray image through a convolution layer, which is the decoded gray image.

[0057] In the decoding stage of the pixel decoder, the pixel decoder has fully fused feature information from different scales, so it can generate a clear and realistic gray image. The output decoded gray image not only matches the original image in shape and structure, but also contains detailed texture information, which provides a sufficient image basis for the subsequent color decoder.

[0058] That is, through the pixel decoder, the details of the image can be gradually restored, and the information of each pixel can be accurately recovered. The final output is a gray image, a decoded gray image, with finely processed edges, textures and structural details, providing a reliable basis for the subsequent coloring process.

[0059] The main task of the pixel decoder is to gradually restore the multi-scale features extracted by the gray image encoder into image pixel information, and finally generate a decoded gray image that matches the structure and texture of the original image. The pixel decoder and the gray image encoder form a structure similar to U-Net, which gradually recovers the detail information of the original image through the decoding process, especially the low-level features such as edges, textures and shapes. The pixel decoder uses a layer-by-layer upsampling and refinement strategy to gradually improve the resolution of the image from multiple different scales, so that the output of each layer can more accurately restore the pixel information of the image, and the decoded decoded gray image is more accurate.

[0060] S130, color decoding based on the decoded gray image and the preset knowledge information to obtain color information of the gray image.

[0061] The preset knowledge information is used to indicate recommended color information of the at least one target entity in multiple common sense scenes respectively. The grayscale image at least includes the target entity. Thus, when coloring the target entity, the preset knowledge information can be referred to, so that the coloring effect of the target entity is better. The target entity can be an animal, a plant, a building, or the like.

[0062] The common sense scene can be a location scene, a time scene, a weather scene, and a climate scene, or the like. For example, the common sense scene can be outdoor, outdoor with clouds, winter, summer, and rainy day, or the like. The preset knowledge information can include “in an outdoor scene, if there are clouds, the sky can be gray or blue”, and for another example, the preset knowledge information can include “a banana in a mature state is yellow”.

[0063] For an example, the method for obtaining the preset knowledge information includes: obtaining a sample image, text description information of the sample image, and configured color usage common sense for the sample image; performing recognition processing based on the sample image and the text description information to obtain a target entity recognized in the sample image and attribute information of the target entity; the attribute information at least includes color information of the target entity; determining a color frequency distribution of the target entity based on the sample image, the text description information, and the color usage common sense; the color frequency distribution indicates a possibility that the target entity belongs to multiple color information respectively; determining color rule template information corresponding to the target entity based on the target entity, the attribute information of the target entity, and the color frequency distribution of the target entity; the color rule template information is used to indicate recommended color information of the target entity in multiple common sense scenes respectively; performing construction of a triple graph structure or construction of a decision tree based on the color rule template information to obtain the preset knowledge information.

[0064] The sample image can be a visual sample from a large-scale labeled color image dataset (such as ImageNet, COCO, ADE20K, or the like), which contains rich object labels, region segmentation information, and color distribution, and provides a basis for statistical corresponding relationship between objects and colors.

[0065] The text description information of the sample image refers to text description information related to the content of the sample image, including image captions, scene descriptions, user annotations, and open corpus (such as Wikipedia, encyclopedia knowledge platform, or the like), which can be used to extract object attributes and context semantic relationship.

[0066] The color usage common sense refers to color usage common sense summarized in domain expert experience, industry application rules, and professional guidance documents in art, medicine, and the like, which usually has strong prior rationality and can be used as important supplementary information for model reasoning. The three types of data together constitute the core input basis for determining the preset knowledge information.

[0067] The semantic target entities and their attribute information can be automatically extracted from image labels, region annotations, and corresponding textual descriptions through computer vision and natural language processing techniques. For example, "dog", "road", and "sky" in a sample image can be identified as target entities, and their associated colors (such as "blue" and "gray"), materials (such as "fur" and "concrete"), and positions (such as "top area" and "foreground") can be extracted as attribute information of the target entities. This process typically uses object detection models (such as Mask R-CNN, YOLO, etc.) to identify target entities, and pre-trained language models (such as BERT, T5, etc.) for text analysis and named entity recognition.

[0068] After extracting the target entities, the color distribution of each type of target entity in a large number of sample images is clustered and analyzed to extract high-frequency color patterns. Specifically, the K-Means or Mean-Shift algorithm is used to analyze the region pixels of the target entities in the sample images to obtain their main color information. Then, by combining the text description information and color usage common sense, the color change trend in different scenarios (such as spring, autumn, and winter) is determined. The output of this process is the color frequency distribution, which will be one of the core elements for determining the preset knowledge information.

[0069] Subsequently, based on the extracted target entities, attribute information, and color frequency distribution, and in combination with the text description, reasoning rules with causal logic are constructed to obtain color rule template information. For example: "In an outdoor scene, if there are clouds, the sky may be gray or blue", "Bananas are yellow when they are ripe, and green when they are unripe". This process performs syntactic analysis and knowledge fusion on expressions from text corpora or industry specifications to convert language descriptions into computer usable rule templates (e.g., in the form of regular expressions). The rule templates are the color rule template information.

[0070] After sufficient color rule template information is sorted out, two types of structured knowledge outputs are constructed: First, a knowledge graph, which expresses "object-attribute-color" triples in a graph structure, with nodes representing specific target entities or attribute information and edges representing semantic relationships such as "belongs to", "corresponds to", and "inferred". Second, a color decision tree, which constructs a decision path from top to bottom based on scene categories or semantic context, and gives color recommendations at the leaf nodes.

[0071] Both of these structured outputs can serve as preset knowledge information, which not only supports visual understanding but also can be directly called by the coloring process.

[0072] However, as aforementioned, the preset knowledge information is in a graph structure (triples in graph structure or decision tree), which is difficult to be directly used, therefore, the system can employ a graph neural network (such as GAT, R-GCN) to embed the preset knowledge information, and convert the original structured knowledge into a trainable vector representation, which is the knowledge vector corresponding to the preset knowledge information. These knowledge vectors will be used as external semantic supplements, and will be input into the coloring model together with the decoded grayscale image, so that the coloring model can refer to the semantic constraints and prior knowledge indicated by the knowledge vectors when making color selection, thereby improving the accuracy and consistency of the coloring process.

[0073] The coloring model can be input with the knowledge vector corresponding to the preset knowledge information and the decoded grayscale image, and can realize coloring by the coloring model to obtain color information corresponding to the grayscale image, wherein the color information includes target color values of each pixel point in the grayscale image.

[0074] The coloring model receives the output from the pixel decoder (usually a structured grayscale image, i.e., the decoded grayscale image, or a pixel feature map containing edges, contours, and textures, i.e., the decoded feature map output by the first step of the pixel decoder), and also receives the knowledge vector corresponding to the preset knowledge information, which is extracted by a graph neural network or other mechanisms and contains high-level semantic information such as main entity categories, spatial positions, typical color attributes, and co-occurrence rules in the preset knowledge information. These two kinds of information constitute the semantic and pixel two channels of the input of the coloring model.

[0075] Optionally, S130 can include: C1, performing color decision processing based on the decoded grayscale image and the preset knowledge information to obtain a color decision weight map of the grayscale image; the color decision weight map includes weights of each pixel point in the grayscale image under multiple color values; C2, predicting color values based on the grayscale image to obtain initial color values of each pixel point in the grayscale image; C3, adjusting the initial color values of each pixel point in the grayscale image based on the color decision weight map to obtain target color values of each pixel point in the grayscale image; the color information of the grayscale image includes the target color values of each pixel point in the grayscale image The coloring model can perform color decision processing based on the decoded grayscale image and the preset knowledge information to obtain a color decision weight map of the grayscale image. Specifically, the knowledge vector after encoding the preset knowledge information and the decoded grayscale image are fused to obtain a fused color feature; the fused color feature is subjected to convolution and attention mechanism operation to obtain the color decision weight map of the grayscale image.

[0076] However, the dimensions of the preset knowledge information encoded knowledge vector and the decoded gray image may not be consistent, therefore, the knowledge vector can be expanded to the dimensions of the decoded gray image, then the expanded knowledge vector is fused with the decoded gray image, and a semantic guide image is generated through a series of convolution and attention calculation, which can be used as a color decision weight map to guide the subsequent coloring calculation, so that the coloring model can rely more on knowledge guidance in key areas, enhance the context judgment in areas with uneven texture or ambiguous semantics, and avoid color errors or style fragmentation.

[0077] Then, the coloring model predicts the color value based on the gray image to obtain the initial color value of each pixel point in the gray image. Specifically, the coloring model uses self-attention combined with cross-domain attention to construct a coloring block, which takes the pixel information of the decoded gray image obtained from the pixel decoder as guidance and takes the given color (for example, 300 given colors) as query, and finally obtains the initial color of each pixel pair (which is one of the 300 given colors).

[0078] Finally, the initial color value of each pixel point in the gray image is adjusted in combination with the color decision weight map to obtain the target color value of each pixel point in the gray image, and the target color value of each pixel point in the gray image is summarized as the color information of the gray image.

[0079] In some embodiments, for each pixel point in the gray image, if the weight of the initial color value of the pixel point in the color decision weight map does not satisfy the target condition, the color value corresponding to the highest weight of the pixel point in the color decision weight map is obtained as the target color value of the pixel point.

[0080] The target condition can include that the weight of the initial color value of the pixel point in the color decision weight map is the highest value of the weights of all colors corresponding to the pixel point in the color decision weight map, or the weight of the initial color value of the pixel point in the color decision weight map is not less than a weight threshold, for example, 0.5.

[0081] For example, the weight of the initial color value y1 of the pixel point s1 in the color decision weight map is 0.45, and the highest value of the weights of all colors (100 colors) corresponding to the pixel point s1 in the color decision weight map is 0.7, then it is determined that the initial color value y1 of the pixel point s1 does not satisfy the target condition, and the color value s11 corresponding to the weight 0.7 of the pixel point s1 in the color decision weight map is obtained as the target color value of the pixel point s1.

[0082] For example, if the weight threshold is 0.6 and the weight of the initial color value y2 of the pixel point s2 in the color decision weight map is 0.3, it is determined that the initial color value y2 of the pixel point s2 does not meet the target condition, and the color value s21 corresponding to the highest weight 0.55 of the pixel point s2 in the color decision weight map is obtained as the target color value of the pixel point s2.

[0083] Correspondingly, for each pixel point in the grayscale image, if the weight of the initial color value of the pixel point in the color decision weight map meets the target condition, the initial color value of the pixel point is obtained as the target color value of the pixel point.

[0084] Of course, in order to improve the coloring efficiency, the initial color value determined can be the values of the a and b channels in the Lab color space, so that only 2 color values need to be determined, thereby improving the coloring efficiency.

[0085] In the coloring process, the coloring model not only considers the local context information of each pixel, but also optimizes the color consistency of the entire region to prevent unnatural color block jumps or cross-border miscoloring problems. In the coloring process, the color decision weight map guided by the preset knowledge information is also utilized, and the color information is controlled by the knowledge guidance, thereby achieving a higher level in style, semantic consistency and visual naturalness, and achieving better coloring effect.

[0086] S140, coloring processing is performed on the grayscale image based on the color information to obtain a target image corresponding to the grayscale image.

[0087] After obtaining the color information of the grayscale image, each pixel of the grayscale image is colored by using the target color value of each pixel point in the color information.

[0088] As described above, the target color value of each pixel point in the color information is the value of the a and b channels in the Lab color space, and the pixel value of the grayscale image is the L value in the Lab color space. Therefore, the L, a and b values of each pixel point can be directly used as the pixel value of the pixel point, thereby realizing the coloring processing of the grayscale image.

[0089] Of course, after obtaining the target image in the LAB format, it can also be converted into a color image in the RGB format or a color image in the CMYK format for use in different scenarios.

[0090] In some embodiments, S140 can include performing coloring processing on the grayscale image based on the grayscale image and the color information of the grayscale image by using a color fusion model to perform detailed optimization processing on the target color value of each pixel point in the grayscale image, thereby obtaining a target image corresponding to the grayscale image. Since the gray image contains the structure, texture and boundary information of the image, and the decoded color information provides the color distribution based on semantic inference, they are complementary in information. Therefore, the task of the color fusion model is to reasonably combine the structure characteristics of the gray image and the color distribution according to the structure characteristics of the gray image and the color distribution, so as to ensure that the finally generated target image retains the details while the color is natural and conforms to common sense.

[0091] Specifically, the color fusion model combines the gray information in the gray image with the color information provided by the coloring model through a weighted fusion strategy. The color fusion model dynamically adjusts the weight according to the semantic weight of each pixel point, the context information and the boundary details, so that the details of the gray image are retained in the detail area (such as the edge of the object), and the coloring result of the color decoder is preferentially used in the large area (such as the sky, grassland, etc.), so as to achieve a balance. Finally, a finely fused color target image is output, which not only maintains the clarity of the structure, but also meets the actual and visual requirements in color.

[0092] In this embodiment, first, the gray image is sequentially subjected to feature extraction and pixel decoding to obtain a decoded gray image, and then color decoding is performed based on the decoded gray image and the preset knowledge information to obtain color information of the gray image. Finally, color decoding is performed based on the decoded gray image and the preset knowledge information to obtain color information of the gray image. Thus, the coloring process is guided by the preset knowledge information of the recommended color information of the target entity in multiple common sense scenarios, and the coloring model considers the recommended color information of the target entity in different common sense scenarios in the gray image, so that the color information determined for the gray image is more consistent with the common sense scenario, avoiding the situation that the coloring model is difficult to learn the hidden layer knowledge of "recommended color information of the target entity in different common sense scenarios", resulting in that the color information determined for the gray image by the coloring model does not conform to the common sense scenario, and the accuracy of the color information of the gray image is poor. The accuracy and rationality of the color information determined for the gray image are improved, and the accuracy and rationality of the target image after coloring based on the color information of the gray image are improved, and the situation of distortion of the target image after coloring is reduced.

[0093] Secondly, the time information, location information, object color knowledge and the like related to the image are used to introduce a triple graph structure or a decision tree knowledge graph as the preset knowledge information to improve the understanding ability of the image content, so as to realize more accurate, natural and semantically consistent automatic coloring of the image.

[0094] In some embodiments, as aforementioned, the coloring process involves a grayscale image encoder, a pixel decoder, a coloring model, and a color fusion model, and the training process of the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model comprises: S210, feature extraction is performed on the training grayscale image by the grayscale image encoder to obtain training image features of the training grayscale image.

[0095] The training grayscale image refers to a grayscale image used for training the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model, and the acquisition process is as aforementioned in S110.

[0096] The encoding process of S210 refers to the aforementioned encoding process of S110 and will not be repeated here.

[0097] S220, pixel decoding is performed on the training image features by the pixel decoder to obtain a training decoded grayscale image corresponding to the training grayscale image.

[0098] The decoding process of S220 refers to the aforementioned decoding process of S120 and will not be repeated here.

[0099] S230, color decoding is performed on the training decoded grayscale image and the preset knowledge information by the coloring model to obtain training color information of the training grayscale image.

[0100] The training color information includes target color values of each pixel point in the training grayscale image, and the coloring process of S230 refers to the aforementioned coloring process of S130 and will not be repeated here.

[0101] S240, coloring processing is performed on the training grayscale image and the training color information of the training grayscale image by the color fusion model to perform detailed optimization processing on the target color values of each pixel point in the training grayscale image to obtain a target prediction image corresponding to the training grayscale image.

[0102] The fusion process of S240 refers to the aforementioned fusion process of S140 and will not be repeated here.

[0103] S250, the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model are trained based on the difference between the labeled color image corresponding to the training grayscale image and the target prediction image.

[0104] The labeled color image refers to a color image with high accuracy obtained by labeling the training grayscale image. Of course, to make the labeled color image more accurate, a color image can be obtained as a labeled color image, and then the labeled color image is subjected to grayscale processing to obtain a grayscale image as a training grayscale image, thereby making the labeled color image more accurate.

[0105] The image coloring loss can be determined based on the difference between the labeled color image and the target predicted image, and then the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model are trained through the image coloring loss to obtain the trained grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model.

[0106] Of course, after the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model are trained, the model can also be subjected to lightweight processing in the following manner: 1. Pruning processing: First, the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model are subjected to structural analysis, and channels, convolution kernels, or hierarchical modules that have less influence on the final output or are redundant are identified. On this basis, a structured pruning method is applied to remove network connections with low importance layer by layer and reduce unnecessary computational burden. The pruning process can use a sparsity index (such as L1 norm) or a gradient-based importance evaluation method to ensure that the complexity of the model is reduced while maintaining the integrity of core feature transmission.

[0107] 2. Weight quantization processing: After pruning the model (including the grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model), the size and structural complexity of the model are further combined to perform low-bit quantization processing on the weights and activation values in the model, such as quantizing from 32-bit floating point to 8-bit integer (INT8). In this way, the model storage volume is greatly compressed, and the inference speed on hardware accelerators (such as NPU, TPU, embedded GPU) is improved. During the quantization process, according to the characteristics of the model, an asymmetric quantization strategy is adopted in combination with quantization-aware training (QAT) to effectively reduce the accuracy loss.

[0108] 3. Inference graph optimization and fusion acceleration: The simplified model (including the simplified grayscale image encoder, the pixel decoder, the coloring model, and the color fusion model) will be subjected to graph structure optimization, including convolution and activation function fusion, BN layer folding, channel rearrangement, and other operations to further simplify the inference graph of the model. The optimized inference graph of the model is not only more compact in terms of parameter quantity, but also significantly improves the computational efficiency during hardware execution.

[0109] 4. Cross-platform deployment adaptation Finally, the pruned model (including the pruned grayscale image encoder, pixel decoder, coloring model, and color fusion model) can be exported in various deployment formats, such as ONNX, MNN, NCNN, etc., supporting flexible deployment on mobile terminals, embedded platforms, and different devices in the cloud. The final model consumes only about 400 MB of memory on each device, meeting the performance requirements of most devices. The model structure can also be adaptively adjusted according to the computing power characteristics of different platforms to ensure efficient operation of the model on ARM architecture, edge AI (Artificial Intelligence) chips, or low-power devices.

[0110] In this embodiment, the grayscale image encoder, pixel decoder, coloring model, and color fusion model are trained to obtain grayscale image encoders, pixel decoders, coloring models, and color fusion models with higher capabilities. Thus, when using the grayscale image encoder, pixel decoder, coloring model, and color fusion model for coloring, the coloring effect is better, and the resulting colored target image is more accurate.

[0111] Furthermore, techniques such as quantization and pruning are used to optimize the structure of the model, enabling efficient deployment on mobile dual systems. These methods can effectively improve the implementation effect and coloring efficiency of the image coloring technology, reduce costs, and enable efficient use on a variety of devices.

[0112] Moreover, the model is lightweight, with a small volume and low space occupation. The computational resources required for the model inference process are greatly reduced, thereby greatly improving the coloring efficiency and reducing the coloring cost.

[0113] In an exemplary embodiment of the present application, the image coloring process is as shown in Figure 4 .

[0114] The sample image, text description information of the sample image, and configured color usage common sense for the sample image are obtained. The knowledge information is organized based on the sample image, the text description information of the sample image, and the configured color usage common sense for the sample image, to obtain a triple graph structure or a decision tree as preset knowledge information.

[0115] After obtaining the input grayscale image, the grayscale image is encoded by the grayscale image encoder to obtain image features of the grayscale image. Then, the decoded grayscale image is obtained by the pixel decoder based on the image features of the grayscale image.

[0116] Subsequently, the coloring model performs coloring processing based on the decoded grayscale image and the preset knowledge information to obtain color information of the grayscale image.

[0117] Finally, the color fusion model is used to perform color fusion processing based on the original gray image and color information of the gray image, so that a color target image output by the color fusion model is obtained, and thus, the coloring processing of the gray image is realized.

[0118] The method improves the rationality and stability of color generation and reduces distortion by fusing explicit preset knowledge information and color decoding mechanism of image feature coding, and by guiding the distribution selection of the model in the color space.

[0119] Referring to Figure 5 , Figure 5 A block diagram of an image coloring device is shown, and the image coloring device 800 includes: The feature extraction module 810 is configured to perform feature extraction on the gray image to be processed to obtain image features of the gray image. The decoding module 820 is configured to perform pixel decoding on the image features to obtain a decoded gray image corresponding to the gray image. The color decoding module 830 is configured to perform color decoding based on the decoded gray image and preset knowledge information to obtain color information of the gray image; the preset knowledge information is used to indicate recommended color information of at least one target entity in multiple common sense scenarios; and the gray image includes at least one target entity. The coloring module 840 is configured to perform coloring processing on the gray image based on the color information to obtain a target image corresponding to the gray image.

[0120] Optionally, the feature extraction module 810 is further configured to perform convolution processing on the gray image to obtain an initial feature map of the gray image; and perform multi-scale feature extraction based on the initial feature map to obtain the image features of the gray image.

[0121] Optionally, the feature extraction module 810 is further configured to encode the input feature map of the Nth step by the Nth encoding module to obtain an encoded feature map of the Nth step; wherein N is a natural number greater than zero, the input feature map of the first step is the initial feature map, the input feature map of the Nth step and the encoded feature map of the Nth step are different in scale, if N is less than M, the encoded feature map of the Nth step is obtained as the input feature map of the N+1th step, and N is accumulated by 1; M is the total number of the encoding modules; the step of encoding the input feature map of the Nth step by the Nth encoding module to obtain the encoded feature map of the Nth step is returned to be executed until N is equal to M, M encoded feature maps are obtained; and the encoded feature maps other than the encoded feature map of the first step are obtained as the image features of the gray image.

[0122] Optionally, the feature extraction module 810 is further configured to sequentially perform, by the encoding module of the Nth step, deep convolution operation, feature normalization processing, channel mapping processing and residual connection processing on the input feature map of the Nth step to obtain the encoded feature map of the Nth step.

[0123] Optionally, the decoding module 820 is further configured to perform decoding processing based on the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step; P is a natural number greater than 0; the decoding input feature map of the first step is the encoded feature map of the Mth step; the decoding input feature map of the Pth step and the decoding feature map of the Pth step have different sizes; if P is less than M, the decoding feature map of the Pth step is obtained as the decoding input feature map of the P+1th step, and P is accumulated by 1; the step of performing decoding processing based on the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step is returned until P is equal to M, and the decoding feature map of the Mth step is obtained as the decoded gray-scale image.

[0124] Optionally, the decoding module 820 is further configured to perform up-sampling processing on the decoding input feature map of the Pth step to obtain the up-sampled image of the Pth step; and perform fusion processing based on the up-sampled image of the Pth step and the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step.

[0125] Optionally, the decoding module 820 is further configured to perform feature fusion on the up-sampled image of the Pth step and the decoding input feature map of the Pth step to obtain the intermediate fusion feature map of the Pth step; and perform feature fusion on the encoded feature map of the M-Pth step and the intermediate fusion feature map of the Pth step to obtain the decoding feature map of the Pth step.

[0126] Optionally, the color decoding module 830 is further configured to perform color decision processing based on the decoded gray-scale image and the preset knowledge information to obtain a color decision weight map of the gray-scale image; the color decision weight map includes weights of each pixel point in the gray-scale image under multiple color values; perform prediction of the color values based on the gray-scale image to obtain initial color values of each pixel point in the gray-scale image; adjust the initial color values of each pixel point in the gray-scale image based on the color decision weight map to obtain target color values of each pixel point in the gray-scale image; and the color information of the gray-scale image includes the target color values of each pixel point in the gray-scale image.

[0127] Optionally, the color decoding module 830 is further configured to, for each pixel point in the gray-scale image, if the weight of the initial color value of the pixel point in the color decision weight map does not satisfy a target condition, obtain a color value corresponding to the highest value of the weight of the pixel point in the color decision weight map as the target color value of the pixel point.

[0128] Optionally, the color decoding module 830 is further configured to fuse the preset knowledge information coded knowledge vector and the decoded grayscale image to obtain a fused color feature; and perform convolution and attention mechanism operation on the fused color feature to obtain a color decision weight map of the grayscale image.

[0129] Optionally, the color information of the grayscale image includes a target color value of each pixel point in the grayscale image; and the coloring module 840 is further configured to perform coloring processing on the grayscale image based on the grayscale image and the color information of the grayscale image through a color fusion model, to perform detailed optimization processing on the target color value of each pixel point in the grayscale image, and obtain a target image corresponding to the grayscale image.

[0130] Optionally, the device further includes a knowledge determining module configured to: obtain a sample image, text description information of the sample image, and configured color usage common sense for the sample image; perform recognition processing based on the sample image and the text description information to obtain a target entity recognized in the sample image and attribute information of the target entity; the attribute information at least includes color information of the target entity; determine a color frequency distribution of the target entity based on the sample image, the text description information, and the color usage common sense; the color frequency distribution indicates a possibility that the target entity belongs to multiple color information respectively; determine color rule template information corresponding to the target entity based on the target entity, the attribute information of the target entity, and the color frequency distribution of the target entity; the color rule template information is used to indicate recommended color information of the target entity in multiple common sense scenarios respectively; and construct a triple graph structure or construct a decision tree based on the color rule template information to obtain the preset knowledge information.

[0131] It should be noted that the device embodiments in the present application correspond to the foregoing method embodiments, and the specific principles of the device embodiments can be referred to the contents in the foregoing method embodiments, which will not be described herein.

[0132] Figure 6 A structural block diagram of an electronic device for performing the image coloring method according to the embodiments of the present application is shown. The electronic device can be a terminal 110 or a server 120, etc. Figure 1 It should be noted that the electronic device in the present application can be a terminal 110 or a server 120, etc. Figure 6 The computer system 1200 of the electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0133] As shown in FIG. 12, the computer system 1200 can include one or more processors 1202, one or more memory devices 1204, and one or more communication interfaces 1206. Figure 6As shown, the computer system 1200 includes a central processing unit (CPU) 1201 that can perform various suitable actions and processes in accordance with programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage section 1208, such as performing the methods in the above-described embodiments. Various programs and data required for the operation of the system are also stored in the RAM 1203. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0134] Connected to the I / O interface 1205 are an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable recording medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed into the storage section 1208 as necessary.

[0135] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable recording medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are performed.

[0136] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate or transport a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0137] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0138] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0139] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable storage medium carries computer readable instructions, which, when executed by a processor, implement the method in any of the above embodiments.

[0140] According to an aspect of the embodiments of the present application, a computer program product is provided, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the method in any of the above embodiments.

[0141] In the embodiments of the present application, the term "module" or "unit" refers to a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof, and similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit of the function of the module or unit.

[0142] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0143] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the present application.

[0144] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0145] It should be noted that the above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image coloring method characterized by, The method comprises: feature extraction is performed on the to-be-processed gray image to obtain image features of the gray image; pixel decoding is performed on the image features to obtain a decoded gray image corresponding to the gray image; color decoding is performed based on the decoded gray image and preset knowledge information to obtain color information of the gray image; the preset knowledge information is used to indicate recommended color information of at least one target entity in multiple common sense scenes; the gray image comprises at least one target entity; coloring processing is performed on the gray image based on the color information to obtain a target image corresponding to the gray image.

2. The method of claim 1, wherein, The feature extraction performed on the to-be-processed gray image to obtain image features of the gray image comprises: convolution processing is performed on the gray image to obtain an initial feature map of the gray image; multi-scale feature extraction is performed based on the initial feature map to obtain image features of the gray image.

3. The method of claim 2, wherein, The multi-scale feature extraction performed based on the initial feature map to obtain image features of the gray image comprises: encoding is performed on an input feature map of an Nth step by an encoding module of the Nth step to obtain an encoded feature map of the Nth step; N is a natural number greater than zero, and an input feature map of a first step is the initial feature map; a size of the input feature map of the Nth step is different from that of the encoded feature map of the Nth step; if N is less than M, the encoded feature map of the Nth step is acquired as an input feature map of an N+1th step, and N is accumulated by 1; M is a total number of the encoding modules; the step of performing encoding on the input feature map of the Nth step by the encoding module of the Nth step to obtain the encoded feature map of the Nth step is returned to be executed until N is equal to M, and M encoded feature maps are obtained; other encoded feature maps except the encoded feature map of the first step are acquired as the image features of the gray image.

4. The method of claim 3, wherein, The encoding performed on the input feature map of the Nth step by the encoding module of the Nth step to obtain the encoded feature map of the Nth step comprises: the input feature map of the Nth step is sequentially subjected to a depth convolution operation, feature normalization processing, channel mapping processing and residual connection processing by the encoding module of the Nth step to obtain the encoded feature map of the Nth step.

5. The method of claim 3, wherein, The pixel decoding performed on the image features to obtain the decoded gray image corresponding to the gray image comprises: decoding processing is performed based on a decoding input feature map of a Pth step to obtain a decoding feature map of the Pth step; P is a natural number greater than 0; a decoding input feature map of a first step is the encoded feature map of the Mth step; a size of the decoding input feature map of the Pth step is different from that of the decoding feature map of the Pth step; if P is less than M, the decoding feature map of the Pth step is acquired as a decoding input feature map of a P+1th step, and P is accumulated by 1; the step of performing decoding processing based on the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step is returned to be executed until P is equal to M, and the decoding feature map of the Mth step is acquired as the decoded gray image.

6. The method of claim 5, wherein, The decoding processing performed based on the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step comprises: Upsample the decoding input feature map of the Pth step to obtain an upsampled image of the Pth step; Fuse the upsampled image of the Pth step and the decoding input feature map of the Pth step to obtain a decoding feature map of the Pth step.

7. The method of claim 6, wherein, The fusion processing based on the upsampled image of the Pth step and the decoding input feature map of the Pth step to obtain the decoding feature map of the Pth step comprises: Fuse the upsampled image of the Pth step and the decoding input feature map of the Pth step to obtain an intermediate fusion feature map of the Pth step; Fuse the encoding feature map of the M-Pth step and the intermediate fusion feature map of the Pth step to obtain the decoding feature map of the Pth step.

8. The method of claim 1, wherein, The color decoding based on the decoded gray image and the preset knowledge information to obtain the color information of the gray image comprises: Perform color decision processing on the decoded gray image and the preset knowledge information to obtain a color decision weight map of the gray image; the color decision weight map comprises weights of each pixel point in the gray image under multiple color values; Predict a color value based on the gray image to obtain an initial color value of each pixel point in the gray image; Adjust the initial color value of each pixel point in the gray image based on the color decision weight map to obtain a target color value of each pixel point in the gray image; the color information of the gray image comprises the target color value of each pixel point in the gray image.

9. The method of claim 8, wherein, The adjustment of the initial color value of each pixel point in the gray image based on the color decision weight map to obtain the target color value of each pixel point in the gray image comprises: For each pixel point in the gray image, if the weight of the pixel point in the color decision weight map does not satisfy a target condition, obtain a color value corresponding to the highest weight of the pixel point in the color decision weight map as the target color value of the pixel point.

10. The method of claim 8, wherein, The color decision processing based on the decoded gray image and the preset knowledge information to obtain the color decision weight map of the gray image comprises: Fuse the knowledge vector after encoding the preset knowledge information and the decoded gray image to obtain a fused color feature; Perform convolution and attention mechanism operation on the fused color feature to obtain the color decision weight map of the gray image.

11. The method of claim 1, wherein, The color information of the gray image comprises the target color value of each pixel point in the gray image; The coloring processing based on the color information on the gray image to obtain a target image corresponding to the gray image comprises: Perform coloring processing on the gray image and the color information of the gray image based on a color fusion model to perform detailed optimization processing on the target color value of each pixel point in the gray image to obtain a target image corresponding to the gray image.

12. The method of claim 1, wherein, Before the color decoding based on the decoded gray image and the preset knowledge information to obtain the color information of the gray image, the method further comprises: obtaining a sample image, text description information of the sample image, and configured color use common sense for the sample image; performing recognition processing based on the sample image and the text description information to obtain a target entity recognized in the sample image and attribute information of the target entity; the attribute information at least includes color information of the target entity; determining a color frequency distribution of the target entity based on the sample image, the text description information, and the color use common sense; the color frequency distribution indicates a possibility that the target entity belongs to multiple color information respectively; determining color rule template information corresponding to the target entity based on the target entity, the attribute information of the target entity, and the color frequency distribution of the target entity; the color rule template information is used to indicate recommended color information of the target entity in multiple common sense scenarios respectively; performing construction of a triple graph structure or construction of a decision tree based on the color rule template information to obtain the preset knowledge information.

13. An image colorizing apparatus, characterized by comprising: The device comprises: a feature extraction module configured to perform feature extraction on a to-be-processed grayscale image to obtain image features of the grayscale image; a decoding module configured to perform pixel decoding on the image features to obtain a decoded grayscale image corresponding to the grayscale image; a color decoding module configured to perform color decoding based on the decoded grayscale image and preset knowledge information to obtain color information of the grayscale image; the preset knowledge information is used to indicate recommended color information of at least one target entity in multiple common sense scenarios respectively; the grayscale image at least includes one target entity; a coloring module configured to perform coloring processing on the grayscale image based on the color information to obtain a target image corresponding to the grayscale image.

14. An electronic device, comprising: comprise: a processor; a memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method of any one of claims 1-12.

15. A computer readable storage medium, characterized in that, having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method of any one of claims 1-12.

Images