An image coloring method, electronic device, storage medium, and program product

Abstract

This disclosure relates to an image colorization method, electronic device, storage medium, and program product. The image colorization method includes: acquiring a grayscale image to be colorized; scaling the grayscale image to obtain multiple grayscale images at different scales; extracting image semantic features from the multiple grayscale images at different scales respectively; fusing the image semantic features from the multiple grayscale images at different scales; and generating a color image based on the fused image semantic features. This disclosure can improve the color saturation of the generated color image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image coloring method, electronic device, storage medium, and program product. Background Technology

[0002] With the development of image processing technology, colorization techniques that can convert grayscale images into color images have been applied in various fields. For example, they are used in colorizing old photos and video restoration.

[0003] Among related technologies, image colorization techniques support fully automatic image colorization and user-interactive image colorization. User-interactive image colorization tasks often require the introduction of additional semantic information as guidance; in practice, the final colorization result and the introduced semantic information show a strong correlation. Fully automatic image colorization, because it does not introduce additional semantic information, presents greater challenges in solving colorization problems. Existing fully automatic image colorization methods are mainly based on deep learning, including various methods such as cascaded convolutional neural networks and generative adversarial networks.

[0004] However, the image coloring results obtained by using the image coloring methods provided in the related technologies are generally grayish, the image saturation is insufficient, and there are also problems such as large-scale color overflow. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides an image coloring method, an electronic device, a storage medium, and a program product.

[0006] According to a first aspect of the present disclosure, an image coloring method is provided, comprising:

[0007] Obtain a grayscale image to be colored; adjust the scale of the grayscale image to obtain multiple grayscale images of different scales; extract the image semantic features of the multiple grayscale images of different scales respectively, fuse the image semantic features of the multiple grayscale images of different scales, and generate a color image based on the fused image semantic features.

[0008] In one embodiment, the step of extracting image semantic features from the plurality of grayscale images at different scales, fusing the image semantic features from the plurality of grayscale images at different scales, and generating a color image based on the fused image semantic features includes:

[0009] The multiple grayscale images at different scales are input into a multi-scale convolutional neural network, which extracts the semantic features of the multiple grayscale images at different scales. The semantic features of the multiple grayscale images at different scales are then fused. The fused semantic features are then input into a color image generation network, which generates a color image of the grayscale images.

[0010] In one embodiment, the step of extracting image semantic features from the multiple grayscale images at different scales using the multi-scale convolutional neural network includes:

[0011] The multi-scale convolutional neural network performs residual processing and pooling processing on the multiple grayscale images at different scales to extract the image semantic features of the multiple grayscale images at different scales.

[0012] The process of generating a color image from the grayscale image using the color image generation network includes:

[0013] The color image generation network performs upsampling, regularization, and activation processing on the semantic features of the fused image to obtain the color image of the grayscale image.

[0014] In one embodiment, the image colorization method further includes: performing color enhancement processing on the color image to obtain a color-enhanced color image.

[0015] In one embodiment, performing color enhancement processing on the color image to obtain a color-enhanced color image includes:

[0016] The color image is subjected to downsampling and upsampling processes, and a cross-layer connection is made between the downsampling and upsampling processes to perform color enhancement processing on the color image, resulting in a color-enhanced color image.

[0017] In one embodiment, scaling the grayscale image to obtain multiple grayscale images of different scales includes:

[0018] A grayscale image with the original scale is used as a first-scale grayscale image; the first-scale grayscale image is downsampled to obtain a grayscale image with a different scale than the first-scale grayscale image.

[0019] In one embodiment, downsampling is performed on the grayscale image at the first scale to obtain a grayscale image at a different scale than the grayscale image at the first scale, including:

[0020] The first-scale grayscale image is downsampled to obtain a second-scale grayscale image, the scale of which is half the scale of the first-scale grayscale image; the second-scale grayscale image is downsampled to obtain a third-scale grayscale image, the scale of which is half the scale of the second-scale grayscale image.

[0021] According to a second aspect of the present disclosure, an image coloring apparatus is provided, comprising:

[0022] An acquisition unit is used to acquire a grayscale image to be colored; a processing unit is used to scale the grayscale image to obtain multiple grayscale images of different scales; extract the image semantic features of the multiple grayscale images of different scales respectively, fuse the image semantic features of the multiple grayscale images of different scales, and generate a color image based on the fused image semantic features.

[0023] In one embodiment, the processing unit extracts the image semantic features of the plurality of grayscale images at different scales respectively, fuses the image semantic features of the plurality of grayscale images at different scales, and generates a color image based on the fused image semantic features:

[0024] The multiple grayscale images at different scales are input into a multi-scale convolutional neural network, which extracts the semantic features of the multiple grayscale images at different scales. The semantic features of the multiple grayscale images at different scales are then fused. The fused semantic features are then input into a color image generation network, which generates a color image of the grayscale images.

[0025] In one embodiment, the multi-scale convolutional neural network extracts image semantic features from the plurality of grayscale images at different scales, including: performing residual processing and pooling processing on the plurality of grayscale images at different scales respectively by the multi-scale convolutional neural network to extract image semantic features from the plurality of grayscale images at different scales.

[0026] The step of generating a color image from the grayscale image by the color image generation network includes: performing upsampling, regularization, and activation processing on the fused image semantic features by the color image generation network to obtain the color image from the grayscale image.

[0027] In one embodiment, the processing unit is further configured to: perform color enhancement processing on the color image to obtain a color-enhanced color image.

[0028] In one embodiment, the processing unit performs color enhancement processing on the color image in the following manner to obtain a color-enhanced color image: performing downsampling and upsampling processing on the color image, and performing cross-layer connections between the downsampling and upsampling processing to perform color enhancement processing on the color image to obtain a color-enhanced color image.

[0029] In one embodiment, the processing unit scales the grayscale image as follows to obtain multiple grayscale images of different scales: the grayscale image with the original scale is used as the first scale grayscale image; the first scale grayscale image is downsampled to obtain a grayscale image with a different scale from the first scale grayscale image.

[0030] In one embodiment, the processing unit downsamples the grayscale image at the first scale in the following manner to obtain a grayscale image with a different scale from the grayscale image at the first scale:

[0031] The first-scale grayscale image is downsampled to obtain a second-scale grayscale image, the scale of which is half the scale of the first-scale grayscale image; the second-scale grayscale image is downsampled to obtain a third-scale grayscale image, the scale of which is half the scale of the second-scale grayscale image.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect or any embodiment of the first aspect.

[0033] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the method described in the first aspect or any embodiment of the first aspect.

[0034] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect or any embodiment of the first aspect.

[0035] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, multiple grayscale images of different scales are obtained by scaling the grayscale image. The local detail information reflected by the semantic information corresponding to the grayscale images of different scales is different. Therefore, the image semantic features of multiple grayscale images of different scales are extracted and fused. A color image is generated based on the fused image semantic features, which can reflect the different local detail information of the grayscale image. In this way, the color saturation of the generated color image can be improved, and the problems of grayness and color overflow in the color image obtained by colorizing the grayscale image can be improved.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0038] Figure 1 This is a flowchart illustrating an image coloring method according to an exemplary embodiment.

[0039] Figure 2 This is a flowchart illustrating an exemplary embodiment of a process for extracting semantic features from grayscale images at different scales and fusing image semantic features from multiple grayscale images at different scales to generate a color image.

[0040] Figure 3 This illustration shows a schematic diagram of a process for generating a color image based on grayscale images of different scales, as shown in an exemplary embodiment of this disclosure.

[0041] Figure 4 This illustration shows a schematic diagram of a process for generating a color image based on grayscale images of different scales, as shown in an exemplary embodiment of this disclosure.

[0042] Figure 5 This is a flowchart illustrating a method for image colorization using a two-stage processing approach, according to an exemplary embodiment.

[0043] Figure 6 A schematic diagram illustrating the above-described two-stage image coloring process shown in an exemplary embodiment of this disclosure is presented.

[0044] Figure 7 This illustration shows a schematic diagram of an image processing procedure for color enhancement via cross-layer connectivity in an exemplary embodiment of the present disclosure.

[0045] Figure 8This is a block diagram illustrating an image coloring apparatus according to an exemplary embodiment.

[0046] Figure 9 This is a block diagram of an electronic device for image coloring according to an exemplary embodiment. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0048] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this disclosure. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0049] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0050] With the development of image processing technology, image colorization technology has been applied in various fields. Image colorization technology can convert grayscale images into color images, and can also be understood as a technology that realizes the color style conversion of image display. For example, image colorization technology can be applied in fields such as colorizing old photos and video restoration.

[0051] Among related technologies, fully automatic image colorization based on deep learning is supported. This includes image colorization techniques based on Convolutional Neural Networks (CNNs) and those based on Generative Adversarial Networks (GANs). CNN-based and GAN-based image colorization techniques can convert grayscale images to color images without requiring additional semantic guidance from user interaction. However, because the neural network generating the color image in a fully automatic deep learning-based image colorization scheme processes the average hue of the input original grayscale image, the generated color image suffers from insufficient saturation, resulting in an overall grayish tone and widespread color bleeding.

[0052] In view of this, the present disclosure provides an image coloring method. In this image coloring method, semantic features extracted from grayscale images of different scales are fused to obtain more local detail information of the grayscale images. Then, image coloring is performed based on the fused semantic features, which can improve image saturation and improve the phenomenon of overall grayness and large-scale color overflow in the current image coloring results.

[0053] The image coloring method provided in this embodiment can be executed by a device with image processing capabilities, such as a terminal or a computer.

[0054] Figure 1 This is a flowchart illustrating an image coloring method according to an exemplary embodiment, such as... Figure 1 As shown, the image coloring method includes the following steps.

[0055] In step S11, a grayscale image to be colored is obtained.

[0056] The grayscale image to be colored can be understood as a single, single-channel grayscale image.

[0057] In step S12, the grayscale image is scaled to obtain multiple grayscale images of different scales.

[0058] Scaling a grayscale image can be understood as adjusting its size. The image information in a grayscale image shifts with scale adjustment, and different scales of grayscale images convey different semantic information. Larger image scales better reflect local semantic information, while smaller image scales better reflect global semantic information.

[0059] In step S13, image semantic features of multiple grayscale images at different scales are extracted, image semantic features of multiple grayscale images at different scales are fused, and a color image is generated based on the fused image semantic features.

[0060] In this embodiment of the disclosure, the ability of multi-scale algorithms to perceive image semantics can be utilized to extract image semantic features of grayscale images at multiple different scales, thereby focusing on image semantic information at different scales and fusing image semantic features of grayscale images at multiple different scales. This achieves the acquisition of image semantic information at different levels of grayscale images, thereby enabling the generated color image to reflect more detailed semantic information of the original grayscale image and improve the color saturation of the generated color image.

[0061] In one embodiment, the multiple grayscale images of different scales in this disclosure may include a grayscale image with the original scale and a grayscale image after scaling the original grayscale image.

[0062] For ease of description, the grayscale image at the original scale is referred to as the grayscale image at the first scale.

[0063] In this embodiment of the disclosure, a grayscale image of the first scale can be downsampled to obtain a grayscale image of a different scale from the grayscale image of the first scale.

[0064] In one embodiment, the grayscale image at the first scale is downsampled to obtain two grayscale images at different scales than the first scale grayscale image, referred to as the second scale grayscale image and the third scale grayscale image, respectively.

[0065] In one example, in this embodiment of the present disclosure, a first-scale grayscale image is downsampled to obtain a second-scale grayscale image, the scale of which is half the scale of the first-scale grayscale image. The second-scale grayscale image is then downsampled to obtain a third-scale grayscale image, the scale of which is half the scale of the second-scale grayscale image.

[0066] Based on the above examples, the grayscale images of different scales involved in the embodiments of this disclosure include a first-scale grayscale image, a second-scale grayscale image, and a third-scale grayscale image. In one example, the scale of the first-scale grayscale image is 256×256, the scale of the second-scale grayscale image is 128×128, and the scale of the third-scale grayscale image is 64×64.

[0067] In this embodiment of the disclosure, semantic features of the obtained grayscale images at different scales can be extracted separately, and the semantic features of the images at different scales can be fused to generate a color image.

[0068] Figure 2 This is a flowchart illustrating an exemplary embodiment of a process for extracting semantic features from grayscale images at different scales and fusing these features to generate a color image. Figure 2 As shown, it includes the following steps:

[0069] In step S21, multiple grayscale images at different scales are input into a multi-scale convolutional neural network, which extracts the semantic features of the multiple grayscale images at different scales.

[0070] In step S22, the image semantic features of multiple grayscale images at different scales are fused.

[0071] In step S23, the fused image semantic features are input into the color image generation network, which generates a color image of the grayscale image.

[0072] Figure 3This illustration shows a schematic diagram of a process for generating a color image based on grayscale images of different scales, as illustrated in an exemplary embodiment of this disclosure. (See also...) Figure 3 As shown, a first-scale grayscale image of size 256×256 is input to a CNN network that matches the first-scale grayscale image, resulting in the grayscale semantic features of the first-scale grayscale image of size 256×256. A second-scale grayscale image of size 128×128 is input to a CNN network that matches the second-scale grayscale image, resulting in the grayscale semantic features of the second-scale grayscale image of size 128×128. A third-scale grayscale image of size 64×64 is input to a CNN network that matches the third-scale grayscale image, resulting in the grayscale semantic features of the third-scale grayscale image of size 64×64. The grayscale semantic features of the first-scale grayscale image, the second-scale grayscale image, and the third-scale grayscale image are fused and then input into a CNN network (color image generation network) to obtain a three-channel color image.

[0073] In this embodiment of the disclosure, image semantic feature extraction, fusion, and color image generation are performed using a convolutional neural network, which enables fully automatic image coloring, reduces image coloring time, and is convenient and fast.

[0074] In this embodiment of the disclosure, to improve the color saturation of the generated color image, a multi-scale convolutional neural network can be used for downsampling to obtain grayscale image semantic features. These grayscale image semantic features are then fused to obtain a grayscale image semantic feature fusion result. The grayscale image semantic feature fusion result is then upsampled to generate a color image.

[0075] In one implementation, when a multi-scale convolutional neural network (CNN) extracts image semantic features from multiple grayscale images at different scales, the CNN includes a residual processing (Resblk) layer and a pooling processing (pooling) layer. The CNN performs residual processing and pooling processing on the multiple grayscale images at different scales respectively, to extract image semantic features from the grayscale images through downsampling. When a color image generation network generates a color image from a grayscale image, the network includes an upsampling layer, a regularization layer, and an activation function layer. The network performs upsampling, regularization, and activation processing on the fused image semantic features to obtain the color image corresponding to the grayscale image.

[0076] In this embodiment of the present disclosure, the method of extracting grayscale image semantic features through downsampling processing using a multi-scale convolutional neural network and a color image generation network, and fusing the grayscale image semantic features, and the network architecture for upsampling and fusing the grayscale image semantic features, can be found in [reference needed]. Figure 4 As shown. Figure 4 This illustration shows a schematic diagram of a process for generating a color image based on grayscale images of different scales, as illustrated in an exemplary embodiment of this disclosure. (See also...) Figure 4 As shown, the multi-scale convolutional neural network includes convolutional neural networks with three scales: 256×256, 128×128, and 64×64. In this embodiment, the generated color image is a three-channel color image; therefore, the grayscale semantic features of the grayscale image in each of the three channels are extracted. The first-scale grayscale image of 256×256×3 is input into the CNN network matching the first-scale grayscale image. After residual processing and pooling processing (image sizes are 128×128×16, 64×64×64, 32×32×128, and 16×16×256 respectively), the grayscale semantic features of the first-scale grayscale image of 256×256 are obtained. A CNN network matching a second-scale grayscale image (128×128) is input and processed with residuals and pooling (image sizes: 64×64×16, 32×32×64, 16×16×128, 16×16×256) to obtain the grayscale semantic features of the second-scale grayscale image (128×128). Similarly, a CNN network matching a third-scale grayscale image (64×64) is input and processed with residuals and pooling (image sizes: 32×32×16, 16×16×64, 16×16×128, 16×16×256) to obtain the grayscale semantic features of the third-scale grayscale image (64×64). The grayscale semantic features of the first-scale grayscale image, the second-scale grayscale image, and the third-scale grayscale image are fused to obtain a 16×16×768 image semantic feature fusion result. This fusion result is then input into a CNN network (color image generation network) that performs upsampling processing to obtain a three-channel color image. The image sizes involved in the upsampling process are 16×16×512, 32×32×256, 64×64×128, and 128×128×64, respectively.

[0077] In the embodiments of this disclosure Figure 3 and Figure 4 The image colorization method involved in the text can use the loss function in the CNN network, which can be the MSE loss function. The MSE loss function constrains the CNN network by calculating the difference between the true value and the output result.

[0078] In this embodiment of the present disclosure, the scheme of extracting semantic features of grayscale images at different scales and performing fusion processing based on multi-scale convolutional neural networks can achieve more detailed image colorization. However, there may be situations where the image colorization effect is still insufficient due to insufficient network performance. Therefore, in another embodiment of this disclosure, color enhancement processing can be performed on the color images obtained by the image colorization methods provided in the above embodiments to obtain color-enhanced color images.

[0079] Based on the above examples, the overall processing of the image colorization method provided in this embodiment can be understood as including two stages. The first stage obtains a coarse colorization result through a multi-scale convolutional neural network. The second stage further enhances the coarsely colorized image obtained in the first stage through a neural network, resulting in a vibrant overall image without color bleeding.

[0080] In this embodiment of the disclosure, the process of color enhancement processing for a color image can be performed based on any of the above embodiments. For ease of description, this embodiment of the disclosure will be used in... Figure 2 The following explanation uses the method shown as an example.

[0081] Figure 5 This is a flowchart illustrating a method for image colorization using a two-stage processing approach, according to an exemplary embodiment. Figure 5 The image coloring method shown is in Figure 2 In addition to the steps shown, the method also includes the following steps:

[0082] In step S24, the color image is subjected to color enhancement processing to obtain a color image with enhanced color.

[0083] in, Figure 6 A schematic diagram illustrating the two-stage image colorization process described above, as shown in an exemplary embodiment of this disclosure, is provided. (See also...) Figure 6 As shown, for Figure 3 The obtained color image is further input into a CNN network for color enhancement processing to obtain a color image with higher contrast and higher color saturation. In this embodiment, the second stage of color enhancement processing of the color image obtained in the first stage can be understood as a process of repairing the color image obtained in the first stage, so that the final output color image is further enhanced in visual effect, so as to obtain a color image with more vivid colors in visual effect.

[0084] In one embodiment, when performing color enhancement processing on the color image in the second stage to obtain a color-enhanced color image, the color image can be downsampled and upsampled, and a cross-layer connection can be made between the downsampling and upsampling processes to perform color enhancement processing on the color image to obtain a color-enhanced color image.

[0085] Figure 7 This illustration shows a schematic diagram of an image processing procedure for color enhancement via cross-layer connectivity in an exemplary embodiment of this disclosure. (See also...) Figure 7 As shown, downsampling of a color image can be performed using residual processing and pooling. In the residual processing, each ResBlk block consists of a 1×1 convolutional layer, a BatchNorm layer, a ReLU activation layer, a 3×3 convolutional layer, a BatchNorm layer, a ReLU activation layer, and another 1×1 convolutional layer, a BatchNorm layer, and a ReLU activation layer, connected between the input and output via cross-layer connections. The pooling process can be max pooling. The resulting 3×3 convolutional layer, BatchNorm layer, and ReLU activation layer after downsampling, followed by upsampling, yields a color-enhanced color image.

[0086] The image colorization method provided in this disclosure utilizes the semantic perception capability of a multi-scale convolutional neural network to extract and fuse semantic features of grayscale images at different scales to achieve more detailed image colorization. The multi-scale approach includes three scales with input image sizes of 256×256, 128×128, and 64×64, respectively. Larger scales are used to capture detailed image information, while smaller scales are used to capture overall semantic information. Furthermore, the image colorization process comprises two steps: a first step to obtain a coarser colorization result, and a second step to refine the result of the first step, further enhancing the final visual effects such as saturation, resulting in a vibrant and color-accurate image without color bleeding.

[0087] The image coloring method provided in this disclosure enables the obtained fully automatically colored image to have better color saturation, and the colored image does not have obvious color overflow. It also solves the problem of overall grayness in current image coloring algorithms.

[0088] Based on the same concept, embodiments of this disclosure also provide an image coloring apparatus.

[0089] It is understood that the image coloring apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0090] Figure 8 This is a block diagram illustrating an image coloring apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The image coloring device 100 includes an acquisition unit 101 and a processing unit 102.

[0091] The acquisition unit 101 is used to acquire a grayscale image to be colored. The processing unit 102 is used to scale the grayscale image to obtain multiple grayscale images of different scales. The semantic features of the multiple grayscale images of different scales are extracted respectively, the semantic features of the multiple grayscale images of different scales are fused, and a color image is generated based on the fused semantic features.

[0092] In one embodiment, the processing unit 102 extracts image semantic features from multiple grayscale images at different scales in the following manner, fuses the image semantic features of the multiple grayscale images at different scales, and generates a color image based on the fused image semantic features:

[0093] Multiple grayscale images at different scales are input into a multi-scale convolutional neural network (CNN), which extracts semantic features from these images. These semantic features are then fused. Finally, the fused semantic features are input into a color image generation network, which generates a color version of the grayscale images.

[0094] In one embodiment, the semantic features of multiple grayscale images at different scales are extracted by a multi-scale convolutional neural network, including: performing residual processing and pooling processing on multiple grayscale images at different scales by the multi-scale convolutional neural network to extract the semantic features of multiple grayscale images at different scales.

[0095] The process of generating a color image from a grayscale image using a color image generation network includes: upsampling, regularization, and activation processing of the fused image semantic features by the color image generation network to obtain the color image from the grayscale image.

[0096] In one embodiment, the processing unit 102 is further configured to: perform color enhancement processing on the color image to obtain a color-enhanced color image.

[0097] In one embodiment, the processing unit 102 performs color enhancement processing on the color image in the following manner to obtain a color-enhanced color image: performing downsampling processing and upsampling processing on the color image, and performing cross-layer connection between the downsampling processing and the upsampling processing to perform color enhancement processing on the color image to obtain a color-enhanced color image.

[0098] In one embodiment, the processing unit 102 scales the grayscale image as follows to obtain multiple grayscale images of different scales: The grayscale image with the original scale is used as the first-scale grayscale image. The first-scale grayscale image is then downsampled to obtain a grayscale image with a different scale from the first-scale grayscale image.

[0099] In one embodiment, the processing unit 102 downsamples the grayscale image at the first scale in the following manner to obtain a grayscale image at a different scale than the grayscale image at the first scale:

[0100] The first-scale grayscale image is downsampled to obtain the second-scale grayscale image, which is half the scale of the first-scale grayscale image. The second-scale grayscale image is then downsampled to obtain the third-scale grayscale image, which is half the scale of the second-scale grayscale image.

[0101] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] Figure 9 This is a block diagram of an electronic device 200 for image coloring, according to an exemplary embodiment.

[0103] like Figure 9As shown, one embodiment of this disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 stores instructions. The processor 202 executes the image coloring method of this disclosure embodiment by calling the instructions stored in the memory 201. The processor 202 is connected to both the memory 201 and the I / O interface 203, for example, via a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program for the image coloring method involved in the embodiments of this disclosure. The processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0104] In this embodiment of the disclosure, the processor 202 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 202 may be one or a combination of several of the following: central processing unit (CPU) or other processing units with data processing capability and / or instruction execution capability.

[0105] The memory 201 in this embodiment may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0106] In this embodiment of the disclosure, the I / O interface 203 can be used to receive input instructions (such as numeric or character information, and to generate key signal inputs related to user settings and function control of the electronic device 200), and can also output various information (such as images or sounds) to the outside. In this embodiment of the disclosure, the I / O interface 203 may include one or more of the following: a physical keyboard, function keys (such as volume control keys, power buttons, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0107] In some embodiments, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, perform any of the methods described above.

[0108] In some embodiments, this disclosure provides a computer program product comprising a computer program that, when executed by a processor, performs any of the methods described above.

[0109] Although the operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0110] The methods and apparatus disclosed herein can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement various method steps. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0111] Any step, operation, or procedure described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

[0112] The foregoing description of embodiments of this disclosure has been provided for purposes of illustration and description. The foregoing description is not exhaustive and is not intended to limit this disclosure to the exact form disclosed; various modifications and variations may be made in accordance with the foregoing teachings, or may be derived from practice of this disclosure. These embodiments were chosen and described to illustrate the principles of this disclosure and its practical application, enabling those skilled in the art to utilize this disclosure in various implementations and modifications suitable for the particular purpose conceived.

[0113] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0114] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0115] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0116] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0117] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0118] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following scope of claims.

[0119] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image coloring method, characterized in that, The image coloring method includes: Obtain the grayscale image to be colored; The grayscale image is scaled to obtain multiple grayscale images of different scales; The semantic features of the multiple grayscale images at different scales are extracted separately, the semantic features of the multiple grayscale images at different scales are fused, and a color image is generated based on the fused semantic features; The step of extracting image semantic features from the multiple grayscale images at different scales, fusing the image semantic features from the multiple grayscale images at different scales, and generating a color image based on the fused image semantic features includes: The multiple grayscale images at different scales are input into a multi-scale convolutional neural network, which then extracts the semantic features of the multiple grayscale images at different scales. The image semantic features of the multiple grayscale images at different scales are fused together; The fused image semantic features are input into a color image generation network, which then generates a color image of the grayscale image.

2. The image coloring method according to claim 1, characterized in that, The extraction of image semantic features from the multiple grayscale images at different scales by the multi-scale convolutional neural network includes: The multi-scale convolutional neural network performs residual processing and pooling processing on the multiple grayscale images at different scales to extract the image semantic features of the multiple grayscale images at different scales. The process of generating a color image from the grayscale image using the color image generation network includes: The color image generation network performs upsampling, regularization, and activation processing on the semantic features of the fused image to obtain the color image of the grayscale image.

3. The image coloring method according to claim 1 or 2, characterized in that, The image coloring method further includes: The color image is then subjected to color enhancement processing to obtain a color image with enhanced color.

4. The image coloring method according to claim 3, characterized in that, The step of performing color enhancement processing on the color image to obtain a color-enhanced color image includes: The color image is subjected to downsampling and upsampling processes, and a cross-layer connection is made between the downsampling and upsampling processes to perform color enhancement processing on the color image, resulting in a color-enhanced color image.

5. The image coloring method according to claim 1, characterized in that, The scaling of the grayscale image is adjusted to obtain multiple grayscale images of different scales, including: The grayscale image with the original scale is used as the grayscale image with the first scale. The grayscale image at the first scale is downsampled to obtain a grayscale image at a different scale than the grayscale image at the first scale.

6. The image coloring method according to claim 5, characterized in that, Downsampling the grayscale image at the first scale to obtain a grayscale image at a different scale than the grayscale image at the first scale includes: The first-scale grayscale image is downsampled to obtain a second-scale grayscale image, and the scale of the second-scale grayscale image is half the scale of the first-scale grayscale image. The second-scale grayscale image is downsampled to obtain a third-scale grayscale image, the scale of which is half the scale of the second-scale grayscale image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-6.

Images