Image coloring method, electronic device, storage medium and product

Abstract

The present disclosure relates to an image colorization method, electronic device, storage medium, and product. The image colorization method includes: obtaining a single-channel grayscale image; extracting image features from the single-channel grayscale image to obtain image features of the single-channel grayscale image; extracting semantic features from the single-channel grayscale image to obtain semantic features of the single-channel grayscale image; fusing the image features and the semantic features, and obtaining a multi-channel color image based on the fused features. The present disclosure can improve the quality of image colorization.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image coloring method, electronic device, storage medium, and product. Background Art

[0002] With the development of image processing technology, colorization technology that can convert grayscale images into color images has been applied in various fields, such as colorization of old photos and video restoration.

[0003] Related technologies support fully automatic image colorization using deep learning approaches, such as those utilizing convolutional neural networks (CNNs) and generative adversarial networks (GANs). CNNs often first understand the semantic information of an image and then perform targeted colorization based on this information. GANs, on the other hand, primarily learn the probability distribution of images in a high-dimensional data space.

[0004] However, methods based on convolutional neural networks generally employ supervised learning, which places high demands on data labeling and processing. Furthermore, training requires learning the color distribution of each pixel, often requiring higher computational resources and storage space. Methods based on generative adversarial networks often lack semantic perception networks, which limits the performance of image colorization models. Furthermore, due to the inherent limitations of generative adversarial networks, convergence is often difficult to achieve without a good balance between various factors during training.

[0005] Therefore, the image coloring method in the related art needs to be further improved. Summary of the Invention

[0006] To overcome the problems existing in the related art, the present disclosure provides an image coloring method, an electronic device, a storage medium and a product.

[0007] According to a first aspect of an embodiment of the present disclosure, there is provided an image coloring method, the image coloring method comprising:

[0008] Acquire a single-channel grayscale image; extract image features of the single-channel grayscale image to obtain image features of the single-channel grayscale image, and extract semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image; fuse the image features and the semantic features, and obtain a multi-channel color image based on the fused features.

[0009] In one embodiment, extracting the image features of the single-channel grayscale image to obtain the image features of the single-channel grayscale image, and extracting the semantic features of the single-channel grayscale image to obtain the semantic features of the single-channel grayscale image include:

[0010] Inputting the single-channel grayscale image to a first neural network model in a first model included in an image colorization model, and obtaining image features of the single-channel grayscale image through the first neural network model; inputting the single-channel grayscale image to a second neural network model in a second model included in the image colorization model, and obtaining semantic features of the single-channel grayscale image through the second neural network model;

[0011] The fusing of the image features and the semantic features and obtaining a multi-channel color image based on the fused features includes:

[0012] The feature fusion network included in the first model fuses the semantic features with the image features, and the integration network included in the first model obtains a multi-channel color image based on the fused features;

[0013] The image colorization model is pre-trained based on target loss.

[0014] In one embodiment, the image colorization model is pre-trained based on target loss in the following manner:

[0015] A single-channel grayscale training image is input into a first neural network model in a first model included in an image colorization model, and image features of the single-channel grayscale training image are obtained through the first neural network model. The single-channel grayscale training image is input into a second neural network model in a second model included in the image colorization model, and semantic features of the single-channel grayscale training image are obtained through the second neural network model. The semantic features of the single-channel grayscale training image are fused with the image features of the single-channel grayscale training image by a feature fusion network included in the first model, and a multi-channel color prediction image is obtained based on the fused features by an integration network included in the first model. A first target loss is determined based on the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image. The image colorization model is trained based on the first target loss until the image colorization model converges.

[0016] In one embodiment, the method further comprises:

[0017] Acquire a multi-channel color test image; input the multi-channel color test image and the multi-channel predicted image into a discriminator, and use the discriminator to identify the difference between the multi-channel color predicted image and the multi-channel color test image; determine a second target loss based on the class distribution vector, the multi-channel color predicted image and the multi-channel color test image; and train the image colorization model based on the second target loss and the difference until the image colorization model converges.

[0018] In one embodiment, determining the second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image includes:

[0019] Based on the multi-channel color prediction image and the multi-channel color test image, determine the color correction loss and the structural similarity loss for constraining the first model; based on the class distribution vector, determine the class distribution loss for constraining the second model; based on the multi-channel color prediction image and the multi-channel color test image, determine the adversarial loss for constraining the multi-channel color model and the discriminator; based on the color correction loss, the structural similarity loss, the class distribution loss and the adversarial loss, determine the second target loss.

[0020] In one embodiment, determining the color correction loss and the structural similarity loss for constraining the first model based on the multi-channel color prediction image and the multi-channel color test image includes:

[0021] For the pixels in the multi-channel color prediction image and the pixels in the multi-channel color test image, the absolute value of the pixel value difference of each corresponding pixel pair is determined respectively; if the absolute value of the pixel value difference is greater than the first parameter value, the difference between the absolute value of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model, and the first parameter value is a set multiple of the second parameter value; if the absolute value of the pixel value difference is less than the first parameter value, the product between the square of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model; determine the SSIM index of the multi-channel color, and determine the SSIM index of the multi-channel color test image; determine the norm of the difference between the SSIM index of the multi-channel color test image and the SSIM index of the multi-channel color prediction image as the structural similarity loss for constraining the first model.

[0022] In one embodiment, determining a class distribution loss for constraining the second model includes:

[0023] Determine the class distribution vector of the multi-channel color test image; and determine the norm of the difference between the class distribution vector of the single-channel grayscale image and the class distribution vector of the multi-channel color test image as the class distribution loss for constraining the second model.

[0024] According to a second aspect of an embodiment of the present disclosure, there is provided an image coloring device, comprising:

[0025] An acquisition unit is used to acquire a single-channel grayscale image; a processing unit is used to extract image features of the single-channel grayscale image to obtain image features of the single-channel grayscale image, and extract semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image; and fuse the image features and the semantic features, and obtain a multi-channel color image based on the fused features.

[0026] In one embodiment, the processing unit extracts image features of the single-channel grayscale image to obtain image features of the single-channel grayscale image, and extracts semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image in the following manner:

[0027] Inputting the single-channel grayscale image to a first neural network model in a first model included in an image colorization model, and obtaining image features of the single-channel grayscale image through the first neural network model; inputting the single-channel grayscale image to a second neural network model in a second model included in the image colorization model, and obtaining semantic features of the single-channel grayscale image through the second neural network model;

[0028] The fusing of the image features and the semantic features and obtaining a multi-channel color image based on the fused features includes:

[0029] The feature fusion network included in the first model fuses the semantic features with the image features, and the integration network included in the first model obtains a multi-channel color image based on the fused features;

[0030] The image colorization model is pre-trained based on target loss.

[0031] In one embodiment, the processing unit is further configured to pre-train the image colorization model based on the target loss in the following manner:

[0032] A single-channel grayscale training image is input into a first neural network model in a first model included in an image colorization model, and image features of the single-channel grayscale training image are obtained through the first neural network model. The single-channel grayscale training image is input into a second neural network model in a second model included in the image colorization model, and semantic features of the single-channel grayscale training image are obtained through the second neural network model. The semantic features of the single-channel grayscale training image are fused with the image features of the single-channel grayscale training image by a feature fusion network included in the first model, and a multi-channel color prediction image is obtained based on the fused features by an integration network included in the first model. A first target loss is determined based on the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image. The image colorization model is trained based on the first target loss until the image colorization model converges.

[0033] In one embodiment, the processing unit is further configured to:

[0034] Acquire a multi-channel color test image; input the multi-channel color test image and the multi-channel predicted image into a discriminator, and use the discriminator to identify the difference between the multi-channel color predicted image and the multi-channel color test image; determine a second target loss based on the class distribution vector, the multi-channel color predicted image and the multi-channel color test image; and train the image colorization model based on the second target loss and the difference until the image colorization model converges.

[0035] In one embodiment, the processing unit determines the second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image in the following manner:

[0036] Based on the multi-channel color prediction image and the multi-channel color test image, determine the color correction loss and the structural similarity loss for constraining the first model; based on the class distribution vector, determine the class distribution loss for constraining the second model; based on the multi-channel color prediction image and the multi-channel color test image, determine the adversarial loss for constraining the multi-channel color model and the discriminator; based on the color correction loss, the structural similarity loss, the class distribution loss and the adversarial loss, determine the second target loss.

[0037] In one embodiment, the processing unit determines the color correction loss and the structural similarity loss for constraining the first model based on the multi-channel color prediction image and the multi-channel color test image in the following manner:

[0038] For the pixels in the multi-channel color prediction image and the pixels in the multi-channel color test image, the absolute value of the pixel value difference of each corresponding pixel pair is determined respectively; if the absolute value of the pixel value difference is greater than the first parameter value, the difference between the absolute value of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model, and the first parameter value is a set multiple of the second parameter value; if the absolute value of the pixel value difference is less than the first parameter value, the product between the square of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model; determine the SSIM index of the multi-channel color, and determine the SSIM index of the multi-channel color test image; determine the norm of the difference between the SSIM index of the multi-channel color test image and the SSIM index of the multi-channel color prediction image as the structural similarity loss for constraining the first model.

[0039] In one embodiment, the processing unit determines the class distribution loss for constraining the second model in the following manner:

[0040] Determine the class distribution vector of the multi-channel color test image; and determine the norm of the difference between the class distribution vector of the single-channel grayscale image and the class distribution vector of the multi-channel color test image as the class distribution loss for constraining the second model.

[0041] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect or any one of the embodiments of the first aspect.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by a processor, the method described in the first aspect or any one of the embodiments of the first aspect is implemented.

[0043] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program is used to implement the method described in the first aspect or any one of the embodiments of the first aspect when executed by a processor.

[0044] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by fusing the semantic features of a single-channel grayscale image with the image features of the single-channel grayscale image, and obtaining a multi-channel color image based on the fused features, the multi-channel color image generation process is no longer based solely on image features, and the color distribution of each single pixel can be better learned in combination with semantic information, so that the generated multi-channel color image can be accurately colored.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0047] Figure 1 The figure is a flowchart of an image coloring method according to an exemplary embodiment.

[0048] Figure 2 The present invention is a flowchart of a method for coloring an image by fusing image features and semantic features according to an exemplary embodiment.

[0049] Figure 3 A flowchart of a method for training an image colorization model is shown in an exemplary embodiment of the present disclosure.

[0050] Figure 4 The figure is a flowchart of another method for training an image colorization model according to an exemplary embodiment.

[0051] Figure 5 The present invention is a flowchart of a method for determining target loss based on a class distribution vector of a single-channel grayscale image, a multi-channel color prediction image, and a multi-channel color test image according to an exemplary embodiment.

[0052] Figure 6 The present invention is a flowchart of a method for determining a color correction loss for constraining a first model based on a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment.

[0053] Figure 7 The present invention is a flowchart of a method for determining a structural similarity loss for constraining a first model based on a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment.

[0054] Figure 8 The present invention is a flowchart of a method for determining a class distribution loss for constraining a second model based on a class distribution vector of a single-channel grayscale image according to an exemplary embodiment.

[0055] Figure 9 The present invention is a flowchart showing a method for identifying the difference between a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment.

[0056] Figure 10The figure is a schematic diagram showing a process of coloring an image using a trained image coloring model according to an exemplary embodiment.

[0057] Figure 11 The figure is a block diagram of an image coloring device according to an exemplary embodiment.

[0058] Figure 12 The figure is a block diagram of an electronic device for image colorization model training or image colorization according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure.

[0060] In the accompanying drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be understood as limiting the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure. The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0061] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0062] With the development of image processing technology, colorization technology that can convert grayscale images into color images has been applied in various fields, such as colorization of old photos and video restoration.

[0063] Related technologies support fully automated image colorization using deep learning approaches, such as those utilizing convolutional neural networks (CNNs) and generative adversarial networks (GANs). CNNs often first understand the semantic information of an image and then perform targeted colorization based on this information. GANs, on the other hand, primarily learn the probability distribution of images in a high-dimensional data space.

[0064] However, methods based on convolutional neural networks generally employ supervised learning, which places high demands on data labeling and processing. Furthermore, training requires learning the color distribution of each pixel, often requiring higher computational resources and storage space. Methods based on generative adversarial networks often lack semantic perception networks, which limits the performance of image colorization models. Furthermore, due to the inherent limitations of generative adversarial networks, convergence is often difficult to achieve without a good balance between various factors during training.

[0065] In view of this, the present disclosure provides an image colorization method, in which the image features and semantic features of a single-channel grayscale image are fused, and a multi-channel color image is generated based on the fused features, so that the generated multi-channel color image can be accurately colored.

[0066] Figure 1 is a flow chart of an image coloring method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.

[0067] In step S11, a single-channel grayscale image is acquired.

[0068] In step S12, image features of the single-channel grayscale image are extracted to obtain image features of the single-channel grayscale image, and semantic features of the single-channel grayscale image are extracted to obtain semantic features of the single-channel grayscale image.

[0069] In step S13, the image features and the semantic features are fused, and a multi-channel color image is obtained based on the fused features.

[0070] The image colorization method provided by the embodiments of the present disclosure extracts image features and semantic features respectively for a single-channel grayscale image to be colored, fuses the image features and semantic features, and generates a multi-channel color image based on the fused features. This method can accurately understand the semantic information of the image and avoid incorrect coloring. In addition, the image colorization process based on image features is combined with semantic information to improve the quality of image coloring and the image coloring effect.

[0071] The image colorization method provided in the embodiment of the present disclosure can perform image colorization based on an image colorization model. The image colorization model includes a first model and a second model. The first model is used to realize the mapping of a single-channel grayscale image to a multi-channel color image, and the second model is used for semantic perception. In the embodiment of the present disclosure, the first model can be understood as an adversarial network for generating color images, and the second model can be understood as a semantic perception network. In the embodiment of the present disclosure, the advantages of the semantic perception network and the generative adversarial network are combined to realize the mapping between a single-channel grayscale image and a multi-channel color image.

[0072] In the disclosed embodiments, the extraction of image features and semantic features from a single-channel grayscale image can be performed based on different neural network models. For ease of description, the neural network model for extracting image features from a single-channel grayscale image is referred to as the first neural network model, and the neural network model for extracting semantic features from a single-channel grayscale image is referred to as the second neural network model. In other words, the first model includes the first neural network model, and the second model includes the second neural network model.

[0073] Among them, the first model in the embodiment of the present disclosure also includes a feature fusion network for fusing semantic features and image features, and a subsequent integration network for obtaining a multi-channel color image based on the fused features.

[0074] Figure 2 This is a flow chart showing a method for coloring an image by fusing image features and semantic features according to an exemplary embodiment. Figure 2 As shown, the process includes the following steps S21 to S22.

[0075] In step S21, the single-channel grayscale image is input into the first neural network model in the first model included in the image colorization model, and the image features of the single-channel grayscale image are obtained through the first neural network model, and the single-channel grayscale image is input into the second neural network model in the second model included in the image colorization model, and the semantic features of the single-channel grayscale image are obtained through the second neural network model.

[0076] Among them, the image colorization model is pre-trained based on the target loss.

[0077] In step S22, the feature fusion network included in the first model fuses the semantic features with the image features, and the integration network included in the first model obtains a multi-channel color image based on the fused features.

[0078] In an embodiment of the present disclosure, image features of a single-channel grayscale image are extracted through a first neural network model, and semantic features of the single-channel grayscale image are extracted through a second neural network model, wherein the first neural network model and the second neural network model can also be understood as different generators. Therefore, the present disclosure can realize the extraction of image features and semantic features based on different generators, and realize the coloring of images using multiple generators.

[0079] In one embodiment, the single-channel grayscale image in the embodiment of the present disclosure can be understood as the L channel image in the Lab color space. The multi-channel color image can be understood as a three-channel color image obtained by splicing the L channel image and the ab channel image. It can be understood that through the present disclosure, the user inputs a grayscale image L, and the mapping relationship L→(a, b) can be learned through the image colorization model, and multi-channel image splicing can be performed to achieve fully automatic coloring of a single image. Among them, the image in the Lab color space used in the embodiment of the present disclosure is more helpful in perceiving the semantic information and color information of the image than other methods, such as the RGB color space.

[0080] In the embodiment of the present disclosure, the image colorization model for realizing fully automatic colorization of a single grayscale image can be pre-trained based on target loss.

[0081] An embodiment of the present disclosure provides a method for training an image colorization model. Figure 3 A flowchart of a method for training an image coloring model in an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, the method for training the image colorization model includes the following steps S31 to S36.

[0082] In step S31, a single-channel grayscale training image is obtained.

[0083] The single-channel grayscale training image in the embodiment of the present disclosure can be understood as a training sample set composed of single-channel grayscale images.

[0084] In step S32, the single-channel grayscale training image is input into the first neural network model in the first model included in the image colorization model, and the image features of the single-channel grayscale training image are obtained through the first neural network model, and the single-channel grayscale training image is input into the second neural network model in the second model included in the image colorization model, and the semantic features of the single-channel grayscale training image are obtained through the second neural network model.

[0085] In step S33, the feature fusion network included in the first model fuses the semantic features of the single-channel grayscale training image with the image features of the single-channel grayscale training image.

[0086] In step S34, the integration network included in the first model obtains a multi-channel color prediction image based on the fused features.

[0087] For example, the fused single-channel grayscale training image can be mapped to a multi-channel color image using an image colorization model to obtain a two-channel mapped image. The two-channel image of the mapped image can be understood as the ab channel image in the Lab color space. The mapped image and the single-channel grayscale training image (the single-channel grayscale training image can be regarded as the L channel image in the Lab color space) are channel-joined to obtain a three-channel image, which is a multi-channel color prediction image in Lab format.

[0088] For example, a multi-channel color image in Lab format can be converted into a multi-channel color image in RGB format to facilitate subsequent target loss calculation.

[0089] In step S35 , a first target loss is determined based on the class distribution vector of the single-channel grayscale training image and the multi-channel color prediction image.

[0090] Among them, the class distribution vector of the single-channel grayscale training image can be obtained by outputting the semantic features of the single-channel grayscale image extracted by the second neural network model after passing through the fully connected layer.

[0091] In step S36, the image colorization model is trained based on the first target loss until the image colorization model converges.

[0092] The image colorization model training method provided by the disclosed embodiments can fuse the semantic features of a single-channel grayscale training image with the image features of a single-channel grayscale training image. Subsequently, multi-channel color image mapping is performed on the fused single-channel grayscale training image to obtain a multi-channel color prediction image. The target loss is then determined using the class distribution vector of the single-channel grayscale training image and the multi-channel color prediction image. Based on this target loss, the image colorization model is trained to obtain a converged image colorization model.

[0093] In the method for training an image colorization model provided by an embodiment of the present disclosure, the first model can be understood as an adversarial network for generating color images, and the second model can be understood as a semantic perception network. Therefore, the present disclosure can generate the first neural network model in the adversarial network model and the second neural network model in the semantic perception network model to extract features of the single-channel grayscale image, respectively, to obtain the image features of the single-channel grayscale image, the single-channel grayscale image semantic features and the class distribution vector of the single-channel grayscale image. Furthermore, the semantic features of the single-channel grayscale image can be fused with the image features of the single-channel grayscale image to obtain a fused single-channel grayscale image, and the fused single-channel grayscale image can be mapped to a multi-channel color image based on the image colorization model to obtain a multi-channel color prediction image. On this basis, the target loss can be determined by the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image, and the image colorization model can be trained with the target loss until the image colorization model converges. Since the image colorization model trained by the present disclosure uses a generative adversarial network and a semantic perception network for the image colorization model at the same time. Therefore, the disclosed embodiments can better combine semantic information to learn the color distribution of each single pixel, avoiding the problem of incorrect coloring caused by not introducing semantic information. In addition, because the disclosed embodiments determine the target loss through the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image, a clear loss can be obtained, and the training process can obtain a converged image colorization model, which can improve the effect of the image colorization model on the colorization of single-channel grayscale images.

[0094] Furthermore, in order to further identify the degree of real naturalness of the multi-channel color image generated by the image colorization model in the embodiment of the present disclosure, the embodiment of the present disclosure can be based on the model used to identify whether the multi-channel color prediction image generated by the first model is real and natural, and reverse training of the image colorization model can be performed to constrain the image colorization model.

[0095] In one embodiment, the present disclosure can use a real color image, hereinafter referred to as a multi-channel color test image, to identify whether the multi-channel color prediction image generated by the first model is realistic and natural. In one approach, a discriminator identifies the difference between the multi-channel color prediction image and the multi-channel color test image, and determines a second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image. Based on the second target loss and the difference between the multi-channel color prediction image and the multi-channel color test image, the image colorization model is trained until the image colorization model converges.

[0096] In the embodiment of the present disclosure, the image colorization model is trained based on the first model, the second model and the discriminator, which can improve the image colorization effect.

[0097] Figure 4 is a flowchart of another method for training an image coloring model according to an exemplary embodiment. Figure 4 As shown, it includes steps S41 to S48. Among them, the execution methods of steps S41, S42, S43, and S44 in the embodiment of the present disclosure are similar to those of steps S31, S32, S33, and S34, and are not described here in detail. Only the differences are described below.

[0098] In step S45, a multi-channel color test image of a single-channel grayscale image is obtained, and the multi-channel color test image and the multi-channel prediction image are input into a discriminator, through which the difference between the multi-channel color prediction image and the multi-channel color test image is identified.

[0099] The input of the discriminator is a multi-channel color test image and a multi-channel color prediction image, and the output is the difference between the multi-channel color test image and the multi-channel color prediction image.

[0100] In step S46 , a second target loss is determined based on the class distribution vector of the single-channel grayscale image, the multi-channel color prediction image, and the multi-channel color test image.

[0101] In step S47 , the image colorization model is trained based on the second target loss and the difference between the multi-channel color prediction image and the multi-channel color test image until the image colorization model converges.

[0102] Based on the difference between the multi-channel color prediction image and the multi-channel color test image, the image colorization model is reversely trained until the image colorization model converges.

[0103] For example, reverse training of the image colorization model can be understood as training the image colorization model by back propagation.

[0104] The discriminator can model the high-frequency information of the image to compensate for the high-frequency structure that was not captured. Similar to the feature extraction network in deep learning, the discriminator uses a convolutional neural network to map image data into a high-dimensional space and identify whether the input image is a true color image or a colorized image obtained by coloring using an image colorization model. This method focuses on how to achieve more universal data reconstruction capabilities through multiple iterations. The resulting image colorization model has superior colorization capabilities.

[0105] The method for training an image colorization model provided in the embodiments of the present disclosure determines a target loss using a class distribution vector of a single-channel grayscale image, a multi-channel color prediction image, and a multi-channel color test image. The target loss is then used to further train the image colorization model until the model converges. The resulting converged image colorization model can then be used to perform image colorization tasks on single-channel grayscale images. Given an input single-channel grayscale image, the model can output a multi-channel color image with high color reproduction.

[0106] For example, when determining the class distribution vector of a single-channel grayscale image, a multi-channel color prediction image, and a multi-channel color test image, the following can be used: Figure 5 The second target loss is determined by

[0107] Figure 5 is a flow chart of a method for determining target loss based on a class distribution vector of a single-channel grayscale image, a multi-channel color prediction image, and a multi-channel color test image according to an exemplary embodiment. Figure 5 As shown, the following steps are included.

[0108] In step S51a, based on the multi-channel color prediction image and the multi-channel color test image, a color correction loss and a structural similarity loss for constraining the first model are determined.

[0109] In step S51b, based on the class distribution vector of the single-channel grayscale image, a class distribution loss for constraining the second model is determined.

[0110] In step S51c, based on the multi-channel color prediction image and the multi-channel color test image, an adversarial loss for constraining the multi-channel color model and the discriminator is determined.

[0111] In step S52 , a second target loss is determined based on the color correction loss, the structural similarity loss, the class distribution loss, and the adversarial loss.

[0112] In one embodiment, the color correction loss, structural similarity loss, class distribution loss, and adversarial loss can be superimposed by a loss stacking method to obtain the target loss. For example, L(G1, G2, D) = L e (G1)+τ1L sm (G1)+τ2L g (G,D)+τ3L s (G2), determine the target loss. Among them, L(G1,G2,D) is the target loss, L e (G1) is the color correction loss, L sm (G1) is the structural similarity loss, Lg (G,D) is the adversarial loss, L s (G2) is the class distribution loss, G1 represents the first model, G2 represents the second model, D represents the discriminator, and τ1, τ2, and τ3 are weight coefficients. Specifically, τ1, τ2, and τ3 can be set to τ1 = 0.1, τ2 = 0.1, and τ3 = 0.0025, respectively.

[0113] For example, the color correction loss for constraining the first model may be determined as follows: For ease of description, the parameter values ​​used in determining the color correction loss of the first model are referred to as first parameter values ​​and second parameter values, respectively.

[0114] Figure 6 is a flow chart of a method for determining a color correction loss for constraining a first model based on a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment. Figure 6 As shown, the following steps are included.

[0115] In step S61 , for each pixel in the multi-channel color prediction image and each pixel in the multi-channel color test image, the absolute value of the pixel value difference between each corresponding pixel pair is determined.

[0116] In step S62a, if the absolute value of the pixel value difference is greater than the first parameter value, the difference between the absolute value of the pixel value difference and the second parameter value is determined as the color correction loss constraining the first model.

[0117] In step S62b, if the absolute value of the pixel value difference is less than the first parameter value, the product of the square of the pixel value difference and the second parameter value is determined as the color correction loss constraining the first model.

[0118] The first parameter value is a set multiple of the second parameter value.

[0119] For example, the first parameter value can be set to 1, and the second parameter value can be set to 0.5. Determine the color correction loss used to constrain the first model. e (G1) is the color correction loss, I GT is the pixel value of the corresponding pixel in the multi-channel color prediction image, I GENis the pixel value of the corresponding pixel in the multi-channel color test image. In addition, the above embodiment can be understood as determining the color correction loss for constraining the first model through Smooth L1 Loss. This method can enable the chromaticity value of the a-channel image and / or the b-channel image in each pixel to be learned through the L-channel image in the Lab color space. Compared with the conventional loss calculation method that has problems such as zero point non-differentiability and / or oscillation caused by excessive gradients at a distance from the origin, this method is universal. By training the image colorization model with the color correction loss obtained by this method, a converged image colorization model can be obtained.

[0120] For example, the structural similarity (SSIM) loss for constraining the first model may be determined as follows.

[0121] Figure 7 is a flow chart of a method for determining a structural similarity loss for constraining a first model based on a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment. Figure 7 As shown, the following steps are included.

[0122] In step S71 , the SSIM index of the multi-channel color prediction image is determined, and the SSIM index of the multi-channel color test image is determined.

[0123] In step S72, the norm of the difference between the SSIM index of the multi-channel color test image and the SSIM index of the multi-channel color prediction image is determined as the structural similarity loss for constraining the first model.

[0124] For example, you can use L sm (G1)=||SSIM(I GT )-SSIM(I GEN )||1, is determined as the structural similarity loss used to constrain the first model. sm (G1) is the structural similarity loss used to constrain the first model, SSIM(I GT ) is the SSIM index of the multi-channel color test image, SSIM(I GEN ) is the SSIM index of the multi-channel color prediction image.

[0125] In the disclosed embodiment, the SSIM indicator is used to evaluate the difference in SSIM between the colored color image and the original color image, and it is expected to minimize this difference. The structural similarity loss obtained by the SSIM indicator is used to train the image colorization model, so that the image colorization model can output a multi-channel color image that is consistent with the color information such as brightness and / or saturation of the original color image.

[0126] For example, the class distribution loss for constraining the second model can be determined as follows.

[0127] Figure 8 is a flow chart of a method for determining a class distribution loss for constraining a second model based on a class distribution vector of a single-channel grayscale image according to an exemplary embodiment. Figure 8 As shown, the following steps are included.

[0128] In step S81, a class distribution vector of a multi-channel color test image is determined based on .

[0129] In step S82, the norm of the difference between the class distribution vector of the single-channel grayscale image and the class distribution vector of the multi-channel color test image is determined as the class distribution loss for constraining the second model.

[0130] For example, you can use L s (G2)=||KL(y)-KL(y GT )||1 determines the class distribution loss used to constrain the second model. s (G2) is the class distribution loss used to constrain the second model, KL(y GT ) is the class distribution vector of the multi-channel color test image, and KL(y) is the class distribution vector of the single-channel grayscale image.

[0131] Because the class distribution loss is an asymmetric measure of the difference between two probability distributions, it can characterize the KL divergence (Kullback-Leibler divergence) between the class distribution vector of a single-channel grayscale image and the true distribution vector (i.e., the class distribution vector of a multi-channel color image). Therefore, the class distribution loss further enhances the interpretability of semantic information in the image colorization model, further improving the model's training effectiveness.

[0132] In one embodiment, the multi-channel color test image can be processed by using ResNet50 on mini-ImageNet to obtain a class distribution vector of the multi-channel color image.

[0133] In the disclosed embodiments, the discriminator can identify the difference between a multi-channel color prediction image and a multi-channel color test image, either locally or globally. The following describes the implementation process for identifying the difference between a multi-channel color prediction image and a multi-channel color test image, using local image sampling as an example.

[0134] Figure 9 FIG. 1 is a flow chart showing a method for identifying the difference between a multi-channel color prediction image and a multi-channel color test image according to an exemplary embodiment. Figure 9As shown, the following steps are included.

[0135] In step S91 , a multi-channel color test image of a single-channel grayscale image is acquired.

[0136] In step S92 , local sampling is performed on the multi-channel color prediction image and the multi-channel color test image respectively.

[0137] In step S93 , the locally sampled sampled image is input to a discriminator to discriminate the difference between the multi-channel color prediction image and the multi-channel color test image.

[0138] For example, for a multi-channel color test image and a multi-channel color prediction image, a square with a side length of one-eighth the original image size can be randomly selected from the image, and the area within this area can be used as a sampling sample to identify the difference between the multi-channel color prediction image and the multi-channel color test image. It will be understood that the sampled area is the same for the multi-channel color test image and the multi-channel color prediction image. Of course, the specific method for identifying the difference between the multi-channel color prediction image and the multi-channel color test image can be set according to actual needs and is not specifically limited in this disclosure.

[0139] The image colorization model obtained by the image colorization model training method provided by the embodiments of the present disclosure can be used to perform image colorization tasks. For example, a single-channel grayscale image is input into the image colorization model. The image colorization model is pre-trained using the image colorization model method described in any of the above embodiments. Based on the output of the image colorization model, a multi-channel color image is determined after the single-channel grayscale colorization.

[0140] Figure 10 The figure is a schematic diagram showing a process of coloring an image using a trained image coloring model according to an exemplary embodiment.

[0141] For example, Figure 10As shown, in the training process of the image colorization model, the first model, the second model and the discriminator participate together. Specifically, the single-channel grayscale image is input into the first neural network model of the first model and the second neural network model of the second model respectively. The first neural network model extracts the image features of the single-channel grayscale image. The second neural network model extracts the semantic features of the single-channel grayscale image. The image features of the single-channel grayscale image and the semantic features of the single-channel grayscale image are fused through a feature fusion network. The fused features are input into the integration network to obtain a multi-channel color prediction image. The multi-channel color prediction image can be converted into a color space to obtain a multi-channel color prediction image in RGB space. Based on the discriminator, the difference between the multi-channel color prediction image and the multi-channel color test image is identified. Specifically, the second neural network model extracts the semantic features of the single-channel grayscale image and outputs the class distribution vector of the single-channel grayscale image after passing through the fully connected layer. The class distribution vector can be used to constrain the target loss.

[0142] The obtained single-channel grayscale image may be an L-channel image obtained by converting the single-channel grayscale image into an L-channel image in a Lab color space, and adjusting the pixel size of the L-channel image to be smaller than a pixel threshold.

[0143] For example, the L-channel image can be input into the first model and the second model included in the image colorization model respectively. In this process, the first model can map the L-channel image to obtain an a-channel image and a b-channel image, and the second model can extract the semantic feature information and class distribution vector of the L-channel image, and fuse it with the output of the first model through convolution and upsampling to obtain a multi-channel color image of the single-channel grayscale image. Among them, the color image output by the image colorization model is a multi-channel color image in the Lab color space. Since in the embodiment of the present disclosure, the obtained multi-channel color image is a multi-channel color image in the Lab color space. Therefore, it is also necessary to perform color space conversion on the multi-channel color image in the Lab color space to obtain a color image in the RGB color space.

[0144] For example, a multi-channel color image in Lab color space can be converted into a multi-channel color image in XYZ color space, and then the multi-channel color image in XYZ color space can be converted into a multi-channel color image in RGB color space. The X channel image, Y channel image and Z channel image in the XYZ color space are obtained in this way. Furthermore, in the case of obtaining a multi-channel color image in an XYZ color space through an X-channel image, a Y-channel image, and a Z-channel image, In this way, the R channel image, G channel image and B channel image in the RGB color space are obtained.

[0145] The image colorization method provided by the embodiment of the present disclosure uses a generative adversarial network method with dual generators to obtain the final fully automatic colorized image. In the process of using the generative adversarial network to obtain the fully automatic colorized image, a semantic perception extraction network is introduced to obtain a more refined colorization result. Taking various factors into comprehensive consideration, a loss function that can converge and obtain better results is designed, thereby improving the image colorization effect.

[0146] Based on the same concept, an embodiment of the present disclosure also provides an image coloring device.

[0147] It is understandable that the image coloring device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0148] Figure 11 FIG. 1 is a block diagram of an image coloring device according to an exemplary embodiment. Figure 11 The device 100 includes an acquisition unit 101 and a processing unit 102.

[0149] The acquisition unit 101 is used to acquire a single-channel grayscale image; the processing unit 102 is used to extract image features of the single-channel grayscale image to obtain image features of the single-channel grayscale image, and extract semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image; fuse the image features and the semantic features, and obtain a multi-channel color image based on the fused features.

[0150] In one embodiment, the processing unit 102 extracts image features of a single-channel grayscale image to obtain image features of the single-channel grayscale image, and extracts semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image in the following manner:

[0151] Inputting the single-channel grayscale image to a first neural network model in a first model included in the image colorization model, and obtaining image features of the single-channel grayscale image through the first neural network model, and inputting the single-channel grayscale image to a second neural network model in a second model included in the image colorization model, and obtaining semantic features of the single-channel grayscale image through the second neural network model;

[0152] The processing unit 102 fuses the image features and the semantic features in the following manner, and obtains a multi-channel color image based on the fused features: a feature fusion network included in the first model fuses the semantic features with the image features, and an integration network included in the first model obtains the multi-channel color image based on the fused features;

[0153] The image colorization model is pre-trained based on the target loss.

[0154] In one embodiment, the processing unit 102 is further configured to pre-train the image colorization model based on the target loss in the following manner:

[0155] A single-channel grayscale training image is input into a first neural network model in a first model included in an image colorization model, and image features of the single-channel grayscale training image are obtained through the first neural network model. The single-channel grayscale training image is input into a second neural network model in a second model included in the image colorization model, and semantic features of the single-channel grayscale training image are obtained through the second neural network model. The semantic features of the single-channel grayscale training image are fused with the image features of the single-channel grayscale training image by the first model, and a multi-channel color prediction image is obtained based on the fused features. A first target loss is determined based on the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image. The image colorization model is trained based on the first target loss until the image colorization model converges.

[0156] In one embodiment, the processing unit 102 is further used to obtain a multi-channel color test image; input the multi-channel color test image and the multi-channel predicted image into a discriminator, and use the discriminator to identify the difference between the multi-channel color predicted image and the multi-channel color test image; determine a second target loss based on the class distribution vector, the multi-channel color predicted image and the multi-channel color test image; and train the image colorization model based on the second target loss and the difference until the image colorization model converges.

[0157] In one embodiment, the processing unit 102 determines the second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image in the following manner:

[0158] Based on the multi-channel color prediction image and the multi-channel color test image, the color correction loss and the structural similarity loss for constraining the first model are determined; based on the class distribution vector, the class distribution loss for constraining the second model is determined; based on the multi-channel color prediction image and the multi-channel color test image, the adversarial loss for constraining the multi-channel color model and the discriminator is determined; based on the color correction loss, the structural similarity loss, the class distribution loss and the adversarial loss, the second target loss is determined.

[0159] In one embodiment, the processing unit 102 determines the color correction loss and the structural similarity loss for constraining the first model based on the multi-channel color prediction image and the multi-channel color test image in the following manner:

[0160] For the pixels in the multi-channel color prediction image and the pixels in the multi-channel color test image, the absolute value of the pixel value difference of each corresponding pixel pair is determined respectively; if the absolute value of the pixel value difference is greater than the first parameter value, the difference between the absolute value of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model, and the first parameter value is a set multiple of the second parameter value; if the absolute value of the pixel value difference is less than the first parameter value, the product between the square of the pixel value difference and the second parameter value is determined as the color correction loss for constraining the first model; the SSIM index of the multi-channel color is determined, and the SSIM index of the multi-channel color test image is determined; the norm of the difference between the SSIM index of the multi-channel color test image and the SSIM index of the multi-channel color prediction image is determined as the structural similarity loss for constraining the first model.

[0161] In one embodiment, the processing unit 102 determines the class distribution loss for constraining the second model in the following manner:

[0162] Determine the class distribution vector of the multi-channel color test image; and determine the norm of the difference between the class distribution vector of the single-channel grayscale image and the class distribution vector of the multi-channel color test image as the class distribution loss for constraining the second model.

[0163] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0164] Figure 12 FIG2 is a block diagram of an electronic device 200 for image colorization model training or image colorization according to an exemplary embodiment.

[0165] like Figure 12As shown, an embodiment of the present disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the image coloring model training method or image coloring method of the embodiment of the present disclosure. The processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the programs of the image coloring model training method or image coloring method involved in the embodiment of the present disclosure, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the programs stored in the memory 201.

[0166] In the embodiment of the present disclosure, the processor 202 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 202 can be a central processing unit (CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0167] The memory 201 in the embodiment of the present disclosure may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0168] In the embodiment of the present disclosure, the I / O interface 203 may be used to receive input commands (e.g., digital or character information, and to generate key signal input related to user settings and function control of the electronic device 200), and may also output various information (e.g., images or sounds) to the outside. In the embodiment of the present disclosure, the I / O interface 203 may include one or more of a physical keyboard, function keys (e.g., volume control keys, power keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0169] In some embodiments, the present disclosure provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, any of the methods described above is performed.

[0170] In some embodiments, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, any of the methods described above is performed.

[0171] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0172] The methods and apparatus of the present disclosure can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement the various method steps. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0173] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations or procedures described.

[0174] The foregoing description of embodiments of the present disclosure has been provided for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the disclosure to the precise form disclosed, and various variations and modifications are possible in light of the foregoing teachings or from practice of the disclosure. These embodiments have been chosen and described to illustrate the principles of the disclosure and its practical application, so as to enable those skilled in the art to utilize the disclosure in various embodiments and with various modifications as appropriate for the particular use contemplated.

[0175] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0176] It is understood that in this disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of related objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0177] It will be further understood that the terms "first," "second," and the like are used to describe various types of information, but such information should not be limited to these terms. These terms are used solely to distinguish information of the same type from one another and do not indicate a particular order or level of importance. In fact, the terms "first," "second," and the like are fully interchangeable. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of this disclosure.

[0178] It is further understood that, unless otherwise specified, “connection” includes a direct connection where there are no other components between the two elements, and also includes an indirect connection where there are other elements between the two elements.

[0179] It is further understood that although operations are described in a particular order in the drawings in the embodiments of the present disclosure, this should not be construed as requiring that the operations be performed in the particular order shown or in a serial order, or that all of the operations shown be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0180] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0181] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image coloring method, characterized in that: The image coloring method comprises: Get a single-channel grayscale image; Extracting image features of the single-channel grayscale image to obtain image features of the single-channel grayscale image, and extracting semantic features of the single-channel grayscale image to obtain semantic features of the single-channel grayscale image; fusing the image features and the semantic features, and obtaining a multi-channel color image based on the fused features; The extracting the image features of the single-channel grayscale image to obtain the image features of the single-channel grayscale image, and extracting the semantic features of the single-channel grayscale image to obtain the semantic features of the single-channel grayscale image, includes: Inputting the single-channel grayscale image to a first neural network model in a first model included in an image colorization model, and obtaining image features of the single-channel grayscale image through the first neural network model; inputting the single-channel grayscale image to a second neural network model in a second model included in the image colorization model, and obtaining semantic features of the single-channel grayscale image through the second neural network model; The image colorization model is pre-trained based on the target loss in the following way: Inputting a single-channel grayscale training image into a first neural network model in a first model included in an image colorization model, and obtaining image features of the single-channel grayscale training image through the first neural network model; inputting the single-channel grayscale training image into a second neural network model in a second model included in the image colorization model, and obtaining semantic features of the single-channel grayscale training image through the second neural network model; The feature fusion network included in the first model fuses the semantic features of the single-channel grayscale training image with the image features of the single-channel grayscale training image, and the integration network included in the first model obtains a multi-channel color prediction image based on the fused features; Determining a first target loss based on the class distribution vector of the single-channel grayscale image and the multi-channel color prediction image; The image colorization model is trained based on the first target loss until the image colorization model converges.

2. The method according to claim 1, characterized in that The fusing of the image features and the semantic features and obtaining a multi-channel color image based on the fused features includes: The feature fusion network included in the first model fuses the semantic features with the image features, and the integration network included in the first model obtains a multi-channel color image based on the fused features.

3. The method according to claim 1, characterized in that The method further comprises: Acquire a multi-channel color test image; Inputting the multi-channel color test image and the multi-channel color prediction image into a discriminator, and using the discriminator to discriminate the difference between the multi-channel color prediction image and the multi-channel color test image; determining a second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image; The image colorization model is trained based on the second target loss and the difference until the image colorization model converges.

4. The method according to claim 3, characterized in that The determining of a second target loss based on the class distribution vector, the multi-channel color prediction image, and the multi-channel color test image includes: Determining, based on the multi-channel color prediction image and the multi-channel color test image, a color correction loss and a structural similarity loss for constraining the first model; Determining a class distribution loss for constraining the second model based on the class distribution vector; Determining an adversarial loss for constraining the image colorization model and the discriminator based on the multi-channel color prediction image and the multi-channel color test image; A second target loss is determined based on the color correction loss, the structural similarity loss, the class distribution loss, and the adversarial loss.

5. The method according to claim 4, characterized in that The determining, based on the multi-channel color prediction image and the multi-channel color test image, a color correction loss and a structural similarity loss for constraining the first model comprises: For each pixel in the multi-channel color prediction image and each pixel in the multi-channel color test image, determining an absolute value of a pixel value difference between each corresponding pixel pair; If the absolute value of the pixel value difference is greater than a first parameter value, determining the difference between the absolute value of the pixel value difference and a second parameter value as the color correction loss constraining the first model, and the first parameter value is a set multiple of the second parameter value; If the absolute value of the pixel value difference is less than the first parameter value, multiplying the square of the pixel value difference by the second parameter value to determine the color correction loss constraining the first model; Determining the SSIM index of the multi-channel color and determining the SSIM index of the multi-channel color test image; The norm of the difference between the SSIM index of the multi-channel color test image and the SSIM index of the multi-channel color prediction image is determined as the structural similarity loss for constraining the first model.

6. The method according to claim 4, characterized in that The determining, based on the class distribution vector, a class distribution loss for constraining the second model comprises: Determining a class distribution vector of the multi-channel color test image; The norm of the difference between the class distribution vector of the single-channel grayscale image and the class distribution vector of the multi-channel color test image is determined as the class distribution loss for constraining the second model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images