Image color processing method, processing model training method, device, and electronic equipment

Abstract

The present disclosure provides an image color processing method, a training method, a device, an electronic device and a medium for an image color processing model, and relates to the fields of image processing and artificial intelligence, and in particular to the fields of deep learning and computer vision. The implementation scheme is: obtaining a grayscale image to be color processed and a reference image; obtaining an intermediate image by superimposing the grayscale image with noise; obtaining a color description vector based on the reference image; and obtaining a colored output image with noise removed from the intermediate image through an image generation network based on the color description vector and the intermediate image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing and artificial intelligence technology, in particular to deep learning and computer vision, and specifically to an image color processing method, a training method for an image color processing model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides an image color processing method, a training method for an image color processing model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, there is provided an image color processing method, comprising: obtaining a grayscale image to be color processed and a reference image; obtaining an intermediate image by superimposing the grayscale image with noise; obtaining a color description vector based on the reference image; and obtaining a colored output image in which noise in the intermediate image is removed through an image generation network based on the color description vector and the intermediate image.

[0006] According to another aspect of the present disclosure, a training method for an image color processing model is provided, wherein the image color processing model includes a noise superposition network, a color description network and an image generation network, and the method includes: obtaining a color image and a grayscale image corresponding to the color image; based on the grayscale image, obtaining an intermediate image generated by superimposing the grayscale image and noise through the noise superposition network; based on a reference image, obtaining a color description vector through the color description network; based on the color description vector and the intermediate image, obtaining a colored output image in which noise in the intermediate image is removed through the image generation network; calculating a loss based on the output image and the color image; and adjusting parameters of at least a portion of the image color processing model based on the calculated loss.

[0007] According to another aspect of the present disclosure, there is provided an image color processing device, comprising: an obtaining unit for obtaining a grayscale image to be color processed and a reference image; a superimposing unit for obtaining an intermediate image by superimposing the grayscale image with noise; a reference unit for obtaining a color description vector based on the reference image; and a coloring unit for obtaining, through an image generation network, a colored output image in which noise in the intermediate image is removed based on the color description vector and the intermediate image.

[0008] According to another aspect of the present disclosure, a training device for an image color processing model is provided, wherein the image color processing model includes a noise superposition network, a color description network and an image generation network, and the device includes: an acquisition unit, for obtaining a color image and a grayscale image corresponding to the color image; a superposition unit, for obtaining, based on the grayscale image, an intermediate image generated by superimposing the grayscale image with noise through the noise superposition network; a reference unit, for obtaining, based on the reference image, a color description vector through the color description network; a coloring unit, for obtaining, based on the color description vector and the intermediate image, an output image with noise removed from the intermediate image through the image generation network; a loss unit, for calculating the loss based on the output image and the color image; and an adjustment unit, for adjusting the parameters of at least a portion of the image color processing model based on the calculated loss.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image color processing method or the training method for the image color processing model according to one or more embodiments of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute an image color processing method or a training method for an image color processing model according to one or more embodiments of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements an image color processing method or a training method for an image color processing model according to one or more embodiments of the present disclosure.

[0012] According to one or more embodiments of the present disclosure, a grayscale image may be effectively colored or a model for coloring an image may be effectively obtained.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A flowchart of an image color processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart of a method for training an image color processing model according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A data flow diagram of image color processing according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of an image color processing device according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A structural block diagram of a training device for an image color processing model according to an embodiment of the present disclosure is shown;

[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0024] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0027] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the image color processing method or the training method for an image color processing model according to the present disclosure.

[0028] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0029] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0030] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to view images, process images, perform image color processing, or train image color processing models, etc. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0031] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0032] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0033] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0034] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0036] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0037] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0038] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0039] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0040] Reference below Figure 2 An image color processing method 200 according to an exemplary embodiment of the present disclosure is described.

[0041] In step S201, a grayscale image to be color processed and a reference image are obtained.

[0042] In step S202, an intermediate image is obtained by superimposing the grayscale image and the noise.

[0043] In step S203, a color description vector is obtained based on the reference image.

[0044] At step S204, based on the color description vector and the intermediate image, a colored output image with noise removed from the intermediate image is obtained through an image generation network.

[0045] According to the method described in the embodiment of the present disclosure, the grayscale image can be effectively colored. According to the embodiment of the present disclosure, by superimposing noise and then removing the noise, and taking the color to be added as the description vector and inputting it in the noise removal stage, the image can be effectively colored.

[0046] The process of superimposing noise and removing noise can utilize a diffusion model, which can adopt a structure or a variation as known to those skilled in the art, and will not be described in detail herein. Exemplarily, the intermediate image and the color description vector can be input into a denoising network of the diffusion model as an image generation network to obtain an output image with the noise in the intermediate image removed.

[0047] According to some embodiments, obtaining an output image with noise removed from the intermediate image based on the color description vector and the intermediate image may include: obtaining an output image with noise removed from the intermediate image based on the color description vector, the intermediate image and the grayscale image.

[0048] In such an example, the grayscale image can be input again in the denoising process so that the output result can be more consistent in structure with the original grayscale image.

[0049] According to some embodiments, based on the color description vector, the intermediate image and the grayscale image, obtaining an output image with noise removed from the intermediate image may include: scaling the grayscale image to obtain a scaled grayscale image; splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and inputting the spliced ​​intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

[0050] In such an example, the grayscale image can be reduced and then spliced ​​with the noise image (for example, adding a channel for splicing), thereby enhancing the characteristics of the original image and ensuring that the output result is consistent with the original result.

[0051] According to some embodiments, obtaining a color description vector based on the reference image may include: obtaining an embedding vector of the reference image based on the reference image; and obtaining the color description vector based on the embedding vector of the reference image so that a dimension of the color description vector is the same as a dimension of a description vector required by the image generation network.

[0052] In such an example, the reference image may be converted into a description vector and mapped to a corresponding dimension so that a color feature corresponding to the reference image can be generated.

[0053] In some embodiments, the dimension of the description vector required by the image generation network may be the dimension of the text description vector in the diffusion model.

[0054] Reference below Figure 3 A training method 300 for an image color processing model according to an exemplary embodiment of the present disclosure is described. The image color processing model may include a noise superposition network, a color description network, and an image generation network.

[0055] In step S301, a color image and a grayscale image corresponding to the color image are obtained.

[0056] At step S302, based on the grayscale image, an intermediate image is obtained by superimposing the grayscale image and noise through the noise superposition network.

[0057] In step S303, a color description vector is obtained through the color description network based on the reference image.

[0058] At step S304, based on the color description vector and the intermediate image, a colored output image is obtained by the image generation network, in which noise in the intermediate image is removed.

[0059] At step S305, a loss is calculated based on the output image and the color image.

[0060] At step S306 , based on the calculated loss, parameters of at least a portion of the image color processing model are adjusted.

[0061] According to the method described in the embodiment of the present disclosure, a model for coloring an image can be effectively obtained.

[0062] It is understood that the reference image can be an image associated with the color image, such as an image with which a color is associated, whereby the model can learn to learn color description information from the reference image and colorize the grayscale image. Exemplarily, the method can include obtaining a color image, a grayscale image, and a reference image associated with the color image.

[0063] Exemplarily, the color of the output image may correspond to the color description vector, for example, be consistent with the color description vector. As can be understood by those skilled in the art, the denoising network of the diffusion model may denoise the input image with superimposed noise based on the description vector, so that the output image is denoised and consistent with the description of the description vector. Exemplarily, the color description vector according to an embodiment of the present disclosure may play the role of the description vector in the denoising network of the original diffusion model.

[0064] According to some embodiments, based on the color description vector and the intermediate image, obtaining, through the image generation network, an output image in which the noise in the intermediate image is removed may include: based on the color description vector, the intermediate image and the grayscale image, obtaining, through the image generation network, an output image in which the noise in the intermediate image is removed.

[0065] In such an example, the grayscale image can be input again in the denoising process so that the output result can be more consistent in structure with the original grayscale image.

[0066] According to some embodiments, based on the color description vector, the intermediate image and the grayscale image, obtaining, through the image generation network, an output image with noise removed from the intermediate image may include: scaling the grayscale image to obtain a scaled grayscale image; splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and inputting the spliced ​​intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

[0067] In such an example, the grayscale image can be reduced and then concatenated with the noise image (eg, adding a channel) to ensure that the output result is consistent with the original result.

[0068] According to some embodiments, obtaining a color image and a grayscale image corresponding to the color image may include: obtaining a color image; and performing a grayscale operation on the color image to obtain the grayscale image corresponding to the color image.

[0069] In such an example, the color image (output true value) and the grayscale image (input) can be the same image and its grayscale result, so that the colorization effect of the grayscale image can be guaranteed.

[0070] According to some embodiments, the color description network may include an encoding network and a mapping network. Based on a reference image, obtaining a color description vector through the color description network may include: obtaining an embedding vector of the reference image through the encoding network based on the reference image; and obtaining the color description vector through the mapping network based on the embedding vector of the reference image, so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

[0071] In such an example, the reference image may be converted into a description vector and mapped to a corresponding dimension so that a color feature corresponding to the reference image can be generated.

[0072] According to some embodiments, adjusting parameters of at least a portion of the image color processing model may include adjusting parameters of the color description network.

[0073] In such an example, the color description network (or mapping network) may include one or more fully connected layers, and the training process may include adjusting parameters of the fully connected layers.

[0074] According to some embodiments, adjusting parameters of at least a portion of the image color processing model may include adjusting parameters of the image generation network.

[0075] In such an example, for example, only the first layer of the image generation network (denoising model) can be adjusted. For example, a fully trained diffusion model can be used, and only the first layer of the denoising part can be adjusted, thereby reducing the time used for training and improving the convergence speed.

[0076] According to some embodiments, the color image and the reference image may be associated video frames from the same video.

[0077] In such an example, associated video frames may be used as constraints, so as to facilitate the network to learn the association between features. For example, the associated video frames may be adjacent or non-adjacent video frames in the video but with similar scenes, pictures, and colors.

[0078] According to some embodiments, the noise superposition network may be based on a forward denoising network of a pre-trained diffusion model, and the image generation network may be based on a backward denoising network of the pre-trained diffusion model.

[0079] According to one or more embodiments of the present disclosure, a method for image color processing based on a diffusion model is proposed. For example, after a user uploads a grayscale image to be processed, the model can be used as a reference by inputting a reference image of the desired color scheme, and finally a high-quality color result can be obtained. The method of one or more embodiments of the present disclosure can be applied to various scenarios that require image color processing, and users can design modifications to grayscale images according to their needs to obtain corresponding high-quality color generation results. Exemplarily, a user can generate different coloring results corresponding to the same grayscale image by inputting different reference images, thereby realizing user-interactive image color processing.

[0080] Reference below Figure 4 The data flow 400 according to a specific non-limiting embodiment of the present disclosure is described. As described in the data flow 400, a grayscale image with a size of W*H can be input to the encoder 401, and then passed through the diffusion model forward noise adding part 402 to obtain a noise image, and the noise image is combined with the grayscale image scaled to W / 8*H / 8. Figure 1 The vectorized representation of the reference image with color is obtained through the image encoding part 404 and the coding mapping part 405, and is also input to the diffusion model forward denoising part 403. After that, through the decoder 406, the finally generated color image can be obtained.

[0081] Exemplarily, a diffusion model that has been pre-trained with a large amount of data can be used as a base. As a specific example, the text used in the original diffusion model as cross-attention to direct image generation can be improved. Exemplarily, a reference image can be used to extract features through an image encoder, and then the image code can be mapped to the text code space of the original model through a coding mapping layer to participate in the final result generation of the model. In addition, according to an exemplary embodiment of the present disclosure, in the denoising stage of the diffusion model, the grayscale or black and white input can be scaled to the feature map size and input into the denoising model together, so that the model can refer to the grayscale input image during denoising, and finally achieve coloring of the grayscale input image.

[0082] The encoding mapping part 405 may also be referred to as an encoding mapping layer or an image feature mapping layer, which may correspond to the color depiction network described in the embodiments of the present disclosure, which may be (for example, two) fully connected layers, which may be used to map the extracted image features to the dimensions of the text encoding in the original diffusion model. Exemplarily, a pre-trained model based on training with a large amount of data may be used, and only the parameters of the newly added image feature mapping layer and the first layer of the denoising network may be retrained, thereby greatly reducing the training workload, and utilizing the characteristics of the pre-trained diffusion model that can restore the image to obtain a more accurate coloring effect. For example, a pre-trained diffusion model that has been trained and can output the original image may be directly used with a small amount of adjustment.

[0083] For example, during training, two frames of images at different times in the same video can be extracted as input images and reference images, which can be adjacent frames or not, but image frames of the same picture are selected as much as possible to obtain similar colors. When applied, the reference image can be input by the user, for example, the user selects the desired style as the reference image.

[0084] As a more specific non-limiting example, a public video dataset can be used in the training phase. For example, two frames of images at different times in the same video can be extracted as input images and reference images. The input image can be black and white or grayscaled, and the grayscale image of the input image is input into the network as a grayscale image input, and its original color version is used as the final generation target or true value of the network. During training, the network can learn to add corresponding colors to the input grayscale image through the color information of the reference image. After the model training is completed, when testing, you only need to input a grayscale image and then input a color image as a reference image to get the coloring result of the input image generated according to the reference image.

[0085] According to one or more embodiments of the present disclosure, the user only needs to input the grayscale image to be colored and the color image to be referenced when using the system. After the network processing is completed, the coloring result according to the reference color image can be obtained.

[0086] As a more specific non-limiting example, the self-attention module in the diffusion model can be changed to an inter-frame attention module, and the grayscale video can be generated into a stable color video based on the reference color frame without modifying the model parameters. In such an example, the video can be colored. For example, black-and-white movies and documentaries from the last century can be turned into color to help people better recall the scene at that time. For videos, if a single frame is processed, there may be a gap between consecutive frames, causing the picture to flash all the time. Therefore, the original self-attention module in the diffusion model can be changed to an inter-frame attention module.

[0087] Understandably, reference Figure 4 The data flows described can be applied to reference Figure 2 and Figure 3 The described application-side method, training-side method and related variations, and their advantages are not repeated here.

[0088] Reference now Figure 5 An image color processing apparatus 500 according to an embodiment of the present disclosure is described. The image color processing apparatus 500 may include an obtaining unit 501 , a superimposing unit 502 , a reference unit 503 , and a coloring unit 504 .

[0089] The obtaining unit 501 may be used to obtain a grayscale image to be color processed and a reference image. The superposition unit 502 may be used to obtain an intermediate image by superimposing the grayscale image with noise. The reference unit 503 may be used to obtain a color description vector based on the reference image. The coloring unit 504 may be used to obtain a colored output image in which noise in the intermediate image is removed through an image generation network based on the color description vector and the intermediate image.

[0090] According to the device described in the embodiment of the present disclosure, grayscale images can be effectively colored.

[0091] According to some embodiments, the colorization unit may include: a unit for obtaining an output image with noise removed from the intermediate image based on the color description vector, the intermediate image and the grayscale image.

[0092] According to some embodiments, the unit for obtaining an output image with noise removed from the intermediate image based on the color description vector, the intermediate image and the grayscale image may include: a unit for scaling the grayscale image to obtain a scaled grayscale image; a unit for splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and a unit for inputting the spliced ​​intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

[0093] According to some embodiments, the reference unit may include: a unit for obtaining an embedding vector of the reference image based on the reference image; and a unit for obtaining the color description vector based on the embedding vector of the reference image so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

[0094] Reference now Figure 6A training device 600 for an image color processing model according to an embodiment of the present disclosure is described. The image color processing model may include a noise superposition network, a color description network, and an image generation network. The training device 600 for an image color processing model may include an acquisition unit 601, a superposition unit 602, a reference unit 603, a coloring unit 604, a loss unit 605, and an adjustment unit 606.

[0095] The obtaining unit 601 can be used to obtain a color image and a grayscale image corresponding to the color image. The superposition unit 602 can be used to obtain an intermediate image generated by superimposing the grayscale image and noise based on the grayscale image through the noise superposition network. The reference unit 603 can be used to obtain a color description vector based on the reference image through the color description network. The coloring unit 604 can be used to obtain an output image in which noise in the intermediate image is removed through the image generation network based on the color description vector and the intermediate image. The loss unit 605 can be used to calculate the loss based on the output image and the color image. The adjustment unit 606 can be used to adjust the parameters of at least a part of the image color processing model based on the calculated loss.

[0096] According to the device described in the embodiment of the present disclosure, a model for coloring an image can be effectively obtained.

[0097] According to some embodiments, the coloring unit may include: a unit for obtaining, based on the color description vector, the intermediate image and the grayscale image, through the image generation network, an output image in which noise in the intermediate image is removed.

[0098] According to some embodiments, the unit for obtaining, through the image generation network, an output image with noise removed from the intermediate image based on the color description vector, the intermediate image and the grayscale image may include: a unit for scaling the grayscale image to obtain a scaled grayscale image; a unit for splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and a unit for inputting the spliced ​​intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

[0099] According to some embodiments, the obtaining unit may include: a unit for obtaining a color image; and a unit for performing a grayscale operation on the color image to obtain the grayscale image corresponding to the color image.

[0100] According to some embodiments, the reference unit may include: a unit for obtaining an embedding vector of the reference image through the encoding network based on the reference image; and a unit for obtaining the color description vector through the mapping network based on the embedding vector of the reference image so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

[0101] According to some embodiments, the adjusting unit may include a unit for adjusting parameters of the color description network.

[0102] According to some embodiments, the adjusting unit may comprise a unit for adjusting parameters of the image generation network.

[0103] According to some embodiments, the color image and the reference image may be associated video frames from the same video.

[0104] According to some embodiments, the noise superposition network may be based on a forward denoising network of a pre-trained diffusion model, and the image generation network may be based on a backward denoising network of the pre-trained diffusion model.

[0105] In the technical solution disclosed herein, the collection, acquisition, storage, use, processing, transmission, provision and public application of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0106] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0107] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0108] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0109] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700. The input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0110] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as methods 200 and / or 300 and their variants, etc. For example, in some embodiments, methods 200 and / or 300 and their variants, etc. may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods 200 and / or 300 and their variants, etc. described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the methods 200 and / or 300 and their variations, etc., in any other appropriate manner (eg, by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image color processing method, comprising: Obtaining a grayscale image to be color processed and a reference image; A forward denoising process based on a diffusion model is performed to obtain an intermediate image by superimposing the grayscale image with the noise; Obtaining a color description vector based on the reference image; as well as Based on the color description vector and the intermediate image, a colored output image with noise removed from the intermediate image is obtained through an image generation network, wherein the image generation network is based on an inverse denoising network of the diffusion model, and the inverse denoising network can denoise an input image with superimposed noise based on the description vector so that the output image is denoised and consistent with the description of the description vector, and wherein the color description vector is input into the inverse denoising network as the description vector of the inverse denoising network.

2. The method according to claim 1, wherein: Based on the color description vector and the intermediate image, obtaining an output image with noise removed from the intermediate image comprises: Based on the color description vector, the intermediate image and the grayscale image, an output image is obtained in which noise in the intermediate image is removed.

3. The method according to claim 2, wherein: Based on the color description vector, the intermediate image and the grayscale image, obtaining an output image in which noise in the intermediate image is removed comprises: Scaling the grayscale image to obtain a scaled grayscale image; splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and The spliced ​​intermediate image and the color description vector are input into the image generation network to obtain an output image with noise removed from the intermediate image.

4. The method according to claim 3, wherein: Obtaining a color description vector based on the reference image includes: Obtaining an embedding vector of the reference image based on the reference image; and The color description vector is obtained based on the embedding vector of the reference image so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

5. The method according to any one of claims 1 to 4, wherein: The color of the output image corresponds to the color description vector.

6. A training method for an image color processing model, the image color processing model comprising a noise superposition network, a color description network and an image generation network, the method comprising: Obtaining a color image and a grayscale image corresponding to the color image; Based on the grayscale image, obtaining an intermediate image generated by superimposing the grayscale image and noise through the noise superposition network, wherein the noise superposition network is based on a forward noise addition network of a pre-trained diffusion model; Based on the reference image, obtaining a color description vector through the color description network; Based on the color description vector and the intermediate image, a colored output image with noise removed from the intermediate image is obtained through the image generation network, wherein the image generation network is based on an inverse denoising network of the pre-trained diffusion model, the inverse denoising network can denoise an input image with superimposed noise based on the description vector so that the output image is denoised and consistent with the description of the description vector, and wherein the color description vector is input into the inverse denoising network as the description vector of the inverse denoising network; determining a loss based on the output image and the color image; and Based on the loss, parameters of at least a portion of the image color processing model are adjusted.

7. The method according to claim 6, wherein: Based on the color description vector and the intermediate image, obtaining, through the image generation network, an output image in which noise in the intermediate image is removed comprises: Based on the color description vector, the intermediate image and the grayscale image, an output image with noise removed from the intermediate image is obtained through the image generation network.

8. The method according to claim 7, wherein: Based on the color description vector, the intermediate image and the grayscale image, obtaining an output image with noise removed from the intermediate image through the image generation network includes: Scaling the grayscale image to obtain a scaled grayscale image; splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and The spliced ​​intermediate image and the color description vector are input into the image generation network to obtain an output image with noise removed from the intermediate image.

9. The method according to any one of claims 6 to 8, wherein: Obtaining a color image and a grayscale image corresponding to the color image comprises: obtaining a color image; and A grayscale operation is performed on the color image to obtain the grayscale image corresponding to the color image.

10. The method according to any one of claims 6 to 9, wherein: The color description network includes an encoding network and a mapping network. Based on a reference image, obtaining a color description vector through the color description network includes: Obtaining an embedding vector of the reference image through the encoding network based on the reference image; and Based on the embedding vector of the reference image, the color description vector is obtained through the mapping network so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

11. The method according to any one of claims 6 to 10, wherein: Adjusting parameters of at least a portion of the image color processing model includes adjusting parameters of the color description network.

12. The method according to any one of claims 6 to 11, wherein: Adjusting parameters of at least a portion of the image color processing model includes adjusting parameters of the image generation network.

13. The method according to any one of claims 6 to 12, wherein: The color image and the reference image are associated video frames from the same video.

14. The method according to any one of claims 6 to 13, wherein: The color of the output image corresponds to the color description vector.

15. An image color processing device, comprising: An obtaining unit, used for obtaining a grayscale image to be color processed and a reference image; A superposition unit, used for a forward noise addition process based on a diffusion model, to obtain an intermediate image by superimposing the grayscale image with the noise; A reference unit, configured to obtain a color description vector based on the reference image; as well as A coloring unit is used to obtain a colored output image with noise removed from the intermediate image based on the color description vector and the intermediate image through an image generation network, wherein the image generation network is based on an inverse denoising network of the diffusion model, and the inverse denoising network can denoise an input image with superimposed noise based on the description vector so that the output image is denoised and consistent with the description of the description vector, and wherein the color description vector is input into the inverse denoising network as the description vector of the inverse denoising network.

16. The device according to claim 15, wherein: The coloring unit comprises: A unit for obtaining, based on the color description vector, the intermediate image and the grayscale image, an output image with noise removed from the intermediate image.

17. The device according to claim 16, wherein: The unit for obtaining an output image with noise removed from the intermediate image based on the color description vector, the intermediate image and the grayscale image comprises: a unit for scaling the grayscale image to obtain a scaled grayscale image; means for splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and A unit for inputting the stitched intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

18. The device according to claim 17, wherein: The reference unit comprises: means for obtaining an embedding vector of the reference image based on the reference image; and A unit for obtaining the color description vector based on the embedding vector of the reference image so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

19. The device according to any one of claims 15 to 18, wherein: The color of the output image corresponds to the color description vector.

20. A training device for an image color processing model, the image color processing model comprising a noise superposition network, a color description network and an image generation network, the device comprising: An obtaining unit, used for obtaining a color image and a grayscale image corresponding to the color image; A superposition unit, configured to obtain, based on the grayscale image, an intermediate image generated by superimposing the grayscale image and noise through the noise superposition network, wherein the noise superposition network is based on a forward noise addition network of a pre-trained diffusion model; A reference unit, used to obtain a color description vector through the color description network based on a reference image; A coloring unit, configured to obtain, based on the color description vector and the intermediate image, an output image with noise removed from the intermediate image through the image generation network, wherein the image generation network is based on an inverse denoising network of the pre-trained diffusion model, the inverse denoising network can denoise an input image with superimposed noise based on the description vector so that the output image is denoised and consistent with the description of the description vector, and wherein the color description vector is input into the inverse denoising network as the description vector of the inverse denoising network; a loss unit for calculating a loss based on the output image and the color image; and An adjusting unit is used to adjust parameters of at least a part of the image color processing model based on the calculated loss.

21. The device according to claim 20, wherein: The coloring unit comprises: A unit for obtaining, based on the color description vector, the intermediate image and the grayscale image, an output image with noise removed from the intermediate image through the image generation network.

22. The device according to claim 21, wherein The unit for obtaining, based on the color description vector, the intermediate image and the grayscale image, an output image with noise removed from the intermediate image through the image generation network comprises: a unit for scaling the grayscale image to obtain a scaled grayscale image; means for splicing the scaled grayscale image with the intermediate image to obtain a spliced ​​intermediate image; and A unit for inputting the stitched intermediate image and the color description vector into the image generation network to obtain an output image with noise removed from the intermediate image.

23. The device according to any one of claims 20 to 22, wherein: The obtaining unit comprises: A unit for obtaining a color image; A unit for performing a grayscale operation on the color image to obtain the grayscale image corresponding to the color image.

24. The device according to any one of claims 20 to 23, wherein: The reference unit comprises: a unit for obtaining an embedding vector of the reference image through the encoding network based on the reference image; and A unit for obtaining the color description vector through the mapping network based on the embedding vector of the reference image so that the dimension of the color description vector is the same as the dimension of the description vector required by the image generation network.

25. The device according to any one of claims 20 to 24, wherein: The adjustment unit comprises a unit for adjusting parameters of the color description network.

26. The device according to any one of claims 20 to 25, wherein: The adjustment unit comprises a unit for adjusting parameters of the image generation network.

27. The device according to any one of claims 20 to 26, wherein: The color image and the reference image are associated video frames from the same video.

28. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5 or 6-14.

29. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5 or 6-14.

30. A computer program product comprising a computer program, wherein: The computer program implements the method of any one of claims 1-5 or 6-14 when executed by a processor.

Images