Image color migration method and device, computer readable medium and electronic device

Abstract

The present disclosure provides an image color migration method and device, a computer-readable medium, and an electronic device, and relates to the field of image processing technology. The method comprises: acquiring a content image and a reference image; performing parameterized processing on the content image and the reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image; iteratively optimizing the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image; performing color migration processing on the content image according to the color mapping relationship to obtain a target content image having similar color distribution characteristics to the reference image. The present disclosure can make the target content image obtained by color migration processing more consistent with the color distribution of the reference image, and improve the image quality of the target content image.

Description

Background Art

[0002] With the continuous improvement of people's living standards, image processing technology has attracted more and more attention. Among them, color migration is an important research direction in digital image processing. Color migration is the color transfer between two images. In layman's terms, one image is used as the source of color and the other is used as the image to be processed. Then, through the algorithm, the image to be processed has the color information of the source image while retaining its own structural shape information.

[0003] At present, the related color transfer schemes either directly apply the task objectives of style transfer to achieve image color transfer, or directly achieve image color adjustment through Generative Adversarial Networks (GAN). However, these two technical solutions do not match the task objectives of color transfer, resulting in the processed images either not retaining the original shape and structure information, or having more noise and poor image quality, and failing to achieve the color transfer task well. Summary of the invention

[0004] The purpose of the present disclosure is to provide an image color migration method, an image color migration device, a computer-readable medium and an electronic device, thereby improving the image quality of the target content image after color migration processing at least to a certain extent, so that it is more consistent with the color distribution of the reference image.

[0005] According to a first aspect of the present disclosure, there is provided an image color migration method, comprising:

[0006] Obtaining a content image and a reference image;

[0007] Performing parameterization processing on the content image and the reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image;

[0008] Iteratively optimizing the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image;

[0009] The content image is subjected to color migration processing according to the color mapping relationship to obtain a target content image having a color distribution feature similar to that of the reference image.

[0010] According to a second aspect of the present disclosure, there is provided an image color migration device, comprising:

[0011] An image acquisition module, used to acquire a content image and a reference image;

[0012] An image parameterization module, used for performing parameterization processing on the content image and the reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image;

[0013] an optimization fitting module, configured to iteratively optimize the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image;

[0014] The image color migration module is used to perform color migration processing on the content image according to the color mapping relationship to obtain a target content image having similar color distribution characteristics to the reference image.

[0015] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0016] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0017] Processor; and

[0018] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0019] An image color migration method provided by an embodiment of the present disclosure performs parameterized processing on an acquired content image and a reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image; then the first statistical feature and the second statistical feature are iteratively optimized to obtain a color mapping relationship between the content image and the reference image, and then color migration processing is performed on the content image according to the color mapping relationship to obtain a target content image having similar color distribution characteristics to the reference image. On the one hand, by calculating the statistical features of the content image and the reference image, and then iteratively optimizing the statistical features to obtain the color mapping relationship, and then realizing the color transfer task according to the color mapping relationship, compared with the style transfer processing based on the style transfer model of the transfer learning task or the color adjustment task based on the adversarial generative network, it can not only make the target content image after the color transfer processing less noisy and achieve more precise texture details, but also improve the image quality of the target content image; on the other hand, compared with the color transfer based on style transfer or adversarial generative network, the color mapping relationship obtained by iterative fitting is used for color transfer processing, so that the color distribution of the target content image is more in line with the color distribution characteristics of the reference image, and the image texture of the content image will not be changed, but only the color transfer is realized, which is more in line with the goal of the color transfer task and improves the image quality of the target content image after the color transfer.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A schematic diagram showing an exemplary system architecture to which the embodiments of the present disclosure may be applied;

[0023] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0024] Figure 3 A flowchart schematically illustrates an image color migration method in an exemplary embodiment of the present disclosure;

[0025] Figure 4 A flowchart for fitting and obtaining a color mapping relationship between a content image and a reference image in an exemplary embodiment of the present disclosure is schematically shown;

[0026] Figure 5 Schematically illustrates a flow chart of generating a target content image through a color mapping relationship in an exemplary embodiment of the present disclosure;

[0027] Figure 6 A flowchart for implementing color migration between a content image and a reference image in an exemplary embodiment of the present disclosure is schematically shown;

[0028] Figure 7 A flowchart of training a color migration model in an exemplary embodiment of the present disclosure is schematically shown;

[0029] Figure 8 The figure schematically shows the composition of the image color migration device in the exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which an image color migration method and device according to an embodiment of the present disclosure can be applied is shown.

[0033] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0034] The image color migration method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the image color migration device is generally provided in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the image color migration method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the image color migration device can also be provided in the server 105, which is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the user may obtain the content image and the reference image through the terminal devices 101, 102, and 103, and then upload the content image and the reference image to the server 105. After the server generates the target content image through the image color migration method provided in the embodiment of the present disclosure, the target content image is transmitted to the terminal devices 101, 102, and 103, etc.

[0035] An exemplary embodiment of the present disclosure provides an electronic device for implementing an image color migration method, which may be Figure 1The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the image color migration method by executing the executable instructions.

[0036] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0037] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0038] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0039] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the mobile terminal 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0040] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0041] The charging management module 240 is used to receive charging input from the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.

[0042] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 200; modulation and demodulation processor can include modulator and demodulator; wireless communication module 260 can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 200. In some embodiments, antenna 1 of mobile terminal 200 is coupled with mobile communication module 250, and antenna 2 is coupled with wireless communication module 260, so that mobile terminal 200 can communicate with network and other devices through wireless communication technology.

[0043] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0044] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs.

[0045] The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0046] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0047] The mobile terminal 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the earphone interface 274 and the application processor.

[0048] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be arranged in the camera module 291.

[0049] The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2802 can be arranged on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0050] The gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 2803. The gyro sensor 2803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.

[0051] In addition, sensors with other functions may be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0052] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the key 294 includes a power key, a volume key, etc., and the user can generate key signal input related to the user settings and function control of the mobile terminal 200 through key input. For example, the indicator 292, the motor 293, the SIM card interface 295, etc.

[0053] In the related art, in order to achieve color migration between images, either the training method of the style migration task is directly adopted, for example, the Visual Geometry Group Network (VGG) is used to construct content loss and style loss. However, since the optimization goals of the style migration task and the color migration task are different, the shape structure information in the content image is destroyed and the image texture information is changed during the color migration process, and the output image is blurry and noisy. In another scheme, the trained generative adversarial network is directly used to directly generate the color-adjusted image. However, the color migration using the generative adversarial network can only process a single color distribution feature, and a large amount of data on the color distribution feature needs to be collected. The color distribution feature in the output image cannot be well approximated to the color distribution of the reference image, resulting in poor color migration effect.

[0054] Based on one or more problems discovered, the present disclosure first proposes an image color migration method. The image color migration method according to an exemplary embodiment of the present disclosure is described in detail below.

[0055] Figure 3 The flowchart of an image color migration method in this exemplary embodiment includes the following steps S310 to S340:

[0056] In step S310, a content image and a reference image are acquired.

[0057] In an exemplary embodiment, a content image refers to a destination image used for color migration processing. For example, a content image may be a landscape image or a person image taken by an image acquisition device, or a line drawing with or without color. This exemplary embodiment does not specifically limit this. A reference image refers to a source image used for color migration processing. For example, a reference image may be a landscape image or a person image taken by an image acquisition device, or an image with a specific color matching scheme drawn in advance by a designer. This exemplary embodiment does not specifically limit this.

[0058] For example, suppose color migration is based on image A and image B, and a new image C is synthesized so that it has both the color distribution of B and the structural distribution of A. That is, image A learns the overall color tone of image B without changing the shape information it expresses. Image A can be called a content image, also called a shape image or a destination image, and image B can be called a reference image, also called a color image or a source image.

[0059] The image captured by the image acquisition unit can be obtained as the content image or reference image, or the content image or reference image input by the user can be obtained through a preset interface. Of course, the image selected by the user can also be obtained from the cloud or database as the content image or reference image. This example embodiment does not make any special limitations on this.

[0060] In step S320, the content image and the reference image are parameterized to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image.

[0061] In an exemplary embodiment, parameterized processing refers to the process of materializing the color distribution characteristics in a content image or a reference image into statistical data. For example, parameterized processing can be a process of obtaining the color probability distribution of an image by fitting a Gaussian mixture model, or a process of obtaining sampling information by performing histogram sampling on an image. This exemplary embodiment does not specifically limit this.

[0062] The first statistical feature refers to data obtained by parameterizing the color distribution feature in the content image. For example, the first statistical feature may be a probability distribution corresponding to the content image, or may be histogram sampling information corresponding to the content image. The present exemplary embodiment is not limited thereto. The second statistical feature refers to data obtained by parameterizing the color distribution feature in the reference image. The second statistical feature and the first statistical feature are statistical features obtained by the same parameterization process, and will not be described in detail herein.

[0063] In step S330, the first statistical feature and the second statistical feature are iteratively optimized to obtain a color mapping relationship between the content image and the reference image.

[0064] In an exemplary embodiment, the color mapping relationship refers to a mapping relationship between the color distribution of the content image and the color distribution of the reference image obtained through iterative fitting, and the color mapping relationship can be used to migrate the color distribution in the reference image to the content image.

[0065] In step S340, color migration processing is performed on the content image according to the color mapping relationship to obtain a target content image having a color distribution feature similar to that of the reference image.

[0066] In an exemplary embodiment, the target content image refers to an image in which the color distribution in the content image is converted into an image that is compatible with the color distribution of the reference image, that is, a content image that has undergone color migration processing.

[0067] After obtaining the color mapping relationship between the content image and the reference image, the color of the content image is adjusted through the color mapping relationship so that the color distribution of the content image is the same or similar to the color distribution of the reference image, thereby migrating the color distribution in the reference image to the content image to obtain the target content image.

[0068] Steps S310 to S340 are explained in detail below.

[0069] At present, attempts to use deep learning methods in color migration tasks are becoming more and more common, but compared with older traditional algorithms, there is no significant improvement in the presented results. The main reason is that the existing schemes have not seriously analyzed the goals of the color migration task, and have not established a mathematical model of color migration. Instead, they simply applied the style migration method that has developed rapidly in recent years. As a result, a large number of schemes that use style migration loss to train color migration have been proposed. However, due to the wrong loss, the color migration scheme cannot achieve the expected results.

[0070] The applicant found that the goal of the color migration task can be better described by equation (1):

[0071] ∫ u f(x)d(x)=∫ t(u) g(x)d(x) (1)

[0072] Among them, f(x) can represent the probability density function of the color distribution in the content image, g(x) can represent the probability density function of the color distribution in the reference image, u can represent any pixel value in the content image, t(u) can represent the color adjustment result of the pixel value u, and t(.) can represent the color mapping relationship (i.e., the color adjustment function).

[0073] For example, suppose that any pixel value u is (0.5, 0.5, 0.5) representing an RGB value between 0 and 1, ∫ u f(x)d(x) can be understood as the value of calculating the number of pixel values ​​in the content image (r<0.5, g<0.5, b<0.5) divided by the total number of pixels in the content image. v=t(u) can be understood as the new pixel value v corresponding to the pixel value u after color migration. The RGB value of v can be expressed as (v r ,v g ,v b ), ∫ t(u)g(x)d(x) represents the reference image (r<v r ,g<v g ,b<v b ) divided by the total number of pixels in the reference image.

[0074] By taking the derivative of equation (1), we can obtain f(u) = |t'(u)|g(t(u)), where |t'(u)| can represent the determinant of the derivative of t at u, which means that the probability distribution f(u) is the same as the probability distribution p(u) = |t'(u)|g(t(u)). Therefore, we can conclude that when achieving the color migration goal, the KL divergence (Kullback–Leibler divergence, KLD, also known as relative entropy) can be used as the loss, that is, equation (2):

[0075]

[0076] Among them, u can represent any pixel value in the content image, f(.) can represent the probability distribution of color distribution in the content image, p(u)=|t'(u)|g(t(u)), |t'(u)| can represent the determinant of the derivative of t(.) at u, and g(.) can represent the probability distribution of color distribution in the reference image.

[0077] In an exemplary embodiment, the statistical features may include a probability distribution of an image. Specifically, a mixed Gaussian fitting may be performed on a content image based on a Gaussian mixture model to obtain a first probability distribution of the content image; and a mixed Gaussian fitting may be performed on a reference image based on a Gaussian mixture model to obtain a second probability distribution of the reference image.

[0078] Among them, Gaussian Mixture Model (GMM) refers to the use of Gaussian probability density function (normal distribution curve) to accurately quantify things and decompose things into several models based on Gaussian probability density function (normal distribution curve). Since KL divergence requires that the probability distribution is greater than 0 everywhere, the color distribution in the image can be parameterized by GMM (Gaussian Mixture Model), and the probability distribution is obtained after GMM fitting the content image, that is, f(u), and the probability distribution is obtained after GMM fitting the reference image, that is, g(u).

[0079] In an exemplary embodiment, the Figure 4 The steps in the above embodiment implement iterative optimization of the first statistical feature and the second statistical feature to obtain the color mapping relationship between the content image and the reference image. Figure 4 As shown, it may specifically include:

[0080] Step S410, constructing a distribution fitting model, wherein the distribution fitting model is a convolutional neural network model, and the loss function of the distribution fitting model is a relative entropy loss function;

[0081] Step S420: performing nonlinear optimization on the first statistical feature and the second statistical feature based on the distribution fitting model to minimize the relative entropy loss function and obtain a color mapping relationship between the content image and the reference image.

[0082] The distribution fitting model refers to a neural network model used to fit the color mapping relationship between the content image and the reference image according to the first statistical feature and the second statistical feature. For example, the distribution fitting model can be a convolutional neural network model (CNN). The CNN network is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. The convolutional neural network model has representation learning capabilities and can perform shift-invariant classification of input information according to its hierarchical structure. Therefore, it is also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0083] For the KL divergence, i.e., the relative entropy loss function, equation (1), since the functions f(u) and g(u) have been obtained in the GMM fitting step, only the function t(.) is unknown. Therefore, the function t(.) can be fitted through a neural network model such as a CNN network, and by using a nonlinear optimization method to minimize equation (1), the function t(.) can be obtained. Function t(.) is the color mapping relationship between the content image and the reference image.

[0084] For example, the model structure of the distribution fitting model corresponding to the function t(.) can be expressed as equation (3):

[0085] t(u)=A'*tanh(A*u+b1)+b2 (3)

[0086] Among them, A, b1, and b2 are model parameters, A can represent a 32*3 matrix, and A' can represent the transposed matrix of matrix A. It can be understood that the model structure of the distribution fitting model can include two 1*1 convolution kernels, and the two convolution kernels are transposed of each other. One of the convolution kernels 1 has a weight of A and a bias of b1, and the other convolution kernel 2 has a weight of A' and a transposed bias of b2, and tanh is the activation function.

[0087] In an exemplary embodiment, after the color mapping relationship t(.) between the content image and the reference image is obtained by fitting, the content image can be subjected to color migration processing according to the color mapping relationship t(.) to obtain a target content image having a color distribution feature similar to that of the reference image. Figure 5 As shown, it may specifically include:

[0088] Step S510, converting original pixel values ​​in the content image into target pixel values ​​according to the color mapping relationship;

[0089] Step S520: Generate a target content image having a similar color distribution feature as the reference image through the target pixel value.

[0090] Among them, the original pixel value refers to the pixel value in the content image before color migration processing, and the target pixel value refers to the pixel value after color migration processing. After fitting the color mapping relationship, that is, the function t(.), for the original pixel value u in the content image, v=t(u) is calculated, where the new pixel value v is the target pixel value obtained after the original pixel value u is processed by color migration.

[0091] Figure 6 A flow chart for implementing color migration between a content image and a reference image in an exemplary embodiment of the present disclosure is schematically shown.

[0092] refer to Figure 6 As shown, in step S610, the content image 601 is fitted based on the Gaussian mixture model to calculate and obtain the first statistical feature (ie, the first probability density distribution function f(u)) corresponding to the content image;

[0093] Step S620, fitting the reference image 602 based on the Gaussian mixture model, and calculating and obtaining a second statistical feature (i.e., a second probability density distribution function g(u)) corresponding to the reference image;

[0094] Step S630, iteratively fitting the first probability density distribution function f(u) and the second probability density distribution function g(u) through a distribution fitting model using a relative entropy loss function (KL divergence) as a loss function to minimize the relative entropy loss function, thereby obtaining a color mapping relationship (i.e., function t(u)) between the content image 601 and the reference image 602;

[0095] Step S640 , converting the original pixel value in the content image 601 according to the fitted color mapping relationship t(u) to obtain the target pixel value, which is the toning result after the color migration process, and generating the target content image 603 according to the target pixel value.

[0096] In another exemplary embodiment, in the color migration task, if the color migration is achieved by performing GMM fitting and optimizing KL divergence for each pair of content images and reference images, the color migration process may be inefficient and labor-intensive. Figure 6 The color migration method in , through the three steps of GMM fitting, optimizing KL divergence and color adjustment, obtains multiple sets of {content image, reference image, target content image} sample data, so that the deep learning model can be trained with the generated sample data, so as to achieve fast and efficient color migration tasks through the trained deep learning model. Figure 7 The training of the color migration model can be implemented by:

[0097] Step S710, taking the content image, the reference image and the target content image as a sample image set;

[0098] Step S720: Perform supervised training on the pre-built color migration model based on the sample image set to obtain a trained color migration model.

[0099] The sample image set refers to a sample image pair consisting of a reference image, a content image, and a target content image obtained after color migration processing based on three steps of GMM fitting, optimizing KL divergence, and color adjustment, which is used to train the color migration model.

[0100] When training the color transfer model, the content image and the reference image in the sample image set {content image, reference image, target content image} can be input as input samples to the color transfer model, and the target content image can be used as a sample label to supervise the training of the color transfer model until the loss function of the color transfer model converges to obtain a trained color transfer model. When in use, the real content image and the real reference image can be input into the trained color transfer model, and the target content image corresponding to the real content image can be generated quickly and efficiently, so as to realize the migration of the color distribution in the real reference image to the real content image.

[0101] The applicant's research found that in the color migration task, the color adjustment problem can be described as r', g', b'=f(r, g, b), that is, the output RGB pixel value is a function of the input R'g'b' pixel value, and the same RGB value will get the same R'g'b' value. Therefore, the color migration task essentially only needs to focus on the current pixel value and the global statistical value, without the need for pixel area information. On this basis, the color migration task may only require a 1*1 receptive domain, and the reduced requirements for the receptive domain also result in the model structure of the color adjustment task not requiring downsampling. Therefore, if a 1*1 convolution kernel is used in the color migration model to process the color distribution characteristics in the image, the image details can be effectively maintained, making the quality of the image after color migration processing higher, and because the structure of the model is simpler, the processing efficiency of the model can be improved.

[0102] Specifically, the color migration model in this exemplary embodiment may include at least two convolution layers, one of which may be used to extract the color distribution features of the input real content image or the real reference image, and the other convolution layer may be used for the final output, and at least two convolution layers may include 1*1 convolution kernels. The content image and the reference image may be input into the pre-built color migration model to respectively extract the first image features in the content image and the second image features in the reference image through the 1*1 convolution kernel in the convolution layer.

[0103] Furthermore, the color migration model may also include an adaptive instance normalization module (AdaIN), the function of which is to calculate the mean and variance of the image features (featuremap) of the content image (content) and the reference image (style), by subtracting the variance of the content image from the image features of the content image, divided by the mean of the content image, and then multiplied by the variance of the reference image plus the mean of the content image. The adaptive instance normalization module can make the mean and variance of the image features corresponding to the content image consistent with the mean and variance of the image features corresponding to the reference image, thereby capturing the statistical information of the entire reference image. That is, the adaptive example normalization module can be used to align the mean and variance of the first image feature to the mean and variance of the second image feature, so as to achieve the migration of the color distribution features in the reference image to the content image.

[0104] To summarize, in this exemplary embodiment, on the one hand, by calculating the statistical features of the content image and the reference image, and then iteratively optimizing the statistical features to obtain a color mapping relationship, and then implementing a color migration task based on the color mapping relationship, compared to performing style migration processing through a style migration model based on a transfer learning task or performing a color adjustment task based on a generative adversarial network, it is possible to not only make the target content image after color migration processing have less noise and achieve more precise texture details, but also improve the image quality of the target content image; on the other hand, compared to color migration based on style migration or a generative adversarial network, color migration processing is performed through the color mapping relationship obtained by iterative fitting, so that the color distribution of the target content image is more closely aligned with the color distribution characteristics of the reference image, without changing the image texture of the content image, but only realizing color migration, which is more in line with the goal of the color migration task and improves the image quality of the target content image after color migration.

[0105] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0106] For further reference, Figure 8 As shown, in the embodiment of this example, an image color migration device 800 is also provided, which may include an image acquisition module 810, an image parameterization module 820, an optimization fitting module 830 and an image color migration module 840. Among them:

[0107] The image acquisition module 810 is used to acquire content images and reference images;

[0108] The image parameterization module 820 is used to perform parameterization processing on the content image and the reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image;

[0109] The optimization fitting module 830 is used to iteratively optimize the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image;

[0110] The image color migration module 840 is used to perform color migration processing on the content image according to the color mapping relationship to obtain a target content image having a color distribution feature similar to that of the reference image.

[0111] In an exemplary embodiment, the image parameterization module 820 may be used to:

[0112] Performing mixed Gaussian fitting on the content image based on a Gaussian mixture model to obtain a first probability distribution of the content image; and

[0113] A mixed Gaussian fitting is performed on the reference image based on a Gaussian mixture model to obtain a second probability distribution of the reference image.

[0114] In an exemplary embodiment, the optimization fitting module 830 may be used to:

[0115] Constructing a distribution fitting model, wherein the distribution fitting model is a convolutional neural network model, and the loss function of the distribution fitting model is a relative entropy loss function;

[0116] Based on the distribution fitting model, nonlinear optimization is performed on the first statistical feature and the second statistical feature to minimize the relative entropy loss function, thereby obtaining a color mapping relationship between the content image and the reference image.

[0117] In an exemplary embodiment, the image color migration module 840 may be used to:

[0118] Converting original pixel values ​​in the content image into target pixel values ​​according to the color mapping relationship;

[0119] A target content image having a similar color distribution feature to the reference image is generated by using the target pixel value.

[0120] In an exemplary embodiment, the image color migration apparatus 800 further includes a color migration model training module, and the color migration model training module can be used to:

[0121] Taking the content image, the reference image and the target content image as a sample image set;

[0122] The pre-built color migration model is supervisedly trained based on the sample image set to obtain a trained color migration model.

[0123] In an exemplary embodiment, the color migration model may include a convolution layer, which may include a 1*1 convolution kernel; the color migration model training module may also be used for:

[0124] The content image and the reference image are input into the pre-built color migration model to respectively extract the first image feature in the content image and the second image feature in the reference image through the 1*1 convolution kernel.

[0125] In an exemplary embodiment, the color transfer model may include an adaptive example normalization module; the color transfer model training module may also be used to:

[0126] The adaptive example normalization module aligns the mean and variance of the first image feature to the mean and variance of the second image feature, so as to achieve the migration of the color distribution feature in the reference image to the content image.

[0127] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0128] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0129] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figures 3 to 7 Any one or more steps in .

[0130] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0131] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0132] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0133] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0134] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image color migration method, characterized in that: include: Obtaining a content image and a reference image; The content image and the reference image are parameterized to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image; wherein, a mixed Gaussian fitting is performed on the content image based on a Gaussian mixture model to obtain a first probability distribution of the content image as the first statistical feature, and a mixed Gaussian fitting is performed on the reference image based on a Gaussian mixture model to obtain a second probability distribution of the reference image as the second statistical feature; Iteratively optimizing the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image; Performing color migration processing on the content image according to the color mapping relationship to obtain a target content image having a similar color distribution feature to the reference image; The iterative optimization of the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image includes: Constructing a distribution fitting model, wherein the distribution fitting model is a neural network model for fitting a color mapping relationship between the content image and the reference image according to the first statistical feature and the second statistical feature, and a loss function of the distribution fitting model is a relative entropy loss function; Based on the distribution fitting model, nonlinear optimization is performed on the first statistical feature and the second statistical feature to minimize the relative entropy loss function, thereby obtaining a color mapping relationship between the content image and the reference image.

2. The method according to claim 1, characterized in that The distribution fitting model is a convolutional neural network model.

3. The method according to claim 1, characterized in that Performing color migration processing on the content image according to the color mapping relationship to obtain a target content image having a similar color distribution feature as the reference image includes: Converting original pixel values ​​in the content image into target pixel values ​​according to the color mapping relationship; A target content image having a similar color distribution feature to the reference image is generated by using the target pixel value.

4. The method according to claim 1, characterized in that: The method further comprises: Taking the content image, the reference image and the target content image as a sample image set; The pre-built color migration model is supervisedly trained based on the sample image set to obtain a trained color migration model.

5. The method according to claim 4, characterized in that The color migration model includes a convolution layer, and the convolution layer includes a 1*1 convolution kernel; the method further includes: The content image and the reference image are input into the pre-built color migration model to respectively extract the first image feature in the content image and the second image feature in the reference image through the 1*1 convolution kernel.

6. The method according to claim 5, characterized in that The color migration model includes an adaptive example normalization module; the method further includes: The adaptive example normalization module aligns the mean and variance of the first image feature to the mean and variance of the second image feature, so as to achieve the migration of the color distribution feature in the reference image to the content image.

7. An image color migration device, characterized in that: include: An image acquisition module, used to acquire a content image and a reference image; an image parameterization module, configured to perform parameterization processing on the content image and the reference image to obtain a first statistical feature corresponding to the content image and a second statistical feature of the reference image; wherein, a mixed Gaussian fitting is performed on the content image based on a Gaussian mixture model to obtain a first probability distribution of the content image as the first statistical feature, and a mixed Gaussian fitting is performed on the reference image based on a Gaussian mixture model to obtain a second probability distribution of the reference image as the second statistical feature; an optimization fitting module, configured to iteratively optimize the first statistical feature and the second statistical feature to obtain a color mapping relationship between the content image and the reference image; An image color migration module, configured to perform color migration processing on the content image according to the color mapping relationship to obtain a target content image having a color distribution feature similar to that of the reference image; Wherein, the optimization fitting module is configured as follows: Constructing a distribution fitting model, where the distribution fitting model is a neural network model for fitting a color mapping relationship between the content image and the reference image according to the first statistical feature and the second statistical feature, and a loss function of the distribution fitting model is a relative entropy loss function; Based on the distribution fitting model, nonlinear optimization is performed on the first statistical feature and the second statistical feature to minimize the relative entropy loss function, thereby obtaining a color mapping relationship between the content image and the reference image.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

Images