Video Image Processing Method and Video Image Processing Device

Abstract

The present disclosure provides a video image processing method and a video image processing apparatus. The video image processing method includes: obtaining a template image and a content image corresponding to an original video; calculating color transfer conversion information between the template image and the content image based on the template image and the content image; and performing color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information, thereby generating a target video.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a video image processing method and a video image processing apparatus. Background Art

[0002] With the development of technology, the functions of terminals are becoming increasingly rich, and users can use terminals for work, study, entertainment, etc. For example, users can use terminals to publish videos. Users can publish videos they have taken themselves, or they can choose existing videos for publication.

[0003] Users can also perform color transfer on videos according to their own needs. For example, users can select a style image they want to transfer and transfer the color tone of the style image to the target video. In related technologies, when performing color transfer on a video, a global color transfer method is adopted, that is, only the matching of image mean and image variance is performed. For some style images with rich colors, if only the matching of image mean and image variance is performed, the colors of the style image cannot be effectively mapped to the video frames, and some colors in the style image will be lost. Therefore, in related technologies, the effect of color transfer on videos is poor. Summary of the Invention

[0004] The present disclosure provides a video image processing method and a video image processing apparatus to at least solve the problem in related technologies that when performing color transfer on a video, for some style images with rich colors, the colors of the style image cannot be effectively mapped to the video frames, some colors in the style image are lost, and the effect of color transfer on the video is poor. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video image processing method is provided, including: obtaining a template image and a content image corresponding to an original video; calculating color transfer conversion information between the template image and the content image based on the template image and the content image; and performing color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information, so as to generate a target video.

[0006] Optionally, the performing color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information, so as to generate a target video includes: calculating histogram linear transformation parameters based on the color transfer conversion information; and performing color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, so as to generate the target video.

[0007] Optionally, calculating a histogram linear transformation parameter based on the color transfer conversion information includes: performing color transfer on pixel values of pixels in the content image based on the color transfer conversion information; obtaining a first histogram corresponding to the content image after pixel value color transfer and a second histogram corresponding to the template image; and calculating the histogram linear transformation parameter for matching the first histogram to the second histogram.

[0008] Optionally, calculating the histogram linear transformation parameter for matching the first histogram to the second histogram includes: finding a first left endpoint and a first right endpoint of the first histogram and finding a second left endpoint and a second right endpoint of the second histogram; and calculating the histogram linear transformation parameter when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint.

[0009] Optionally, performing color transfer on pixel values of pixels in each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameter to generate the target video includes: performing color transfer on pixel values of pixels in each video frame of the original video based on the color transfer conversion information; and performing a linear transformation on the pixel values after color transfer of pixels in each video frame of the original video based on the histogram linear transformation parameter to obtain the target video.

[0010] Optionally, performing color transfer on pixel values of pixels in each video frame of the original video based on the color transfer conversion information includes: generating a first color lookup table according to the color transfer conversion information, where the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value calculated by the input through the color transfer conversion information; and using the pixel values of pixels in each video frame of the original video as inputs to look up the first color lookup table to obtain the pixel values after color transfer of pixels in each video frame.

[0011] Optionally, performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters includes: generating a first color lookup table according to the color transfer conversion information, where the first color lookup table includes inputs and corresponding outputs, the input is the input pixel value, and the output is the pixel value obtained by calculating the input through the color transfer conversion information; performing a linear transformation on the inputs and corresponding outputs in the first color lookup table based on the histogram linear transformation parameters to obtain a second color lookup table; using the pixel values of each video frame of the original video as inputs to look up the second color lookup table to obtain the pixel values of the pixels of the target video.

[0012] Optionally, the color transfer conversion information is a color transfer conversion matrix, and color transfer is performed using the following formula:

[0013] I′ p = T × [I p - mean(I c )] + mean(I s )

[0014] where T is the color transfer conversion matrix, mean(I c ) is the mean value of the pixel values of the content image, mean(I s ) is the mean value of the pixel values of the template image; in the case of performing color transfer on the pixel values of each video frame, I′ p is the pixel value of each video frame after pixel value color transfer, I p is the pixel value of each video frame; in the case of performing color transfer on the pixel values of the content image, I′ p is the pixel value of the content image after pixel value color transfer, I p is the pixel value of the content image.

[0015] Optionally, calculating the color transfer conversion information between the template image and the content image based on the template image and the content image includes: calculating the covariance matrix of the content image and the covariance matrix of the template image; calculating a color transfer conversion matrix based on the covariance matrix of the content image and the covariance matrix of the template image as the color transfer conversion information.

[0016] Optionally, calculating the color transfer conversion matrix based on the covariance matrix of the content image and the covariance matrix of the template image includes: calculating the color transfer conversion matrix through the following formula:

[0017]

[0018] wherein, T is the color transfer conversion matrix, C u is the covariance matrix of the content image, C v is the covariance matrix of the template image.

[0019] Optionally, obtaining the content image corresponding to the original video includes: extracting a predetermined number of video frames from a plurality of video frames included in the original video; splicing the predetermined number of video frames into a single image as the content image.

[0020] According to a second aspect of the embodiments of the present disclosure, there is provided a video image processing apparatus, including: an acquisition module configured to acquire a template image and a content image corresponding to an original video; a calculation module configured to calculate color transfer conversion information between the template image and the content image based on the template image and the content image; a color transfer module configured to perform color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information, so as to generate a target video.

[0021] Optionally, the color transfer module is configured to: calculate histogram linear transformation parameters based on the color transfer conversion information; perform color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, so as to generate the target video.

[0022] Optionally, the color transfer module is configured to: perform color transfer on pixel values of pixels of the content image based on the color transfer conversion information; acquire a first histogram corresponding to the content image after pixel value color transfer and a second histogram corresponding to the template image; calculate the histogram linear transformation parameters for matching the first histogram to the second histogram.

[0023] Optionally, the color transfer module is configured to: find a first left endpoint and a first right endpoint of the first histogram and find a second left endpoint and a second right endpoint of the second histogram; calculate the histogram linear transformation parameters when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint.

[0024] Optionally, the color transfer module is configured to: perform color transfer on pixel values of pixels of each video frame of the original video based on the color transfer conversion information; perform a linear transformation on the pixel values after color transfer of pixels of each video frame of the original video based on the histogram linear transformation parameters to obtain the target video.

[0025] Optionally, the color transfer module is configured to: generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value obtained by calculating the input through the color transfer conversion information; use the pixel values of each video frame of the original video as the input, look up the first color lookup table, and obtain the pixel values after color transfer of the pixels of each video frame.

[0026] Optionally, the color transfer module is configured to: generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value obtained by calculating the input through the color transfer conversion information; perform a linear transformation on the input and the corresponding output in the first color lookup table based on the histogram linear transformation parameter to obtain a second color lookup table; use the pixel values of each video frame of the original video as the input, look up the second color lookup table, and obtain the pixel values of the pixels of the target video.

[0027] Optionally, the color transfer conversion information is a color transfer conversion matrix, and the color transfer module is configured to perform color transfer using the following formula:

[0028] I′ p = T × [I p - mean(I c )] + mean(I s )

[0029] where T is the color transfer conversion matrix, mean(I c ) is the average pixel value of the content image, mean(I s ) is the average pixel value of the template image; when performing color transfer on the pixel values of each video frame, I′ p is the pixel value of each video frame after pixel value color transfer, and I p is the pixel value of each video frame; when performing color transfer on the pixel values of the content image, I′ p is the pixel value of the content image after pixel value color transfer, and I p is the pixel value of the content image.

[0030] Optionally, the calculation module is configured to: calculate the covariance matrix of the content image and the covariance matrix of the template image; calculate a color transfer transformation matrix based on the covariance matrix of the content image and the covariance matrix of the template image as the color transfer transformation information.

[0031] Optionally, the calculation module is configured to calculate the color transfer transformation matrix by the following formula:

[0032]

[0033] where T is the color transfer transformation matrix, C u is the covariance matrix of the content image, and C v is the covariance matrix of the template image.

[0034] Optionally, the acquisition module is configured to: extract a predetermined number of video frames from the multiple video frames included in the original video; splice the predetermined number of video frames into a single image as the content image.

[0035] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to execute the instructions to implement the video image processing method according to the present disclosure.

[0036] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the video image processing method according to the present disclosure.

[0037] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, including a computer program, where the computer program implements the video image processing method according to the present disclosure when executed by a processor.

[0038] The technical solution provided by the embodiment of the present disclosure at least brings the following beneficial effects:

[0039] It is possible to perform conversion on the pixel values of each pixel of the original video based on the color transfer transformation information calculated from the template image and the content image, so that the colors of the template image can be effectively mapped to the video frames, avoiding the situation of losing some colors in the template image.

[0040] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0041] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0042] Figure 1 is a flowchart showing a video image processing method according to an exemplary embodiment of the present disclosure;

[0043] Figure 2 is an implementation flowchart showing a video image processing method according to an exemplary embodiment of the present disclosure;

[0044] Figure 3 is a block diagram showing a video image processing device according to an exemplary embodiment of the present disclosure;

[0045] Figure 4 is a block diagram showing an electronic device according to an exemplary embodiment of the present disclosure. Detailed implementation manners

[0046] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above accompanying drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments of the present disclosure described here can be implemented in an order different from those illustrated or described here. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] Currently, the color grading of some movies is achieved by designers manually adjusting using professional software. The desired color tone can be transferred to the video clip to be edited using a color transfer algorithm. Color transfer methods include Histogram Matching, Reinhard, Welsh, etc.

[0049] The histogram matching method adjusts the histogram of the content image to that of the style image. This method is prone to generating mosaics, blocking effects, etc. The Reinhard method converts the image to the LAB color space and then matches the means and variances in the LAB color space. This method only matches the means and variances, which is likely to result in a relatively low contrast and reduced overall sharpness in the matched image. The Welsh method performs color matching of color images on grayscale images. This algorithm mainly uses finding matching pixels to achieve color transfer of grayscale images. Based on Reinhard, through the brightness value matching of grayscale images, the coloring of grayscale images is achieved. Since this method requires sample point matching, it is slow and has a high requirement for the correlation of scene brightness.

[0050] For the Reinhard method, first convert the image to the LAB color space. The L channel represents brightness, and the AB channels represent color. The L and AB channels are independent. The means and variances of each layer of the style image and the content image can be calculated separately, and then the means and variances can be directly transformed. The formula is:

[0051] I′ c =[std(I s ) / std(I c )]*[I c -mean(I c )]+mean(I s ) (1)

[0052] where I′ c represents the pixel value of the pixel of the content image after pixel value transformation, Ic represents the pixel value of the pixel of the content image, mean(I c ) is the mean of the pixel values of the content image, mean(I s ) is the mean of the pixel values of the style image, std(Ic) represents the standard deviation of the pixel values of the content image, and std(Is) represents the standard deviation of the pixel values of the style image. Finally, the content image after pixel value transformation can be converted back from the LAB color space to the RGB color space.

[0053] It should be noted that in the related art, when performing color transfer on a video, a global color transfer method is adopted, that is, only the matching of the image mean and the image variance is performed. For some style images with rich colors, if only the matching of the image mean and the image variance is performed, the colors of the style image will not be effectively mapped to the video frames, resulting in the loss of some colors in the style image. In addition, after performing color transfer on the video frames, the overall tone of the video frames is similar to that of the style image, but some details of the video frames will be lost, making the overall contrast of the video frames relatively low. Moreover, the above global color transfer method is only applicable to the color transfer of images. For videos, color transfer needs to be performed on each video frame. When the color distribution changes significantly between the current frame and the previous frame, the color distribution of the video frames obtained after transfer will not be continuous, easily resulting in the phenomenon of video flickering.

[0054] To solve the problem in the related art that some colors in the style image are lost when performing color transfer on a video, the video image processing method proposed in this disclosure regards the content image and the style image as two multivariate Gaussian distributions, and the goal is to transform one Gaussian distribution into another Gaussian distribution to the greatest extent. This problem can be regarded as a multivariate Monge-Kantorovich transportation problem. For example, transporting the goods in one city to another city, each city has different provinces, and it is necessary to minimize the transportation cost and find the optimal path. At this time, the cities correspond to different Gaussian distributions, the transportation cost represents the similarity between the two Gaussian distributions, and the optimal path is the color transfer transformation matrix from one Gaussian distribution to another Gaussian distribution. This color transfer transformation matrix can be applied to color transfer, and the pixel values of each pixel of the video frames of the video can be transformed, enabling the colors of the style image to be effectively mapped to the video frames and avoiding the loss of some colors in the style image.

[0055] To solve the problem in the related art that the video flickers after color transfer when performing color transfer on a video, the video image processing method proposed in this disclosure can splice the key video frames extracted from the video to obtain a content image, and perform color transfer using this content image as the representative of the entire video, which can ensure the continuity of the color distribution of the video frames obtained after color transfer and prevent the phenomenon that the previous video frame has a darker color and the subsequent video frame has a lighter color. Therefore, the phenomenon of flickering can be avoided.

[0056] To solve the problem in the related art that the contrast of a video frame after color transfer is relatively low when performing color transfer on a video, the video image processing method proposed by the present disclosure can match the histogram distribution of the content image of the video to the histogram distribution of the style image to obtain histogram linear transformation parameters, and perform a linear transformation on the pixel values of the pixels of the video frame, so that the contrast of each video frame matches the contrast of the style image, avoiding the situation of low contrast after the video frame undergoes color transfer.

[0057] Figure 1 FIG. is a flowchart showing a video image processing method according to an exemplary embodiment of the present disclosure.

[0058] Referring to Figure 1 , in step 101, a template image and a content image corresponding to the original video can be obtained. Here, the template image refers to an image whose color tone the user hopes to transfer to the target video. For example, the template image can be a landscape photo, a movie poster, etc. The content image is an image generated based on the image content of the video frames in the original video.

[0059] According to an exemplary embodiment of the present disclosure, the template image can be obtained from a local memory or a local database as needed or received from an external data source (such as the Internet, a server, a database, etc.) through an input device or a transmission medium.

[0060] According to an exemplary embodiment of the present disclosure, a predetermined number of video frames included in the original video can be extracted, and the predetermined number of video frames can be spliced into a single image as the content image.

[0061] In a specific implementation process, when extracting a predetermined number of video frames from the original video, it can be randomly extracted, extracted at a fixed time interval, extracted at a fixed position, extracted according to the image content, etc. For example, three video frames in the multiple video frames included in the original video can be extracted: the 10% video frame, the 50% video frame, and the 90% video frame. Then, these three video frames can be spliced to obtain a content image. In this way, the key video frames extracted from the video can be spliced to obtain a content image, and color transfer can be performed using this content image as a representative of the entire video, which can ensure that the color distribution of the video frames obtained after color transfer is continuous and there will be no phenomenon that the previous video frame has a darker color and the subsequent video frame has a lighter color. Therefore, the phenomenon of flickering can be avoided.

[0062] According to an exemplary embodiment of the present disclosure, after obtaining the content image, it is also necessary to compress the content image, and then perform subsequent processing on the compressed content image. Therefore, in order to ensure that the degree of deformation of the content image after compression is not too serious, that is, in order to ensure that the distortion degree of the content image after compression is not too serious, multiple key video frames can be spliced according to the width value and height value of the extracted key video frames. For example, when the width value of the key video frame is greater than the height value, if the key video frames are spliced horizontally, the content image obtained by splicing will be too wide, which may lead to a relatively serious distortion degree of the content image after compression. Therefore, at this time, multiple key video frames can be spliced vertically, which can reduce the distortion degree of the content image after compression; or, when the width value of the key video frame is less than the height value, if the key video frames are spliced vertically, the content image obtained by splicing will be too high, which may lead to a relatively serious distortion degree of the content image after compression. Therefore, at this time, multiple key video frames can be spliced horizontally, which can reduce the distortion degree of the content image after compression.

[0063] In step 102, based on the template image and the content image, the color transfer conversion information between the template image and the content image can be calculated.

[0064] According to an exemplary embodiment of the present disclosure, in order to solve the problem that colors will be lost after color transfer of a template image with rich colors, the color transfer of the content image can be regarded as a matching problem of a multivariate Gaussian distribution. The content image and the template image are regarded as two multivariate Gaussian distributions, and the goal is to change one Gaussian distribution to the other Gaussian distribution to the greatest extent. This problem can be regarded as a multivariate Monge-Kantorovich transportation problem to calculate the color transfer conversion information. For example, the covariance matrix of the content image and the covariance matrix of the template image can be calculated; based on the covariance matrix of the content image and the covariance matrix of the template image, a linear Monge-Kantorovich color transfer conversion matrix is calculated as the color transfer conversion information. In this way, when performing color transfer on a video, it is not just a simple matching of image mean and image variance, but rather a color transfer conversion matrix is calculated according to the covariance matrix of the content image and the covariance matrix of the template image, and the color transfer of the video is completed using the color transfer conversion matrix. This can effectively map the colors of the template image onto the video frames and avoid losing some colors in the template image.

[0065] In step 103, based on the color transfer conversion information, color transfer can be performed on the pixel values of each video frame of the original video to generate the target video.

[0066] According to an exemplary embodiment of the present disclosure, histogram linear transformation parameters can be calculated based on color transfer conversion information. Furthermore, based on the color transfer conversion information and the histogram linear transformation parameters, color transfer can be performed on the pixel values of each video frame of the original video, thereby generating a target video. In this way, the color transfer conversion information can avoid losing some colors in the template image. At the same time, the histogram linear transformation parameters can make the contrast of each video frame match the contrast of the template image, avoiding the situation of low contrast after the video frame undergoes color transfer.

[0067] According to an exemplary embodiment of the present disclosure, color transfer can be performed on the pixel values of the pixels of the content image based on the color transfer conversion information. Then, a first histogram corresponding to the content image after the pixel value color transfer and a second histogram corresponding to the template image can be obtained. Next, histogram linear transformation parameters for matching the first histogram to the second histogram can be calculated.

[0068] According to an exemplary embodiment of the present disclosure, a first left endpoint and a first right endpoint of the first histogram can be found, and a second left endpoint and a second right endpoint of the second histogram can be found. Then, the histogram linear transformation parameters when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint can be calculated.

[0069] According to an exemplary embodiment of the present disclosure, color transfer can be performed on the pixel values of each video frame of the original video based on the color transfer conversion information. Next, based on the histogram linear transformation parameters, a linear transformation can be performed on the pixel values after color transfer of each video frame of the original video to obtain a target video. In this way, performing color transfer on the pixel values of the video frame based on the color transfer conversion information can avoid losing some colors in the template image. At the same time, using the histogram linear transformation parameters to perform a linear transformation on the pixel values after color transfer of the video frame can make the contrast of each video frame match the contrast of the template image, avoiding the situation of low contrast after the video frame undergoes color transfer.

[0070] According to an exemplary embodiment of the present disclosure, a first color look-up table (LUT) may be generated based on color transfer conversion information. The first color look-up table may include inputs and corresponding outputs. The input may be an input pixel value, and the output may be a pixel value calculated from the input through the color transfer conversion information. Next, the pixel values of each video frame of the original video may be used as inputs to look up the first color look-up table to obtain the pixel values after color transfer for the pixels of each video frame. In this way, a first color look-up table including inputs and corresponding outputs may be generated in advance, and the color transfer for each video frame may be implemented by looking up the first color look-up table. Compared with the method of performing color transfer frame by frame, this look-up table method for implementing color transfer is faster and more efficient.

[0071] According to an exemplary embodiment of the present disclosure, a first color look-up table may be generated based on color transfer conversion information. The first color look-up table may include inputs and corresponding outputs. The input may be an input pixel value, and the output may be a pixel value calculated from the input through the color transfer conversion information. Then, a linear transformation may be performed on the inputs and corresponding outputs in the first color look-up table based on histogram linear transformation parameters to obtain a second color look-up table. Next, the pixel values of each video frame of the original video may be used as inputs to look up the second color look-up table to obtain the pixel values of the pixels of the target video. In this way, for each video frame in the original video, the second color look-up table may be used for color mapping, which can avoid losing some colors in the template image and can also make the contrast of each video frame match the contrast of the template image, avoiding the situation of low contrast after color transfer of the video frame. Moreover, compared with the method of separately calculating a color transfer conversion matrix for each video frame to implement color mapping, this look-up table method for implementing color mapping is faster and more efficient, and also avoids the problem of flicker in the video obtained after color mapping caused by performing color mapping frame by frame.

[0072] Next, reference will be made to Figure 2 to specifically describe the specific implementation process of the video image processing method according to an exemplary embodiment of the present disclosure. Figure 2 FIG. is a flowchart showing the implementation of the video image processing method according to an exemplary embodiment of the present disclosure. In Figure 2 this, the implementation process of the video image processing method according to an exemplary embodiment of the present disclosure mainly includes four parts: video frame extraction, color transfer, LUT generation, and video rendering.

[0073] Referring to Figure 2 , in step 201, the original video may be input. The original video may include multiple video frames.

[0074] In step 202, a template image can be input. This template image can also be referred to as a style image, and the template image can include multiple colors.

[0075] In step 203, video frames can be extracted. Multiple key video frames in the original video can be extracted. For example, as in the previous embodiment, three video frames can be extracted from the multiple video frames included in the original video: the 10% video frame, the 50% video frame, and the 90% video frame. Then, the extracted key video frames can be spliced together to obtain a content image.

[0076] In step 204, a color transfer conversion matrix is calculated. For example, as described in the previous embodiment, the color transfer conversion matrix can be calculated based on the covariance matrix of the content image and the covariance matrix of the template image. This color transfer conversion matrix is used to transfer the hue of the original video to the hue of the template image. Using this color transfer conversion matrix, the pixel values of the pixels of the content image or each video frame of the original video can be color transferred.

[0077] According to an exemplary embodiment of the present disclosure, the color transfer conversion matrix can be calculated by the following formula:

[0078]

[0079] where T is the color transfer conversion matrix, C u is the covariance matrix of the content image, and C v is the covariance matrix of the template image. In this way, using the color transfer conversion matrix to complete the color transfer of the video can effectively map the colors of the template image onto the video frames and avoid losing some colors in the template image. The present disclosure is not limited to the above formula, and any available color transfer conversion matrix can also be used.

[0080] According to an exemplary embodiment of the present disclosure, color transfer can be performed using the following formula:

[0081] I′ p = T × [I p - mean(I c )] + mean(I s ) (3)

[0082] where T is the color transfer conversion matrix, mean(I c ) is the mean of the pixel values of the content image, and mean(I s ) is the mean of the pixel values of the template image. In the case of performing color transfer on the pixel values of the pixels of each video frame, I′ p is the pixel value of the pixel of each video frame after pixel value color transfer, and I pThe pixel value of each pixel in a video frame; in the case of performing color transfer on the pixel values of the pixels in the content image, I′ p Is the pixel value of the pixel in the content image after pixel value color transfer, I p Is the pixel value of the pixel in the content image.

[0083] In step 205, the histogram linear transformation parameters can be calculated. The distribution of the histogram of the color-transferred content image can be stretched into the distribution of the histogram of the style image, so that the left endpoint of the histogram of the content image coincides with the left endpoint of the histogram of the style image, and the right endpoint of the histogram of the content image coincides with the right endpoint of the histogram of the style image, to obtain the histogram linear transformation parameters a, b.

[0084] For example, assume that the first left endpoint and the first right endpoint of the first histogram corresponding to the content image are Pc1 and Pc2 respectively; the second left endpoint and the second right endpoint of the second histogram corresponding to the template image are Ps1 and Ps2 respectively, and these two points can be coordinate points with a color level of 0 - 255. For example, Ps1 may be 2 and Ps2 may be 200.

[0085] A linear transformation Y = aX + b is required to move Pc1 to the position of Ps1 and Pc2 to the position of Ps2. That is, Pc1 and Pc2 can be regarded as X, and Ps1 and Ps2 can be regarded as Y, and substituting them into the equation Y = aX + b, the histogram linear transformation parameters can be obtained:

[0086] a = (Ps1 - Ps2) / (Pc1 - Pc2); (4)

[0087] b = Ps1 – (Ps1 - Ps2) / (Pc1 - Pc2)*Pc1 (5)

[0088] In step 206, a color lookup table can be generated, that is, a 3D LUT can be generated. Given a pixel input value, the corresponding pixel output value can be obtained by looking up the table. Based on the color transfer conversion matrix calculated in step 204 and the histogram linear transformation parameters calculated in step 205, the standard color lookup table LUT 标准 Is transformed to obtain a 3D LUT. In the field of color correction, the color lookup table has three components: R, G, and B. The input can be an RGB value. Corresponding to the input RGB value, the output pixel value can be obtained by looking up the color lookup table. The input and output of the color lookup table can be in a one-to-one correspondence relationship, and according to the corresponding relationship in the color lookup table, the pixel output value can be found for each pixel input value, thus completing the color mapping.

[0089] As described in the previous embodiment, a first color lookup table can be generated. Among them, the first color lookup table can be obtained through the following formula:

[0090] LUT 第一 = T × [LUT 标准 - mean(I c )] + mean(I s ) (6)

[0091] Wherein, LUT 第一 is the first color look-up table, LUT 标准 is the standard color look-up table, mean(I c ) is the average pixel value of the content image, mean(I s ) is the average pixel value of the template image, and T is the color transfer conversion matrix.

[0092] It should be noted that by using a LUT, a set of RGB values can be output as another set of RGB values, thereby changing the exposure and color of the image. For the standard color look-up table LUT 标准 , if the RGB of a pixel is (1, 2, 3), then its output value after applying this standard color look-up table LUT 标准 is still RGB(1, 2, 3), that is, the pixel input value of the standard color look-up table LUT 标准 is equal to the pixel output value.

[0093] Alternatively, as described in the previous embodiment, a second color look-up table can also be generated. The second color look-up table can be obtained through the following formula:

[0094] LUT 第二 = a * LUT 第一 + b (7)

[0095] Wherein, LUT 第二 is the second color look-up table, and a and b are the above-mentioned histogram linear transformation parameters.

[0096] In step 207, video rendering can be performed. The color of the template image can be mapped to each video frame of the original video by using the 3D LUT generated in step 206, that is, the color mapping of each video frame can be completed by looking up the table. Compared with the method of separately calculating the color transfer conversion matrix for each video frame to achieve color mapping, this method of looking up the table to achieve color mapping is faster, more efficient, and also avoids the problem of flicker in the video obtained after color mapping due to frame-by-frame color mapping.

[0097] In step 208, the target video can be output, that is, the target video obtained by performing color mapping on each video frame of the original video can be output.

[0098] Figure 3It is a block diagram showing a video image processing device according to an exemplary embodiment of the present disclosure.

[0099] Referring to Figure 3 , the device 300 may include an acquisition module 301, a calculation module 302, and a color transfer module 303.

[0100] The acquisition module 301 is configured to acquire a template image and a content image corresponding to the original video;

[0101] The calculation module 302 is configured to calculate color transfer conversion information between the template image and the content image based on the template image and the content image;

[0102] The color transfer module 303 is configured to perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information, thereby generating a target video.

[0103] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0104] Calculate histogram linear transformation parameters based on the color transfer conversion information;

[0105] Perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, thereby generating the target video.

[0106] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0107] Perform color transfer on the pixel values of the pixels of the content image based on the color transfer conversion information;

[0108] Obtain a first histogram corresponding to the content image after pixel value color transfer and a second histogram corresponding to the template image;

[0109] Calculate the histogram linear transformation parameters for matching the first histogram to the second histogram.

[0110] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0111] Find a first left endpoint and a first right endpoint of the first histogram and find a second left endpoint and a second right endpoint of the second histogram;

[0112] Calculate the histogram linear transformation parameters when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint.

[0113] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0114] Perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information;

[0115] Perform linear transformation on the pixel values after color transfer of the pixels of each video frame of the original video based on the histogram linear transformation parameters to obtain the target video.

[0116] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0117] Generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value calculated by the input through the color transfer conversion information;

[0118] Use the pixel values of the pixels of each video frame of the original video as the input, look up the first color lookup table, and obtain the pixel values after color transfer of the pixels of each video frame.

[0119] According to an exemplary embodiment of the present disclosure, the color transfer module 303 is configured to:

[0120] Generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value calculated by the input through the color transfer conversion information;

[0121] Perform linear transformation on the input and the corresponding output in the first color lookup table based on the histogram linear transformation parameters to obtain a second color lookup table;

[0122] Use the pixel values of the pixels of each video frame of the original video as the input, look up the second color lookup table, and obtain the pixel values of the pixels of the target video.

[0123] According to an exemplary embodiment of the present disclosure, the color transfer conversion information is a color transfer conversion matrix, and the color transfer module 303 is configured to:

[0124] Perform color transfer using the following formula:

[0125] I′ p = T × [I p - mean(I c )] + mean(I s )

[0126] where T is the color transfer conversion matrix, and mean(I c ) is the mean pixel value of the content image, and mean(I s ) is the mean pixel value of the template image;

[0127] When performing color transfer on the pixel values of the pixels of each video frame, I′ p is the pixel value of the pixel of each video frame after pixel value color transfer, and I p is the pixel value of the pixel of each video frame;

[0128] When performing color transfer on the pixel values of the pixels of the content image, I′ p is the pixel value of the pixel of the content image after pixel value color transfer, and I p is the pixel value of the pixel of the content image.

[0129] According to an exemplary embodiment of the present disclosure, the calculation module 302 is configured to:

[0130] Calculate the covariance matrix of the content image and the covariance matrix of the template image;

[0131] Based on the covariance matrix of the content image and the covariance matrix of the template image, calculate the color transfer conversion matrix as the color transfer conversion information.

[0132] According to an exemplary embodiment of the present disclosure, the calculation module 302 is configured to:

[0133] Calculate the color transfer conversion matrix through the following formula:

[0134]

[0135] where T is the color transfer conversion matrix, C u is the covariance matrix of the content image, and C v is the covariance matrix of the template image.

[0136] According to an exemplary embodiment of the present disclosure, the acquisition module 301 is configured to:

[0137] Extract a predetermined number of video frames from the multiple video frames included in the original video;

[0138] Stitch the predetermined number of video frames into a single image as the content image.

[0139] Figure 4 FIG. is a block diagram showing an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0140] Reference Figure 4 As shown in Figure 4 , the electronic device 400 includes at least one memory 401 and at least one processor 402. Instructions are stored in the at least one memory 401, and when the instructions are executed by the at least one processor 402, a video image processing method according to an exemplary embodiment of the present disclosure is executed.

[0141] As an example, the electronic device 400 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instructions. Here, the electronic device 400 does not have to be a single electronic device, but may also be a collection of devices or circuits that can execute the above instructions (or instruction sets) individually or jointly. The electronic device 400 may also be a part of an integrated control system or a system manager, or may be configured as a portable electronic device that can be interconnected locally or remotely (e.g., via wireless transmission).

[0142] In the electronic device 400, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0143] The processor 402 may run instructions or code stored in the memory 401. Here, the memory 401 may also store data. The instructions and data may also be sent and received via a network interface device over a network, where the network interface device may employ any known transmission protocol.

[0144] The memory 401 may be integrated with the processor 402. For example, RAM or flash memory may be arranged within an integrated circuit microprocessor, etc. In addition, the memory 401 may include a separate device, such as an external disk drive, a storage array, or other storage devices that can be used by any database system. The memory 401 and the processor 402 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, etc., such that the processor 402 can read files stored in the memory.

[0145] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0146] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above video image processing method. Examples of the computer-readable storage medium here include: read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc memory, hard disk drive (HDD), solid state drive (SSD), cartridge memory (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the computer program. The computer program in the above computer-readable storage medium may run in an environment deployed in computer devices such as clients, hosts, proxy devices, servers, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0147] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program which, when executed by a processor, implements the video image processing method according to the present disclosure.

[0148] According to the video image processing method and video image processing device of the present disclosure, various styles of color grading can be automatically performed. Just by providing a style image, the color tone of the content video can be automatically changed to the color tone of the style image, avoiding the cumbersome color grading process. Moreover, based on the color transfer conversion matrix calculated from the template image and the content image, the pixel values of the pixels of each video frame of the original video can be transformed. For example, applying the linear Monge-Kantorovich color transfer conversion matrix to the color transfer can effectively map the colors of the template image onto the video frame, avoiding the situation of losing some colors in the template image. In addition, the key video frames extracted from the video can be spliced together to obtain a content image, and color transfer can be performed with this content image as the representative of the entire video, which can ensure that the color distribution of the video frames obtained after color transfer is continuous, and there will be no phenomenon that the color of the previous video frame is darker and the color of the subsequent video frame is lighter, thus avoiding the phenomenon of flickering. In addition, the histogram distribution of the content image of the video can be matched to the histogram distribution of the style image to obtain the histogram linear transformation parameters, and the pixel values of the pixels of the video frame can be linearly transformed to make the contrast of each video frame match the contrast of the template image, avoiding the situation of low contrast after the video frame undergoes color transfer.

[0149] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0150] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A video image processing method, characterized in that, comprising: obtaining a template image and a content image corresponding to the original video; calculating color transfer conversion information between the template image and the content image based on the template image and the content image; performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information, thereby generating a target video; wherein the color transfer conversion information is a color transfer conversion matrix, and color transfer is performed using the following formula: Among them, is the color transfer conversion matrix, is the average pixel value of the content image, is the average pixel value of the template image; In the case of performing color transfer on the pixel values of the pixels of each video frame, is the pixel value of the pixel of each video frame after pixel value color transfer, is the pixel value of the pixel of each said video frame; In the case of performing color transfer on the pixel values of the pixels of the content image, is the pixel value of the pixel of the content image after pixel value color transfer, is the pixel value of the pixel of the content image.

2. The video image processing method according to claim 1, characterized in that, the performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information, thereby generating a target video, includes: calculating histogram linear transformation parameters based on the color transfer conversion information; performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, thereby generating the target video.

3. The video image processing method according to claim 2, characterized in that, the calculating histogram linear transformation parameters based on the color transfer conversion information, includes: performing color transfer on the pixel values of the pixels of the content image based on the color transfer conversion information; obtaining a first histogram corresponding to the content image after pixel value color transfer and a second histogram corresponding to the template image; calculating the histogram linear transformation parameters for matching the first histogram to the second histogram.

4. The video image processing method according to claim 3, characterized in that, the calculating the histogram linear transformation parameters for matching the first histogram to the second histogram, includes: finding a first left endpoint and a first right endpoint of the first histogram and finding a second left endpoint and a second right endpoint of the second histogram; calculating the histogram linear transformation parameters when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint.

5. The video image processing method according to claim 2, characterized in that, the performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, thereby generating the target video, includes: performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information; performing a linear transformation on the pixel values after color transfer of the pixels of each video frame of the original video based on the histogram linear transformation parameters, to obtain the target video.

6. The video image processing method according to claim 1, characterized in that, the performing color transfer on the pixel values of each video frame of the original video based on the color transfer conversion information, includes: Generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes inputs and corresponding outputs, the inputs are input pixel values, and the outputs are pixel values calculated from the inputs through the color transfer conversion information; Use the pixel values of the pixels of each video frame of the original video as inputs, and look up the first color lookup table to obtain the pixel values after color transfer of the pixels of each video frame.

7. The video image processing method according to claim 2, wherein, performing color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters includes: Generate a first color lookup table according to the color transfer conversion information, where the first color lookup table includes inputs and corresponding outputs, the inputs are input pixel values, and the outputs are pixel values calculated from the inputs through the color transfer conversion information; Perform a linear transformation on the inputs and corresponding outputs in the first color lookup table based on the histogram linear transformation parameters to obtain a second color lookup table; Use the pixel values of the pixels of each video frame of the original video as inputs, and look up the second color lookup table to obtain the pixel values of the pixels of the target video.

8. The video image processing method according to claim 1 or 2, wherein, calculating the color transfer conversion information between the template image and the content image based on the template image and the content image includes: Calculating the covariance matrix of the content image and the covariance matrix of the template image; Calculating a color transfer conversion matrix based on the covariance matrix of the content image and the covariance matrix of the template image as the color transfer conversion information.

9. The video image processing method according to claim 8, wherein, calculating the color transfer conversion matrix based on the covariance matrix of the content image and the covariance matrix of the template image includes: Calculating the color transfer conversion matrix through the following formula: Among them, is the color transfer conversion matrix, is the covariance matrix of the content image, is the covariance matrix of the template image.

10. The video image processing method according to claim 1 or 2, wherein, obtaining the content image corresponding to the original video includes: Extracting a predetermined number of video frames from the multiple video frames included in the original video; Stitching the predetermined number of video frames into a single image as the content image.

11. A video image processing apparatus, wherein, comprising: An acquisition module configured to acquire a template image and a content image corresponding to an original video; A calculation module configured to calculate color transfer conversion information between the template image and the content image based on the template image and the content image; A color transfer module configured to perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information to generate a target video; wherein the color transfer conversion information is a color transfer conversion matrix, and the color transfer module is configured to: Perform color transfer using the following formula: Among them, is the color transfer conversion matrix, is the average pixel value of the content image, is the average pixel value of the template image; In the case of performing color transfer on the pixel values of the pixels of each video frame, is the pixel value of the pixel of each video frame after pixel value color transfer, is the pixel value of the pixel of each said video frame; In the case of performing color transfer on the pixel values of the pixels of the content image, is the pixel value of the pixel of the content image after pixel value color transfer, is the pixel value of the pixel of the content image.

12. The video image processing device according to claim 11, wherein, the color transfer module is configured to: calculate histogram linear transformation parameters based on the color transfer conversion information; perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information and the histogram linear transformation parameters, so as to generate the target video.

13. The video image processing device according to claim 12, wherein, the color transfer module is configured to: perform color transfer on the pixel values of the pixels of the content image based on the color transfer conversion information; acquire a first histogram corresponding to the content image after pixel value color transfer and a second histogram corresponding to the template image; calculate the histogram linear transformation parameters for matching the first histogram to the second histogram.

14. The video image processing device according to claim 13, wherein, the color transfer module is configured to: find a first left endpoint and a first right endpoint of the first histogram and find a second left endpoint and a second right endpoint of the second histogram; calculate the histogram linear transformation parameters when the first left endpoint coincides with the second left endpoint and the first right endpoint coincides with the second right endpoint.

15. The video image processing device according to claim 12, wherein, the color transfer module is configured to: perform color transfer on the pixel values of the pixels of each video frame of the original video based on the color transfer conversion information; perform linear transformation on the pixel values after color transfer of the pixels of each video frame of the original video based on the histogram linear transformation parameters to obtain the target video.

16. The video image processing device according to claim 11, wherein, the color transfer module is configured to: generate a first color lookup table according to the color transfer conversion information, wherein the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value calculated by the input through the color transfer conversion information; use the pixel values of the pixels of each video frame of the original video as the input, and look up the first color lookup table to obtain the pixel values after color transfer of the pixels of each video frame.

17. The video image processing device according to claim 12, wherein, the color transfer module is configured to: generate a first color lookup table according to the color transfer conversion information, wherein the first color lookup table includes an input and a corresponding output, the input is an input pixel value, and the output is a pixel value calculated by the input through the color transfer conversion information; perform linear transformation on the input and the corresponding output in the first color lookup table based on the histogram linear transformation parameters to obtain a second color lookup table; use the pixel values of the pixels of each video frame of the original video as the input, and look up the second color lookup table to obtain the pixel values of the pixels of the target video.

18. The video image processing device according to claim 11 or 12, wherein, the calculation module is configured to: calculate the covariance matrix of the content image and the covariance matrix of the template image; calculate a color transfer conversion matrix based on the covariance matrix of the content image and the covariance matrix of the template image as the color transfer conversion information.

19. The video image processing device according to claim 18, wherein, the calculation module is configured to: calculate the color transfer conversion matrix through the following formula: Among them, is the color transfer conversion matrix, is the covariance matrix of the content image, is the covariance matrix of the template image.

20. The video image processing device according to claim 11 or 12, wherein, the acquisition module is configured to: extract a predetermined number of video frames from the multiple video frames included in the original video; stitch the predetermined number of video frames into a single image as the content image.

21. An electronic device, wherein, it includes: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to execute the instructions to implement the video image processing method according to any one of claims 1 to 10.

22. A computer-readable storage medium, wherein, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video image processing method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, wherein, the computer program, when executed by a processor, implements the video image processing method according to any one of claims 1 to 10.

Images