Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium, and the method comprises the steps: obtaining a first image and a second image, the first image being a content reference image, and the second image being a color reference image; performing first processing on the first image to obtain a plurality of sub-images, the plurality of sub-images having different frequency band information, the plurality of sub-images including a first sub-image and at least one second sub-image, the first sub-image having low frequency information of the first image; performing color style migration processing on the second image and the first sub-image to obtain a color migration image; and performing second processing on the color migration image and the at least one second sub-image to obtain an output image. According to the invention, while the quality of the output image is ensured, the operation complexity is reduced, and the image processing time consumption is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image editing, and in particular to an image processing method, device, electronic device, chip, and storage medium. Background Art

[0002] Color transfer, a computer vision technique, is an image editing and processing technique. Its primary purpose is to apply the color style of one image to another, making the latter resemble the original while preserving the image's content and details. Related color transfer solutions suffer from low computational efficiency and long processing times. Summary of the Invention

[0003] The present disclosure provides an image processing method, device, electronic device, chip and storage medium, which can reduce the complexity of calculation and improve the calculation efficiency by performing color migration on low-frequency images.

[0004] A first aspect embodiment of the present disclosure proposes an image processing method, which includes: acquiring a first image and a second image, where the first image is a content reference reference and the second image is a color reference reference; performing a first processing on the first image to obtain a plurality of sub-images, where the plurality of sub-images have different frequency band information, and the plurality of sub-images include a first sub-image and at least one second sub-image, where the first sub-image has low-frequency information of the first image; performing a color style transfer processing on the second image and the first sub-image to obtain a color transferred image; and performing a second processing on the color transferred image and the at least one second sub-image to obtain an output image.

[0005] In some embodiments of the present disclosure, performing a first processing on the first image to obtain multiple sub-images includes: performing discrete wavelet transform and / or average pooling processing on the first image to obtain multiple sub-images, and the length and width of the multiple sub-images are half the length and width of the first image.

[0006] In some embodiments of the present disclosure, color style migration is performed on the second image and the first sub-image to obtain a color migrated image, including: adjusting the size of the second image to be consistent with the size of the first sub-image; using a block histogram matching algorithm to perform color style migration on the second image and the first sub-image to obtain an initial migration image; using an image gradient recovery algorithm to perform noise and fault processing on the initial migration image to obtain a color migrated image.

[0007] In some embodiments of the present disclosure, the method further includes: aligning the pixel mean and pixel standard deviation of the color migration image to the pixel mean and pixel standard deviation of the second image; and performing sharpening processing on the color migration image.

[0008] In some embodiments of the present disclosure, performing a second processing on the color migration image and the at least one second sub-image to obtain an output image includes: performing discrete wavelet inverse transform and / or nearest neighbor interpolation processing on the color migration image and the at least one second sub-image to obtain an output image, and the size of the output image is consistent with the size of the first image.

[0009] In some embodiments of the present disclosure, the method also includes: post-processing the output image using a trained image enhancement model, wherein the image enhancement model is trained using a first training set, the training data in the first training set includes a first image, an output image, and a target image corresponding to the first image, and the image quality of the target image meets preset requirements.

[0010] A second aspect embodiment of the present disclosure proposes an image processing device, which includes: an acquisition unit, used to acquire a first image and a second image, the first image being a content reference reference and the second image being a color reference reference; a first processing unit, used to perform a first processing on the first image to obtain a plurality of sub-images, the plurality of sub-images having different frequency band information, the plurality of sub-images including a first sub-image and at least one second sub-image, the first sub-image having low-frequency information of the first image; a second processing unit, used to perform a color style transfer processing on the second image and the first sub-image to obtain a color transferred image; and a third processing unit, used to perform a second processing on the color transferred image and the at least one second sub-image to obtain an output image.

[0011] In some embodiments of the present disclosure, the first processing unit may also be configured to perform discrete wavelet transform and / or average pooling processing on the first image to obtain multiple sub-images, where the length and width of the multiple sub-images are half of the length and width of the first image.

[0012] In some embodiments of the present disclosure, the second processing unit can also be used to adjust the size of the second image to be consistent with the size of the first sub-image; use the block histogram matching algorithm to perform color style migration processing on the second image and the first sub-image to obtain an initial migration image; use the image gradient recovery algorithm to perform noise and fault processing on the initial migration image to obtain a color migration image.

[0013] In some embodiments of the present disclosure, the image processing apparatus further includes a fourth processing unit configured to align the pixel mean and pixel standard deviation of the color migration image to the pixel mean and pixel standard deviation of the second image; and perform sharpening processing on the color migration image.

[0014] In some embodiments of the present disclosure, the third processing unit can also be used to perform discrete wavelet inverse transform and / or nearest neighbor interpolation processing on the color migration image and at least one second sub-image to obtain an output image, and the size of the output image is consistent with the size of the first image.

[0015] In some embodiments of the present disclosure, the image processing device also includes a fifth processing unit, which can be used to post-process the output image using a trained image enhancement model, wherein the image enhancement model is trained using a first training set, and the training data in the first training set includes a first image, an output image, and a target image corresponding to the first image, and the image quality of the target image meets preset requirements.

[0016] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect embodiment of the present disclosure.

[0017] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0018] The fifth aspect embodiment of the present disclosure proposes a chip, which includes at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, and the processor communicates with the communication interface and implements the method described in the first aspect embodiment of the present disclosure through logic circuits or executing code instructions.

[0019] In summary, the image processing method, device, electronic device, chip and storage medium proposed in the present disclosure can reduce the complexity of calculation and improve the calculation efficiency by performing color migration on the low-frequency first sub-image to obtain a color migrated image, and obtain an output image based on the color migrated image and the second sub-image.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0022] Figure 1 A flowchart of an image processing method provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of an image processing method provided in an embodiment of the present disclosure;

[0024] Figure 3A flowchart of an image processing method provided in an embodiment of the present disclosure;

[0025] Figure 4 A flowchart of an image processing method provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a discrete wavelet transform and an inverse discrete wavelet transform provided in an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of an input image and an output image of an image enhancement model provided in an embodiment of the present disclosure;

[0028] Figure 7a A local magnified image of an output image of the prior art provided in an embodiment of the present disclosure;

[0029] Figure 7b A partial magnified image of an output image provided by the embodiment of the present disclosure;

[0030] Figure 8 An output image comparison diagram provided by an embodiment of the present disclosure;

[0031] Figure 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present disclosure;

[0032] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0033] Figure 11 A schematic diagram of the chip structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0035] Color transfer belongs to the field of computer vision and is an image editing and processing technique. Its primary purpose is to apply the color style of one image to another, making the latter image similar in color style to the former while preserving the image content and details. Related technologies include color transfer algorithms based on traditional image processing techniques, such as histogram matching. With the development of deep learning technology, color transfer technology has also made significant progress. Deep learning-based color transfer algorithms can automatically learn the color characteristics of an image and apply them to other images, achieving more accurate and natural color transfer effects.

[0036] For example, a related image processing scheme may first use a color transfer algorithm based on a cumulative distribution function to take the input as a content reference. Figure X The color distribution of the image is transferred to the color distribution of the input image R, which serves as a color style reference. This results in an output image Y with content that is essentially consistent with X and style that is essentially consistent with R. An image gradient restoration algorithm called regrain is then used on Y to remove noise or color discontinuities introduced during the color transfer process. However, this solution suffers from low computational efficiency and a long algorithmic time consumption.

[0037] In summary, in order to solve the technical problems in the related art, the embodiment of the present disclosure provides an image processing method. For example, the method can be applied in a single-machine scene during human-computer interaction. When the user uses the image color migration function, two images are uploaded, one of which is used as the input for the image content reference. Figure X , a grayscale image R is used as a color style reference. By running the color style transfer algorithm on the grayscale reference image on a single machine, the final grayscale image output Z is obtained, where the image details of Z are consistent with X and the image style is consistent with R.

[0038] The specific contents of the plan are as follows.

[0039] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method can be executed by an electronic device, optionally, the method can be executed by a network device, optionally, the method can be executed by a server. The method can include the following steps.

[0040] Step 101: Acquire a first image and a second image.

[0041] In some embodiments, the first image is a content reference reference, and the second image is a color reference reference. For example, the first image can be represented as image X, and the second image can be represented as image R. The first image and the second image are used as reference references respectively, so that the output image content is consistent with the first image, and the color style of the output image is consistent with the second image.

[0042] In some embodiments, the first image and the second image may be two images with the same content but different color styles. The first image and the second image may be of the same size. If they are not the same, pre-processing may be performed after the two images are acquired.

[0043] In some embodiments, the first image and the second image may be grayscale images. For example, a color RGB image serving as a content reference image may be converted into a grayscale image to obtain the grayscale image as the first image, and a color RGB image serving as a color reference image may be converted into a grayscale image to obtain the grayscale image as the second image.

[0044] Step 102: Perform a first process on the first image to obtain a plurality of sub-images.

[0045] In some embodiments, the first processing may include performing discrete wavelet transform (DWT) and / or average pooling processing on the first image to obtain a plurality of sub-images.

[0046] In some embodiments, the length and width of the multiple sub-images are smaller than the length and width of the first image. For example, the length and width of the multiple sub-images may be half of the length and width of the first image, that is, four sub-images are obtained after the first processing of the first image.

[0047] In some embodiments, the multiple sub-images have information in different frequency bands, and the multiple sub-images include a first sub-image and at least one second sub-image, wherein the first sub-image has low-frequency information of the first image. For example, the first image can be processed according to different frequency bands (high frequency, mid-frequency, and low frequency) to obtain sub-images in different frequency bands. Optionally, the low-frequency information contains most of the color information of the first image.

[0048] For example, the first sub-image can be represented by X_LL. Since most of the image information is low-frequency information, and the first sub-image has the low-frequency information of the first image, that is, the first sub-image contains most of the color information of the first image, color migration based on the first sub-image can reduce the amount of calculation and improve the color style migration rate while ensuring the color style migration effect.

[0049] For example, at least one second sub-image contains intermediate frequency information and high frequency information of the first image. The second sub-image can be, for example, X_LH, X_HL, and X_HH, where X_LH and X_HL are second sub-images containing intermediate frequency information of the first image, and X_HH is a second sub-image containing high frequency information of the first image.

[0050] In some embodiments, the first sub-image can be obtained by discrete wavelet transform, and its specific formula is:

[0051] Img_LL ij =0.5×(1 mg 2i,2j +Img 2i,2j+1 +Img 2i+1,2j +Img 2i+1,2j+1 )

[0052] =2×(0.25×(1 mg 2i,2j +Img 2i,2j+1 +Img 2i+1,2j +Img 2i+1,2j+1 )

[0053] =2×[mean_pool(Img)] ij

[0054] Among them, Img_LL ij is the pixel in the i-th row and j-th column of the first sub-image, i.e., i and j represent the row and column numbers of the first sub-image, respectively. Since the size of the first sub-image is one-fourth of the first image (the length and width are half of the first image), 2i and 2j represent the row and column numbers of the first image, respectively.

[0055] In particular, the first sub-image can also be obtained through average pooling processing. For example, mean_pool is the average pooling processing, which not only achieves the same processing effect as the wavelet transform, but also makes the calculation process faster and simpler. It can complete color migration without running the complete wavelet algorithm, greatly saving computing time.

[0056] Step 103: Perform color style transfer processing on the second image and the first sub-image to obtain a color-transferred image.

[0057] In some embodiments, the color style of the second sub-image may be transferred to the first sub-image to obtain a color-transferred image.

[0058] In some embodiments, the color style transfer process may include a block histogram matching algorithm, an image gradient recovery algorithm, etc., which is not limited by the present disclosure.

[0059] In some embodiments, before performing the color style transfer process, the second image and the first sub-image may be processed so that the sizes of the two are consistent.

[0060] By performing color style transfer on the second image and the first sub-image containing only low-frequency information of the first image, the color information of the first image is retained, the amount of computation for color style transfer is reduced, and the processing speed is improved.

[0061] Step 104: Perform a second process on the color migration image and the at least one second sub-image to obtain an output image.

[0062] In some embodiments, the second processing may be performing discrete wavelet inverse transform and / or nearest neighbor interpolation processing on the at least one second sub-image to obtain an output image.

[0063] In some embodiments, in order to ensure the integrity of the image content, after the second image and the first sub-image are color migrated, the migrated image is processed with at least one second sub-image to obtain an output image that is consistent with the content of the first image and the color style of the second image.

[0064] In some embodiments, the output image maintains the same size as the original image content (the first image and / or the second image).

[0065] In summary, the above-mentioned embodiments of the present application process the first image to obtain a first sub-image having low-frequency information of the first image, and perform color style transfer on the first sub-image according to the second image to obtain a color transferred image, which can reduce the amount of calculation and improve the color transfer rate. By performing a second processing on the color transferred image and the second image, the content of the output image can be made consistent with the second image, thereby ensuring the quality of the output image.

[0066] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, based on Figure 1 In the embodiment shown, this embodiment specifically describes the above step 103. The method may include the following steps.

[0067] Step 201: Adjust the size of the second image to be consistent with the size of the first sub-image.

[0068] In some embodiments, when performing color style transfer processing on the second image and the first sub-image, the size of the second image can be first adjusted to be consistent with the size of the first sub-image. Since the second image is a reference image of the color style, when the color style of the second image is transferred to the first sub-image, the color styles between each pixel point of the second image and the first sub-image should correspond to each other, so their sizes need to be adjusted to be consistent to facilitate corresponding processing.

[0069] Step 202: Perform color style transfer processing on the second image and the first sub-image using a block histogram matching algorithm to obtain an initial transferred image.

[0070] In some embodiments, a block histogram matching algorithm can be used to obtain an initial migration image based on the second image and the first sub-image. The image content of the initial migration image is consistent with the first sub-image, and the color style of the initial migration image is consistent with the second image.

[0071] Step 203: Using an image gradient restoration algorithm, perform noise and fault processing on the initial migration image to obtain a color migration image.

[0072] In some embodiments, for example, the color migration image can be represented as Y_LL, and the image gradient recovery algorithm Regrain can be used to eliminate the noise and color faults in the color migration image Y_LL that are amplified by the histogram matching algorithm, ensuring that the color migration image and the first image are as consistent as possible in image details and structure.

[0073] In some embodiments, aligning the pixel mean and pixel standard deviation of the color-shifted image to the pixel mean and pixel standard deviation of the second image can reduce the difference in color distribution between the color-shifted image and the second image.

[0074] In some embodiments, the color migration image can be sharpened according to the pixel mean and pixel standard deviation of the second image, which can make the details of the color migration image clearer and further improve the consistency of the color style of Y_LL and the color style of the second image. The specific calculation formula for sharpening is as follows:

[0075] Y_LL=sharpen[Y_LL-mean(Y_LL)) / std(Y_LL)1std(R)+mean(R)]

[0076] In the above formula, sharpen means sharpening, mean means taking the average value, and std means taking the standard deviation.

[0077] In summary, the above-mentioned embodiments of the present application, by using the second image and the first sub-image to perform color style transfer, can reduce the amount of calculation, reduce the complexity of calculation, and improve the efficiency of color style transfer. By utilizing the image gradient recovery algorithm, the initial transfer image is subjected to noise and fault processing, and the color transfer image is sharpened according to the pixel mean and pixel standard deviation of the second image, the color transfer image and the first image can be made as consistent as possible in image details and structure.

[0078] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, based on Figure 1 In the embodiment shown, this embodiment specifically describes the above step 104. The method may include the following steps.

[0079] Step 301 : Perform inverse discrete wavelet transform (IDWT) and / or nearest neighbor interpolation processing on the color migration image and at least one second sub-image to obtain an output image.

[0080] In some embodiments, the color migration image and at least one second sub-image may be subjected to an inverse discrete wavelet transform and / or nearest neighbor interpolation process to obtain an output image. Specifically, the color migration image and the second sub-images containing other frequency band information may be combined to perform an inverse discrete wavelet transform and / or nearest neighbor interpolation process to obtain an output image. Optionally, the output image may be represented by Y.

[0081] In some embodiments, the size of the output image obtained by the above discrete wavelet inverse transform and / or nearest neighbor interpolation process is consistent with the size of the first image. The inverse wavelet transform is the inverse process of the wavelet transform. When the first image is processed by the wavelet transform, the inverse wavelet transform can be used to adjust the size of the color migration image to be consistent with the first image. The specific process of the inverse wavelet transform is as follows:

[0082] Img 2i,2j =0.5×(Img_LL ij +Img_HL ij +Img_LH ij +Img_HH ij )

[0083] Img 2i,2j+1 =0.5×(Img_LL ij -Img_HL ij +Img_LH ij -Img_HH ij )

[0084] Img 2i+1,2j =0.5×(Img_LL ij +Img_HL ij -Img_LH ij -Img_HH ij )

[0085] Img 2i+1,2j+1 =0.5×(Img_LL ij -Img_HL ij -Img_LH ij +Img_HH ij )

[0086] In some embodiments, since this solution only performs color migration on the first sub-image X_LL of low-frequency information to obtain the output image color migration image, other frequency band information remains unchanged, and the difference between the final output image and the first image is also only related to the difference between the color migration image Y_LL and the first sub-image X_LL, that is,

[0087] (YX) 2i,2j =(YX) 2i,2j+1 =(YX) 2i+1,2j =(YX) 2i+1,2j+1 =0.5×(Y_LL ij -X_LL ij )

[0088] (YX) i,j =0.5×[upsample_nearest(Y_LL-X_LL)]i,j

[0089] Here, upsample_nearest indicates upsampling the neighboring points of the pixel. According to either of the above two formulas, the output image Y can be obtained according to the difference between the color migration image Y_LL and the first sub-image X_LL, as well as the first image X.

[0090] For example, when average pooling is used to process the first image, a nearest neighbor interpolation algorithm can be used to obtain the output image Y. The nearest neighbor interpolation algorithm specifically includes subtracting the color migration image from the first sub-image to obtain a difference value △_LL. The difference value △_LL is then interpolated using the nearest neighbor interpolation algorithm to obtain a difference image △ with the same size as the first image. The difference image is then superimposed with the first image, that is, X and △ are added to obtain the output image Y. The specific formula is as follows:

[0091] (YX) i,j =0.5×[upsample_nearest(Y_LL-X_LL)] i,j =0.5×[upsample_nearest(2×[mean_pool(Y)]-2×[mean_pool(X)]) i,j =[upsample_nearest(mean_pool(Y)-mean_pool(X))] i,j

[0092] Y=X+[upsample_nearest(color_transfer(mean_pool(X))-mean_pool(X))] i,j

[0093] According to the above formula, the average pooling process and the nearest neighbor interpolation algorithm can be used to obtain the output image Y.

[0094] In summary, the above-mentioned embodiments of the present application obtain an output image by performing discrete wavelet inverse transform and / or nearest neighbor interpolation processing on the color migration image and at least one second sub-image, which can keep the output image consistent with the content and size of the first image, thereby ensuring the effect of the output image.

[0095] Figure 4 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, based on Figure 3 In the embodiment shown, the method may further include the following steps.

[0096] Step 401: Post-process the output image using the trained image enhancement model.

[0097] In some embodiments, in order to further eliminate the noise introduced by the color style transfer algorithm and wavelet transform and inverse wavelet transform, as well as to enhance image details and contrast, and improve image quality and visual effects, a trained image enhancement model can be used to post-process the output image.

[0098] In some embodiments, the image enhancement model is trained using a first training set, where the training data in the first training set includes a first image, an output image, and a target image corresponding to the first image, where the image quality of the target image meets preset requirements. For example, the image enhancement model may utilize a neural network model based on a U-Net architecture.

[0099] In some embodiments, the target image can be used as a training target for the model. By comparing the output image with the target image, the image loss can be calculated, and the Adam optimizer is used for iterative updates to train the model.

[0100] In some embodiments, the image enhancement model can perform denoising and detail enhancement on the output image obtained in the above embodiment to obtain a grayscale image with the same size as the above output image.

[0101] In summary, the above embodiments of the present application, by utilizing a trained image enhancement model to denoise and enhance the output image, can enhance image details and contrast, improve image quality and visual effects, so that the solution can reduce the amount of computation while ensuring the quality of the output image.

[0102] For the sake of simplicity, the aforementioned method embodiments are all expressed as a series of action combinations. However, those skilled in the art should know that the present disclosure is not limited to the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0103] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0104] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.

[0105] This example proposes a color migration solution for grayscale reference images. By combining multiple algorithms, including wavelet transform principles, histogram matching, image gradient recovery algorithms, and deep learning-based image enhancement, the solution ensures that the details of the output image Z are consistent with the content reference image X and are less noisy, while the color style is consistent with the style reference image R. By combining the strengths of multiple algorithms, the algorithm's efficiency is optimized, the overall algorithm time is reduced, and the output quality is high and stable.

[0106] The specific content of this example scenario is as follows.

[0107] 1. First, convert the RGB input image used as the content reference to grayscale Figure X , perform a discrete wavelet transform on X to obtain the original length and width Figure 1 The four images in the middle record information of different frequency bands and are denoted as X_LL, X_LH, X_HL, and X_HH, where X_LL records the low-frequency information of the input image X. The grayscale input image R, which serves as a color style reference, is resized to be consistent with X_LL. Figure 5 Schematic diagram of discrete wavelet transform and inverse discrete wavelet transform, where 2dDWT represents a two-dimensional discrete wavelet transform of X, and 2dIDWT represents a two-dimensional inverse discrete wavelet transform of X.

[0108] 2. Perform color style transfer on a small-scale image X_LL: Use block histogram matching to obtain a color transfer result Y_LL whose color style is consistent with the reference grayscale image R. Then, use the image gradient recovery algorithm Regrain to remove the noise and color discontinuities in Y_LL amplified by the histogram matching algorithm, ensuring that Y_LL and X_LL maintain the greatest possible consistency in image detail and structure.

[0109] 3. Combine Y_LL with the other frequency band information X_LH, X_HL, and X_HH obtained in the first step to perform inverse discrete wavelet transform to obtain the color style transfer result Y. By taking advantage of the fact that color information is basically low-frequency information, the color style of Y is guaranteed to remain consistent with the color style of the image Y_LL that records low-frequency information, which means it is consistent with the color style of the input image R. The introduction of other frequency band information X_LH, X_HL, and X_HH ensures that the content of image Y is consistent with the input image R. Figure X The content is consistent.

[0110] 4. To further eliminate the noise introduced by the color style transfer algorithm, discrete wavelet transform, and inverse discrete wavelet transform, as well as to enhance image detail and contrast, and improve image quality and visual effects, this solution trains and uses an image enhancement model to post-process image Y to obtain the final color style transfer result image Z.

[0111] However, since we do not modify the three sub-band images X_LH, X_HL, and X_HH obtained by wavelet transform, we can use the average pooling algorithm and the nearest neighbor interpolation algorithm to equivalent the discrete wavelet transform and discrete wavelet inverse transform process, saving algorithm time. The specific equivalent process is described as follows:

[0112] The calculation formula for obtaining low-frequency information through wavelet transform is as follows:

[0113] Img_LLij =0.5×(1 mg 2i,2j +Img 2i,2j+1 +Img 2i+1,2j +Img 2i+1,2j+1 )

[0114] =2×(0.25×(1 mg 2i,2j +Img 2i,2j+1 +Img 2i+1,2j +Img 2i+1,2j+1 )

[0115] =2×[mean_pool(Img)] ij

[0116] The inverse wavelet transform process is shown in the following formula:

[0117] Img 2i,2j =0.5×(Img_LL ij +Img_HL ij +Img_LH ij +Img_HH ij )

[0118] Img 2i,2j+1 =0.5×(Img_LL ij -Img_HL ij +Img_LH ij -Img_HH ij )

[0119] Img 2i+1,2j =0.5×(Img_LL ij +Img_HL ij -Img_LH ij -Img_HH ij )

[0120] Img 2i+1,2j+1 =0.5×(Img_LL ij -Img_HL ij -Img_LH ij +Img_HH ij )

[0121] Since we only perform color migration on the low-frequency information X_LL to obtain Y_LL, the other frequency band information remains unchanged. The final difference between Y and X is only related to the difference between Y_LL and X_LL.

[0122] (YX) 2i,2j =(YX) 2i,2j+1 =(YX) 2i+1,2j =(YX) 2i+1,2j+1 =0.5×(Y_LLij -X_LL ij )

[0123] That is, (YX) i,j =0.5×[upsample_nearest(Y_LL-X_LL)] i,j

[0124] Therefore, Y can actually be obtained more quickly from X and the color transfer algorithm (color_transfer) in the second step of the theoretical process through the nearest neighbor interpolation algorithm (upsample_nearest) and the average pooling algorithm (mean_pool), without running the full wavelet transform algorithm. The formula is as follows:

[0125] (YX) i,j =0.5×[upsample_nearest(Y_LL-X_LL)] i,j =0.5×[upsample_nearest(2×[mean_pool(Y)]-2×[mean_pool(X)]) i,j =[upsample_nearest(mean_pool(Y)-mean_pool(X))] i,j

[0126] Y=X+[upsample_nearest(color_transfer(mean_pool(X))-mean_pool(X))] i,j therefore,

[0127] The complete solution of replacing the full wavelet algorithm with the nearest neighbor interpolation algorithm and the average pooling algorithm is as follows:

[0128] 1. First, convert the RGB input image used as the content reference to grayscale Figure X , perform average pooling on X to obtain the low-frequency information X_LL, and then adjust the size of the grayscale input image R as a color style reference to be consistent with X_LL.

[0129] 2. Perform color style transfer on a small-scale image X_LL: Use block histogram matching to create a color transfer result Y_LL whose color style is consistent with the reference grayscale image R. Then, use the image gradient recovery algorithm Regrain to remove the noise and color discontinuities in Y_LL amplified by histogram matching, ensuring that Y_LL and X_LL maintain the greatest possible consistency in image detail and structure.

[0130] 3. To further improve the color style consistency between Y_LL and the color style reference image R, Y_LL is processed using the mean and standard deviation of the reference image R, and Y_LL is sharpened to obtain the final low-frequency image Y_LL:

[0131] Y_LL=sharpen[Y_LL-mean(Y_LL)) / std(Y_LL)1std(R)+mean(R)]

[0132] 4. Use the nearest neighbor interpolation algorithm to interpolate the difference △_LL obtained by Y_LL - X_LL to obtain a difference image △ with the same size as X. Then add X and △ to obtain the color style transfer result Y.

[0133] In order to further eliminate the noise introduced by the color style transfer algorithm and wavelet transform and inverse transform, as well as enhance image details and contrast, and improve image quality and visual effects, this solution trains and uses an image enhancement model to post-process image Y to obtain the final color style transfer result image Z.

[0134] Improve the image quality and visual effects through image enhancement models. Figure 6 As shown in Figure 2, the specific contents of the image enhancement model are as follows:

[0135] This solution uses a neural network model based on the U-Net architecture to train an image enhancement model. The training input is a 256x256 synthetic, noisy, low-quality grayscale image. The output is also a 256x256 grayscale image. The L1 loss is calculated compared with the given label (the clean grayscale image). The Adam optimizer is used for iterative updates. The training data volume is approximately 10,000 pairs of samples. Through this process, the model learns to denoise and enhance details in the input image.

[0136] During use, the trained model is directly used to perform denoising and detail enhancement on grayscale images of any size to obtain an output grayscale image of the same size as the input.

[0137] In summary, the solution proposed in this scheme can shorten the calculation time and achieve better output effect by obtaining the low-frequency information of the image and performing color migration on the low-frequency information. Figure 7a A partial magnified image of the output image of the prior art, Figure 7b This is a partial magnified image of the output image of this solution. Figure 8 The following is a comparison of the output images obtained by the related technology and the technology of this solution. From the above comparison, we can see that the output image of this solution has less noise, better visual effects than the existing algorithm, and the color style is closer to the given color style reference image.

[0138] Figure 9FIG. 9 is a block diagram of an image processing device 900 provided in an embodiment of the present disclosure. Figure 9 As shown, the apparatus 900 includes:

[0139] An acquisition unit 910 is used to acquire a first image and a second image, where the first image is a content reference reference and the second image is a color reference reference; a first processing unit 920 is used to perform a first processing on the first image to obtain a plurality of sub-images, where the plurality of sub-images have different frequency band information, and the plurality of sub-images include a first sub-image and at least one second sub-image, where the first sub-image has low-frequency information of the first image; a second processing unit 930 is used to perform a color style transfer processing on the second image and the first sub-image to obtain a color transferred image; and a third processing unit 940 is used to perform a second processing on the color transferred image and the at least one second sub-image to obtain an output image.

[0140] In some embodiments, the first processing unit 920 may also be configured to perform discrete wavelet transform and / or average pooling processing on the first image to obtain multiple sub-images, where the length and width of the multiple sub-images are half of the length and width of the first image.

[0141] In some embodiments, the second processing unit 930 can also be used to adjust the size of the second image to be consistent with the size of the first sub-image; use the block histogram matching algorithm to perform color style migration processing on the second image and the first sub-image to obtain an initial migration image; use the image gradient recovery algorithm to perform noise and fault processing on the initial migration image to obtain a color migration image.

[0142] In some embodiments, the image processing apparatus 900 further includes a fourth processing unit configured to align the pixel mean and pixel standard deviation of the color migration image to the pixel mean and pixel standard deviation of the second image; and perform sharpening processing on the color migration image.

[0143] In some embodiments, the third processing unit 940 can also be used to perform discrete wavelet inverse transform and / or nearest neighbor interpolation processing on the color migration image and at least one second sub-image to obtain an output image, and the size of the output image is consistent with the size of the first image.

[0144] In some embodiments, the image processing device 900 also includes a fifth processing unit, which can be used to post-process the output image using a trained image enhancement model, wherein the image enhancement model is trained using a first training set, and the training data in the first training set includes a first image, an output image, and a target image corresponding to the first image, and the image quality of the target image meets preset requirements.

[0145] In summary, the device 900 can reduce the computational complexity and time consumption of color migration and improve the efficiency of color migration by obtaining a first sub-image with low-frequency information and performing color migration on the first sub-image; by performing secondary processing, sharpening processing and post-processing on the color migrated image, the image quality of the output image can be improved.

[0146] Figure 10 The block diagram of an electronic device 1000 for implementing the above-mentioned image processing method provided in an embodiment of the present disclosure is shown.

[0147] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present disclosure, the embodiment of the present disclosure also provides an electronic device, such as Figure 10 As shown, the electronic device 1000 includes:

[0148] Communication interface 1001, capable of exchanging information with other devices;

[0149] Processor 1002, connected to communication interface 1001 to implement information exchange with other devices, and configured to execute the methods provided by one or more of the above technical solutions when running a computer program;

[0150] The computer program is stored in the memory 1003 .

[0151] Specifically, the processor 1002 can be used to obtain an image to be processed; perform pre-coding processing on the image to be processed, the pre-coding processing including first entropy coding and second entropy coding; determine a target code stream based on a first code stream corresponding to the first entropy coding and a second code stream corresponding to the second entropy coding; and perform image compression processing on the image to be processed based on a target image coding parameter corresponding to the target code stream.

[0152] It should be noted that the specific processing process of the processor 1002 can be understood by referring to the above method.

[0153] Of course, in actual application, the various components in the electronic device 1000 are coupled together through the bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 10 Various buses are labeled as bus system 1004.

[0154] The memory 1003 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 1000. Examples of such data include: any computer program used to operate on the electronic device 1000.

[0155] The methods disclosed in the above embodiments of the present application can be applied to processor 1002 or implemented by processor 1002. Processor 1002 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in processor 1002 or by instructions in the form of software. The above-mentioned first processor 1002 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1002 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in memory 1003. Processor 1002 reads the information in memory 1003 and completes the steps of the above method in combination with its hardware.

[0156] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0157] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.

[0159] Figure 11 FIG1 is a schematic diagram showing the structure of a chip 1100 for implementing the above method according to an exemplary embodiment. Figure 11The chip 1100 includes a communication interface 1101 and at least one processor 1102. The communication interface 1101 is used to receive signals input to the chip 1100 or signals output from the above chip 1100. The processor 1102 communicates with the communication interface 1101 and implements the method described in the above embodiments of the present disclosure through logic circuits or executing code instructions.

[0160] It should be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0161] It should be understood that the terms "system," "device," "unit," and / or "module" used in this application are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0162] As used in this application and the claims, unless the context clearly indicates an exception, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular and may include the plural, unless the context clearly indicates otherwise. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements. The phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus that includes the elements.

[0163] In the description of the embodiments of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "And / or" in this article is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application, "plurality" means two or more than two.

[0164] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0165] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0166] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" indicate that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0167] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0168] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (control method), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0169] It should be understood that various parts of the embodiments of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0170] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0171] Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0172] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and a second image, wherein the first image is a content reference image and the second image is a color reference image; performing a first processing on the first image to obtain a plurality of sub-images, the plurality of sub-images having different frequency band information, the plurality of sub-images including a first sub-image and at least one second sub-image, the first sub-image having low-frequency information of the first image; performing color style transfer processing on the second image and the first sub-image to obtain a color transferred image; A second process is performed on the color migration image and the at least one second sub-image to obtain an output image.

2. The method according to claim 1, characterized in that The performing a first processing on the first image to obtain a plurality of sub-images includes: Perform discrete wavelet transform and / or average pooling processing on the first image to obtain the multiple sub-images, where the length and width of the multiple sub-images are half of the length and width of the first image.

3. The method according to claim 1, characterized in that The performing color style transfer processing on the second image and the first sub-image to obtain a color-transferred image includes: Adjusting the size of the second image to be consistent with the size of the first sub-image; Performing color style migration on the second image and the first sub-image using a block histogram matching algorithm to obtain an initial migrated image; The image gradient restoration algorithm is used to perform noise and fault processing on the initial migration image to obtain the color migration image.

4. The method according to claim 3, characterized in that The method further comprises: aligning the pixel mean and pixel standard deviation of the color-shifted image to the pixel mean and pixel standard deviation of the second image; Sharpen color-shifted images.

5. The method according to claim 1, wherein The performing a second process on the color migration image and the at least one second sub-image to obtain an output image comprises: The color migration image and the at least one second sub-image are subjected to inverse discrete wavelet transform and / or nearest neighbor interpolation processing to obtain the output image, where the size of the output image is consistent with the size of the first image.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Post-processing the output image using the trained image enhancement model, The image enhancement model is trained using a first training set, the training data in the first training set includes the first image, the output image, and a target image corresponding to the first image, and the image quality of the target image meets preset requirements.

7. An image processing device, characterized in that: The device comprises: An acquiring unit, configured to acquire a first image and a second image, wherein the first image is a content reference image and the second image is a color reference image; a first processing unit, configured to perform a first processing on the first image to obtain a plurality of sub-images, wherein the plurality of sub-images have information of different frequency bands, the plurality of sub-images including a first sub-image and at least one second sub-image, and the first sub-image has low-frequency information of the first image; a second processing unit, configured to perform color style migration processing on the second image and the first sub-image to obtain a color migrated image; The third processing unit is configured to perform a second processing on the color migration image and the at least one second sub-image to obtain an output image.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

10. A chip, characterized in that: The method comprises at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method as described in any one of claims 1 to 6 through a logic circuit or executing code instructions.

Images