Image enhancement or repair processing method and device, and electronic device

Abstract

The present disclosure provides a processing method, device and electronic equipment for enhancing or repairing an image. The method comprises: obtaining an original image and an enhanced or repaired image; determining whether the enhanced or repaired image needs color correction based on the original image and color deviation judgment of the enhanced or repaired image; in response to determining that the enhanced or repaired image needs color correction, performing multiple Gaussian difference decompositions on at least part of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and a plurality of first high-frequency detail layers corresponding to the at least part of the original image, and a second low-frequency residual layer and a plurality of second high-frequency detail layers corresponding to the at least part of the enhanced or repaired image; and performing reconstruction based on the first low-frequency residual layer and the second high-frequency detail layer to obtain the enhanced or repaired image after color correction. By fusing the low-frequency color information of the original image and the high-frequency details of the enhanced or repaired image, the color deviation problem in the enhanced or repaired image can be effectively corrected.

Description

Technical Field

[0001] This disclosure relates to the field of image enhancement or restoration processing, particularly to image enhancement, image restoration, and image color correction, and especially to a method, apparatus, electronic circuit, electronic device, computer-readable storage medium, and computer program product for image enhancement or restoration processing. Background Technology

[0002] With the development of computer vision and artificial intelligence technologies, image enhancement and restoration have become important techniques for improving image quality, and are widely used in low-light imaging, security monitoring, medical imaging, autonomous driving, and video content production. Existing image enhancement and restoration techniques typically improve image clarity and recognizability by adjusting or reconstructing image brightness, contrast, and detail information.

[0003] Traditional image enhancement methods (such as histogram equalization and Retinex theory) mainly adjust the pixel distribution based on preset rules. While improving brightness and contrast, they often change the coupling relationship between brightness and chromaticity in the original image, thereby introducing changes in color distribution during the enhancement process, resulting in overall tone shift or local color distortion in the enhanced image.

[0004] With the development of deep learning technology, image enhancement and restoration methods, represented by convolutional neural networks (CNNs) and generative adversarial networks (GANs), have achieved significant results in brightness enhancement, texture restoration, and noise suppression by learning the nonlinear mapping relationship between degraded images and target images. However, because the models jointly model brightness and chromaticity information during training and inference, the enhancement or restoration process itself often introduces new color changes, causing the color distribution of the output image to deviate from the original scene.

[0005] Specifically, enhanced or repaired images often exhibit problems such as an overall warm or cool tone, inconsistent colors in local areas compared to the original image, abnormal saturation changes, and an imbalance in the color relationship between the foreground and background. These color cast phenomena are closely related to color space conversion, uneven brightness and chromaticity processing, statistical biases in color distribution in training data, the emphasis of the loss function on structural and perceptual quality, and error accumulation during nonlinear model mapping and resampling processes. They are all results introduced by enhancement or repair algorithms.

[0006] In generative image inpainting, especially in face restoration and enhancement, color shift issues are more pronounced. When generative models reconstruct missing areas based on prior knowledge of facial structure or texture, they often synthesize new color information, causing skin tone, lip color, overall lighting, or local shadow areas to align with the model's statistical distribution, thereby further amplifying the problems of overall tone drift and local color inconsistencies.

[0007] Therefore, while existing image or video enhancement and restoration technologies improve brightness, clarity, and visual detail, they inevitably introduce color shifts during processing. This causes the enhanced or restored image to deviate from the real scene in terms of color, resulting in color distortion. This color cast problem caused by the algorithm mechanism has become a significant factor affecting the realism and consistency of image enhancement and restoration results. Summary of the Invention

[0008] This disclosure provides a method, apparatus, electronic circuit, electronic device, computer-readable storage medium, and computer program product for enhancing or repairing images.

[0009] According to one aspect of this disclosure, a processing method for enhancing or repairing an image is provided, wherein the enhanced or repaired image is obtained by enhancing or repairing an original image. The method includes: obtaining an original image and an enhanced or repaired image; performing a color cast determination on the enhanced or repaired image based on the original image to determine whether the enhanced or repaired image needs color correction; in response to determining that the enhanced or repaired image needs color correction, performing multiple Gaussian difference decompositions on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to at least a portion of the original image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to at least a portion of the enhanced or repaired image; and reconstructing based on the first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image, wherein the first low-frequency residual layer has color and illumination information of at least a portion of the original image, the multiple second high-frequency detail layers have detail and texture information of at least a portion of the enhanced or repaired image, and the color-corrected enhanced or repaired image has color and illumination information of at least a portion of the original image and detail and texture information of at least a portion of the enhanced or repaired image.

[0010] In some embodiments, reconstructing based on a first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image includes: generating a mask based on the enhanced or repaired image, the mask being used to extract the region to be color-corrected in the enhanced or repaired image; applying the mask to the first low-frequency residual layer to obtain an updated first low-frequency residual layer; and reconstructing based on the updated first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image.

[0011] In some embodiments, the mask is a structure-sensitive gated mask, and generating a mask based on an enhanced or repaired image includes: using an edge detection algorithm on the enhanced or repaired image to generate a structure map of the enhanced or repaired image; and generating a structure-sensitive gated mask based on the structure map.

[0012] In some embodiments, determining whether an enhanced or repaired image needs color correction based on the original image includes: extracting multiple first color feature parameters of the original image and multiple second color feature parameters of the enhanced or repaired image; calculating multiple color difference values ​​between the original image and the enhanced or repaired image based on the multiple first color feature parameters of the original image and the multiple second color feature parameters of the enhanced or repaired image; and comparing the multiple color difference values ​​with a color deviation threshold to determine whether the enhanced or repaired image needs color correction.

[0013] In some embodiments, multiple Gaussian difference decomposition includes: performing Gaussian filtering on the image to be processed to obtain a first intermediate image; subtracting the image to be processed from the first intermediate image to obtain a high-frequency detail layer; and continuing to apply multiple Gaussian difference decompositions to the first intermediate image until a preset number of times is reached to obtain multiple high-frequency detail layers and a low-frequency residual layer.

[0014] In some embodiments, the method further includes: in response to determining that the enhanced or repaired image requires color correction, segmenting the enhanced or repaired image using an image segmentation algorithm to obtain a plurality of first local images of the enhanced or repaired image; filtering the plurality of first local images to determine at least one first local image that needs color correction; extracting at least one second local image from the original image at a position corresponding to the position of the at least one first local image; performing multiple Gaussian difference decompositions on the at least one second local image and the at least one first local image to obtain a first low-frequency residual layer and a plurality of first high-frequency detail layers corresponding to the at least one second local image, and a second low-frequency residual layer and a plurality of second high-frequency detail layers corresponding to the at least one first local image; and reconstructing based on the first low-frequency residual layer, the plurality of second high-frequency detail layers, and other first local images among the plurality of first local images except for the at least one first local image to obtain the color-corrected enhanced or repaired image.

[0015] In some embodiments, the number of the plurality of first high-frequency detail layers and the number of the plurality of second high-frequency detail layers are the same.

[0016] According to one aspect of this disclosure, a processing apparatus for enhancing or repairing an image is provided, wherein the enhanced or repaired image is obtained by enhancing or repairing an original image. The apparatus includes: a first module configured to obtain an original image and an enhanced or repaired image; a second module configured to perform color cast determination on the enhanced or repaired image based on the original image to determine whether the enhanced or repaired image requires color correction; and a third module configured to, in response to determining that the enhanced or repaired image requires color correction, perform multiple Gaussian difference decompositions on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer corresponding to at least a portion of the original image and multiple... A first high-frequency detail layer and a second low-frequency residual layer and a plurality of second high-frequency detail layers corresponding to at least a portion of the enhanced or repaired image; and a fourth module configured to reconstruct based on the first low-frequency residual layer and the plurality of second high-frequency detail layers to obtain a color-corrected enhanced or repaired image, wherein the first low-frequency residual layer has color and illumination information of at least a portion of the original image, the plurality of second high-frequency detail layers have detail and texture information of at least a portion of the enhanced or repaired image, and the color-corrected enhanced or repaired image has color and illumination information of at least a portion of the original image and detail and texture information of at least a portion of the enhanced or repaired image.

[0017] According to another aspect of this disclosure, an electronic circuit is provided, including circuitry configured to perform the steps of the methods provided above.

[0018] According to another aspect of this disclosure, an electronic device is provided, including a processor; and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the steps of the method provided above in this disclosure.

[0019] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0021] According to one or more embodiments of this disclosure, by fusing low-frequency color information of the original image with high-frequency details of the enhanced or repaired image, color cast problems in the enhanced or repaired image can be effectively corrected to restore true colors, while also preserving rich details and textures in the enhanced or repaired image, thus improving the overall visual experience.

[0022] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0023] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0024] Figure 1 This is a flowchart illustrating a processing method for enhancing or repairing an image according to an exemplary embodiment.

[0025] Figure 2 This is a flowchart illustrating a processing method for enhancing or repairing an image according to another exemplary embodiment.

[0026] Figure 3 This is a flowchart illustrating a portion of the processing method for enhancing or repairing an image according to an exemplary embodiment.

[0027] Figure 4 This is a block diagram illustrating a processing apparatus for enhancing or repairing images according to an exemplary embodiment.

[0028] Figure 5 This is a block diagram illustrating an exemplary electronic device that can be applied to exemplary embodiments. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0031] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0032] Current image enhancement or restoration algorithms can adaptively learn the mapping from degraded images to high-quality images, achieving significant results in brightness enhancement, detail restoration, and noise suppression. However, they neglect the color cast problem inherent in the algorithms themselves. With the widespread application of deep learning image enhancement technology, enhanced images are often accompanied by color distortion (color cast), which seriously affects the visual experience and subsequent processing results.

[0033] During image or video enhancement, restoration, and super-resolution processing, the output image may exhibit color distortion or shift, specifically manifested as an overall warm or cool tone, inconsistencies between local colors and the original image, abnormal saturation, and disharmony between background and foreground colors (see reference). Figure 3 The input image in the image enhancement algorithm is an enhanced image, with an overall warm tone and a bluish tint to the sky and water. This is mainly related to color space conversion errors caused by the image enhancement algorithm, uneven brightness and chromaticity processing, color distribution deviation in the training data, loss function design, interpolation resampling, and the nonlinear effects of the algorithm.

[0034] In particular, color distortion is more pronounced in generative face restoration or enhancement. Specifically, the face priors introduced in face restoration may cause skin color to be biased towards the model's preferences, while generative reconstruction generates new color information during texture completion, leading to overall tone shift and local color inconsistencies.

[0035] Therefore, the process of enhancing or restoring images or videos often suffers from color cast issues, deviating from the real scene and causing color distortion.

[0036] The embodiments of this disclosure provide a processing method for enhancing or repairing images. By fusing the low-frequency color information of the original image with the high-frequency details of the enhanced or repaired image, the aforementioned color cast problem in the enhanced or repaired image is effectively corrected to restore true colors. At the same time, the rich details and textures in the enhanced or repaired image are preserved, thereby improving the overall visual experience.

[0037] Figure 1 This is a flowchart illustrating a processing method 100 for enhancing or repairing an image according to an exemplary embodiment.

[0038] like Figure 1 As shown, this disclosure proposes a processing method 100 for enhancing or repairing images. The enhanced or repaired image is obtained by enhancing or repairing the original image, including the following steps: S102, obtaining the original image and the enhanced or repaired image; S104, performing color cast judgment on the enhanced or repaired image based on the original image to determine whether the enhanced or repaired image needs color correction; S106, in response to determining that the enhanced or repaired image needs color correction, performing multiple Gaussian difference decompositions on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency layers corresponding to at least a portion of the original image. The image consists of a detail layer, a second low-frequency residual layer corresponding to at least a portion of the enhanced or repaired image, and multiple second high-frequency detail layers; and S108, reconstruction is performed based on the first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image, wherein the first low-frequency residual layer has color and illumination information of at least a portion of the original image, the multiple second high-frequency detail layers have detail and texture information of at least a portion of the enhanced or repaired image, and the color-corrected enhanced or repaired image has color and illumination information of at least a portion of the original image and detail and texture information of at least a portion of the enhanced or repaired image.

[0039] In step S102, the original image and the enhanced or repaired image are obtained.

[0040] In the example, the original image can be a human-taken image, in which the colors of the scene and objects are distributed naturally and realistically, conforming to visual naturalness.

[0041] In some embodiments, the enhanced or repaired image has color distortion caused by the image enhancement or repair.

[0042] In the example, while the enhanced or repaired image can enrich the details and textures of the original image compared to the original image, it can also easily introduce color distortion. Color distortion can manifest as: the image as having an overall reddish, greenish, or yellowish tint; the colors of the scene and objects deviating from their true distribution, affecting visual naturalness; impacting the accuracy of downstream tasks such as object detection, semantic segmentation, and 3D reconstruction; and reducing the usability and credibility of the enhanced or repaired image.

[0043] In the examples, the enhanced or repaired images may also include other color distortions not caused by the enhancement or repair, which is not limited here.

[0044] In step S104, the color cast of the enhanced or repaired image is judged based on the original image to determine whether the enhanced or repaired image needs color correction.

[0045] In some embodiments, step S104 may include: extracting a plurality of first color feature parameters of the original image and a plurality of second color feature parameters of the enhanced or repaired image; calculating a plurality of color difference values ​​between the original image and the enhanced or repaired image based on the plurality of first color feature parameters of the original image and the plurality of second color feature parameters of the enhanced or repaired image; and comparing the plurality of color difference values ​​with a color deviation threshold to determine whether the enhanced or repaired image needs color correction.

[0046] In the example, color cast can be determined by comparing the color features of the original image with the enhanced or corrected image to determine whether the enhanced or corrected image needs color correction. Specifically, this could include:

[0047] Extract color feature parameters of global and key semantic regions from the original image and the enhanced or corrected image respectively; color feature parameters may include, but are not limited to, the mean, variance, color gamut range, and proportion of dominant color in the RGB three channels, as well as the statistical values ​​of the L channel (luminance), a channel (red and green components), and b channel (blue and yellow components) in the Lab color space;

[0048] Based on the extracted color feature parameters, such as using Euclidean distance or cosine similarity, calculate the color difference between the original image and the enhanced or corrected image at the global level and in corresponding key semantic regions (such as potential skin and sky regions); and

[0049] A global color deviation threshold and a key area color deviation threshold are preset. The calculated color difference values ​​are compared with the corresponding thresholds. If any difference value is greater than the corresponding threshold, it is determined that color correction is needed for the enhanced or corrected image. If all difference values ​​are less than or equal to the corresponding thresholds, it is determined that color correction is not needed.

[0050] In some other embodiments, only the color feature parameters of key semantic regions (such as the skin region of a portrait image or the sky region of a landscape image) can be extracted, the color difference value between the original image and the enhanced or corrected image in the corresponding key semantic regions can be calculated, and if the color difference value is greater than a threshold, it can be determined that color correction of the enhanced or corrected image is required, and if it is less than the threshold, it can be determined that color correction is not required.

[0051] In step S106, in response to determining that the enhanced or repaired image requires color correction, multiple Gaussian difference decompositions are performed on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to at least a portion of the original image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to at least a portion of the enhanced or repaired image.

[0052] In some embodiments, multiple Gaussian difference decomposition may include: applying Gaussian filtering to the image to be processed to obtain a first intermediate image; subtracting the image to be processed from the first intermediate image to obtain a high-frequency detail layer; and continuing to apply multiple Gaussian difference decompositions to the first intermediate image until a preset number of times is reached to obtain multiple high-frequency detail layers and a low-frequency residual layer.

[0053] Performing multiple Gaussian difference decompositions on an image yields a low-frequency residual layer and multiple high-frequency detail layers. The low-frequency residual layer can contain the image's color and lighting information, while the multiple high-frequency detail layers can contain the image's detail and texture information.

[0054] The image to be processed may include at least a portion of the original image and the enhanced or modified image, such as the entire original image and the enhanced or modified image, or one or more local parts of the enhanced or modified image to be color corrected and corresponding local parts of the original image. The resulting plurality of high-frequency detail layers and one low-frequency residual layer may include a plurality of first high-frequency detail layers and a first low-frequency residual layer corresponding to at least a portion of the original image, and a plurality of second high-frequency detail layers and a second low-frequency residual layer corresponding to at least a portion of the enhanced or repaired image.

[0055] In some embodiments, multiple second high-frequency detail layers enhance or repair the detail and texture information of the image, providing richer detail and texture information compared to multiple first high-frequency detail layers. Second low-frequency residual layers enhance or repair the color and lighting information of the image, but introduce color distortion compared to first low-frequency residual layers.

[0056] In step S108, reconstruction is performed based on the first low-frequency residual layer and multiple second high-frequency detail layers to obtain an enhanced or repaired image after color correction.

[0057] In some embodiments, the number of the plurality of first high-frequency detail layers and the number of the plurality of second high-frequency detail layers are the same.

[0058] Therefore, multiple first high-frequency detail layers can correspond one-to-one with multiple second high-frequency detail layers, and the first low-frequency residual layer and the second low-frequency residual layer have the same resolution.

[0059] In the example, since the first low-frequency residual layer has the color and lighting information of the original image and multiple second high-frequency detail layers have the detail and texture information of the enhanced or repaired image, reconstruction based on the first low-frequency residual layer and multiple second high-frequency detail layers can yield a color-corrected enhanced or repaired image.

[0060] In some embodiments, the color-corrected enhanced or repaired image has color and lighting information of the original image and detail and texture information of the enhanced or repaired image.

[0061] In some embodiments, reconstructing based on a first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image may include: generating a mask based on the enhanced or repaired image, the mask being used to extract the region to be color-corrected in the enhanced or repaired image; applying the mask to the first low-frequency residual layer to obtain an updated first low-frequency residual layer; and reconstructing based on the updated first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image.

[0062] In some embodiments, the mask is a structure-sensitive gated mask, and generating a mask based on an enhanced or repaired image may include: using an edge detection algorithm on the enhanced or repaired image to generate a structure map of the enhanced or repaired image; and generating a structure-sensitive gated mask based on the structure map.

[0063] In the example, the mask is used to select the part of the enhanced or repaired image to be color corrected. For example, the part to be color corrected can be a local image that meets the conditions of "single color, large area and color deviation greater than a threshold". For example, the mask can be a structure-sensitive gated mask, and an edge detection algorithm can be used to generate a structure-sensitive gated mask for the enhanced or repaired image.

[0064] In the example, the edge detection algorithm could be Sobel gradient algorithm, Prewitt gradient algorithm, Roberts cross gradient algorithm, feature attention or Canny edge detection, etc., to identify structural regions of the image to be enhanced or repaired.

[0065] Therefore, since structure-sensitive gating masks can suppress color perturbations in structural regions, they can ensure that the structure (e.g., texture or edges) of the enhanced or repaired image is not damaged.

[0066] Figure 2 This is a flowchart illustrating a processing method 200 for enhancing or repairing an image according to another exemplary embodiment.

[0067] refer to Figure 2 According to another exemplary embodiment, the processing method 200 for enhancing or repairing an image may include:

[0068] S202. Obtain the original image and the enhanced or repaired image;

[0069] S204. Based on the original image, determine the color cast of the enhanced or repaired image to determine whether the enhanced or repaired image needs color correction.

[0070] S206. In response to determining that the enhanced or repaired image requires color correction, perform multiple Gaussian difference decompositions on the local areas of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to the local areas of the original image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to the local areas of the enhanced or repaired image, including: S2061. Segmenting the enhanced or repaired image using an image segmentation algorithm to obtain multiple first local images of the enhanced or repaired image; S2062. Filtering the multiple first local images to determine at least one first local image that needs color correction; S2063. Extracting at least one second local image from the original image at a position corresponding to the position of at least one first local image; S2064. Performing multiple Gaussian difference decompositions on the at least one second local image and at least one first local image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to the at least one second local image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to the at least one first local image; and

[0071] S208. Reconstruction is performed based on the first low-frequency residual layer, multiple second high-frequency detail layers, and other first local images (excluding at least one first local image) to obtain a color-corrected enhanced or repaired image.

[0072] In some examples, in addition to using the masks described above, image segmentation can be used to select local areas of the enhanced or repaired image for color correction. For example, in response to determining that the enhanced or repaired image requires color correction, a region extraction operation can be performed to accurately locate the local image area requiring color correction that meets the criteria of "uniform color, large area occupied, and color deviation greater than a threshold." Specifically, this can include:

[0073] Image segmentation algorithms are used to segment enhanced or repaired images to obtain multiple independent local images. Dedicated segmentation strategies are adapted for different image types. For example, semantic segmentation based on skin color models (such as YCbCr skin color threshold) is used for portrait images, region growing algorithm based on edge detection is used for landscape images, and K-means segmentation based on color clustering is used for still life images.

[0074] For each local image, two core indicators are calculated and selected: First, the color uniformity indicator: represented by the variance value of the pixel colors within the local image. The smaller the variance value, the more uniform the color. Local images with variance values ​​less than a preset color uniformity threshold are selected. Second, the region area ratio indicator: the proportion of the number of pixels in the local image to the total number of pixels in the enhanced or corrected image is calculated. Regions with a proportion greater than a preset area ratio threshold (e.g., 5%) are selected. Regions that meet both of the above conditions are selected as at least one first local image that needs color correction.

[0075] Extract at least one second local image from the original image that completely corresponds to the location of at least one first local image.

[0076] It should be noted that this disclosure does not limit the order of steps S102-S108 and S202-208, nor does it limit the order of obtaining the various consideration factors. For example, in steps S106 and S208, the order in which the original image and the enhanced or repaired image are subjected to multiple Gaussian difference decompositions can be arbitrary and is not limited here.

[0077] Therefore, by fusing the low-frequency color information of the original image with the high-frequency details of the enhanced or repaired image, color cast problems in the enhanced or repaired image can be effectively corrected to restore true colors, while also preserving the rich details and textures of the enhanced or repaired image, thus improving the overall visual experience. Furthermore, the above method is simple to implement, suitable for various image enhancement or repair applications, and possesses good versatility.

[0078] Figure 3 This is a detailed flowchart illustrating a portion of the process 300 of a processing method 100 for enhancing or repairing an image according to an exemplary embodiment. The following is in conjunction with... Figure 3 A detailed description of process 300 is provided below:

[0079] refer to Figure 3 The figure illustrates the process of multiple Gaussian difference decomposition of an enhanced or repaired image according to some embodiments, and the process of reconstructing the color-corrected enhanced or repaired image (the process of multiple Gaussian difference decomposition of the original image is not shown).

[0080] The input image is an enhanced or repaired image, which suffers from color distortion (e.g., the overall tone is noticeably warm, and the sky and water are noticeably bluish). Filtering this enhanced or repaired image yields the Blur_0 image, which is slightly blurrier than the enhanced or repaired image because filtering resulted in the loss of some details and texture information. Subtracting the enhanced or repaired image from the Blur_0 image yields a second high-frequency detail layer, Detail_0, which contains the lost details and texture information. Similarly, filtering the Blur_0 image yields the Blur_1 image, which is even blurrier than the Blur_0 image because filtering resulted in the loss of another portion of details and texture information. Subtracting the Blur_0 image from the Blur_1 image yields another second high-frequency detail layer, Detail_1, which contains the remaining lost details and texture information. Similarly, filtering the Blur_1 image yields the Blur_2 image, which is even more blurred than the Blur_1 image because it loses some more detail and texture information after filtering. Subtracting the Blur_1 and Blur_2 images yields another second high-frequency detail layer, Detail_2, which contains the aforementioned lost detail and texture information. Ultimately, the input enhanced or repaired image is decomposed into a Blur_2 image (i.e., the second low-frequency residual layer) and multiple second high-frequency detail layers: Detail_0, Detail_1, and Detail_2.

[0081] In the example, the filtering used in the multiple Gaussian difference decomposition processes can be Gaussian filtering, or an extension or improvement of Gaussian filtering, such as Gabor directional filtering or bilateral filtering.

[0082] The multiple Gaussian difference decomposition process of the original image is similar to the multiple Gaussian difference decomposition process of the enhanced or repaired image described above. For example, the Blur_2_org image is the first low-frequency residual layer obtained from the original image through multiple Gaussian difference decompositions (the specific process is not shown, but it is similar to the one described above). Figure 3 The process of multiple Gaussian difference decompositions for enhancing or repairing images is similar to that described.

[0083] During image reconstruction, reconstruction is performed based on the Blur_2_org image and multiple second high-frequency detail layers Detail_0, Detail_1, and Detail_2 to obtain the output image. The output image is the color-corrected enhanced or repaired image. (See reference...) Figure 3 As can be seen, the output image retains the color foundation of the original image, and the overall color is no longer yellowish, which is more in line with visual naturalness. At the same time, it retains the high-frequency details of the enhanced or repaired image, improving the overall visual experience of the image.

[0084] In some examples, a mask can be applied to the Blur_2_org image to obtain the updated first low-frequency residual layer Blur_2_org_masked (e.g., the calculation formula could be Blur_2_org_masked = (1 - mask)). blur_2 + mask Blur_2_org).

[0085] In some examples, a mask can be applied directly to the output image after global color correction (or the color-corrected enhanced or repaired image) (e.g., output image _masked = (1-mask)). Input image + mask Output image).

[0086] For the process of multiple Gaussian difference decompositions of a local image, and Figure 3 The process of multiple Gaussian difference decompositions shown is similar and will not be repeated here.

[0087] It should be understood that, Figure 3 The specific flowcharts shown are for illustrative purposes only and not for limiting purposes. For example, multiple high-frequency detail layers may not be three layers, but may be two, four, or more layers; this is not a limitation.

[0088] Embodiments of this disclosure also provide a processing apparatus for enhancing or repairing images.

[0089] Figure 4 This is a schematic block diagram illustrating a processing apparatus 400 for enhancing or repairing images according to an exemplary embodiment.

[0090] like Figure 4As shown, in some embodiments, the apparatus 400 includes: a first module 410 configured to obtain an original image and an enhanced or repaired image; a second module 420 configured to perform color cast determination on the enhanced or repaired image based on the original image to determine whether the enhanced or repaired image needs color correction; and a third module 430 configured to, in response to determining that the enhanced or repaired image needs color correction, perform multiple Gaussian difference decompositions on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to at least a portion of the original image and the enhanced or repaired image. The image includes at least a second low-frequency residual layer and multiple second high-frequency detail layers; and a fourth module 440 configured to reconstruct based on the first low-frequency residual layer and multiple second high-frequency detail layers to obtain a color-corrected enhanced or repaired image, wherein the first low-frequency residual layer has color and illumination information of at least a portion of the original image, the multiple second high-frequency detail layers have detail and texture information of at least a portion of the enhanced or repaired image, and the color-corrected enhanced or repaired image has at least a portion of color and illumination information of the original image and at least a portion of detail and texture information of the enhanced or repaired image.

[0091] The operations of the first module 410, the second module 420, the third module 430, and the fourth module 440 described above can be combined with... Figure 1 The operations or combinations of steps S102, S104, S106 and S108 are described above. Figure 2 The operations of steps S202, S204, S206 and S208 are the same, so the details of each aspect will not be repeated here.

[0092] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0093] It should also be understood that the above regarding Figure 4The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0094] According to one aspect of this disclosure, an electronic circuit is also provided, which includes circuitry configured to perform the steps of any of the method embodiments described above.

[0095] According to one aspect of this disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0096] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0097] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0098] In the following text, combined with Figure 5 Illustrative examples describing such electronic devices, non-transitory computer-readable storage media, and computer program products.

[0099] Figure 5 An example configuration of an electronic device 500 that can be used to implement the methods described herein is shown.

[0100] Electronic device 500 can be a variety of different types of devices. Examples of electronic device 500 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices, etc.

[0101] Electronic device 500 may include at least one processor 502, memory 504, multiple communication interfaces 506, display device 508, other input / output (I / O) devices 510, and one or more mass storage devices 512 capable of communicating with each other, such as via system bus 514 or other suitable connections.

[0102] Processor 502 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 502 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 502 may be configured to acquire and execute computer-readable instructions stored in memory 504, mass storage device 512, or other computer-readable media, such as program code of operating system 516, program code of application program 518, program code of other program 520, etc.

[0103] Memory 504 and mass storage device 512 are examples of computer-readable storage media for storing instructions that are executed by processor 502 to perform the various functions described above. For example, memory 504 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 512 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 504 and mass storage device 512 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 502 as a specific machine configured to perform the operations and functions described in the examples herein.

[0104] Multiple programs may be stored on mass storage device 512. These programs include operating system 516, one or more application programs 518, other programs 520, and program data 522, and they may be loaded into memory 504 for execution. Examples of such application programs or program modules may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0105] Although Figure 5 The data is illustrated as being stored in memory 504 of electronic device 500, but operating system 516, one or more application programs 518, other programs 520, and program data 522, or portions thereof, may be implemented using any form of computer-readable medium accessible by electronic device 500. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0106] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by electronic devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0107] One or more communication interfaces 506 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TMInterfaces include near-field communication (NFC) interfaces. Communication interface 506 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 506 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0108] In some examples, a display device 508, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 510 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0109] The technologies described herein can be supported by these various configurations of electronic device 500, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from electronic device 500. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect electronic device 500 to other electronic devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on electronic device 500 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. A processing method for enhancing or repairing images, characterized in that, The enhanced or repaired image is obtained by enhancing or repairing the original image, and the method includes: Obtain the original image and the enhanced or repaired image; Based on the original image, the color cast of the enhanced or repaired image is determined to determine whether the enhanced or repaired image needs color correction. In response to determining that the enhanced or repaired image requires color correction, multiple Gaussian difference decompositions are performed on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to at least a portion of the original image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to at least a portion of the enhanced or repaired image; and Reconstruction is performed based on the first low-frequency residual layer and the multiple second high-frequency detail layers to obtain an enhanced or repaired image after color correction. The first low-frequency residual layer contains at least a portion of the color and illumination information of the original image; the plurality of second high-frequency detail layers contain at least a portion of the detail and texture information of the enhanced or repaired image; and the color-corrected enhanced or repaired image contains at least a portion of the color and illumination information of the original image and at least a portion of the detail and texture information of the enhanced or repaired image. The multiple Gaussian difference decomposition includes: The first intermediate image is obtained by applying Gaussian filtering to the image to be processed. The difference between the image to be processed and the first intermediate image is calculated to obtain a high-frequency detail layer; and The Gaussian difference decomposition is applied to the first intermediate image multiple times until a preset number of times is reached, resulting in multiple high-frequency detail layers and a low-frequency residual layer. The low-frequency residual layer is the image obtained after the image to be processed has undergone the preset number of Gaussian filters.

2. The method according to claim 1, characterized in that, Reconstruction is performed based on the first low-frequency residual layer and the plurality of second high-frequency detail layers to obtain a color-corrected enhanced or repaired image, including: A mask is generated based on the enhanced or repaired image, and the mask is used to extract the region in the enhanced or repaired image that needs color correction. Applying the mask to the first low-frequency residual layer yields an updated first low-frequency residual layer; and Reconstruction is performed based on the updated first low-frequency residual layer and the plurality of second high-frequency detail layers to obtain an enhanced or repaired image with color correction.

3. The method according to claim 2, characterized in that, The mask is a structure-sensitive gated mask, and generating the mask based on the enhanced or repaired image includes: An edge detection algorithm is used on the enhanced or repaired image to generate a structural map of the enhanced or repaired image; and A structure-sensitive gating mask is generated based on the structure graph.

4. The method according to claim 1, characterized in that, Based on the original image, perform color cast determination on the enhanced or repaired image to determine whether the enhanced or repaired image needs color correction, including: Extract multiple first color feature parameters from the original image and multiple second color feature parameters from the enhanced or repaired image; Based on multiple first color feature parameters of the original image and multiple second color feature parameters of the enhanced or repaired image, multiple color difference values ​​between the original image and the enhanced or repaired image are calculated; and The multiple color difference values ​​are compared with a color deviation threshold to determine whether the enhanced or repaired image requires color correction.

5. The method according to claim 1, characterized in that, The method further includes: In response to determining that the enhanced or repaired image requires color correction, the enhanced or repaired image is segmented using an image segmentation algorithm to obtain multiple first local images of the enhanced or repaired image; The plurality of first local images are filtered to determine at least one first local image that requires color correction; Extract at least one second local image from the original image at a position corresponding to the position of the at least one first local image; Multiple Gaussian difference decompositions are performed on the at least one second local image and the at least one first local image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to the at least one second local image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to the at least one first local image; and Reconstruction is performed based on the first low-frequency residual layer, the plurality of second high-frequency detail layers, and other first local images among the plurality of first local images except for the at least one first local image, to obtain an enhanced or repaired image after color correction.

6. The method according to any one of claims 1 to 5, characterized in that, The number of the plurality of first high-frequency detail layers is the same as the number of the plurality of second high-frequency detail layers.

7. A processing apparatus for enhancing or repairing images, characterized in that, The enhanced or repaired image is obtained by enhancing or repairing the original image, and the apparatus includes: The first module is configured to obtain the original image and the enhanced or repaired image; The second module is configured to perform a color cast judgment on the enhanced or repaired image based on the original image, and determine whether the enhanced or repaired image needs color correction. The third module is configured to, in response to determining that the enhanced or repaired image requires color correction, perform multiple Gaussian difference decompositions on at least a portion of the original image and the enhanced or repaired image to obtain a first low-frequency residual layer and multiple first high-frequency detail layers corresponding to at least a portion of the original image, and a second low-frequency residual layer and multiple second high-frequency detail layers corresponding to at least a portion of the enhanced or repaired image; and The fourth module is configured to reconstruct based on the first low-frequency residual layer and the plurality of second high-frequency detail layers to obtain a color-corrected enhanced or repaired image. The first low-frequency residual layer contains at least a portion of the color and illumination information of the original image; the plurality of second high-frequency detail layers contain at least a portion of the detail and texture information of the enhanced or repaired image; and the color-corrected enhanced or repaired image contains at least a portion of the color and illumination information of the original image and at least a portion of the detail and texture information of the enhanced or repaired image. The multiple Gaussian difference decomposition includes: The first intermediate image is obtained by applying Gaussian filtering to the image to be processed. The difference between the image to be processed and the first intermediate image is calculated to obtain a high-frequency detail layer; and The Gaussian difference decomposition is applied to the first intermediate image multiple times until a preset number of times is reached, resulting in multiple high-frequency detail layers and a low-frequency residual layer. The low-frequency residual layer is the image obtained after the image to be processed has undergone the preset number of Gaussian filters.

8. An electronic circuit, characterized in that, The electronic circuit includes: A circuit configured to perform the steps of the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The electronic device includes: Processor; and A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

Images