An image processing method and apparatus

Abstract

This invention relates to the field of image processing technology, and more particularly to an image processing method, apparatus, computing device, and computer-readable storage medium. The method includes: processing a first image using a convolutional neural network to obtain a global semantic information map of the first image; based on the global semantic information map, enlarging the grayscale image of the first image to obtain a first reconstructed image; based on the global semantic information map, performing color enhancement processing on the chroma image of the first image to obtain a second reconstructed image; and fusing the first reconstructed image and the second reconstructed image to obtain a second image. This allows the second image to be reconstructed synchronously with the first image in terms of super-resolution and color enhancement, resulting in a higher degree of consistency between the first and second reconstructed images, thus improving the quality of the second image. Simultaneously, reusing global semantic information also saves computational resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, computing device and computer-readable storage medium. Background Technology

[0002] Image super-resolution refers to the process of acquiring a corresponding high-resolution image from a low-resolution image. Color enhancement typically refers to color gamut expansion or subjective color adjustment, i.e., the mapping process from a narrow color gamut to a wide color gamut, or color correction of images with poor color performance to improve the subjective perception of image colors. Resolution and color are both important factors affecting the subjective visual experience when displaying images at high resolution.

[0003] Current algorithms often treat image super-resolution processing and image color enhancement as separate problems. This leads to discrepancies between the resulting super-resolution and color-enhanced images when processing the same image, resulting in poor compatibility. Furthermore, this separate processing consumes significant computational resources, hindering the improvement of image processing efficiency.

[0004] In summary, there is an urgent need for an image processing method to solve the problem that existing image super-resolution processing and color enhancement cannot be performed simultaneously and have low processing efficiency. Summary of the Invention

[0005] This invention provides an image processing method to solve the problem that existing image super-resolution processing and color enhancement cannot be performed simultaneously and have low processing efficiency.

[0006] This invention provides an image processing method, comprising:

[0007] The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image;

[0008] Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image;

[0009] Based on the global semantic information graph, the chroma graph of the first image is subjected to color enhancement processing to obtain the second reconstructed image;

[0010] The first reconstructed image and the second reconstructed image are fused together to obtain the second image.

[0011] Because there are differences and correlations between spatial resolution and color in images, this invention proposes to process image super-resolution and color enhancement simultaneously using the same lightweight network. Specifically, in the same spatial dimension, the grayscale image of the image is magnified to obtain a first reconstructed image, and the chroma image of the image is color reconstructed to obtain a second reconstructed image. The first and second reconstructed images are then fused to obtain a second image. This allows the second image to be reconstructed synchronously in terms of super-resolution and color enhancement compared to the first image, resulting in a higher degree of consistency between the first and second reconstructed images and thus improving the quality of the second image. Furthermore, the global semantic information of the first image is used to guide the super-resolution and color enhancement processes, which not only improves the quality of the reconstructed image but also saves computational resources by reusing global semantic information.

[0012] Optionally, based on the global semantic information map, color enhancement processing is performed on the chroma map of the first image to obtain a second reconstructed image, including:

[0013] The color control factor of the first image is determined using the global semantic information graph;

[0014] The global semantic information map is fused and convolved with the chroma map of the first image to obtain the chroma output map;

[0015] The chromaticity output image is adjusted based on the color control factor to obtain the second reconstructed image.

[0016] Since different first images have varying color saturation, these differences must be considered when performing color enhancement on different first images. By determining the corresponding color control factor using the global semantic information map of the first image, and then using this color control factor to adjust the chroma output map, the degree of color enhancement can be controlled in a targeted manner.

[0017] Optionally, the method further includes:

[0018] The chroma map of the first image is upsampled to obtain a chroma-enhanced map;

[0019] The second reconstructed image is obtained by adjusting the chromaticity output image based on the color control factor, including:

[0020] The second reconstructed image is obtained by adjusting the chroma output image and the chroma enhancement image based on the color control factor.

[0021] A chroma-enhanced image is obtained by upsampling the chroma map of the first image. The saturation of the chroma-enhanced image is then used to guide the generation of the second reconstructed image. The resulting second reconstructed image takes into account the original saturation of the first image and thus performs targeted color enhancement.

[0022] Optionally, the global semantic information map is fused and convolved with the chroma map of the first image to obtain a chroma output map, including:

[0023] After merging the global semantic information map with the chroma map of the first image, the pixels are mixed after dilated convolution to obtain the chroma output map.

[0024] The global semantic information map is used to guide the color enhancement processing of the first image. Merging the global semantic information map with the chroma map of the first image and then performing dilated convolution reduces computation while obtaining a larger receptive field, thus enhancing the color enhancement effect. Further pixel blending allows for the enlargement of the color-enhanced image, resulting in a chroma output map with resolution and sharpness comparable to the first reconstructed image. The resolution and sharpness of the first reconstructed image are also comparable to the first reconstructed image; merging the two enhances the overall image processing quality.

[0025] Optionally, the first image is processed using a convolutional neural network to obtain a global semantic information map of the first image, including:

[0026] The first image is convolved through the encoder unit to obtain the first encoded image;

[0027] The first encoded image is magnified to obtain a global semantic information map of the first image.

[0028] Convolution calculations are performed using encoder units. During this process, the image resolution is reduced, resulting in a larger receptive field and improved accuracy in extracting global semantic information. After obtaining the first encoded image, upsampling restores the previously reduced resolution, facilitating subsequent merging with the chroma and grayscale images of the first image.

[0029] Optionally, determining the color control factor of the first image through the global semantic information graph includes:

[0030] The color control factor of the first image is obtained by processing the first encoded image through a pooling layer and a fully connected layer.

[0031] By processing the first encoded image with pooling layers and fully connected layers, the first encoded image can be converted into a 1*1 dimensional number to represent the color control factor of the first image. The resulting color control factor can accurately reflect the saturation of the first image.

[0032] Optionally, based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image, including:

[0033] Extract the grayscale image from the first image;

[0034] The grayscale image of the first image is subjected to a first residual processing to obtain a first grayscale feature map;

[0035] After merging the first grayscale feature map and the global semantic information map, a second residual processing is performed to obtain the second grayscale feature map;

[0036] The second grayscale feature map is pixel-wise mixed to obtain the first reconstructed image.

[0037] By extracting the grayscale image of the first image, super-resolution processing can be performed on the grayscale image, thus achieving the goal of improving resolution. The global semantic information map is merged with the first grayscale feature map to guide the super-resolution processing of the first image. Furthermore, after two residual processing steps and pixel blending, the reconstruction quality of the super-resolution image is improved.

[0038] This invention also provides an image processing apparatus, comprising:

[0039] Processing unit, used for:

[0040] The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image;

[0041] Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image;

[0042] Based on the global semantic information graph, the chroma graph of the first image is subjected to color enhancement processing to obtain the second reconstructed image;

[0043] The first reconstructed image and the second reconstructed image are fused together to obtain the second image.

[0044] This invention also provides a computing device, comprising:

[0045] Memory, used to store computer programs;

[0046] The processor is configured to invoke a computer program stored in the memory and execute the image processing method listed in any of the above methods according to the obtained program.

[0047] This invention also provides a computer-readable storage medium storing a computer-executable program for causing a computer to perform any of the image processing methods listed above. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 An example of a possible image processing method provided by an embodiment of the present invention is shown;

[0050] Figure 2 An example is shown of a possible architecture for extracting global semantic information graphs;

[0051] Figure 3 An example is shown of a possible architecture for extracting color control factors;

[0052] Figure 4 An example is shown of a possible architecture for super-resolution processing;

[0053] Figure 5 A schematic diagram of a residual network layer is shown;

[0054] Figure 6 An example is shown of a possible architecture for color enhancement processing;

[0055] Figure 7 This is a schematic diagram of the overall process involved in the embodiments of the present invention;

[0056] Figure 8 This is a possible image processing apparatus shown in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0058] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0059] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0060] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0061] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0062] This invention provides an image processing method that utilizes the differences and similarities between super-resolution and color enhancement. By processing images on the same lightweight network, it achieves joint enhancement of resolution and color within the same space, thereby improving the quality and efficiency of image processing.

[0063] Figure 1 An example is shown of a possible image processing method provided by an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0064] Step 101: Process the first image using a convolutional neural network to obtain the global semantic information map of the first image.

[0065] In this embodiment of the invention, the format and size of the first image used in the global semantic information extraction are not limited; it can be in RGB or YUV format. Global semantic information is extracted from the first image using a convolutional neural network to guide subsequent image resolution and color enhancement.

[0066] Figure 2An example is shown of a possible architecture for extracting global semantic information graphs. For example... Figure 2 As shown, it includes an input layer, a downsampling layer, an encoder unit, an amplification layer, and an output layer.

[0067] The following section uses an RGB image as an example to introduce the method for extracting global semantic information graphs.

[0068] The input layer is a convolutional layer used to increase the number of channels in the input image, thereby enabling more detailed and richer computations in subsequent processing and improving processing quality. For example, if the first input image X_in has a resolution of 128*128 and 3 channels, processing it through a 3*3 convolutional layer with 64*3 kernels will transform the 3-channel image into a 64-channel feature map. Zero-padding is used during the convolution process to ensure that the resolution is not reduced due to the convolution. The resulting image retains a resolution of 128*128 and 64 channels.

[0069] The downsampling layer is used to reduce the resolution of the input feature map, thereby reducing the computational cost in global semantics. Specifically, it can be a bilinear interpolation layer; however, this embodiment of the invention does not limit the downsampling method. The reduced feature map resolution can be arbitrary, and this embodiment of the invention does not impose any restrictions on this. For example, through the downsampling layer, the resolution of the feature map is reduced to 64*64.

[0070] The encoder unit performs convolution calculations on the first image to extract global semantic information, resulting in the first encoded image. During the convolution calculation, the encoder unit reduces the image resolution, thereby obtaining a larger receptive field and more accurately extracting the global semantic information. For example, the encoder unit includes a 3x3 convolutional layer with a stride of 2, a ReLU activation function layer, and a 3x3 convolutional layer with a stride of 1. Through the convolution processing of the encoder unit, the resolution of the first image is further reduced; for example, the image resolution is reduced by a factor of 2 through the processing of a single encoder. Figure 2 The example illustrates four encoder units, which reduces the resolution of a 64*64 image to 4*4. This embodiment of the invention does not limit the structure of each encoder, the reduction factor, or the number of encoders.

[0071] The first encoded image obtained after processing by the encoder unit has a low resolution. In order to facilitate subsequent super-resolution and color enhancement processing, the first encoded image can be enlarged to the same resolution as the first image through an enlargement layer. For example, in the example above, the image resolution is enlarged to 128*128.

[0072] This invention does not limit the structure of the magnification layer or the image magnification method. For example, the magnification layer may include a bilinear interpolation layer, a 3×3 convolutional layer, a ReLU activation function layer, and an upsampling layer. The bilinear interpolation layer magnifies the image resolution from 4*4 to 64*64, then the convolutional layer and activation function layer perform calculations to ensure sharpness, and finally the upsampling layer magnifies it to 128*128.

[0073] After the global semantic information extraction process described above, the image resolution and number of channels remain unchanged; for example, in the previous example, the number of channels is still 64. Therefore, an output layer is set up to transform the 64-channel feature map into a single-channel global semantic information map. The resulting global semantic information map can be used to subsequently merge with the chroma and grayscale images of the first image for joint super-resolution and color enhancement processing.

[0074] Convolution calculations are performed using encoder units. During this process, the image resolution is reduced, resulting in a larger receptive field and improved accuracy in extracting global semantic information. After obtaining the first encoded image, upsampling restores the previously reduced resolution, facilitating subsequent merging with the chroma and grayscale images of the first image.

[0075] Because different first images have varying color saturation, this difference in saturation must be considered when performing color enhancement on different first images. If the first image itself has high color saturation, the degree of color enhancement should be reduced in subsequent color enhancement processes; conversely, if the first image itself has low color saturation, the degree of color enhancement should be increased in subsequent color enhancement processes. Therefore, during the extraction of the global semantic information map, the color control factor of the first image can also be determined using the global semantic information map.

[0076] Figure 3 An example is shown of a possible architecture for extracting color control factors. For example... Figure 3 As shown, it includes a global average pooling layer, a fully connected layer, and a sigmoid layer.

[0077] In this embodiment of the invention, the first encoded image generated during the global semantic information extraction process is reused, thereby greatly reducing computational cost.

[0078] The first encoded image is processed by a pooling layer, reducing the 64-dimensional, 4x4 resolution first encoded image to 1x64 dimensional features. It is then processed by a fully connected layer to reduce the number of channels. More fully connected layers result in greater computation, but more accurate extraction of color control factors. This invention does not limit the number of fully connected layers. For example, in... Figure 3 Four fully connected layers are used. The first three layers have half the number of output channels, resulting in 32, 16, and 8 channels respectively. The last fully connected layer outputs only 1 channel. Therefore, after the pooling and fully connected layer processing described above, the color control factor of the first image is extracted. Finally, a sigmoid layer limits the color control factor to a range, for example, within the interval (0,1). This ensures that the color control factor extracted from any image will be limited to the range (0,1).

[0079] By processing the first encoded image with pooling layers and fully connected layers, the first encoded image can be converted into a 1*1 dimensional number to represent the color control factor of the first image. The resulting color control factor can accurately reflect the saturation of the first image.

[0080] Step 102: Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image.

[0081] Step 102 aims to perform super-resolution processing on the first image, enlarging it by a factor of n. The first image is in YUV format. Therefore, if the first image is in RGB format, to facilitate subsequent processing of grayscale and color information in the Y and UV channels respectively, the input RGB image is first converted to a YUV image in color space.

[0082] The following section uses the first image as an example, which is a YUV image, to introduce the method of super-resolution processing in YUV space.

[0083] First, extract the grayscale image from the YUV image. If the resolution of the first image is 128*128 and the number of channels is 3, then the resolution of the grayscale image is 128*128 and the number of channels is 1.

[0084] Figure 4 An example is shown: a possible architecture for super-resolution processing. For example... Figure 4 As shown, it includes an input layer, a first residual network layer, a merging layer, a second residual network layer, an amplification layer, and an output layer.

[0085] The input layer is a convolutional layer used to increase the dimensionality of the input grayscale image by increasing the number of channels. The specific processing is the same as the input layer mentioned above, and will not be repeated here. After processing by the input layer, the grayscale image is transformed into a 64-channel feature map.

[0086] The first residual network layer performs the first residual processing on the obtained feature map and performs convolution operation to obtain the first grayscale feature map. Figure 5A schematic diagram of a residual network layer is shown, comprising several residual blocks and a global residual connection from the input of the residual blocks to the output of the residual blocks. Only two residual blocks are shown in the diagram; multiple residual blocks can be set to ensure the quality of the reconstructed image.

[0087] Each residual block not only processes the output of the previous convolutional layer, but also processes the input of the previous residual block through global residual connections. This reduces information loss during processing.

[0088] The convolutional layers contained in the residual block can be 3*3 convolutional layers, 1*1 convolutional layers, etc. The embodiments of the present invention do not limit the number of residual blocks or their specific structure.

[0089] For example, in Figure 5 The residual network layer illustrates a possible residual block structure, comprising a 3×1 convolutional layer, a ReLU activation function layer, another 1×3 convolutional layer, and residual connections connecting the two ends. Using linear kernels instead of the 3×3 convolutional kernels in a typical residual network can reduce computational cost.

[0090] After the convolutional computation of the first residual network layer, the resulting first grayscale feature map and global semantic information map are merged in the merging layer. The global semantic information map can provide global semantic information guidance for subsequent super-resolution processing.

[0091] Then, the merged image undergoes a second residual processing in the second residual network layer to obtain a second grayscale feature map. The structure of this second residual network layer is similar to that of the first residual network layer, also including several residual blocks and global residual connections from before the residual block input to after the residual block output. This embodiment of the invention does not limit the number or specific structure of the residual blocks; they can be the same as or different from the first residual network layer.

[0092] Alternatively, only one residual network layer can be used, that is, the grayscale image Y is merged with the global semantic information graph and then convolutional computation is performed through a residual network layer. Multiple residual network layers can also be used. The more residual network layers there are, the more refined and accurate the computation will be, but this will also increase the computational load. This embodiment of the invention does not impose any limitations on this.

[0093] It is worth noting that after the above merging layer processing, the number of channels in the resulting second grayscale feature map actually changes. For example, if the first grayscale feature map has 64 channels, then since the global semantic information map is a single channel, the number of channels in the merged image is 65. After further calculation by the residual network layer, the number of channels remains unchanged.

[0094] Considering that image enlargement will be performed subsequently through pixel blending, a convolutional layer can be added after the second residual network layer for dimensionality reduction of the number of channels, such as converting a 65-channel image into a 64-channel image. The specific convolutional layer structure and processing procedure are not limited in this embodiment of the invention.

[0095] The magnification layer performs pixel blending on the second grayscale feature map, thereby obtaining an image magnified n times to the target resolution. For example, in the example above, a second grayscale feature map with a resolution of 128*128 and 64 channels, if magnified by 2 times, will be transformed into an image with a resolution of 256*256 and 32 channels; if magnified by 4 times, it will be transformed into an image with a resolution of 512*512 and 4 channels. And so on.

[0096] The output layer is used to transform the high-resolution feature map into a single-channel high-resolution output image, i.e., the first reconstructed image. For example, in the example above, if the image is magnified by 2 times to obtain an image with a resolution of 256*256 and 32 channels, it will be transformed into a first reconstructed image with a resolution of 256*256 and 1 channel by the output layer.

[0097] By extracting the grayscale image of the first image, super-resolution processing can be performed on the grayscale image, thus achieving the goal of improving resolution. The global semantic information map is merged with the first grayscale feature map to guide the super-resolution processing of the first image. Furthermore, after two residual processing steps and pixel blending, the reconstruction quality of the super-resolution image is improved.

[0098] Step 103: Based on the global semantic information graph, perform color enhancement processing on the chroma graph of the first image to obtain the second reconstructed image.

[0099] Step 103 aims to enhance the color of the first image, making its color representation more delicate, rich, and vibrant. The first image is in YUV format. Therefore, if the first image is in RGB format, to facilitate subsequent processing of grayscale and color information in the Y and UV channels respectively, the input RGB image is first converted to a YUV image color space.

[0100] The following section uses the first image as an example (a YUV image) to introduce the color enhancement processing method in the YUV space.

[0101] First, extract the UV channel image, i.e., the chromaticity map X, from the YUV image. If the resolution of the first image is 128*128 and the number of channels is 3, then the resolution of the chromaticity map X is 128*128 and the number of channels is 2.

[0102] Figure 6 An example is shown: a possible architecture for color enhancement processing. For example... Figure 6 As shown, it includes a merging layer, an input layer, a downsampling layer, a dilated convolutional layer, a magnification layer, an output layer, and a color suppression layer.

[0103] First, the chroma map X and the global semantic information map are merged in a merging layer to obtain a merged image with a resolution of 128*128 and 3 channels. Global semantic information is then used to supplement global information, improving the quality of the color-enhanced image.

[0104] Optionally, the input layer then undergoes convolutional upscaling, with the specific process being the same as the input layer mentioned above, and will not be repeated here. After processing by the input layer, the number of channels is transformed to 64.

[0105] Optionally, since the UV channels have relatively little high-frequency information, reconstruction can be performed at a further downscaled level to reduce computational cost. For example, a downsampling layer reduces the resolution of the input feature map, facilitating reduced computation in subsequent calculations. This embodiment of the invention does not limit the downsampling method; for example, it can be a bilinear interpolation layer. For instance, through downsampling, the image resolution in the above example is reduced by a factor of 2, transforming it into an image with a resolution of 64*64 and 64 channels.

[0106] In color enhancement computations, dilated convolutional layers can be used instead of encoder units. Encoder units reduce resolution and lose pixel detail, while dilated convolutions increase the receptive field using expanded kernels without significantly increasing computational cost or reducing spatial resolution.

[0107] This invention does not limit the structure or number of dilated convolutional layers. For example, a dilated convolutional layer can be a 3x3 dilated convolution, with four dilated convolutional layers and dilation factors of 2, 3, 5, and 8 respectively. Each dilated convolutional layer is followed by a ReLU activation function layer and a residual connection from before the dilated convolution to after the last dilated convolution.

[0108] To facilitate subsequent merging with the first reconstructed image obtained through super-resolution processing, a magnification layer can be applied after color enhancement using a dilated convolutional layer to enlarge the resulting feature map to the target resolution. For example, enlarging the 64*64 resolution image with 64 channels obtained in the previous example by a factor of 4 will yield a chroma output image with the same resolution as the first reconstructed image.

[0109] Optionally, the magnification layer is a pixel blending layer, so the clarity of the feature map obtained through pixel blending will not be reduced. For example, in the example above, if a feature map with a resolution of 64*64 and 64 channels is to be magnified by 4 times, pixel blending will result in a feature map with a resolution of 256*256 and 4 channels.

[0110] Optionally, an output layer is used to obtain the chroma output map. The output layer is used to perform convolutional dimensionality reduction on the feature map, transforming the high-resolution feature map into a high-resolution image, i.e., the chroma output map. The output layer can be a 3*3 convolutional layer; in the example above, it transforms a feature map with 4 channels into a UV channel map with 2 channels.

[0111] The global semantic information map is used to guide the color enhancement processing of the first image. Merging the global semantic information map with the chroma map of the first image and then performing dilated convolution reduces computation while obtaining a larger receptive field, thus enhancing the color enhancement effect. Further pixel blending allows for the enlargement of the color-enhanced image, resulting in a chroma output map with resolution and sharpness comparable to the first reconstructed image. The resolution and sharpness of the first reconstructed image are also comparable to the first reconstructed image; merging the two enhances the overall image processing quality.

[0112] Optionally, in the color suppression layer, the chroma output map is adjusted based on a color control factor to obtain the second reconstructed image. Since different first images have varying color saturation, this difference in saturation needs to be considered when performing color enhancement on different first images. By determining the corresponding color control factor using the global semantic information map of the first image and applying this color control factor to adjust the chroma output map, the degree of color enhancement can be controlled specifically.

[0113] Optionally, when adjusting the chroma output image based on the color control factor, a chroma enhancement image is also introduced. The chroma enhancement image can be obtained by upsampling the chroma image of the first image. For example, in the above example, the resolution of the chroma image is 128*128, and after upsampling, the resolution of the obtained chroma enhancement image is 256*256.

[0114] The second reconstructed image is obtained by adjusting the chromaticity output map and chromaticity enhancement map based on the color control factor.

[0115] Alternatively, the second reconstructed image can be obtained using the following formula:

[0116] U_HR=αkU_out+(1-αk)U_u

[0117] Where U_HR is the second reconstructed image, U_out is the chroma output image, U_u is the chroma enhancement image, k is the color control factor, and α is the attenuation factor. Because the color distribution of the dataset is often not ideal, oversaturation after enhancement is easily generated in the saturation region of a certain color. By limiting α to 0.85, oversaturation is further avoided.

[0118] The resulting second reconstructed image still has a resolution of 256*256 and 2 channels.

[0119] A chroma-enhanced image is obtained by upsampling the chroma map of the first image. The saturation of the chroma-enhanced image is then used to guide the generation of the second reconstructed image. The resulting second reconstructed image takes into account the original saturation of the first image and thus performs targeted color enhancement.

[0120] Step 104: Fuse the first reconstructed image and the second reconstructed image to obtain the second image.

[0121] The first and second reconstructed images are merged.

[0122] By combining super-resolution processing in the Y channel and color enhancement processing in the UV channel, a high-resolution first reconstructed image and a color-enhanced second reconstructed image were obtained. The two images were then merged and converted to the RGB color space to obtain the final output image with the combined resolution and color enhancement.

[0123] To facilitate understanding, the above image processing methods will be explained in general below. For example... Figure 7 As shown, the first image is an RGB image, which will be used as an example for introduction.

[0124] Step 701: Input the first image.

[0125] Step 702: Extract the global semantic information map of the first image.

[0126] Step 703: Extract the color control factor of the first image.

[0127] Step 704: Convert the RGB image to a YUV image.

[0128] Step 705: Extract the Y channel image from the YUV image, combine it with the global semantic information map, and perform super-resolution processing on the Y channel image to obtain the first reconstructed image.

[0129] Step 706: Extract the UV channel image from the YUV image, combine the global semantic information map and color control factor, and perform color enhancement processing on the UV channel image to obtain the second reconstructed image;

[0130] Step 707: After fusing the first reconstructed image and the second reconstructed image, the second image is obtained and converted into an RGB image for output.

[0131] Furthermore, training for super-resolution and color enhancement using the aforementioned methods requires a large number of training image pairs. Generally, expert color correction is used as the ground truth, and the lightweight network is trained by establishing paired training sets. However, manual color correction methods struggle to generate a sufficient number of training images, and the colors in the training sets often exhibit color casts. This leads to color casts or oversaturation in some flat and highly saturated pure color areas in current color enhancement algorithms, such as grass and sunsets. Consequently, inconsistent enhancement results result in a flickering effect in video playback.

[0132] To address the issue of high manual processing workload in color enhancement datasets, this invention also provides a method for obtaining training samples.

[0133] First, generate professionally paired HDR and SDR high-resolution video sequences for the same video file.

[0134] Then, images are extracted frame by frame from both video sequences, thus obtaining two versions of the same image: HDR and SDR.

[0135] Then, the SDR image was used as the calibration ground truth for color enhancement; the HDR image was subjected to simple tone mapping to achieve color distortion, and further downsampled to serve as a low-resolution, color-distorted input image. This yielded a large number of sample pairs without the need for manual annotation.

[0136] Optionally, the sample pairs can be cut into a large number of pairs of 128×128 image patches for training, using the commonly used L1 loss.

[0137] Optionally, during the training of global semantic information extraction, common scenes such as sky, skin color, trees, and grass in the video dataset are manually labeled, and the global semantic extraction module is pre-trained and fixed, and not updated in subsequent training.

[0138] This invention also provides an image processing apparatus, such as... Figure 8 As shown, it includes:

[0139] Processing unit 801 is used for:

[0140] The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image;

[0141] Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image;

[0142] Based on the global semantic information graph, the chroma graph of the first image is subjected to color enhancement processing to obtain the second reconstructed image;

[0143] The first reconstructed image and the second reconstructed image are fused together to obtain the second image.

[0144] This invention also provides a computing device, comprising:

[0145] Memory, used to store computer programs;

[0146] The processor is configured to invoke a computer program stored in the memory and execute the image processing method listed in any of the above methods according to the obtained program.

[0147] This invention also provides a computer-readable storage medium storing a computer-executable program for causing a computer to perform any of the image processing methods listed above.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized by, include: The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image; Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image; Based on the global semantic information map, color enhancement processing is performed on the chroma map of the first image to obtain a second reconstructed image; the step of performing color enhancement processing on the chroma map of the first image based on the global semantic information map to obtain a second reconstructed image includes: determining the color control factor of the first image through the global semantic information map; performing a fusion convolution processing on the global semantic information map and the chroma map of the first image to obtain a chroma output map; adjusting the chroma output map based on the color control factor to obtain the second reconstructed image; the method further includes: upsampling the chroma map of the first image to obtain a chroma enhancement map; adjusting the chroma output map based on the color control factor to obtain the second reconstructed image, including: adjusting the chroma output map and the chroma enhancement map based on the color control factor to obtain the second reconstructed image; The first reconstructed image and the second reconstructed image are fused together to obtain the second image.

2. The method as described in claim 1, characterized in that, The global semantic information map is fused and convolved with the chroma map of the first image to obtain a chroma output map, including: After merging the global semantic information map with the chroma map of the first image, the pixels are mixed after dilated convolution to obtain the chroma output map.

3. The method according to any one of claims 1 to 2, characterized in that, The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image, including: The first image is convolved through the encoder unit to obtain the first encoded image; The first encoded image is magnified to obtain a global semantic information map of the first image.

4. The method as described in claim 3, characterized in that, Determining the color control factor of the first image using the global semantic information graph includes: The color control factor of the first image is obtained by processing the first encoded image through a pooling layer and a fully connected layer.

5. The method according to any one of claims 1 to 2, characterized in that, Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image, including: Extract the grayscale image from the first image; The grayscale image of the first image is subjected to a first residual processing to obtain a first grayscale feature map; After merging the first grayscale feature map and the global semantic information map, a second residual processing is performed to obtain the second grayscale feature map; The second grayscale feature map is pixel-wise mixed to obtain the first reconstructed image.

6. An image processing apparatus, characterized in that, include: Processing unit, used for: The first image is processed by a convolutional neural network to obtain a global semantic information map of the first image; Based on the global semantic information map, the grayscale image of the first image is magnified to obtain the first reconstructed image; Based on the global semantic information map, the chroma map of the first image is subjected to color enhancement processing to obtain a second reconstructed image; the processing unit is specifically used to: determine the color control factor of the first image through the global semantic information map; perform fusion convolution processing on the global semantic information map and the chroma map of the first image to obtain a chroma output map; adjust the chroma output map based on the color control factor to obtain the second reconstructed image; the processing unit is further used to: perform upsampling processing on the chroma map of the first image to obtain a chroma enhancement map; the processing unit is specifically used to: adjust the chroma output map and the chroma enhancement map based on the color control factor to obtain the second reconstructed image; The first reconstructed image and the second reconstructed image are fused together to obtain the second image.

7. A computing device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method according to any one of claims 1 to 5 in accordance with the obtained program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program for causing a computer to perform the method according to any one of claims 1 to 5.

Images