Image enhancement model training method and device, image processing method and device and medium

Abstract

The embodiment of the invention provides an image enhancement model training method and device, an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: processing an input image through a to-be-trained image enhancement model to obtain an output image; analyzing the input image and the output image to obtain an image enhancement loss value; analyzing the output image and the high-quality image to obtain an image perception loss value; and training the to-be-trained image enhancement model based on a loss function generated by the image enhancement loss value and the image perception loss value to obtain a target image enhancement model. According to the method, the to-be-trained image enhancement model is trained on the basis of the image enhancement loss value and the image perception loss value, so that the target image enhancement model obtained through training can reduce the compression coding burden while ensuring the enhanced image visual quality, and balance between the enhancement quality and the compression efficiency is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image enhancement model training method, an image processing method, a device, an electronic device, a computer program product, and a storage medium. Background Art

[0002] With the rapid advancement of video image technology, viewers can enjoy more vivid and high-quality content, which has greatly promoted the prosperity of the short video industry. In the process from video creation to distribution, the quality of the video directly affects the content quality and user experience. Through image enhancement and restoration technology, the quality of image content can be effectively improved and the user's viewing experience can be enhanced.

[0003] However, as the clarity and texture complexity of image enhancement increase, the burden of video compression and encoding will increase, resulting in an increase in bit rate, which may cause resource waste and playback freezes. Therefore, while pursuing image quality improvement, how to effectively reduce the burden of compression and encoding has become a key issue that needs to be solved to ensure a smooth user experience while improving visual effects. Summary of the invention

[0004] The embodiments of the present disclosure provide an image enhancement model training method, an image processing method, an apparatus, an electronic device, a computer program product, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image enhancement model training method, comprising: obtaining a training data set comprising a plurality of image training samples; the image training samples comprising a corresponding input image and a high-quality image; performing image enhancement processing on the input image through the image enhancement model to be trained to obtain an output image; determining an image enhancement difference based on the input image and the output image; determining an image enhancement loss value based on the output image and the image enhancement difference; determining an image perception loss value based on the output image and the high-quality image; generating a loss function based on the image enhancement loss value and the image perception loss value; and training the image enhancement model to be trained based on the loss function to obtain a target image enhancement model.

[0006] In some exemplary embodiments of the present disclosure, calculating the image enhancement difference based on the input image and the output image includes: determining at least one image sub-difference between the input image and the output image; each of the image sub-difference corresponds to an image attribute dimension; and determining the image enhancement difference based on the at least one image sub-difference.

[0007] In some exemplary embodiments of the present disclosure, determining the image enhancement difference based on the input image and the output image includes: determining a clarity sub-difference and a texture complexity sub-difference between the input image and the output image; and determining the image enhancement difference based on the clarity sub-difference and the texture complexity sub-difference.

[0008] In some exemplary embodiments of the present disclosure, determining the image enhancement loss value based on the output image and the image enhancement difference includes: performing a discrete cosine transform (DCT) on the output image to obtain an image spectrum distribution diagram; determining a high-frequency area in the image spectrum distribution diagram; determining a high-frequency coefficient threshold based on the image enhancement difference; and searching in the high-frequency area based on the high-frequency coefficient threshold to determine the image enhancement loss value.

[0009] In some exemplary embodiments of the present disclosure, determining the high-frequency area in the image spectrum distribution map includes: determining a target area located in a lower right corner of the image spectrum distribution map as the high-frequency area.

[0010] In some exemplary embodiments of the present disclosure, determining the high-frequency coefficient threshold based on the image enhancement difference includes: obtaining the high-frequency coefficients corresponding to each coefficient item in the high-frequency area; and determining the high-frequency coefficient threshold based on the image enhancement difference and the high-frequency coefficients corresponding to each coefficient item.

[0011] In some exemplary embodiments of the present disclosure, the searching in the high frequency area based on the high frequency coefficient threshold to determine the image enhancement loss value includes: searching in the high frequency area based on the high frequency coefficient threshold to obtain at least one first coefficient term; the high frequency coefficient corresponding to the first coefficient term is less than the high frequency coefficient threshold; and determining the image enhancement loss value based on the high frequency coefficient corresponding to the at least one first coefficient term.

[0012] In some exemplary embodiments of the present disclosure, determining the image perceptual loss value based on the output image and the high-quality image includes: performing perceptual loss calculation on the output image and the high-quality image through a perceptual loss model to obtain the image perceptual loss value.

[0013] In some exemplary embodiments of the present disclosure, generating a loss function according to the image enhancement loss value and the image perception loss value includes: performing a weighted summation of the image enhancement loss value and the image perception loss value to obtain the loss function.

[0014] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: acquiring an image to be processed; inputting the image to be processed into an image enhancement model; the image enhancement model is obtained by training through any one of the image enhancement model training methods described; and the image enhancement model performs image enhancement processing on the image to be processed to obtain a target image.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an image enhancement model training device, comprising: a sample acquisition module, configured to acquire a training data set comprising a plurality of image training samples; the image training samples comprise a corresponding input image and a high-quality image; an image enhancement module, configured to perform image enhancement processing on the input image through the image enhancement model to be trained to obtain an output image; a difference calculation module, configured to determine an image enhancement difference according to the input image and the output image; an image enhancement loss module, configured to determine an image enhancement loss value according to the output image and the image enhancement difference; an image perception loss module, configured to determine an image perception loss value according to the output image and the high-quality image; a loss function module, configured to generate a loss function according to the image enhancement loss value and the image perception loss value; and a model training module, configured to train the image enhancement model to be trained according to the loss function to obtain a target image enhancement model.

[0016] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: an image acquisition module, configured to acquire an image to be processed; an image input module, configured to input the image to be processed into an image enhancement model; the image enhancement model is obtained by training through any one of the image enhancement model training methods described; and an image processing module, configured to perform image enhancement processing on the image to be processed by the image enhancement model to obtain a target image.

[0017] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement any one of the image enhancement model training methods or any one of the image processing methods.

[0018] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the image enhancement model training methods or any one of the image processing methods.

[0019] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein a processor executes any one of the image enhancement model training methods or any one of the image processing methods.

[0020] The image enhancement model training method provided by the disclosed embodiment processes the input image through the image enhancement model to be trained to obtain the output image; obtains the image enhancement loss value by analyzing the input image and the output image; obtains the image perception loss value by analyzing the output image and the high-quality image; and trains the image enhancement model to be trained based on the loss function generated by the image enhancement loss value and the image perception loss value to obtain the target image enhancement model. The method characterizes the loss measurement value of the non-important part of the output image through the image enhancement loss value, and characterizes the image quality loss between the output image and the high-quality image through the image perception loss value. By training the image enhancement model to be trained based on the image enhancement loss value and the image perception loss value, the trained target image enhancement model can reduce the compression coding burden while ensuring the visual quality of the enhanced image, thereby achieving a balance between enhanced quality and compression efficiency.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0023] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0024] Figure 2 The figure is a flowchart of an image enhancement model training method according to an exemplary embodiment.

[0025] Figure 3 The figure is a flow chart of a method for calculating image enhancement difference according to an example.

[0026] Figure 4 is a schematic diagram illustrating a process of calculating image enhancement difference according to an example.

[0027] Figure 5 is a flowchart of a method for calculating an image enhancement loss value according to an example.

[0028] Figure 6 is a flowchart of a method for calculating a high frequency coefficient threshold according to an example.

[0029] Figure 7 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0030] Figure 8 It is a block diagram of an image enhancement model training device according to an exemplary embodiment.

[0031] Fig. 9 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0032] Fig.10 It is a schematic diagram showing the structure of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0034] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0035] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0038] Figure 1 A schematic diagram showing an exemplary system architecture to which the method according to an embodiment of the present disclosure can be applied is shown.

[0039] like Figure 1 As shown, the system architecture may include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used to provide a medium for a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The server 101 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal device 103, the terminal device 104, or the terminal device 105. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device 103, the terminal device 104, or the terminal device 105.

[0041] Terminal device 103, terminal device 104 and terminal device 105 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, etc., but are not limited thereto.

[0042] It should be understood that Figure 1 The number of terminal devices 103, terminal devices 104, terminal devices 105, networks 102 and servers 101 are merely illustrative. Server 101 may be a physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, any number of terminal devices, networks and servers may be provided.

[0043] Below, each step of the method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0044] Figure 2 The figure is a flowchart of an image enhancement model training method according to an exemplary embodiment. Figure 2 The method provided in the embodiment can be executed by any electronic device, such as the above Figure 1 The terminal device in Figure 1 Servers in, or Figure 1 The terminal device and the server in the embodiment are jointly executed, but the present disclosure is not limited to this.

[0045] In step S210, a training data set including a plurality of image training samples is obtained; the image training samples include input images and high-quality images in a corresponding relationship.

[0046] In the disclosed embodiment, a large number of image training samples are obtained in advance to train the image enhancement model and construct a training data set. Each image training sample includes at least one set of corresponding input images and high-quality images. The input image is used as the image to be enhanced in the image training sample. The high-quality image is used as the target image in the image training sample.

[0047] In step S220, the input image is enhanced by the image enhancement model to be trained to obtain an output image.

[0048] In the disclosed embodiment, an image enhancement model to be trained is pre-built to perform image enhancement processing on the input image in each image training sample to obtain a corresponding output image. Based on the comparison between the output image and the corresponding high-quality image, the image enhancement model to be trained is trained.

[0049] In step S230, an image enhancement difference is determined according to the input image and the output image.

[0050] In the disclosed embodiment, the image enhancement difference between the input image and the output image processed by the image enhancement model is calculated by comparing the input image and the output image processed by the image enhancement model. The image enhancement difference is used to characterize the difference between the image before and after the image enhancement processing by the image enhancement model.

[0051] It should be noted that the calculation of the image enhancement difference is not limited to the change of a single image attribute dimension, but can also be a comprehensive consideration from multiple image attribute dimensions to fully reflect the difference before and after image enhancement. The image attribute dimensions may include: clarity, texture complexity, contrast, brightness, image size, etc. The change in each dimension can reflect different aspects of image enhancement. For example, the improvement of clarity may mean that the blurred area becomes sharper, while the increase in texture complexity means richer details.

[0052] In an exemplary embodiment, for each selected image attribute dimension, we can independently calculate its change before and after enhancement, assign different weights as needed, and finally integrate it into a comprehensive image enhancement difference. This method can ensure that the difference not only reflects the improvement of a single attribute, but also reflects the improvement of the overall image quality. At the same time, the weights between different dimensions can also be adjusted according to the needs of the application scenario.

[0053] In step S240, an image enhancement loss value is determined according to the output image and the image enhancement difference.

[0054] A frame of image includes both parts that are more important to the user's visual perception and parts that are relatively less important. In the process of image enhancement, image enhancement of important parts can significantly improve the visual effect, while image enhancement of unimportant parts will not bring obvious visual effect improvement, but will increase the image encoding burden. Based on this, the training goal of the image enhancement model should be to minimize the increased encoding burden of unimportant parts of the image due to image enhancement processing and increase the image enhancement amplitude of important parts of the image.

[0055] In the embodiment of the present disclosure, an image enhancement loss value is determined based on the output image and the image enhancement difference. The image enhancement loss value is used to characterize the loss measurement value of the non-important part of the output image. It should be pointed out that the focus of the image enhancement model training method is to train the model by characterizing the image enhancement loss value of the non-important part of the image as part of the loss function, so that the model can reduce the compression coding burden while ensuring that the image enhancement quality is not significantly affected. Under this design concept, the present disclosure does not limit the specific algorithm for calculating the image enhancement loss value based on the output image and the image enhancement difference.

[0056] In an exemplary embodiment, based on the image enhancement difference, an image analysis is performed on the output image to determine a non-important part of the image. The non-important part may be a partial area in the image or a set of pixels with target features in the image. The non-important part is statistically calculated and converted into an image enhancement loss value that can be used for loss measurement.

[0057] In step S250, an image perception loss value is determined according to the output image and the high-quality image.

[0058] As mentioned earlier, the goal of model training is to strike a balance between enhancement quality and compression efficiency. Therefore, in addition to the image enhancement loss value used to characterize the loss of non-important parts, it is also necessary to measure the loss of image enhancement quality.

[0059] In the embodiment of the present disclosure, by comparing the output image and the high-quality image, the image perception loss value between the two is calculated. The image perception loss value is used to characterize the image quality loss between the output image and the high-quality image. The smaller the difference between the output image and the high-quality image in image quality, the smaller the image quality loss. Conversely, the greater the difference between the output image and the high-quality image in image quality, the greater the image quality loss. Under this design concept, the present disclosure does not limit the specific algorithm for calculating the image perception loss value based on the output image and the high-quality image.

[0060] In an exemplary embodiment, a perceptual loss model for estimating image quality loss may be pre-trained. The perceptual loss of the output image and the high-quality image is calculated by the perceptual loss model to obtain the perceptual loss value of the image. It should be noted that there are many model architectures for evaluating image quality loss, which are not specifically limited here. For example, Watson perceptual supervision constraints may be introduced into the perceptual loss model to estimate the perceptual loss between images.

[0061] In step S260, a loss function is generated according to the image enhancement loss value and the image perception loss value.

[0062] In the disclosed embodiment, the image enhancement loss value representing the loss of the non-important part and the image perception loss value representing the loss of the image quality are obtained through the aforementioned steps S250 and S260, respectively. The loss function is generated by integrating the image enhancement loss value and the image perception loss value to achieve a balance between the enhancement quality and the compression efficiency.

[0063] In an exemplary embodiment, the image enhancement loss value and the image perception loss value may be weighted and summed to obtain a loss function. The weight between the two loss values ​​may be adjusted according to the needs of the application scenario to achieve a desired balance effect.

[0064] In step S270, the image enhancement model to be trained is trained according to the loss function to obtain a target image enhancement model.

[0065] In the disclosed embodiment, the image enhancement model to be trained is trained according to the loss function that integrates the image enhancement loss value and the image perception loss value. During the training process, the model parameters are continuously optimized based on the loss function using the back propagation algorithm, so that the image enhancement model to be trained can learn to balance between enhancement quality and compression efficiency, thereby obtaining a target image enhancement model that meets the expected balance.

[0066] The image enhancement model training method provided by the disclosed embodiment processes the input image through the image enhancement model to be trained to obtain the output image; obtains the image enhancement loss value by analyzing the input image and the output image; obtains the image perception loss value by analyzing the output image and the high-quality image; and trains the image enhancement model to be trained based on the loss function generated by the image enhancement loss value and the image perception loss value to obtain the target image enhancement model. The method characterizes the loss measurement value of the non-important part of the output image through the image enhancement loss value, and characterizes the image quality loss between the output image and the high-quality image through the image perception loss value. By training the image enhancement model to be trained based on the image enhancement loss value and the image perception loss value, the trained target image enhancement model can reduce the compression coding burden while ensuring the visual quality of the enhanced image, thereby achieving a balance between enhanced quality and compression efficiency.

[0067] Figure 3 FIG. 1 is a flow chart of a method for calculating image enhancement difference according to an example. Figure 3 As shown, in the embodiment of the present disclosure, the image enhancement difference calculation process of the aforementioned step S230 may include the following steps.

[0068] In step S310, at least one image sub-difference between the input image and the output image is determined; each of the image sub-difference corresponds to an image attribute dimension.

[0069] In the disclosed embodiment, the calculation of the image enhancement difference is not limited to the change of a single image attribute dimension, but can also be a comprehensive consideration from multiple image attribute dimensions to fully reflect the difference before and after image enhancement. A plurality of image difference operators are pre-configured. According to the plurality of image difference operators, each image sub-difference of the image attribute dimension corresponding to the input image and the output image is calculated respectively. Each image difference operator corresponds to a different image attribute dimension. Accordingly, each image sub-difference determined based on different operators corresponds to an image attribute dimension.

[0070] In an exemplary embodiment, at least one image difference operator is obtained from a pre-constructed image difference operator library. Each of the image difference operators corresponds to an image attribute dimension. The image difference operator is used to calculate the corresponding difference value between two images based on the corresponding image attribute dimension. The present disclosure does not limit the specific difference calculation algorithm used by each difference operator, and any algorithm can be selected according to actual needs to calculate the corresponding difference.

[0071] In an exemplary embodiment, difference operator 1, difference operator 2, difference operator 3, and difference operator 4 corresponding to definition, texture complexity, contrast, and brightness are pre-constructed. The difference between the input image and the output image is calculated by the four difference operators to obtain image sub-difference 1, image sub-difference 2, image sub-difference 3, and image sub-difference 4. The four image sub-differences are used to measure the difference between the input image and the output image in definition, texture complexity, contrast, and brightness containment.

[0072] In step S320, the image enhancement difference is determined according to the at least one image sub-difference.

[0073] In the disclosed embodiment, the image enhancement difference is determined based on each image sub-difference determined by the aforementioned image difference operators. The image enhancement difference is obtained by comprehensive consideration of each image sub-difference, and can fully reflect the difference before and after image enhancement. The value of the image enhancement difference should generally be greater than 1.

[0074] In an exemplary embodiment, each image sub-difference may be weighted and summed to obtain the image enhancement difference. The weights between each image sub-difference may be adjusted according to the needs of the application scenario to obtain an image enhancement difference that meets the needs of the scenario.

[0075] In an exemplary embodiment, the arithmetic mean of the differences of the image sub-differencings may be calculated to obtain the image enhancement difference.

[0076] Since, after image enhancement, the improvement of clarity and texture complexity will significantly increase the burden of compression coding, so that the compression bit rate is likely to increase significantly, therefore, in the embodiment of the present disclosure, the image enhancement difference is calculated based on the clarity and texture complexity dimensions.

[0077] Figure 4 FIG. 1 is a schematic diagram showing a process of calculating image enhancement difference according to an example. Figure 4 As shown, in the embodiment of the present disclosure, the image enhancement difference calculation process may include the following steps.

[0078] Determining a sharpness sub-difference and a texture complexity sub-difference between the input image and the output image;

[0079] The image enhancement difference is determined according to the definition sub-difference and the texture complexity sub-difference.

[0080] In the disclosed embodiment, a clarity difference operator and a texture complexity difference operator are pre-set. The clarity difference operator is used to calculate the corresponding difference value between two images based on the clarity dimension. The texture complexity difference operator is used to calculate the difference between two images based on the texture complexity dimension. The difference between the input image and the output image is calculated by the clarity difference operator and the texture complexity difference operator respectively. The clarity sub-difference D1 and the texture complexity sub-difference D2 are obtained by the clarity difference operator and the texture complexity difference operator respectively.

[0081] In an exemplary embodiment, the calculation method of the clarity difference operator may include the following steps.

[0082] Perform edge enhancement processing on input images and output images;

[0083] The edge sharpness information of the input image and the output image after edge enhancement is determined respectively by a sliding window;

[0084] By comparing the edge sharpness information corresponding to the input image and the output image, the definition sub-difference D1 is determined.

[0085] In this exemplary embodiment, first, edge enhancement is performed on the input image and the output image respectively by an edge enhancement algorithm to obtain the edge-enhanced input image and the output image. Specifically, the Laplacian operator can be applied to process the input image and the output image respectively to obtain the second-order gradient, thereby highlighting the edge information in the image.

[0086] Next, edge sharpness calculation can be performed on the edge parts of the input image and the output image after edge enhancement through a preset sliding window to obtain corresponding edge sharpness information. Specifically, for the edge parts of the input image and the output image after edge enhancement, a 3x3 window is used to obtain variance to obtain a second-order variance reflecting the edge sharpness. The second-order variance is used to characterize the corresponding edge sharpness.

[0087] Finally, by comparing the edge sharpness information corresponding to the input image and the output image, the definition sub-difference D1 is determined. Specifically, the definition sub-difference D1 is obtained by calculating the ratio of the second-order variance of the output image to the second-order variance of the input image.

[0088] In an exemplary embodiment, the method for calculating the texture complexity difference operator may include the following steps: extracting edge information of an input image and an output image;

[0089] The edge information of the input image and the output image are evaluated respectively through a sliding window to determine the edge strength information;

[0090] By comparing the edge strength information corresponding to the input image and the output image, the texture complexity sub-difference D2 is determined.

[0091] In this exemplary embodiment, first, edge information in the input image and the output image is extracted. Specifically, the edge information of the input image and the output image is extracted respectively using the Canny edge detection operator. The Canny edge detection operator is used to identify edge information in an image.

[0092] Next, the edge information of the input image and the output image is evaluated by a pre-set sliding window to determine the corresponding edge strength information. Specifically, the variance of the extracted edge information is calculated using a 3x3 window, which reflects the change of the first-order derivative, that is, the first-order variance. The first-order variance is used to characterize the change of edge strength and quantify the complexity of details in the edge area.

[0093] Finally, by comparing the edge strength information corresponding to the input image and the output image, the texture complexity sub-difference D2 is determined. Specifically, the texture complexity sub-difference D2 is obtained by calculating the ratio of the first-order variance of the output image to the first-order variance of the input image.

[0094] In the embodiment of the present disclosure, the image enhancement difference D3 is determined based on the definition sub-difference D1 and the texture complexity sub-difference D2. According to the actual application scenario, the image enhancement difference D3 can be determined based on the definition sub-difference D1 and the texture complexity sub-difference D2 by a variety of algorithms. The present disclosure does not limit the specific algorithm used.

[0095] In an exemplary embodiment, the definition sub-difference D1 and the texture complexity sub-difference D2 may be weightedly summed to obtain the image enhancement difference D3.

[0096] In an exemplary embodiment, the image enhancement difference D3 may be determined based on the ratio of the texture complexity sub-difference D2 to the definition sub-difference D1, taking the definition sub-difference D1 as a reference. Specifically, the image enhancement difference D3 may be calculated according to the following calculation formula.

[0097] D3=(D1 2 +D2 2 ) / D1

[0098] According to the calculation formula, the image enhancement difference D3 is obtained. When the difference between the clarity and texture complexity before and after enhancement is large, the value of D3 is large. Otherwise, the value of D3 is small.

[0099] The image enhancement model training method provided by the embodiment of the present disclosure calculates the difference between the input image and the output image respectively through multiple difference operators corresponding to different image attribute dimensions. Then, the image enhancement difference is determined based on multiple image sub-differences of each attribute dimension. The image enhancement difference determined by this method can comprehensively reflect the difference before and after image enhancement from a comprehensive consideration of multiple image attribute dimensions.

[0100] Figure 5 FIG. 1 is a flowchart of a method for calculating an image enhancement loss value according to an example. Figure 5 As shown, in the embodiment of the present disclosure, the image enhancement loss value calculation process of the aforementioned step S240 may include the following steps.

[0101] In step S510, a discrete cosine transform (DCT) is performed on the output image to obtain an image spectrum distribution diagram.

[0102] As mentioned above, in the process of calculating the image enhancement loss value, it is first necessary to determine the non-important parts of the image by performing image analysis on the output image. Then, the non-important parts are statistically calculated based on the image enhancement difference and converted into image enhancement loss values ​​that can be used for loss measurement.

[0103] In the disclosed embodiment, the output image is subjected to a discrete cosine transform to obtain a corresponding image spectrum distribution diagram. The DCT transform is a mathematical transform in image processing, similar to the discrete Fourier transform, but only uses real numbers. The DCT transform can convert the output image from the spatial domain to the frequency domain to generate a corresponding image spectrum distribution diagram. The image spectrum distribution diagram shows the energy distribution of the image at different frequencies. Among them, in the image spectrum distribution diagram, the low-frequency part is concentrated in the upper left corner of the image, and the high-frequency part is concentrated in the lower right corner of the image. The image spectrum distribution diagram obtained by the transformation is divided into multiple coefficient items. Each coefficient item corresponds to a different frequency. Each coefficient item corresponds to a coefficient value. The coefficient value is used to represent the energy corresponding to the coefficient item.

[0104] In an exemplary embodiment, the output image may be subjected to DCT transformation using a preset window size, which may be 8×8.

[0105] In step S520, a high-frequency area in the image frequency spectrum distribution diagram is determined.

[0106] Since the human eye is more sensitive to low-frequency information and less sensitive to high-frequency information, the high-frequency part based on the image spectrum distribution diagram can be considered as a non-important part of the image.

[0107] In the disclosed embodiment, the target high frequency area is determined from the image spectrum distribution diagram according to the above principle. The target high frequency area can be considered as a non-important part of the image.

[0108] It should be noted that the size of the target high-frequency area or the division threshold of the target high-frequency area can be determined according to the requirements of the application scenario. The present disclosure does not limit the specific method of determining the high-frequency area.

[0109] In an exemplary embodiment, since in the image spectrum distribution diagram, the low frequency part is concentrated in the upper left corner of the image, and the high frequency part is concentrated in the lower right corner of the image. Therefore, the target area located in the lower right corner of the image spectrum distribution diagram is determined as the target high frequency area. The shape of the target area can be a triangle.

[0110] In an exemplary embodiment, we can also use the window selected in the DCT transform as a unit. In each window, the target area located in the lower right corner of the window is the target high-frequency area. The shape of the target area can be a triangle.

[0111] In step S530, a high frequency coefficient threshold is determined according to the image enhancement difference.

[0112] Although the non-important parts of the image are mainly concentrated in its high-frequency parts. However, some high-frequency components also play a relatively important role in the visual quality of the entire image. Therefore, in order to screen the aforementioned high-frequency area, the present disclosure also determines a high-frequency coefficient threshold based on the aforementioned image enhancement difference. Through the high-frequency coefficient threshold, the high-frequency part is further screened to determine the non-important part.

[0113] It should be noted that, according to the requirements of the application scenario, there may be many methods for determining the high-frequency coefficient threshold, and the present disclosure does not further limit the method for determining the high-frequency coefficient threshold.

[0114] In some exemplary embodiments, the high frequency coefficient threshold may be determined by the following method.

[0115] Figure 6 is a flow chart of a method for calculating a high frequency coefficient threshold according to an example. Figure 6 As shown, the high-frequency coefficient threshold calculation method process may include the following steps.

[0116] In step S610, high frequency coefficients corresponding to each coefficient item in the high frequency region are obtained.

[0117] As mentioned above, the image spectrum distribution diagram is divided into a plurality of coefficient items, and each coefficient item corresponds to a coefficient value.

[0118] In the embodiment of the present disclosure, based on the above-mentioned determination of the high-frequency region, the coefficient values ​​corresponding to the coefficient items in the region are obtained as the high-frequency coefficients.

[0119] In step S620, the high-frequency coefficient threshold is determined according to the image enhancement difference and the high-frequency coefficients corresponding to each coefficient item.

[0120] In the disclosed embodiment, the high-frequency coefficient threshold is determined based on the image enhancement difference determined above and the high-frequency coefficients corresponding to each coefficient item in the high-frequency region. It should be noted that the numerical statistical method for determining the high-frequency coefficient threshold based on the image enhancement difference and the high-frequency coefficients corresponding to each coefficient item can be determined in different numerical statistical methods according to the needs of the actual application scenario.

[0121] In an exemplary embodiment, the high frequency coefficient threshold may be determined by calculation:

[0122] Calculate the average value of the absolute values ​​of the high-frequency coefficients corresponding to the coefficient items to obtain the average value of the high-frequency coefficients;

[0123] The high-frequency coefficient threshold is determined according to the image enhancement difference and the high-frequency coefficient average value.

[0124] In this calculation method, the average value of the absolute value of the high-frequency coefficient corresponding to each coefficient item in the high-frequency region is calculated to obtain the high-frequency coefficient average value M. The high-frequency coefficient threshold N is determined according to the image enhancement difference D3 and the high-frequency coefficient average value M. For example, the high-frequency coefficient threshold N is the product of the image enhancement difference D3 and the high-frequency coefficient average value M. That is, N=M·D3.

[0125] In step S540, the high frequency region is searched based on the high frequency coefficient threshold to determine the image enhancement loss value.

[0126] In the embodiment of the present disclosure, a search is performed in the high frequency region based on the high frequency coefficient threshold, and the high frequency region is further screened to determine a non-important part. Based on the non-important part, the image enhancement loss value is determined.

[0127] In an exemplary embodiment, in step S540, the process of determining the image enhancement loss value may include the following steps.

[0128] Searching in the high frequency region based on the high frequency coefficient threshold value to obtain at least one first coefficient term; the high frequency coefficient corresponding to the first coefficient term is less than the high frequency coefficient threshold value;

[0129] The image enhancement loss value is determined according to the high-frequency coefficient corresponding to the at least one first coefficient term.

[0130] In the disclosed embodiment, a search is performed in the high-frequency region based on the high-frequency coefficient threshold to obtain at least one first coefficient term in the high-frequency region. The first coefficient term is a coefficient term whose high-frequency coefficient is less than the high-frequency coefficient threshold. Based on the high-frequency coefficient threshold, it can be considered that the coefficient term whose high-frequency coefficient in the high-frequency region is greater than the high-frequency coefficient threshold is an important part, while the coefficient term whose high-frequency coefficient is less than the high-frequency coefficient threshold is an unimportant part. The high-frequency region is further screened by this method to determine the unimportant part. Based on the high-frequency coefficients corresponding to the multiple first coefficient terms determined by the further screening, data statistics are performed on them to determine the image enhancement loss value. It can be seen that the image enhancement loss value represents the loss measurement value of the unimportant part in the output image. Therefore, in the model training process, the goal should be to approach the image enhancement loss value to zero to minimize the bit rate loss caused by the unimportant part.

[0131] It should be noted that the numerical statistical method for determining the image enhancement loss value based on the high-frequency coefficients corresponding to the multiple first coefficient items can be determined in different numerical statistical methods according to the needs of the actual application scenario.

[0132] In an exemplary embodiment, the image enhancement loss value may be determined by calculation:

[0133] The absolute values ​​of the high-frequency coefficients corresponding to the at least one first coefficient term are accumulated to obtain the image enhancement loss value.

[0134] In this calculation method, the absolute values ​​of the corresponding high-frequency coefficients are accumulated, and the accumulated value is used as the image enhancement loss value.

[0135] The image enhancement model training method provided by the embodiment of the present disclosure converts the output image into a corresponding image spectrum distribution map through DCT transformation. Based on the high-frequency area in the image spectrum distribution map, the non-important part in the image is determined. Based on the image enhancement difference, the high-frequency coefficient threshold is determined. The high-frequency area is searched through the high-frequency coefficient threshold to further screen the high-frequency area to more accurately determine the non-important part in the image. Based on the determined non-important part, the image enhancement loss value is statistically determined, which can characterize the loss measurement value of the non-important part in the output image.

[0136] Figure 7 The figure is a flowchart of an image processing method according to an exemplary embodiment. Figure 7 The method provided in the embodiment can be executed by any electronic device, such as the above Figure 1 The terminal device in Figure 1 Servers in, or Figure 1 The terminal device and the server in the embodiment are jointly executed, but the present disclosure is not limited to this.

[0137] In step S710, an image to be processed is obtained.

[0138] In the embodiment of the present disclosure, an image enhancement request of a user is obtained. The image enhancement request at least includes: an image to be processed. The image to be processed is an input image to be subjected to image enhancement processing.

[0139] In step S720, the image to be processed is input into an image enhancement model; the image enhancement model is obtained by training using any of the aforementioned image enhancement model training methods.

[0140] In step S730, the image enhancement model performs image enhancement processing on the image to be processed to obtain a target image.

[0141] In the disclosed embodiment, the image to be processed in the image enhancement request is input into the image enhancement model trained by any of the aforementioned image enhancement model training methods. The image enhancement model can perform image enhancement processing on the image to be processed based on the model parameters adjusted in the aforementioned training process to achieve a balance between enhancement quality and compression efficiency, thereby obtaining a target image.

[0142] The image processing method provided by the embodiment of the present disclosure inputs the image to be processed into the image enhancement model trained by the aforementioned image enhancement model training method, and performs image enhancement processing on the image to be processed through the image enhancement model to obtain a target image. Since the model takes into account the balance between enhancement quality and compression efficiency during the training process, the target image obtained by performing image enhancement processing based on the image enhancement model reduces the compression coding burden while ensuring the visual quality of the enhanced image.

[0143] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0144] Figure 8 is a block diagram of an image enhancement model training device according to an exemplary embodiment. Figure 8 The device 800 may include: a sample acquisition module 810, an image enhancement module 820, a difference calculation module 830, an image enhancement loss module 840, an image perception loss module 850, a loss function module 860 and a model training module 870.

[0145] The sample acquisition module 810 is configured to acquire a training data set including a plurality of image training samples; the image training samples include input images and high-quality images in a corresponding relationship.

[0146] The image enhancement module 820 is configured to perform image enhancement processing on the input image through the image enhancement model to be trained to obtain an output image.

[0147] The difference calculation module 830 is configured to determine the image enhancement difference according to the input image and the output image.

[0148] The image enhancement loss module 840 is configured to determine an image enhancement loss value according to the output image and the image enhancement difference.

[0149] The image perception loss module 850 is configured to determine an image perception loss value according to the output image and the high-quality image.

[0150] The loss function module 860 is configured to generate a loss function according to the image enhancement loss value and the image perception loss value.

[0151] The model training module 870 is configured to train the image enhancement model to be trained according to the loss function to obtain a target image enhancement model.

[0152] In some exemplary embodiments of the present disclosure, the difference calculation module 830 is further configured to determine at least one image sub-difference between the input image and the output image; each of the image sub-difference corresponds to an image attribute dimension; and determine the image enhancement difference based on the at least one image sub-difference.

[0153] In some exemplary embodiments of the present disclosure, the difference calculation module 830 is further configured to determine a clarity sub-difference and a texture complexity sub-difference between the input image and the output image; and determine the image enhancement difference based on the clarity sub-difference and the texture complexity sub-difference.

[0154] In some exemplary embodiments of the present disclosure, the image enhancement loss module 840 is further configured to perform a discrete cosine transform (DCT) on the output image to obtain an image spectrum distribution diagram; determine a high-frequency area in the image spectrum distribution diagram; determine a high-frequency coefficient threshold based on the image enhancement difference; and search in the high-frequency area based on the high-frequency coefficient threshold to determine the image enhancement loss value.

[0155] In some exemplary embodiments of the present disclosure, the image enhancement loss module 840 is further configured to determine that the target area located at the lower right corner of the image spectrum distribution diagram is the high-frequency area.

[0156] In some exemplary embodiments of the present disclosure, the image enhancement loss module 840 is further configured to obtain high-frequency coefficients corresponding to each coefficient item in the high-frequency area; and determine the high-frequency coefficient threshold according to the image enhancement difference and the high-frequency coefficients corresponding to each coefficient item.

[0157] In some exemplary embodiments of the present disclosure, the image enhancement loss module 840 is also configured to search in the high-frequency area based on the high-frequency coefficient threshold to obtain at least one first coefficient term; the high-frequency coefficient corresponding to the first coefficient term is less than the high-frequency coefficient threshold; and the image enhancement loss value is determined based on the high-frequency coefficient corresponding to the at least one first coefficient term.

[0158] In some exemplary embodiments of the present disclosure, the image perceptual loss module 850 is further configured to perform perceptual loss calculation on the output image and the high-quality image through a perceptual loss model to obtain the image perceptual loss value.

[0159] In some exemplary embodiments of the present disclosure, the loss function module 860 is further configured to perform a weighted summation of the image enhancement loss value and the image perception loss value to obtain the loss function.

[0160] Fig. 9 is a block diagram of an image processing device according to an exemplary embodiment. Fig. 9 The device 900 may include: an image acquisition module 910, an image input module 920 and an image processing module 930.

[0161] The image acquisition module 910 is configured to acquire an image to be processed.

[0162] The image input module 920 is configured to input the image to be processed into an image enhancement model; the image enhancement model is obtained by training through any one of the image enhancement model training methods described above.

[0163] The image processing module 930 is configured to perform image enhancement processing on the image to be processed using the image enhancement model to obtain a target image.

[0164] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0165] Refer to the following Fig.10 hereinafter, an electronic device 1000 according to such an embodiment of the present disclosure is described. Fig.10 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0166] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), and a display unit 1040.

[0167] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the following steps: Figure 2 The steps shown.

[0168] For example, electronic devices can achieve Figure 2 or Figure 7 The steps shown.

[0169] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1021 and / or a cache memory unit 1022 , and may further include a read-only memory unit (ROM) 1023 .

[0170] The storage unit 1020 may also include a program / utility 1024 having a set (at least one) of program modules 1025, such program modules 1025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0171] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0172] The electronic device 1000 may also communicate with one or more external devices 1070 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0173] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0174] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0175] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in the above embodiment is implemented.

[0176] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0177] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an image enhancement model, characterized in that: include: Obtaining a training data set including a plurality of image training samples; The image training samples include an input image and a high-quality image in a corresponding relationship; Performing image enhancement processing on the input image by using the image enhancement model to be trained to obtain an output image; Determining an image enhancement difference according to the input image and the output image; Determining an image enhancement loss value according to the output image and the image enhancement difference; Determining an image perception loss value according to the output image and the high-quality image; Generate a loss function according to the image enhancement loss value and the image perception loss value; According to the loss function, the image enhancement model to be trained is trained to obtain a target image enhancement model.

2. The method according to claim 1, characterized in that The step of calculating the image enhancement difference according to the input image and the output image includes: Determining at least one image sub-difference measure between the input image and the output image, each of the image sub-difference measures corresponding to an image attribute dimension; The image enhancement difference is determined according to the at least one image sub-difference.

3. The method according to claim 2, characterized in that The step of determining the image enhancement difference according to the input image and the output image includes: Determining a sharpness sub-difference and a texture complexity sub-difference between the input image and the output image; The image enhancement difference is determined according to the definition sub-difference and the texture complexity sub-difference.

4. The method according to claim 1, characterized in that: The step of determining the image enhancement loss value according to the output image and the image enhancement difference comprises: Performing Discrete Cosine Transform (DCT) on the output image to obtain an image spectrum distribution diagram; Determine a high frequency region in the image frequency spectrum distribution diagram; Determining a high-frequency coefficient threshold according to the image enhancement difference; The high frequency region is searched based on the high frequency coefficient threshold to determine the image enhancement loss value.

5. The method according to claim 4, characterized in that The determining of the high-frequency area in the image spectrum distribution diagram comprises: The target area located at the lower right corner of the image spectrum distribution diagram is determined as the high-frequency area.

6. The method according to claim 4, characterized in that The step of determining a high-frequency coefficient threshold according to the image enhancement difference comprises: Obtain high-frequency coefficients corresponding to each coefficient item in the high-frequency region; The high-frequency coefficient threshold is determined according to the image enhancement difference and the high-frequency coefficients corresponding to each coefficient item.

7. The method according to claim 4, characterized in that The searching in the high frequency region based on the high frequency coefficient threshold to determine the image enhancement loss value comprises: Searching in the high frequency region based on the high frequency coefficient threshold value to obtain at least one first coefficient term; the high frequency coefficient corresponding to the first coefficient term is less than the high frequency coefficient threshold value; The image enhancement loss value is determined according to the high-frequency coefficient corresponding to the at least one first coefficient term.

8. The method according to claim 1, characterized in that: The step of determining the image perception loss value according to the output image and the high-quality image comprises: The perceptual loss model is used to calculate the perceptual loss of the output image and the high-quality image to obtain the perceptual loss value of the image.

9. The method according to claim 1, characterized in that: The generating a loss function according to the image enhancement loss value and the image perception loss value comprises: The image enhancement loss value and the image perception loss value are weightedly summed to obtain the loss function.

10. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into an image enhancement model; the image enhancement model is obtained by training the image enhancement model training method according to any one of claims 1 to 9; The image enhancement model performs image enhancement processing on the image to be processed to obtain a target image.

11. An image enhancement model training device, characterized in that: include: A sample acquisition module is configured to acquire a training data set including a plurality of image training samples; The image training samples include an input image and a high-quality image in a corresponding relationship; An image enhancement module is configured to perform image enhancement processing on the input image through the image enhancement model to be trained to obtain an output image; A difference calculation module is configured to determine an image enhancement difference according to the input image and the output image; An image enhancement loss module is configured to determine an image enhancement loss value according to the output image and the image enhancement difference; An image perception loss module is configured to determine an image perception loss value based on the output image and the high-quality image; A loss function module, configured to generate a loss function according to the image enhancement loss value and the image perception loss value; The model training module is configured to train the image enhancement model to be trained according to the loss function to obtain a target image enhancement model.

12. An image processing device, characterized in that: include: An image acquisition module is configured to acquire an image to be processed; An image input module, configured to input the image to be processed into an image enhancement model; The image enhancement model is obtained by training through the image enhancement model training method according to any one of claims 1 to 9; The image processing module is configured to perform image enhancement processing on the image to be processed by the image enhancement model to obtain a target image.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the executable instructions to implement the image enhancement model training method as described in any one of claims 1 to 9 or the image processing method as described in claim 10.

14. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the image enhancement model training method as described in any one of claims 1 to 9 or the image processing method as described in claim 10.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the image enhancement model training method as described in any one of claims 1 to 9 or the image processing method as described in claim 10.

Images