Image enhancement model training method, image enhancement method and related device

Abstract

The present application relates to the technical field of image processing, and provides an image enhancement model training method, an image enhancement method and related devices. The method comprises: obtaining a plurality of low-quality image samples and corresponding high-quality images; then performing high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image samples by using a generator to obtain corresponding reconstructed images, and determining corresponding high-frequency enhancement loss functions; then training the generator based on a preset loss function and the high-frequency enhancement loss function corresponding to each low-quality image sample to obtain a trained generator, and taking the trained generator as an image enhancement model. By using the high-frequency enhancement loss function, the image enhancement model has the ability to adaptively adjust the enhancement degree, and different degrees of enhancement of high and low features in the image are realized, thereby improving the image enhancement effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image enhancement model training method, an image enhancement method and related devices. BACKGROUND

[0002] Textures of different scales in an image correspond to different features. Large-scale textures correspond to low-frequency features, which represent overall trunk information in the image such as background, basic shape and structure, etc. Small-scale textures correspond to high-frequency features, which represent rapidly changing local detail information in the image such as edges, etc. Current image enhancement models often enhance all textures in the image to the same degree, which may result in insufficient enhancement of high-frequency features and redundant enhancement of low-frequency features, thereby causing insufficiently clear detail information, image distortion and poor enhancement effect. SUMMARY

[0003] Therefore, the present application aims to provide an image enhancement model training method, an image enhancement method and related devices.

[0004] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0005] In a first aspect, the present application provides an image enhancement model training method, which comprises:

[0006] obtaining a plurality of low-quality image samples and their corresponding high-quality images;

[0007] For each low-quality image sample, performing high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample by using a generator to obtain a reconstructed image corresponding to the low-quality image sample;

[0008] obtaining a high-frequency enhancement loss function corresponding to the low-quality image sample based on the high-quality image and the reconstructed image corresponding to the low-quality image sample;

[0009] training the generator based on a preset loss function and the high-frequency enhancement loss function corresponding to each low-quality image sample to obtain a trained generator, and taking the trained generator as an image enhancement model.

[0010] In an optional embodiment, the generator comprises a plurality of network layers.

[0011] The high-frequency enhancement operation and the low-frequency enhancement operation on the low-quality image sample by using the generator to obtain the reconstructed image corresponding to the low-quality image sample comprises:

[0012] taking the low-quality image sample as an input feature and taking a first network layer in the generator as a target network layer;

[0013] convolve the input feature by using the target network layer to obtain a first feature map and a second feature map;

[0014] perform high-frequency enhancement operation on the first feature map by using the target network layer to obtain a high-frequency feature map; the high-frequency feature map represents local detailed information of the low-quality image sample;

[0015] perform low-frequency enhancement operation on the second feature map by using the target network layer to obtain a low-frequency feature map; the low-frequency feature map represents overall backbone information of the low-quality image sample;

[0016] perform residual connection on the high-frequency feature map and the low-frequency feature map and the input feature by using the target network layer to obtain an output feature;

[0017] repeat the step of convolving the input feature by using the target network layer to obtain a first feature map and a second feature map, until an output feature of a last network layer in the generator is obtained, to obtain a reconstructed image corresponding to the low-quality image sample.

[0018] In an optional implementation, the network layer comprises a high-frequency branch network;

[0019] the high-frequency enhancement operation on the first feature map by using the target network layer to obtain a high-frequency feature map comprises:

[0020] perform convolution operation and global pooling operation on the first feature map by using the high-frequency branch network in the target network layer to obtain a scaling coefficient of each channel in the first feature map;

[0021] perform fast Fourier transform operation on the first feature map by using the high-frequency branch network in the target network layer to obtain a frequency distribution of each channel in the first feature map;

[0022] calculate a cutoff frequency of each channel based on the scaling coefficient and the frequency distribution of each channel by using the high-frequency branch network in the target network layer, and perform high-pass filtering on the first feature map based on the cutoff frequencies of all channels to obtain an initial high-frequency feature map;

[0023] perform convolution operation, global pooling operation and activation operation on the first feature map by using the high-frequency branch network in the target network layer to obtain a sharpening parameter;

[0024] perform sharpening operation on the first feature map based on the sharpening parameter and the initial high-frequency feature map by using the high-frequency branch network in the target network layer to obtain the high-frequency feature map.

[0025] In an optional implementation, the network layer comprises a low-frequency branch network.

[0026] The low-frequency enhancement operation on the second feature map by using the target network layer comprises:

[0027] The low-frequency feature map is obtained by performing a depth separable convolution operation and a convolution operation on the second feature map by using the low-frequency branch network in the target network layer.

[0028] In an optional implementation, the high-frequency enhancement loss function corresponding to the low-quality image sample is obtained based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, and the high-frequency enhancement loss function comprises:

[0029] The Laplacian gradient transform is performed on the low-quality image sample, the high-quality image corresponding to the low-quality image sample, and the reconstructed image respectively to obtain a low-quality gradient image, a high-quality gradient image, and a reconstructed gradient image;

[0030] The average value of gradient values of all pixel points in the low-quality gradient image is calculated, and the product of the average value and a preset coefficient is calculated to obtain a gradient threshold value;

[0031] All pixel points in the low-quality gradient image with a gradient value greater than or equal to the gradient threshold value are obtained to obtain an initial region, and a Gaussian smoothing operation is performed on the initial region to obtain a target region; the target region represents a region where high-frequency features of the low-quality image sample exist;

[0032] The high-frequency enhancement deficiency region and the high-frequency enhancement excess region in the reconstructed image are determined according to the target region, the high-quality gradient image, and the reconstructed gradient image;

[0033] The excitation loss function corresponding to the low-quality image sample is determined according to the high-frequency enhancement deficiency region, and the penalty loss function corresponding to the low-quality image sample is determined according to the high-frequency enhancement excess region, to obtain the high-frequency enhancement loss function corresponding to the low-quality image sample.

[0034] In an optional implementation, the high-frequency enhancement deficiency region and the high-frequency enhancement excess region in the reconstructed image are determined according to the target region, the high-quality gradient image, and the reconstructed gradient image, and the method comprises:

[0035] Any one pixel point in the target region is taken as a target pixel point;

[0036] Pixel points corresponding to the target pixel point in the high-quality gradient image and the reconstructed gradient image are obtained respectively to obtain a first pixel point and a second pixel point;

[0037] if the gradient value of the first pixel point is greater than the gradient value of the second pixel point, a pixel point corresponding to the second pixel point in the reconstructed image is taken as an enhanced deficiency pixel point;

[0038] if the gradient value of the first pixel point is less than the gradient value of the second pixel point, a pixel point corresponding to the second pixel point in the reconstructed image is taken as an enhanced excess pixel point;

[0039] each pixel point in the target region is traversed, and a high-frequency enhanced deficiency region composed of all the enhanced deficiency pixel points and a high-frequency enhanced excess region composed of all the enhanced excess pixel points in the reconstructed image are obtained.

[0040] In an optional implementation, the determination of the excitation loss function corresponding to the low-quality image sample according to the high-frequency enhanced deficiency region and the determination of the penalty loss function corresponding to the low-quality image sample according to the high-frequency enhanced excess region include:

[0041] an excitation matrix is obtained according to the high-frequency enhanced deficiency region, and the excitation loss function corresponding to the low-quality image sample is obtained according to the excitation matrix;

[0042] each element in the excitation matrix represents as follows:

[0043]

[0044] wherein, M E represents the excitation matrix; M E (i, j) represents an element in the i-th row and the j-th column of the excitation matrix; T HR (i, j) represents a gradient value of a pixel point in the i-th row and the j-th column of the high-quality gradient image; T SR (i, j) represents a gradient value of a pixel point in the i-th row and the j-th column of the reconstructed gradient image; T LR (i, j) represents a gradient value of a pixel point in the i-th row and the j-th column of the low-quality gradient image; and ε represents a gradient threshold value.

[0045] the excitation loss function corresponding to the low-quality image sample is represented as follows:

[0046]

[0047] wherein, L E represents the excitation loss function corresponding to the low-quality image sample; I HR represents a high-quality image corresponding to the low-quality image sample; I SR represents a reconstructed image corresponding to the low-quality image sample; represents multiplication of elements in corresponding positions in matrices; || represents taking an absolute value; and ||2 represents a modulus operation.

[0048] A penalty matrix is obtained according to the high-frequency enhanced excess area, and a penalty loss function corresponding to the low-quality image sample is obtained according to the penalty matrix;

[0049] Each element in the penalty matrix is represented as follows:

[0050]

[0051] Wherein, M P represents a penalty matrix; M P (i,j) represents an element in the i-th row and the j-th column of the penalty matrix; T HR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the high-quality gradient image; T SR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the reconstructed gradient image; T LR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the low-quality gradient image; and ε represents a gradient threshold value.

[0052] The penalty loss function corresponding to the low-quality image sample is represented as follows:

[0053]

[0054] Wherein, L P represents a penalty loss function corresponding to the low-quality image sample; I HR represents a high-quality image corresponding to the low-quality image sample; I SR represents a reconstructed image corresponding to the low-quality image sample. represents multiplication of elements in corresponding positions in matrices; || represents taking an absolute value; and ||2 represents a modulus operation.

[0055] In a second aspect, the present application provides an image enhancement method, which comprises:

[0056] Obtaining a low-quality image to be processed;

[0057] Performing high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image to be processed by using an image enhancement model to obtain a high-quality image corresponding to the low-quality image to be processed;

[0058] Wherein, the image enhancement model is obtained according to the image enhancement model training method in any one of the preceding embodiments.

[0059] In a third aspect, the present application provides an image enhancement model training device, which comprises:

[0060] A sample acquisition module is configured to acquire a plurality of low-quality image samples and high-quality images corresponding thereto;

[0061] a model training module configured to, for each of the low-quality image samples, perform high-frequency enhancement and low-frequency enhancement on the low-quality image sample by using the generator to obtain a reconstructed image corresponding to the low-quality image sample;

[0062] obtain a high-frequency enhancement loss function corresponding to the low-quality image sample based on the high-quality image and the reconstructed image corresponding to the low-quality image sample;

[0063] train the generator based on the preset loss function and the high-frequency enhancement loss function corresponding to each of the low-quality image samples to obtain a trained generator, and use the trained generator as an image enhancement model.

[0064] In a fourth aspect, the present application provides an image enhancement device, and the image enhancement device comprises:

[0065] an image acquisition module configured to acquire a low-quality image to be processed;

[0066] an image enhancement module configured to perform high-frequency enhancement and low-frequency enhancement on the low-quality image to be processed by using an image enhancement model to obtain a high-quality image corresponding to the low-quality image to be processed;

[0067] The image enhancement model is obtained according to the image enhancement model training method in any one of the preceding embodiments.

[0068] In a fifth aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the image enhancement model training method in any one of the preceding embodiments and / or the image enhancement method in the preceding embodiments when executing the computer program.

[0069] In a sixth aspect, the present application provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the image enhancement model training method in any one of the preceding embodiments and / or the image enhancement method in the preceding embodiments.

[0070] The image enhancement model training method, the image enhancement method and the related device provided by the present application first acquire a plurality of low-quality image samples and corresponding high-quality images; then, for each low-quality image sample, a generator is used to perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample to obtain a corresponding reconstructed image, and a corresponding high-frequency enhancement loss function is obtained based on the high-quality image corresponding to the low-quality image sample and the reconstructed image; then, based on a preset loss function and the high-frequency enhancement loss function corresponding to each low-quality image sample, the generator is trained to obtain a trained generator, and the trained generator is used as an image enhancement model. By using the high-frequency enhancement loss function, the image enhancement model has the ability to adaptively adjust the enhancement degree, and the image enhancement model can enhance the high and low features in the image to different degrees, so that the image is more rich and real, and the image enhancement effect is improved.

[0071] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be referred to, as follows. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0073] Figure 1 An example diagram of the existing image enhancement model training method is shown;

[0074] Figure 2 A schematic diagram of an electronic device provided by the embodiment of the present application is shown;

[0075] Figure 3 A flowchart of the image enhancement model training method provided by the embodiment of the present application is shown;

[0076] Figure 4 An example diagram of the image enhancement model training method provided by the embodiment of the present application is shown;

[0077] Figure 5 Another example diagram of the image enhancement model training method provided by the embodiment of the present application is shown;

[0078] Figure 6 A flowchart of the image enhancement method provided by the embodiment of the present application is shown;

[0079] Figure 7 A functional module diagram of the image enhancement model training device provided by the embodiment of the present application is shown;

[0080] Figure 8 A functional module diagram of the image enhancement device provided by the embodiment of the present application is shown.

[0081] Icon: 100-electronic device; 110-bus; 120-processor; 130-memory; 150-I / O module; 170-communication interface; 300-image enhancement model training device; 310-sample acquisition module; 330-model training module; 400-image enhancement device; 410-image acquisition module; 430-image enhancement module. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0083] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0084] It should be noted that the relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations exist in any such actual relationship or order. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0085] Textures of different scales in an image correspond to different features. Large-scale textures correspond to low-frequency features, which represent overall trunk information in the image such as background, basic shape and structure, etc. Small-scale textures correspond to high-frequency features, which represent rapidly changing local detail information in the image such as edges, etc. Current image enhancement models often enhance all scales of textures in the image to the same degree, which will lead to insufficient enhancement of high-frequency features and redundant enhancement of low-frequency features.

[0086] For ease of understanding, an example diagram of an existing image enhancement model training method is given below. Please refer to Figure 1 , the existing implementation is to input a low-quality image to a generator, and use the generator to perform enhancement operation on the low-quality image to obtain a reconstructed image, and then calculate the L1 loss based on the high-quality image corresponding to the low-quality image and the reconstructed image, at the same time, a discriminator is used to judge the true and false of the high-quality image and the reconstructed image. The generator includes an encoder and a decoder, the encoder is responsible for extracting the visual features of the low-quality image, and the decoder is responsible for decoding the visual features into the corresponding reconstructed image.

[0087] But in this way, the generator uses the same processing operation on any scale of texture in the image, so reflected on the reconstructed image is that the enhancement degree of high-frequency features and low-frequency features is the same. Since there are more low-frequency features in the low-quality image, its reconstruction difficulty is smaller, and there are fewer high-frequency features, its reconstruction difficulty is larger, if the same degree of enhancement is performed on these two kinds of features, it will often lead to insufficient enhancement of high-frequency features, and redundant enhancement of low-frequency features, resulting in insufficient clear and obvious details in the reconstructed image. And, since the low-quality image has only a small amount of detailed information, this will also cause the high-frequency details generated by the generator to have a certain degree of distortion, thereby resulting in poor image enhancement effect. Therefore, the embodiment of the present application provides a training method of an image enhancement model to solve the above problems.

[0088] Please refer to Figure 2 , is a schematic diagram of an electronic device 100 provided by the embodiment of the present application. The electronic device 100 includes a bus 110, a processor 120, a memory 130, an I / O module 150, and a communication interface 170.

[0089] The bus 110 can be a circuit that connects the above-mentioned elements to each other and transmits communication (such as control messages) between the above-mentioned elements.

[0090] The processor 120 can receive commands from the above-mentioned other elements (such as the memory 130, the I / O module 150, the communication interface 170, etc.) through the bus 110, can interpret the received commands, and can perform calculations or data processing according to the interpreted commands.

[0091] The processor 120 can be an integrated circuit chip with signal processing capability. The processor 120 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-program gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0092] The memory 130 can store commands or data received from the processor 120 or other elements (e.g., the I / O module 150, the communication interface 170, etc.) or commands or data generated by the processor 120 or other elements.

[0093] The memory 130 can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM).

[0094] The I / O module 150 can receive commands or data input from a user via an input-output means (e.g., a sensor, a keyboard, a touch screen, etc.), and can transmit the received commands or data to the processor 120 or the memory 130 through the bus 110. Also, for displaying various information (e.g., multimedia data, text data) received, stored, processed from the above elements, a video, an image, data, etc. can be displayed to the user.

[0095] The communication interface 170 can be used for signaling or data communication with other node devices.

[0096] It can be understood that, Figure 2 The structure shown is only a structural schematic diagram of the electronic device 100, and the electronic device 100 can further include more or less components than those shown in the description, or have a different configuration from that shown in the description. Figure 2 The components shown in the description can be implemented in hardware, software, or a combination thereof. Figure 2 The components shown in the description can be implemented in hardware, software, or a combination thereof. Figure 2 The components shown in the description can be implemented in hardware, software, or a combination thereof.

[0097] The electronic device provided by the embodiments of the present application can be a smart phone, a personal computer, a tablet computer, a wearable device, an Ultra-Mobile Personal Computer (UMPC), a netbook, a Personal Digital Assistant (PDA), etc. The embodiments of the present application do not make any limitation on this.

[0098] The electronic device 100 described above will be taken as an execution subject to execute each step in each method provided by the embodiments of the present application, and to achieve the corresponding technical effects.

[0099] Referring to Figure 3 , Figure 3 is a flowchart of an image enhancement model training method provided by an embodiment of the present application.

[0100] In step S202, a plurality of low-quality image samples and corresponding high-quality images thereof are obtained.

[0101] In this embodiment, a plurality of low-quality images with low resolution and blurred pictures, i.e., a plurality of low-quality image samples, are obtained, and a high-quality image corresponding to each low-quality image sample is obtained, i.e., a sample set for model training is obtained.

[0102] In step S204, for each low-quality image sample, a generator is used to perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample to obtain a reconstructed image corresponding to the low-quality image sample.

[0103] In step S206, based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, a high-frequency enhancement loss function corresponding to the low-quality image sample is obtained.

[0104] It can be understood that the processing mode for each low-quality image sample is similar, and for brevity, one low-quality image sample is taken as an example for description below.

[0105] In this embodiment, the low-quality image sample is input into the generator, and the generator is used to perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample to obtain a reconstructed image corresponding to the low-quality image sample. It can be understood that the generator enhances the high-frequency features and the low-frequency features in the low-quality image sample to different degrees by performing high-frequency enhancement operation and low-frequency enhancement operation, so that the reconstructed image output by the generator has good picture quality, i.e., has rich and clear detail information.

[0106] Moreover, based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, a high-frequency enhancement loss function corresponding to the low-quality image sample is constructed. It can be understood that, since the enhancement degrees of the high-frequency features and the low-frequency features are different in this embodiment, and the processing of the high-frequency features is more important for improving the picture quality of the reconstructed image, the generator provided by the present application focuses more on the processing of the high-frequency features. The high-frequency enhancement loss function is used to adaptively adjust the enhancement degree of the high-frequency features, so that the enhancement degree of the high-frequency features is adapted to the required improvement degree of the picture quality, and the generated reconstructed image has better visual effect.

[0107] Each low-quality image sample is processed in the above similar manner, i.e., a reconstructed image corresponding to each low-quality image sample and a high-frequency enhancement loss function corresponding to each low-quality image sample are obtained.

[0108] Step S208, based on the preset loss function, the high-frequency enhancement loss function corresponding to each low-quality image sample, the generator is trained to obtain the trained generator, and the trained generator is used as the image enhancement model.

[0109] It can be understood that the preset loss function can include a pixel loss function, a perception loss function and an adversarial loss function, and can also be set according to actual conditions, and the embodiments of the present application are not limited.

[0110] In this embodiment, the loss is calculated based on the high-quality image and the reconstructed image corresponding to each low-quality image sample, that is, the pixel loss, the perception loss, the adversarial loss and the high-frequency enhancement loss are calculated, and the generator is trained to obtain the trained generator, that is, the image enhancement model is obtained.

[0111] As can be seen, based on the above steps, first, a plurality of low-quality image samples and their corresponding high-quality images are obtained; then for each low-quality image sample, the generator is used to perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample to obtain the corresponding reconstructed image, and based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, the corresponding high-frequency enhancement loss function is obtained; then based on the preset loss function, the high-frequency enhancement loss function corresponding to each low-quality image sample, the generator is trained to obtain the trained generator, and the trained generator is used as the image enhancement model. By using the high-frequency enhancement loss function, the image enhancement model has the ability to adaptively adjust the enhancement degree, and the image enhancement model enhances the high and low features in the image to different degrees, making the image more rich and real, and improving the image enhancement effect.

[0112] In order to better understand the present application, the structure of the generator provided by the embodiments of the present application will be introduced. Please refer to Figure 4 , the encoder and the decoder in the existing generator are both composed of multiple convolutional layers. Based on this, the embodiments of the present application provide two improved ways for the structure of the generator.

[0113] The first is to replace the convolutional layer in the existing generator with the network layer provided by the embodiments of the present application, such as the first improved structure shown in Figure 4 ; the second is to insert the network layer provided by the embodiments of the present application between adjacent convolutional layers in the existing generator, such as the second improved structure shown in Figure 4 . It should be noted that the input and output dimensions of the network layer provided by the embodiments of the present application are the same.

[0114] It can be understood that the two structures provided above have the same technical effects and basic principles. For brevity, the first structure will be taken as an example for description.

[0115] Optionally, for the step S204, the embodiment of the present application provides a possible implementation manner.

[0116] Step S204-1, taking the low-quality image sample as the input feature, and taking the first network layer in the generator as the target network layer;

[0117] Step S204-3, performing convolution operation on the input feature by using the target network layer to obtain the first feature map and the second feature map;

[0118] In the embodiment, the generator includes a plurality of network layers, and the output of a previous network layer is taken as the input of a subsequent network layer, and the processing manner of each network layer is the same. For the convenience of understanding, the first network layer in the generator will be taken as an example for description.

[0119] Firstly, the low-quality image sample is taken as the input feature, and the first network layer in the generator is taken as the target network layer. The input feature is input into the target network layer, and the input feature is subjected to convolution operation by using the target network layer, so as to obtain the first feature map and the second feature map.

[0120] For example, the input feature F in ∈R C×H×W , wherein R C×H×W represents the dimension of the input feature, and the input feature F in is subjected to convolution operation by using a 1*1 size convolution kernel Conv1, so as to reduce the channel number by half, and the first feature map is as follows: At the same time, the input feature F in is subjected to similar processing, and the second feature map is as follows: That is, the channel numbers of the first feature map and the second feature map are both half of the input feature.

[0121] Step S204-5, performing high-frequency enhancement operation on the first feature map by using the target network layer to obtain a high-frequency feature map; the high-frequency feature map represents the local detail information of the low-quality image sample;

[0122] Step S204-7, performing low-frequency enhancement operation on the second feature map by using the target network layer to obtain a low-frequency feature map; the low-frequency feature map represents the overall backbone information of the low-quality image sample;

[0123] Continuing to describe the above example, based on the obtained first feature map high-frequency enhancement operation is performed thereon, and a high-frequency feature map is obtained as follows: The high-frequency feature map corresponds to the small-scale texture in the low-quality image sample, that is, represents the local detail information.

[0124] At the same time, based on the obtained second feature map low-frequency enhancement operation is performed thereon, and a low-frequency feature map is obtained as follows: Low-frequency feature maps correspond to large-scale textures in low-quality image samples, representing the overall backbone information.

[0125] Step S204-9: Use the target network layer to concatenate the high-frequency feature map and the low-frequency feature map and perform residual connection with the input features to obtain the output features;

[0126] Continuing with the example above, we obtain the high-frequency feature map. and low-frequency feature maps Then, the two feature maps are concatenated along the channel dimension and then combined with the input feature F. in Perform residual connection to obtain the output feature F. out Then it can be expressed as And output feature F out With input features F in The dimensions are the same, that is, the output feature is F. out ∈R C×H×W .

[0127] Step S204-11 involves using the output features as input features and the next network layer of the target network layer as the target network layer. Then, the steps of using the target network layer to perform convolution operations on the input features to obtain the first feature map and the second feature map are repeated until the output features of the last network layer in the generator are obtained, thus obtaining the reconstructed image corresponding to the low-quality image sample.

[0128] Continuing with the example above, the output features are used as input features, and the next network layer after the target network layer is used as the target network layer. That is, the second network layer in the generator is used as the target network layer. In a similar manner as described above, the output features of the first network layer are processed by the second network layer. This process is repeated until the output features of the last network layer in the generator are obtained, which is the reconstructed image corresponding to the low-quality image sample.

[0129] Optionally, for the above steps S204-5, this embodiment of the invention provides a possible implementation method.

[0130] Step S204-5-1: Perform convolution and global pooling operations on the first feature map using the high-frequency branch network in the target network layer to obtain the scaling factor of each channel in the first feature map.

[0131] In this embodiment, the network layer includes a high-frequency branch network and a low-frequency branch network. For ease of understanding, an example diagram is provided in this embodiment of the invention; please refer to it. Figure 5 The following will combine Figure 5 The processing procedure for high-frequency branch networks is explained.

[0132] like Figure 5As shown in path ①, a 3×3 convolution kernel, Conv3, is used to process the first feature map F. H After performing convolution and global pooling, the scaling factor of each channel in the first feature map is obtained.

[0133] Step S204-5-3: Use the high-frequency branch network in the target network layer to perform a fast Fourier transform operation on the first feature map to obtain the frequency distribution of each channel in the first feature map.

[0134] Step S204-5-5: Calculate the cutoff frequency of each channel based on the scaling factor and frequency distribution of each channel using the high-frequency branch network in the target network layer, and perform high-pass filtering on the first feature map based on the cutoff frequencies of all channels to obtain the initial high-frequency feature map.

[0135] like Figure 5 As shown in path ②, for the first feature map F H Performing a Fast Fourier Transform operation yields the features of each channel, such as... The frequency distribution of each channel is determined, and the maximum and minimum frequencies of each channel are determined based on the frequency distribution of each channel. Then, the cutoff frequency of each channel is calculated based on the scaling factor, maximum frequency, minimum frequency and preset formula of each channel.

[0136] The default formula is expressed as follows:

[0137]

[0138] Among them, W i σ represents the cutoff frequency of the i-th channel; i This represents the scaling factor for the i-th channel. This represents the maximum frequency of the i-th channel; This represents the minimum frequency of the i-th channel.

[0139] Next, components below their cutoff frequency in each channel are filtered out to achieve a high-pass filtering effect on the first feature map, thus obtaining the initial high-frequency feature map as shown below. It can then be expressed as ΔF H =FFT(pooling(Conv3(F H )), F H ).

[0140] Understandably, the scaling factor can adaptively adjust the cutoff frequency automatically based on the input features of the high-frequency branch network. Furthermore, the initial high-frequency feature map obtained after high-pass filtering using the cutoff frequency only contains the high-frequency features from the first feature map.

[0141] Step S204-5-7: Perform convolution, global pooling and activation operations on the first feature map using the high-frequency branch network in the target network layer to obtain sharpening parameters;

[0142] Step S204-5-9: Sharpen the first feature map using the high-frequency branch network in the target network layer based on the sharpening parameters and the initial high-frequency feature map to obtain the high-frequency feature map;

[0143] like Figure 5 As shown in paths ③ and ④, to further enhance high-frequency features, the first high-frequency feature map can be sharpened. This can be done by first sharpening the first feature map F. H Perform convolution, global pooling, and activation operations to obtain the sharpening parameters. This can be expressed as μ = sigmoid(pooling(Conv3(F)). H )).

[0144] Then, based on the sharpening parameter μ and the initial high-frequency feature map ΔF H For the first feature map F H Perform a sharpening operation to obtain a high-frequency feature map. It can then be expressed as

[0145] Optionally, for steps S204-7 above, this embodiment of the invention provides a possible implementation method, namely: using the low-frequency branch network in the target network layer to perform depthwise separable convolution and convolution operations on the second feature map to obtain the low-frequency feature map.

[0146] Continuing with the above Figure 5 To explain, for low-frequency branch networks, the complexity and computational cost of these branch networks can be reduced to decrease the redundancy in enhancing low-frequency features. This can be achieved by modifying the second feature map F. L After performing a depthwise separable convolution operation, a 3×3 Conv3 kernel is then used for convolution to obtain the low-frequency feature map. It can then be expressed as Understandably, using depthwise separable convolutional layers can significantly reduce computational cost compared to traditional convolutional layers.

[0147] Optionally, for step S206 above, the present invention provides a possible implementation method.

[0148] Step S206-1: Perform Laplacian gradient transformation on the low-quality image sample, the high-quality image corresponding to the low-quality image sample, and the reconstructed image, respectively, to obtain the low-quality gradient map, the high-quality gradient map, and the reconstructed gradient map.

[0149] It can be understood that the artifacts usually appear in the high frequency part of the image, such as the edge of various textures, and since the reconstructed image is generated based on the low-quality image samples, the area in the reconstructed image corresponding to the high frequency part of the low-quality image samples is most likely to have artifacts.

[0150] Thus in the present embodiment, the high frequency part in the low-quality image sample can be determined by extracting the Laplacian gradient map of the low-quality image sample, i.e. the low-quality gradient map. And the Laplacian gradient map of the corresponding high-quality image, i.e. the high-quality gradient map, and the Laplacian gradient map of the corresponding reconstructed image, i.e. the reconstructed gradient map, are extracted and compared to determine the degree of enhancement of the high frequency feature.

[0151] Step S206-3, calculate the average value of the gradient values of all pixel points in the low-quality gradient map, and calculate the product of the average value and a preset coefficient to obtain a gradient threshold value;

[0152] Step S206-5, obtain all pixel points in the low-quality gradient map with gradient values greater than or equal to the gradient threshold value to obtain an initial region, and perform a Gaussian smoothing operation on the initial region to obtain a target region; the target region represents the region where the high frequency feature of the low-quality image sample exists;

[0153] It can be understood that since the low-quality image sample is relatively blurred, the gradient values of the pixel points in the low-quality gradient map are generally low, and thus the average value of the gradient values of all pixel points in the low-quality gradient map can be used to determine the region where the high frequency part is located.

[0154] In the present embodiment, the average value of the gradient values of all pixel points in the low-quality gradient map can be calculated first, and the average value multiplied by a preset coefficient, i.e. the gradient threshold value, can be obtained, which can be represented as ε = λ · E(T LR ); where ε represents the gradient threshold value; λ represents the preset coefficient and λ ∈ [0, 1]; T LR represents the low-quality gradient map; E(*) represents the average value. The preset coefficient can be understood as a proportion parameter for controlling the selection of the high frequency part.

[0155] Then, based on the obtained gradient threshold value, all pixel points in the low-quality gradient map with gradient values greater than or equal to the gradient threshold value are selected, i.e. an initial region is obtained. In order to obtain the region where the artifacts are most likely to be generated as much as possible, a Gaussian smoothing operation can be performed on the initial region to slightly expand its range, i.e. a target region is obtained, which represents the region where the high frequency part in the low-quality image sample is located. And the target region obtained based on the Gaussian smoothing operation can ensure the continuity of the high frequency feature.

[0156] Step S206-7, determine the high frequency enhancement deficiency region and the high frequency enhancement excess region in the reconstructed image according to the target region, the high-quality gradient map and the reconstructed gradient map;

[0157] In this embodiment, the target region represents a high-frequency part in the low-quality image sample, and then a region corresponding to the target region in the high-quality gradient map is determined to obtain a high-frequency target region, and a region corresponding to the target region in the reconstructed gradient map is determined to obtain a reconstructed target region.

[0158] Then, the high-frequency target region and the reconstructed target region are compared to determine a high-frequency enhancement deficiency region and a high-frequency enhancement excess region in the reconstructed image. The high-frequency enhancement deficiency region can be understood as a region in which the enhancement degree of the high-frequency feature is not enough. The high-frequency enhancement excess region can be understood as a region in which the enhancement degree of the high-frequency feature is too strong.

[0159] In step S206-9, a high-frequency enhancement loss function corresponding to the low-quality image sample is determined according to the high-frequency enhancement deficiency region, and a high-frequency enhancement loss function corresponding to the low-quality image sample is determined according to the high-frequency enhancement excess region, to obtain a high-frequency enhancement loss function corresponding to the low-quality image sample.

[0160] In this embodiment, the high-frequency enhancement deficiency region and the high-frequency enhancement excess region are both regions that do not meet the standard. For these regions, an incentive mechanism or a penalty mechanism can be used for adjustment. For example, the greater the incentive degree, the greater the enhancement degree; the smaller the incentive degree, the smaller the enhancement degree. The greater the penalty degree, the smaller the enhancement degree; the smaller the penalty degree, the greater the enhancement degree.

[0161] Then, for the high-frequency enhancement deficiency region in which the enhancement degree is not enough, an incentive mechanism can be used, that is, a high-frequency enhancement loss function corresponding to the low-quality image sample is determined to guide the generator to increase the enhancement degree of the high-frequency feature. For the high-frequency enhancement excess region in which the enhancement degree is too strong, a penalty mechanism can be used, that is, a high-frequency enhancement loss function corresponding to the low-quality image sample is determined to guide the generator to reduce the enhancement degree of the high-frequency feature.

[0162] Optionally, for the above step S206-7, the present embodiment provides a possible implementation manner.

[0163] In step S206-7-1, any one pixel point in the target region is taken as a target pixel point.

[0164] In step S206-7-3, pixel points corresponding to the target pixel point are obtained in the high-quality gradient map and the reconstructed gradient map respectively to obtain a first pixel point and a second pixel point.

[0165] In step S206-7-5A, if the gradient value of the first pixel point is greater than the gradient value of the second pixel point, a pixel point corresponding to the second pixel point in the reconstructed image is taken as an enhancement deficiency pixel point.

[0166] Step S206-7-5B, if the gradient value of the first pixel point is less than the gradient value of the second pixel point, then the pixel point in the reconstructed image corresponding to the second pixel point is taken as an over-enhanced pixel point;

[0167] In the embodiment, the pixel point in the target region can be taken as a reference point to compare the pixel point in the high-quality gradient map corresponding to the reference point with the pixel point in the reconstructed gradient map corresponding to the reference point. For the convenience of description, the following takes any one pixel point in the target region as a target pixel point for example.

[0168] Firstly, the pixel point corresponding to the target pixel point in the high-quality gradient map is obtained, which is a first pixel point, and the pixel point corresponding to the target pixel point in the reconstructed gradient map is obtained, which is a second pixel point.

[0169] Then, the gradient value of the first pixel point is compared with the gradient value of the second pixel point, and the comparison result of the two exists the following four cases.

[0170] Case 1: if the gradient value of the first pixel point is greater than the gradient value of the second pixel point, it indicates that the enhancement degree of the pixel point in the reconstructed image corresponding to the second pixel point is not enough, that is, it has not reached the standard of the high-quality image, and the enhancement degree needs to be increased, then the pixel point in the reconstructed image is taken as an under-enhanced pixel point.

[0171] Case 2: if the gradient value of the first pixel point is equal to the gradient value of the second pixel point, it indicates that the enhancement degree of the pixel point in the reconstructed image corresponding to the second pixel point is just right, that is, it just reaches the standard of the high-quality image, then the enhancement degree does not need to be adjusted.

[0172] Case 3: if the gradient value of the first pixel point is less than the gradient value of the second pixel point, and the gradient value of the first pixel point is 0, it indicates that there is no high-frequency feature in the pixel point in the high-quality image corresponding to the first pixel point, that is, there is an artifact in the pixel point in the reconstructed image corresponding to the second pixel point, and the enhancement degree needs to be reduced, then the pixel point in the reconstructed image is taken as an over-enhanced pixel point.

[0173] Case 4: if the gradient value of the first pixel point is less than the gradient value of the second pixel point, and the gradient value of the first pixel point is not 0, it indicates that the enhancement degree of the pixel point in the reconstructed image corresponding to the second pixel point is too strong, and the enhancement degree needs to be reduced, then the pixel point in the reconstructed image is taken as an over-enhanced pixel point.

[0174] Step S206-7-7, each pixel point in the target region is traversed to obtain a high-frequency under-enhanced region composed of all under-enhanced pixel points in the reconstructed image and a high-frequency over-enhanced region composed of all over-enhanced pixel points in the reconstructed image.

[0175] In this embodiment, by processing each pixel in the target region in a similar manner as described above, a high-frequency enhancement-deficient region composed of all enhancement-deficient pixels and a high-frequency enhancement-excess region composed of all enhancement-excess pixels can be obtained in the reconstructed image.

[0176] Optionally, for steps S206-9 above, this embodiment of the invention provides a possible implementation method.

[0177] Step S206-9-1: Obtain the excitation matrix based on the high-frequency enhancement deficient region, and obtain the excitation loss function corresponding to the low-quality image sample based on the excitation matrix;

[0178] Each element in the activation matrix is ​​represented as follows:

[0179]

[0180] Among them, M E M represents the activation matrix; E (i, j) represents the element in the i-th row and j-th column of the activation matrix; T HR (i, j) represents the gradient value of the pixel in the i-th row and j-th column of the high-quality gradient map; T SR (i, j) represents the gradient value of the pixel in the i-th row and j-th column of the reconstructed gradient map; T LR (i, j) represents the gradient value of the pixel in the i-th row and j-th column of a low-quality gradient image; ε represents the gradient threshold.

[0181] The activation loss function for low-quality image samples is expressed as follows:

[0182]

[0183] Among them, L E I represents the activation loss function corresponding to low-quality image samples; HR Indicates the high-quality image corresponding to the low-quality image sample; I SR This represents the reconstructed image corresponding to a low-quality image sample. || represents element-wise multiplication at corresponding positions in the matrix; || represents taking the absolute value; ||2 represents the modulo operation;

[0184] In this embodiment, for pixels in the reconstructed image that belong to regions with insufficient high-frequency enhancement, indicating that their enhancement level is insufficient, an activation mechanism can be used to construct the activation matrix. That is, for case 1 described above, since the gradient value of the first pixel in the high-quality gradient map is greater than the gradient value of the second pixel in the reconstructed gradient map, i.e., T... HR (i, j) > T SR If (i, j), then the difference between their gradient values, i.e., T, can be calculated.HR (i, j)-T SR (i, j), as the excitation parameter of the corresponding pixel point in the reconstructed image, all excitation parameters are obtained, that is, the excitation matrix is obtained.

[0185] Then, the excitation loss function corresponding to the low-quality image sample is determined according to the excitation matrix, that is, the absolute difference of the pixel values of the corresponding pixel points in the high-quality image and the reconstructed image is calculated, and the modulus of the matrix after multiplication with the excitation matrix is calculated.

[0186] Step S206-9-3, obtaining a penalty matrix according to the high-frequency enhancement excessive area, and obtaining a penalty loss function corresponding to the low-quality image sample according to the penalty matrix;

[0187] Each element in the penalty matrix is represented as follows:

[0188]

[0189] Wherein, M P represents the penalty matrix; M P (i, j) represents the element in the i-th row and the j-th column of the penalty matrix; T HR (i, j) represents the gradient value of the i-th row and the j-th column pixel point in the high-quality gradient image; T SR (i, j) represents the gradient value of the i-th row and the j-th column pixel point in the reconstructed gradient image; T LR (i, j) represents the gradient value of the i-th row and the j-th column pixel point in the low-quality gradient image; and ε represents the gradient threshold value.

[0190] The penalty loss function corresponding to the low-quality image sample is represented as follows:

[0191]

[0192] Wherein, L P represents the penalty loss function corresponding to the low-quality image sample; I HR represents the high-quality image corresponding to the low-quality image sample; I SR (i, j) represents the reconstructed image corresponding to the low-quality image sample; represents the multiplication of the elements in the corresponding positions in the matrix; || represents the absolute value; and ||2 represents the modulus operation.

[0193] In this embodiment, for the pixel points in the reconstructed image belonging to the high-frequency enhancement excessive area, it is explained that the enhancement degree is too strong, so the penalty mechanism can be used to construct the penalty matrix, that is, for the above-mentioned cases 3 and 4, since the gradient value of the first pixel point in the high-quality gradient image is less than the gradient value of the second pixel point in the reconstructed gradient image, that is, T SR (i, j) > T HR (i, j), then the difference between the gradient values of the two, that is, TSR (i, j)-T HR (i, j), as a penalty parameter of the corresponding pixel point in the reconstructed image, obtains all the penalty parameters, that is, the penalty matrix.

[0194] Then, the penalty loss function corresponding to the low-quality image sample is determined according to the penalty matrix, that is, the absolute difference of the pixel values of the corresponding pixel points in the high-quality image and the reconstructed image is calculated, and the modulus of the matrix is calculated after being multiplied by the penalty matrix.

[0195] Optionally, based on the image enhancement model obtained in the above manner, the embodiment of the present application further provides an image enhancement method using the image enhancement model. Please refer to Figure 6 .

[0196] Step S212, obtaining a to-be-processed low-quality image;

[0197] Step S214, performing high-frequency enhancement operation and low-frequency enhancement operation on the to-be-processed low-quality image by using the image enhancement model, to obtain a high-quality image corresponding to the to-be-processed low-quality image;

[0198] In this embodiment, the to-be-processed low-quality image represents a low-quality image sample that needs to be processed. The to-be-processed low-quality image can be input into the image enhancement model, and the image enhancement model is used to perform high-frequency enhancement operation and low-frequency enhancement operation on the to-be-processed low-quality image, so as to obtain a high-quality image corresponding to the to-be-processed low-quality image, and the high-quality image has good image quality, that is, has rich and clear detail information.

[0199] It should be understood that the process of the image enhancement model performing high-frequency enhancement operation and low-frequency enhancement operation is similar to the execution process of the generator described above. For brevity, the specific execution process of the image enhancement model can be referred to the corresponding part in the above embodiment.

[0200] It can be understood that the image enhancement model enhances the high-frequency features and low-frequency features in the image to different degrees through high-frequency enhancement operation and low-frequency enhancement operation, that is, the high-frequency features and low-frequency features in the image are enhanced in a targeted manner, thereby improving the image enhancement effect, making the image more rich and real and having good visual effect.

[0201] In order to execute each step in the image enhancement model training method provided by the embodiment of the present application, an implementation manner of an image enhancement model training device is given below. Please refer to Figure 7 , Figure 7A functional module diagram of an image enhancement model training apparatus 300 is provided for an embodiment of the present application. It should be noted that the image enhancement model training apparatus 300 provided in the present embodiment has the same basic principles and technical effects as the above-described embodiments. For brief description, the corresponding content of the above-described embodiments can be referred to for the part not mentioned in the present embodiment. The image enhancement model training apparatus 300 comprises:

[0202] A sample acquisition module 310 is configured to acquire a plurality of low-quality image samples and corresponding high-quality images thereof.

[0203] A model training module 330 is configured to, for each low-quality image sample, perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample by using a generator to obtain a reconstructed image corresponding to the low-quality image sample.

[0204] Based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, a high-frequency enhancement loss function corresponding to the low-quality image sample is obtained.

[0205] Based on the preset loss function and the high-frequency enhancement loss function corresponding to each low-quality image sample, the generator is trained to obtain a trained generator, and the trained generator is used as an image enhancement model.

[0206] Optionally, the model training module 330 is further configured to: take the low-quality image sample as an input feature and take a first network layer in the generator as a target network layer; perform convolution operation on the input feature by using the target network layer to obtain a first feature map and a second feature map; perform high-frequency enhancement operation on the first feature map by using the target network layer to obtain a high-frequency feature map; the high-frequency feature map represents local detailed information of the low-quality image sample; perform low-frequency enhancement operation on the second feature map by using the target network layer to obtain a low-frequency feature map; the low-frequency feature map represents overall trunk information of the low-quality image sample; perform splicing on the high-frequency feature map and the low-frequency feature map by using the target network layer and perform residual connection on the input feature to obtain an output feature; after taking the output feature as the input feature and taking a next network layer of the target network layer as the target network layer, repeat the step of performing convolution operation on the input feature by using the target network layer to obtain the first feature map and the second feature map until the output feature of the last network layer in the generator is obtained, and the reconstructed image corresponding to the low-quality image sample is obtained.

[0207] Optionally, the model training module 330 is further configured to: perform convolution operation and global pooling operation on the first feature map by using the high-frequency branch network in the target network layer, to obtain a scaling coefficient of each channel in the first feature map; perform fast Fourier transform operation on the first feature map by using the high-frequency branch network in the target network layer, to obtain a frequency distribution of each channel in the first feature map; calculate a cut-off frequency of each channel based on the scaling coefficient and the frequency distribution of each channel by using the high-frequency branch network in the target network layer, and perform high-pass filtering on the first feature map based on the cut-off frequencies of all channels, to obtain an initial high-frequency feature map; perform convolution operation, global pooling operation and activation operation on the first feature map by using the high-frequency branch network in the target network layer, to obtain a sharpening parameter; and perform sharpening operation on the first feature map based on the sharpening parameter and the initial high-frequency feature map by using the high-frequency branch network in the target network layer, to obtain a high-frequency feature map.

[0208] Optionally, the model training module 330 is further configured to: perform depth separable convolution operation and convolution operation on the second feature map by using the low-frequency branch network in the target network layer, to obtain a low-frequency feature map.

[0209] Optionally, the model training module 330 is further configured to: perform Laplacian gradient transformation on the low-quality image sample, the high-quality image corresponding to the low-quality image sample and the reconstructed image respectively, to obtain a low-quality gradient map, a high-quality gradient map and a reconstructed gradient map; calculate an average value of gradient values of all pixel points in the low-quality gradient map, and calculate a product of the average value and a preset coefficient to obtain a gradient threshold value; obtain all pixel points with gradient values greater than or equal to the gradient threshold value in the low-quality gradient map to obtain an initial region, and perform Gaussian smoothing operation on the initial region to obtain a target region; the target region represents a region where the high-frequency feature of the low-quality image sample exists; determine a high-frequency enhancement deficiency region and a high-frequency enhancement excess region in the reconstructed image according to the target region, the high-quality gradient map and the reconstructed gradient map; determine an excitation loss function corresponding to the low-quality image sample according to the high-frequency enhancement deficiency region, and determine a penalty loss function corresponding to the low-quality image sample according to the high-frequency enhancement excess region, to obtain a high-frequency enhancement loss function corresponding to the low-quality image sample.

[0210] Optionally, the model training module 330 is further configured to: take any one pixel point in the target region as a target pixel point; obtain a first pixel point and a second pixel point by obtaining pixel points corresponding to the target pixel point in the high-quality gradient map and the reconstructed gradient map, respectively; if a gradient value of the first pixel point is greater than a gradient value of the second pixel point, take a pixel point corresponding to the second pixel point in the reconstructed image as an enhanced deficiency pixel point; if the gradient value of the first pixel point is less than the gradient value of the second pixel point, take a pixel point corresponding to the second pixel point in the reconstructed image as an enhanced excess pixel point; and traverse each pixel point in the target region to obtain a high-frequency enhanced deficiency region composed of all the enhanced deficiency pixel points in the reconstructed image and a high-frequency enhanced excess region composed of all the enhanced excess pixel points.

[0211] Optionally, the model training module 330 is further configured to: obtain an excitation matrix according to the high-frequency enhanced deficiency region, and obtain an excitation loss function corresponding to the low-quality image sample according to the excitation matrix.

[0212] Each element in the excitation matrix is represented as follows:

[0213]

[0214] wherein M E represents the excitation matrix; M E (i,j) represents an element in the i-th row and the j-th column of the excitation matrix; T HR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the high-quality gradient map; T SR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the reconstructed gradient map; T LR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the low-quality gradient image; and ε represents a gradient threshold value.

[0215] The excitation loss function corresponding to the low-quality image sample is represented as follows:

[0216]

[0217] wherein L E represents the excitation loss function corresponding to the low-quality image sample; I HR represents a high-quality image corresponding to the low-quality image sample; I SR represents a reconstructed image corresponding to the low-quality image sample; represents multiplication of elements in corresponding positions in matrices; || represents taking an absolute value; and ||2 represents a modulo operation.

[0218] obtain a penalty matrix according to the high-frequency enhanced excess region, and obtain a penalty loss function corresponding to the low-quality image sample according to the penalty matrix.

[0219] Each element in the penalty matrix is represented as follows:

[0220]

[0221] wherein, M P represents a penalty matrix; M P (i,j) represents an element in the i-th row and the j-th column of the penalty matrix; T HR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the high-quality gradient map; T SR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the reconstructed gradient map; T LR (i,j) represents a gradient value of a pixel point in the i-th row and the j-th column of the low-quality gradient image; and ε represents a gradient threshold value;

[0222] The penalty loss function corresponding to the low-quality image sample is represented as follows:

[0223]

[0224] wherein, L P represents the penalty loss function corresponding to the low-quality image sample; I HR represents a high-quality image corresponding to the low-quality image sample; I SR represents a reconstructed image corresponding to the low-quality image sample; represents multiplication of elements in corresponding positions of matrices; || represents taking an absolute value; and ||2 represents a modulus operation.

[0225] In order to perform each step in the image enhancement method provided by the embodiments of the present application, an implementation manner of an image enhancement device is given below. Please refer to Figure 8 , Figure 8 a function module diagram of an image enhancement device 400 provided by the embodiments of the present application. It should be noted that the basic principle and the generated technical effects of the image enhancement device 400 provided by the present embodiment are the same as those of the above-mentioned embodiments, and for brief description, the part not mentioned in the present embodiment can be referred to the corresponding content in the above-mentioned embodiments. The image enhancement device 400 comprises:

[0226] an image acquisition module 410, configured to acquire a low-quality image to be processed;

[0227] an image enhancement module 430, configured to perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image to be processed by using an image enhancement model, and obtain a high-quality image corresponding to the low-quality image to be processed.

[0228] The embodiments of the present application further provide an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the image enhancement model training method and / or the image enhancement method disclosed by the embodiments of the present application when executing the computer program.

[0229] The embodiment of the present application further provides a storage medium, which stores a computer program, and the computer program is executed by a processor to realize the image enhancement model training method and / or the image enhancement method disclosed by the embodiment of the present application.

[0230] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementation manners, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0231] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0232] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0233] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. An image enhancement model training method, characterized in that, The method comprises: obtaining a plurality of low-quality image samples and corresponding high-quality images thereof; for each of the low-quality image samples, performing high-frequency enhancement and low-frequency enhancement on the low-quality image sample by using a generator to obtain a reconstructed image corresponding to the low-quality image sample; based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, obtaining a high-frequency enhancement loss function corresponding to the low-quality image sample; based on a preset loss function and the high-frequency enhancement loss function corresponding to each of the low-quality image samples, training the generator to obtain a trained generator, and taking the trained generator as an image enhancement model; the generator comprises a plurality of network layers; the use of the generator to perform high-frequency enhancement and low-frequency enhancement on the low-quality image sample to obtain a reconstructed image corresponding to the low-quality image sample comprises: taking the low-quality image sample as an input feature, and taking a first network layer in the generator as a target network layer; using the target network layer to perform convolution operation on the input feature to obtain a first feature map and a second feature map; using the target network layer to perform high-frequency enhancement operation on the first feature map to obtain a high-frequency feature map; the high-frequency feature map represents local detail information of the low-quality image sample; using the target network layer to perform low-frequency enhancement operation on the second feature map to obtain a low-frequency feature map; the low-frequency feature map represents overall trunk information of the low-quality image sample; using the target network layer to splice the high-frequency feature map and the low-frequency feature map and to perform residual connection with the input feature to obtain an output feature; after taking the output feature as an input feature and taking a next network layer of the target network layer as a target network layer, repeating the step of using the target network layer to perform convolution operation on the input feature to obtain a first feature map and a second feature map until the output feature of the last network layer in the generator is obtained, thereby obtaining the reconstructed image corresponding to the low-quality image sample.

2. The image enhancement model training method according to claim 1, characterized in that, the network layer comprises a high-frequency branch network; the use of the target network layer to perform high-frequency enhancement operation on the first feature map to obtain a high-frequency feature map comprises: using the high-frequency branch network in the target network layer to perform convolution operation and global pooling operation on the first feature map to obtain a scaling coefficient of each channel in the first feature map; using the high-frequency branch network in the target network layer to perform fast Fourier transform operation on the first feature map to obtain a frequency distribution of each channel in the first feature map; using the high-frequency branch network in the target network layer to calculate a cutoff frequency of each channel based on the scaling coefficient and the frequency distribution of each channel, and performing high-pass filtering on the first feature map based on the cutoff frequencies of all channels to obtain an initial high-frequency feature map; using the high-frequency branch network in the target network layer to perform convolution operation, global pooling operation and activation operation on the first feature map to obtain a sharpening parameter; using the high-frequency branch network in the target network layer to perform sharpening operation on the first feature map based on the sharpening parameter and the initial high-frequency feature map to obtain the high-frequency feature map.

3. The image enhancement model training method of claim 1, wherein, The network layer comprises a low-frequency branch network; The low-frequency feature map is obtained by performing low-frequency enhancement operation on the second feature map using the target network layer, and the low-frequency branch network in the target network layer comprises: The low-frequency feature map is obtained by performing depth separable convolution operation and convolution operation on the second feature map using the low-frequency branch network in the target network layer.

4. The image enhancement model training method of claim 1, wherein, The high-frequency enhancement loss function corresponding to the low-quality image sample is obtained based on the high-quality image and the reconstructed image corresponding to the low-quality image sample, and the high-frequency enhancement loss function comprises: The low-quality gradient image, the high-quality gradient image and the reconstructed gradient image are obtained by performing Laplacian gradient transformation on the low-quality image sample, the high-quality image corresponding to the low-quality image sample and the reconstructed image respectively; The average value of gradient values of all pixel points in the low-quality gradient image is calculated, and the product of the average value and a preset coefficient is obtained to obtain a gradient threshold value; The initial region is obtained by acquiring all pixel points in the low-quality gradient image whose gradient values are greater than or equal to the gradient threshold value, and the target region is obtained by performing Gaussian smoothing operation on the initial region; the target region represents a region where high-frequency features of the low-quality image sample exist; The high-frequency enhancement deficiency region and the high-frequency enhancement excess region in the reconstructed image are determined according to the target region, the high-quality gradient image and the reconstructed gradient image; The high-frequency enhancement loss function corresponding to the low-quality image sample is obtained according to the high-frequency enhancement deficiency region and the high-frequency enhancement excess region.

5. The image enhancement model training method according to claim 4, characterized in that, The high-frequency enhancement deficiency region and the high-frequency enhancement excess region in the reconstructed image are determined according to the target region, the high-quality gradient image and the reconstructed gradient image, and the method comprises the steps of: Any one pixel point in the target region is taken as a target pixel point; The first pixel point and the second pixel point are obtained by acquiring the pixel points corresponding to the target pixel point in the high-quality gradient image and the reconstructed gradient image respectively; If the gradient value of the first pixel point is greater than the gradient value of the second pixel point, the pixel point corresponding to the second pixel point in the reconstructed image is taken as an enhancement deficiency pixel point; If the gradient value of the first pixel point is less than the gradient value of the second pixel point, the pixel point corresponding to the second pixel point in the reconstructed image is taken as an enhancement excess pixel point; The high-frequency enhancement deficiency region composed of all enhancement deficiency pixel points and the high-frequency enhancement excess region composed of all enhancement excess pixel points in the reconstructed image are obtained by traversing each pixel point in the target region.

6. The image enhancement model training method according to claim 4, characterized in that, The high-frequency enhancement loss function corresponding to the low-quality image sample is obtained according to the high-frequency enhancement deficiency region and the high-frequency enhancement excess region, and the method comprises the steps of: The excitation matrix is obtained according to the high-frequency enhancement deficiency region, and the excitation loss function corresponding to the low-quality image sample is obtained according to the excitation matrix; Each element in the excitation matrix represents as follows: ； in, Represents the activation matrix; Represents the first in the excitation matrix i Line 1 j Column elements; In the high-quality gradient graph, the first... i Line 1 j Gradient values ​​of column pixels; Indicates the first step in the reconstruction gradient graph. i Line 1 j Gradient values ​​of column pixels; Indicating the first low-quality gradient image i Line 1 j Gradient values ​​of column pixels; Indicates the gradient threshold; The excitation loss function corresponding to the low-quality image sample represents as follows: ； wherein, represents the excitation loss function corresponding to the low-quality image sample; represents the high-quality image corresponding to the low-quality image sample; represents the reconstructed image corresponding to the low-quality image sample; represents the multiplication of the elements in the corresponding positions in the matrices; represents the absolute value; represents the modulo operation; obtain a penalty loss function corresponding to the low-quality image sample according to the penalty matrix; each element in the penalty matrix represents the following: ； in, Represents the penalty matrix; Represents the first element in the penalty matrix. i Line 1 j Column elements; In the high-quality gradient graph, the first... i Line 1 j Gradient values ​​of column pixels; Indicates the first step in the reconstruction gradient graph. i Line 1 j Gradient values ​​of column pixels; Indicating the first low-quality gradient image i Line 1 j Gradient values ​​of column pixels; Indicates the gradient threshold; the penalty loss function corresponding to the low-quality image sample represents the following: ； wherein, represents a penalty loss function corresponding to a low-quality image sample; represents a high-quality image corresponding to a low-quality image sample; represents a reconstructed image corresponding to a low-quality image sample; represents multiplication of elements in corresponding positions in matrices; represents taking an absolute value; represents a modulo operation.

7. An image enhancement method characterized by, The method comprises: obtaining a to-be-processed low-quality image; performing high-frequency enhancement operation and low-frequency enhancement operation on the to-be-processed low-quality image by using an image enhancement model to obtain a high-quality image corresponding to the to-be-processed low-quality image; The image enhancement model is obtained according to the image enhancement model training method in any one of claims 1 to 6.

8. An image enhancement model training apparatus, characterized by comprising: The image enhancement model training device comprises: a sample acquisition module configured to acquire a plurality of low-quality image samples and high-quality images corresponding to the low-quality image samples; a model training module configured to, for each low-quality image sample, perform high-frequency enhancement operation and low-frequency enhancement operation on the low-quality image sample by using a generator to obtain a reconstructed image corresponding to the low-quality image sample; obtain a high-frequency enhancement loss function corresponding to the low-quality image sample based on the high-quality image and the reconstructed image corresponding to the low-quality image sample; train the generator based on a preset loss function and the high-frequency enhancement loss function corresponding to each low-quality image sample to obtain a trained generator, and use the trained generator as an image enhancement model; The generator comprises a plurality of network layers; the model training module is specifically configured to: take the low-quality image sample as an input feature, and take a first network layer in the generator as a target network layer; perform convolution operation on the input feature by using the target network layer to obtain a first feature map and a second feature map; perform high-frequency enhancement operation on the first feature map by using the target network layer to obtain a high-frequency feature map; the high-frequency feature map represents local detail information of the low-quality image sample; perform low-frequency enhancement operation on the second feature map by using the target network layer to obtain a low-frequency feature map; the low-frequency feature map represents overall trunk information of the low-quality image sample; and perform stitching on the high-frequency feature map and the low-frequency feature map by using the target network layer, and perform residual connection on the input feature to obtain an output feature; after taking the output feature as an input feature and taking a next network layer of the target network layer as a target network layer, repeat the step of performing convolution operation on the input feature by using the target network layer to obtain a first feature map and a second feature map until an output feature of a last network layer in the generator is obtained, so as to obtain a reconstructed image corresponding to the low-quality image sample.

9. An image enhancement device, characterized by The image enhancement device comprises: an image acquisition module configured to acquire a to-be-processed low-quality image; an image enhancement module configured to perform high-frequency enhancement operation and low-frequency enhancement operation on the to-be-processed low-quality image by using an image enhancement model to obtain a high-quality image corresponding to the to-be-processed low-quality image; The image enhancement model is obtained according to the image enhancement model training method in any one of claims 1 to 6.

10. An electronic device, comprising: The image enhancement model training method and / or the image enhancement method in any one of claims 1-6 are implemented by a processor executing a computer program stored in a memory.

11. A storage medium, characterized by The image enhancement model training method and / or the image enhancement method in any one of claims 1-6 are implemented by a processor executing a computer program stored in a memory.

Images