Image enhancement and model training methods, devices, equipment, media and products

Abstract

The present disclosure provides an image enhancement and model training method, apparatus, device, medium, and product, relating to the fields of artificial intelligence technology, specifically computer vision, deep learning, and other technical fields. The image enhancement method includes: extracting features from an original image to obtain an initial feature map; processing the initial feature map based on a basic block to obtain a target feature map; and acquiring a target image based on the target feature map; wherein the basic block includes an attention network and a feedforward network, wherein the attention network is a residual network composed of attention layers and convolutional layers, and the feedforward network is a residual network composed of gating units.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning and other technical fields, and in particular to an image enhancement and model training method, device, equipment, medium and product. Background Art

[0002] Dark light enhancement is an image enhancement technology that improves image quality by enhancing low-light images.

[0003] How to achieve dark light enhancement efficiently and lightweight is a problem that needs to be solved. Summary of the Invention

[0004] The present disclosure provides an image enhancement method, apparatus, device, medium and product.

[0005] According to one aspect of the present disclosure, an image enhancement method is provided, comprising: extracting features from an original image to obtain an initial feature map; processing the initial feature map based on a basic block to obtain a target feature map; and acquiring a target image based on the target feature map; wherein the basic block comprises: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0006] According to another aspect of the present disclosure, a method for training an image enhancement model is provided, wherein the image enhancement model includes a basic block, and the method includes: performing feature extraction on an image sample to obtain an initial feature map; processing the initial feature map based on the basic block to obtain a target feature map; obtaining a predicted image based on the target feature map; constructing a loss function based on the predicted image and a real image corresponding to the image sample; and adjusting the model parameters of the basic block based on the loss function; wherein the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0007] According to another aspect of the present disclosure, an image enhancement device is provided, comprising: an extraction module for performing feature extraction on an original image to obtain an initial feature map; a processing module for processing the initial feature map according to a basic block to obtain a target feature map; and an acquisition module for acquiring a target image according to the target feature map; wherein the basic block comprises: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0008] According to another aspect of the present disclosure, an image enhancement model training device is provided, wherein the image enhancement model includes a basic block, and the device includes: an extraction module for performing feature extraction on an image sample to obtain an initial feature map; a processing module for processing the initial feature map according to the basic block to obtain a target feature map; an acquisition module for acquiring a predicted image according to the target feature map; a construction module for constructing a loss function according to the predicted image and a real image corresponding to the image sample; an adjustment module for adjusting the model parameters of the basic block according to the loss function; wherein the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the methods described in any one of the above aspects.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the above aspects.

[0012] According to the embodiments of the present disclosure, image enhancement can be achieved efficiently and lightweight.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of an application scenario for implementing an embodiment of the present disclosure;

[0017] Figure 3is a schematic diagram of an image enhancement model provided according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a basic block provided according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of an SGAU provided according to an embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram according to a second embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram according to a third embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0023] Figure 9 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0024] Figure 10 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0025] Figure 11 Schematic diagram of an electronic device used to implement the image enhancement method or image enhancement model training method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In order to achieve image enhancement efficiently and light-weight, the present disclosure provides the following embodiments.

[0028] Figure 1 This embodiment provides an image enhancement method, such as Figure 1 As shown, the method includes:

[0029] 101. Perform feature extraction on the original image to obtain an initial feature map.

[0030] 102. Based on the basic block, process the initial feature map to obtain a target feature map.

[0031] 103. Acquire a target image based on the target feature map.

[0032] Among them, the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0033] The original image refers to the image before processing.

[0034] The target image refers to the processed image.

[0035] For example, in a low-light enhancement scenario, the original image refers to the image before low-light enhancement, that is, the low-light image; the target image is the reconstructed image under normal light illumination.

[0036] The initial feature map is the feature map obtained after feature extraction from the original image. Specifically, the initial feature map can be obtained by processing the original image using a convolutional network.

[0037] The target feature map refers to the feature map after processing the initial feature map.

[0038] Specifically, the initial feature map can be processed based on the basic blocks.

[0039] The basic block is the basic unit for processing feature maps, and the specific number can be set according to actual needs.

[0040] The basic blocks include: attention network and feed-forward network (FFN).

[0041] Both attention network and FFN are residual networks. In addition, attention network is based on attention layer and convolution layer, while FFN is based on gated units.

[0042] Residual Network (ResNet) is a network architecture used in deep learning. It is usually composed of two parts: a shortcut path and a residual path. The shortcut path usually does not contain nonlinear transformations; the residual path usually contains nonlinear transformations such as multiple convolutional layers. The output of the residual path is added to the output of the shortcut path to obtain the final output. It can be expressed as:

[0043] H(x)=F(x)+A(x)

[0044] Where x is the input of the residual network, H(x) is the output of the residual network, F(x) is the output of the residual path, and A(x) is the output of the shortcut path. If the shortcut path is a direct path, then A(x)=x.

[0045] In this embodiment, the attention network is a residual network composed of an attention layer and a convolutional layer, wherein the attention layer serves as a shortcut path and the convolutional layer serves as a residual path.

[0046] FFN is a residual network built based on gating units, whose residual path includes gating units and its shortcut path is a direct path.

[0047] In this embodiment, the attention network is a residual network composed of attention layers and convolutional layers. It can introduce the attention mechanism of the Transformer network into convolutional neural networks (CNNs), thereby improving the network's feature extraction capabilities and enhancing image enhancement. The FFN is a residual network based on gated units, which can simplify the network structure, improve representation capabilities while reducing weight, and enhance image enhancement. As a result, image enhancement can be achieved efficiently and lightweight.

[0048] In order to better understand the present disclosure, the application scenarios involved in the present disclosure are described as follows:

[0049] Figure 2 It is a schematic diagram of an application scenario for implementing the embodiment of the present disclosure.

[0050] like Figure 2 As shown, a user terminal 201 may have an application (APP) for image enhancement installed. The user can use the APP to upload an original image (e.g., a low-light image). For example, using local image enhancement on the user terminal, after acquiring the original image, the user terminal can use a pre-deployed image enhancement model to process the original image to obtain a target image (e.g., a low-light image enhanced by normal light). After acquiring the target image, the APP can display it to the user.

[0051] The above takes the local deployment of the image enhancement model in the user terminal as an example. It can be understood that the image enhancement model can also be deployed on the server. The user terminal interacts with the server and sends the original image to the server. The server enhances the original image based on the image enhancement model to obtain the target image and returns it to the user terminal for display.

[0052] Figure 3 Schematic diagram of an image enhancement model provided according to an embodiment of the present disclosure.

[0053] like Figure 3 As shown in Figure 3, the image enhancement model includes: convolution blocks and basic blocks.

[0054] There are two convolution blocks. The convolution block at the input end is used to convert the image into features (feature map), and the convolution block at the output end is used to convert the features into an image.

[0055] The original image is input into the convolution block at the input end, and the output is the initial feature map of the initial resolution.

[0056] The basic block is used to process the initial feature map to obtain the target feature map.

[0057] The target feature map is input to the convolution block at the output end, and the output is an image, which can be called a difference image. After that, the original image and the difference image are added to obtain the target image.

[0058] The basic block can be divided into encoding function and decoding function in terms of function. For this purpose, the basic block can be divided into encoding block and decoding block. Figure 3 The left part, which is the basic block corresponding to the downsampling module, can be called the encoding block. Figure 3 The right part of , i.e., the basic block of the convolution block corresponding to the upsampling module and the output end, can be called a decoding block.

[0059] The encoding block is used to encode the initial feature map to obtain the intermediate feature map; the decoding block is used to decode the intermediate feature map to obtain the target feature map.

[0060] In order to improve the model effect, feature maps of different resolutions can be obtained and fused. For this purpose, Figure 3 As shown in Figure 1, the image enhancement model also includes a downsampling module and an upsampling module, through which feature maps of various resolutions can be obtained.

[0061] The number of basic blocks and up- and down-sampling modules can be set according to actual needs. These modules can form a U-shaped network (Unet) structure.

[0062] In this embodiment, to ensure real-time performance, there are three downsampling modules and three upsampling modules, and the number of basic blocks under each resolution feature map is 1. In addition, the network width can be selected as 8, and the upsampling and downsampling coefficients are both 2. This can reduce redundant network parameters and thus ensure the real-time performance of the model.

[0063] Assuming that the initial resolution (resolution of the initial feature map) is 1, and the upsampling and downsampling coefficients are both 2, then, Figure 3 As shown, feature maps with resolutions of 1, 1 / 2, 1 / 4, and 1 / 8 can be obtained respectively.

[0064] After obtaining feature maps of different resolutions, they can be fused. Figure 3As shown, the feature map from the encoding block and the feature map from the upsampling module of the same resolution are fused, such as element-by-element addition or splicing in the channel dimension, to obtain a fused feature map, and then the fused feature map is decoded by the decoding block of the corresponding resolution.

[0065] In terms of specific composition, the convolution block can adopt 3x3 convolution (Conv2d 3x3).

[0066] The downsampling module can be implemented through 3x3 convolution and the Rectified Linear Unit (ReLU) function. ReLU is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0067] The upsampling module can be implemented through 1x1 convolution and resizing operators. The resizing operator uses nearest neighbor interpolation to improve the real-time performance of the model. Nearest neighbor interpolation uses the values ​​of the pixels closest to a pixel to be calculated as the value of the pixel to be calculated.

[0068] Figure 4 is a schematic diagram of a basic block provided according to an embodiment of the present disclosure.

[0069] like Figure 4 As shown, the basic blocks include: attention network and FFN.

[0070] The input feature map of the basic block is first processed by the attention network, and the feature map output by the attention network is input into the FFN. After processing by the FFN, the output feature map of the basic block is obtained.

[0071] The attention network is a residual network composed of convolutional layers and attention layers.

[0072] like Figure 4 As shown, the attention layer can specifically adopt the channel attention layer.

[0073] Channel attention layer processes feature maps based on the channel attention mechanism.

[0074] The channel attention mechanism assigns a weight to each feature map channel. This weight reflects the importance of the channel to the final task. The input feature map is multiplied by the corresponding weight to obtain a weighted feature map. In this way, important channel features are amplified, while unimportant channel features are suppressed.

[0075] like Figure 4As shown in the figure, the convolution layer can specifically include: a common convolution layer and a depthwise separable convolution layer (Depthwiseconv2d). The common convolution layer is represented by Conv2d 3x3, and the depthwise separable convolution layer is represented by DWConv2d 3x3.

[0076] A standard convolutional layer performs convolution on each channel of the input feature map separately and then accumulates the results to produce the output feature map. Depthwise separable convolution, on the other hand, splits this process into two steps: depthwise convolution and pointwise convolution. Depthwise convolution applies a separate convolution kernel to each input channel, while pointwise convolution applies a 1x1 convolution kernel to the output of the depthwise convolution.

[0077] Compared with ordinary convolution, depth-wise separable convolution can reduce the amount of computation and the number of model parameters, thereby improving the training and inference speed of the model.

[0078] The activation function uses the LReLU function. The LReLU (Leaky ReLU) function is a variant of the rectified linear unit (ReLU) function. Compared with the ReLU function, it can improve the nonlinear representation ability of the network.

[0079] After the input feature map of the attention network is processed by the channel attention layer and the convolution layer, the feature map output by the channel attention layer is added to the feature map output by the convolution layer to obtain the output feature map of the attention network, which is input into the FFN as the input feature map of the FFN.

[0080] FFN is a residual network based on gated units.

[0081] like Figure 4 As shown in Figure 3, the shortcut path of FFN is the direct path, represented by a straight line; the residual path includes the convolutional layer and the gating unit.

[0082] The convolutional layers include: depthwise separable convolutional layer (DWConv2d 3x3) and normal convolutional layer (Conv2d 3x3).

[0083] The gate unit is specifically the Simple Gate Attention Unit (SGAU).

[0084] Figure 5 is a schematic diagram of an SGAU provided according to an embodiment of the present disclosure.

[0085] like Figure 5As shown in the figure, the input feature map of SGAU is of size H*W*C (height is H, width is W, number of channels is C), and is divided into two feature maps along the channel direction. One feature map is activated by the Swish activation function to obtain the activated feature map, and then multiplied element-wise with the other feature map to obtain the output feature map of SGAU, which is of size H*W*C / 2.

[0086] The Swish activation function is a nonlinear activation function that can improve the performance of neural networks and thus improve the efficiency of machine learning.

[0087] SGAU achieves lightweighting by splitting feature maps along the channel direction, and implements the gate mechanism through the Swish activation function, thereby dynamically adjusting the influence of different feature layers to help the network better select useful information. It is essentially an attention mechanism to enhance the network's representation ability.

[0088] In combination with the above application scenarios, the present disclosure also provides the following embodiments.

[0089] Figure 6 is a schematic diagram according to a second embodiment of the present disclosure. This embodiment provides an image enhancement method, the method comprising:

[0090] 601. Perform feature extraction on the original image to obtain an initial feature map.

[0091] For example, see Figure 3 , the convolution block can be used to convolve the original image to obtain the initial feature map.

[0092] 602. Based on the basic block, process the initial feature map to obtain a target feature map.

[0093] Among them, the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0094] Specifically, if Figure 4 As shown, the attention layer can be a channel attention layer; the convolution layer includes: a depth-separable convolution layer.

[0095] The channel attention mechanism assigns a weight to each feature map channel. This weight reflects the importance of the channel to the final task. The input feature map is multiplied by the corresponding weight to obtain a weighted feature map. In this way, important channel features are amplified, while unimportant channel features are suppressed.

[0096] A standard convolutional layer performs convolution on each channel of the input feature map separately and then accumulates the results to produce the output feature map. Depthwise separable convolution, on the other hand, splits this process into two steps: depthwise convolution and pointwise convolution. Depthwise convolution applies a separate convolution kernel to each input channel, while pointwise convolution applies a 1x1 convolution kernel to the output of the depthwise convolution.

[0097] Compared with ordinary convolution, depth-wise separable convolution can reduce the amount of computation and the number of model parameters, thereby improving the training and inference speed of the model.

[0098] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability and further improving the image enhancement effect. Through the depthwise separable convolutional layer, the amount of computation can be reduced and the processing efficiency can be improved.

[0099] The gate control unit can be used specifically for:

[0100] Splitting the input feature map of the gate unit into a first sub-feature map and a second sub-feature map;

[0101] Using a preset activation function, the first sub-feature map is activated to obtain a weighted feature map;

[0102] The second sub-feature map is weighted based on the weighted feature map to obtain an output feature map of the gating unit.

[0103] For example, Figure 5 As shown in FIG, the first sub-feature map and the second sub-feature map are obtained after being split by channel, and the size of both is H*W*C / 2. The feature map obtained after the first sub-feature map is processed by a preset activation function (such as the Swish activation function) is called a weighted feature map. The weighted feature map is weighted with the second sub-feature map, such as element-by-element multiplication, to obtain the output feature map of the gate control unit.

[0104] In this embodiment, the computation can be simplified and lightweight through feature map splitting, and the gate mechanism can be realized through activation function processing to improve the feature representation capability and thus enhance the image enhancement effect.

[0105] The specific process of processing the initial feature map using the basic block to obtain the target feature map can adopt an encoding and decoding method.

[0106] Accordingly, the basic block includes: an encoding block and a decoding block; the processing of the initial feature map based on the basic block to obtain a target feature map includes: encoding the initial feature map based on the encoding block to obtain an intermediate feature map; and decoding the intermediate feature map based on the decoding block to obtain the target feature map.

[0107] Among them, reference Figure 3 , the basic block corresponding to downsampling can be called an encoding block, and the basic block corresponding to the convolution block at the upsampling and output end can be called a decoding block.

[0108] The feature map output by the encoding block can be called an intermediate feature map, the feature map output by the convolution block at the input end is called an initial feature map, and the feature map input by the convolution block at the output end is called a target feature map.

[0109] In this way, through the encoding and decoding process, the target feature map can be efficiently obtained based on the initial feature map.

[0110] Furthermore, in order to improve the feature representation capability, feature maps of different resolutions can be fused.

[0111] Accordingly, there are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions;

[0112] There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution;

[0113] The decoding block based on the intermediate feature map is decoded to obtain the target feature map, comprising:

[0114] For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution;

[0115] Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution;

[0116] The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

[0117] For example, reference Figure 3 , multiple resolutions are represented by 1, 1 / 2, 1 / 4, and 1 / 8 respectively. During decoding, processing starts from the lowest resolution. For example, the bottom basic block is used to process the feature map with a resolution of 1 / 8, and then the processed feature map is upsampled by a factor of 2 to obtain a feature map with a resolution of 1 / 4.

[0118] The current resolution refers to any resolution corresponding to the fused feature map. In this embodiment, the current resolutions are 1 / 4, 1 / 2, and 1, respectively.

[0119] The previous resolution is adjacent to and smaller than the current resolution, while the next resolution is adjacent to and larger than the current resolution. For example, if the current resolution is 1 / 4, the previous resolution is 1 / 8 and the next resolution is 1 / 2.

[0120] Taking the current resolution of 1 / 4 as an example, the intermediate feature map with a resolution of 1 / 4 and the upsampled feature map with a resolution of 1 / 4 are fused. After adding them, a fused feature map with a resolution of 1 / 4 is obtained. After processing the fused feature map with the corresponding decoding block, a decoded feature map with a resolution of 1 / 4 is obtained.

[0121] By analogy, we can finally obtain a decoding feature map with a resolution of 1, that is, the feature map output by the rightmost basic block, as the target feature map.

[0122] In this embodiment, by fusing feature maps of different resolutions, the feature representation capability can be improved, thereby enhancing the image enhancement effect.

[0123] 603. Perform convolution on the target feature map to obtain a difference image.

[0124] 604. Add the original image and the difference image to obtain a target image.

[0125] For example, reference Figure 3 , the convolution block at the output end is used to perform convolution processing on the target feature map. The output image is called the difference image. The difference image and the original image are then added to obtain the target image.

[0126] In this embodiment, the target image is obtained based on the difference image and the original image, which can be obtained simply and efficiently.

[0127] Figure 7 FIG3 is a schematic diagram of a third embodiment of the present disclosure, which provides an image enhancement method. This embodiment takes image enhancement based on an image enhancement model as an example. The method includes:

[0128] 701. Use the input convolution block in the image enhancement model to extract features from the original image to obtain an initial feature map.

[0129] For example, reference Figure 3 , the original image is input into the input convolution block, and the output is the initial feature map.

[0130] 702. Use multiple encoding blocks in the image enhancement model to encode the initial feature map to obtain intermediate feature maps of multiple resolutions.

[0131] For example, reference Figure 3 The basic block corresponding to downsampling can be called a coding block. Through multiple coding blocks and their corresponding downsampling processing, intermediate feature maps of various resolutions can be obtained. For example, various resolutions are represented by 1, 1 / 2, 1 / 4, and 1 / 8 respectively.

[0132] 703. For a current resolution among the multiple resolutions, fuse the intermediate feature map and the upsampled feature map of the current resolution to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution.

[0133] For example, reference Figure 3 Taking the fusion as an example, assuming that the current resolution is 1 / 2, the intermediate feature map with a resolution of 1 / 2 and the upsampled feature map are added to obtain a fused feature map with a resolution of 1 / 2.

[0134] 704. Use the decoding block corresponding to the current resolution in the image enhancement model to decode the fused feature map to obtain a decoded feature map of the current resolution.

[0135] For example, reference Figure 3 The basic block corresponding to the upsampling can be called a decoding block. Taking the current resolution of 1 / 2 as an example, the decoding block corresponding to the resolution of 1 / 2 is used to decode the fused feature map of resolution 1 / 2 to obtain a decoded feature map of resolution 1 / 2.

[0136] The decoded feature map is input into the upsampling module, and the output is the upsampled feature map. For example, after upsampling the decoded feature map with a resolution of 1 / 2, a feature map with a resolution of 1 is obtained.

[0137] 705. Use the decoded feature map when the current resolution is the initial resolution of the initial feature map as the target feature map.

[0138] For example, reference Figure 3 After processing by the rightmost basic block, a decoding feature map with a resolution of 1 is obtained. Since the initial resolution is represented by 1, the decoding feature map at this time, that is, the output feature map of the rightmost basic block, is used as the target feature map.

[0139] 706. Perform convolution on the target feature map to obtain a difference image.

[0140] For example, reference Figure 3 , the target feature map is input into the output convolution block, and the output of the convolution block is the difference image.

[0141] 707. Add the original image and the difference image to obtain a target image.

[0142] For example, reference Figure 3 , the original image and the difference image output by the output convolution block are added to obtain the target image.

[0143] The above-mentioned overall image enhancement process can achieve lightweight and efficient image enhancement, improve the real-time performance of the image enhancement process, and is suitable for user-side processing scenarios. In addition, it also has a good image enhancement effect on super-resolution images.

[0144] After experiments and verification on public datasets such as LOLv1, LOLv2_real and LOLv2_syn, the above-mentioned image enhancement model was used for dark light enhancement with a small number of parameters and flops, indicating that the computational complexity was reduced; PSNR and SSIM both showed significant improvements compared to related algorithms, indicating that the image enhancement effect was improved.

[0145] In a low-light enhancement and super-resolution validation on the RELLISUR dataset, PSNR and SSIM also showed significant improvements over related algorithms. Therefore, the image enhancement model based on this disclosure performs well in both low-light enhancement and low-light enhancement super-resolution scenarios, demonstrating the versatility of the model.

[0146] LOLv1, LOLv2_real, LOLv2_syn, and RELLISUR are public datasets, flops stands for floating-point operations per second, psnr stands for peak signal-to-noise ratio, and ssim stands for structural similarity index. Dark-light enhancement super-resolution can achieve dark-light enhancement and improve image resolution. For example, an upsampling module is added after the model's summation operation to obtain a target image with a target resolution greater than the original resolution of the original image. This added upsampling module can specifically include a 3x3 convolution and a pixel shuffle operator. The pixel shuffle operator is an operation used in fields such as image super-resolution to convert low-resolution feature maps into high-resolution images.

[0147] Figure 8 is a schematic diagram according to a fourth embodiment of the present disclosure. This embodiment provides an image enhancement model training method. The image enhancement model includes a basic block. The method includes:

[0148] 801. Perform feature extraction on the image sample to obtain an initial feature map.

[0149] 802. Based on the basic block, process the initial feature map to obtain a target feature map.

[0150] 803. Obtain a predicted image based on the target feature map.

[0151] 804. Construct a loss function based on the predicted image and the real image corresponding to the image sample.

[0152] 805. Adjust the model parameters of the basic block based on the loss function.

[0153] Among them, the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0154] Among them, image samples can be obtained from existing sample sets. In the dark light enhancement scenario, the pattern samples are low-light images. In addition, the real images corresponding to the image samples can be obtained from existing sample sets or through annotation, that is, the normal light illumination images corresponding to the low-light images.

[0155] The process of obtaining the predicted image during model training is similar to the process of obtaining the target image during the above-mentioned image enhancement and will not be repeated here.

[0156] After obtaining the predicted image, a loss function, such as the L1 loss function, can be constructed based on the predicted image and the true image. The L1 loss function calculates the average of the absolute differences between the predicted value and the true value to measure the accuracy of the model's prediction.

[0157] After obtaining the loss function, the model parameters of the basic blocks can be adjusted using methods such as backpropagation until a preset end condition is reached, such as reaching a preset number of iterations or model convergence. The model that meets the end condition is used as the final image enhancement model. During the inference phase, this image enhancement model can be used in the image enhancement process.

[0158] In this embodiment, the attention network is a residual network composed of attention layers and convolutional layers. It can introduce the attention mechanism of the Transformer network into convolutional neural networks (CNNs), thereby improving the network's feature extraction capabilities and enhancing the effectiveness of image enhancement models. The FFN is a residual network composed of gated units, which can simplify the network structure, improve representation capabilities while reducing weight, and enhance the effectiveness of image enhancement models. As a result, an efficient and lightweight image enhancement model can be obtained, enabling efficient and lightweight image enhancement.

[0159] In some embodiments, the attention layer is a channel attention layer; and / or,

[0160] The convolutional layer includes: a depth-wise separable convolutional layer; and / or,

[0161] The gate control unit is used for:

[0162] Splitting the input feature map of the gate unit into a first sub-feature map and a second sub-feature map;

[0163] Using a preset activation function, the first sub-feature map is activated to obtain a weighted feature map;

[0164] The second sub-feature map is weighted based on the weighted feature map to obtain an output feature map of the gating unit.

[0165] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability and further improving the image enhancement effect. Through the depthwise separable convolutional layer, the amount of computation can be reduced and the processing efficiency can be improved.

[0166] In this embodiment, the computation can be simplified and lightweight through feature map splitting, and the gate mechanism can be realized through activation function processing to improve the feature representation capability and thus enhance the image enhancement effect.

[0167] In some embodiments, the basic blocks include: an encoding block and a decoding block;

[0168] The processing of the initial feature map based on the basic block to obtain a target feature map includes:

[0169] Encoding the initial feature map based on the encoding block to obtain an intermediate feature map;

[0170] Based on the decoding block, the intermediate feature map is decoded to obtain the target feature map.

[0171] In this embodiment, through the encoding and decoding process, the target feature map can be efficiently obtained based on the initial feature map.

[0172] In some embodiments, there are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions;

[0173] There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution;

[0174] The decoding block based on the intermediate feature map is decoded to obtain the target feature map, comprising:

[0175] For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution;

[0176] Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution;

[0177] The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

[0178] In this embodiment, by fusing feature maps of different resolutions, the feature representation capability can be improved, thereby improving the image enhancement model and image enhancement effect.

[0179] In some embodiments, obtaining a predicted image based on the target feature map includes:

[0180] Convolving the target feature map to obtain a difference image;

[0181] The image samples and the difference image are added to obtain the predicted image.

[0182] In this embodiment, the predicted image is obtained based on the difference image and the image sample, which can easily and efficiently obtain the predicted image.

[0183] Figure 9 is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides an image enhancement device. The device 900 includes: an extraction module 901 , a processing module 902 and an acquisition module 903 .

[0184] The extraction module 901 is used to extract features from the original image to obtain an initial feature map; the processing module 902 is used to process the initial feature map according to the basic block to obtain a target feature map; the acquisition module 903 is used to obtain a target image according to the target feature map;

[0185] Among them, the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0186] In this embodiment, the attention network is a residual network composed of attention layers and convolutional layers. It can introduce the attention mechanism of the Transformer network into convolutional neural networks (CNNs), thereby improving the network's feature extraction capabilities and enhancing image enhancement. The FFN is a residual network based on gated units, which can simplify the network structure, improve representation capabilities while reducing weight, and enhance image enhancement. As a result, image enhancement can be achieved efficiently and lightweight.

[0187] In some embodiments, the attention layer is a channel attention layer; and / or,

[0188] The convolution layer includes: a depth-wise separable convolution layer.

[0189] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability and further improving the image enhancement effect. Through the depthwise separable convolutional layer, the amount of computation can be reduced and the processing efficiency can be improved.

[0190] In some embodiments, the gate control unit is used to:

[0191] Splitting the input feature map of the gate unit into a first sub-feature map and a second sub-feature map;

[0192] Using a preset activation function, the first sub-feature map is activated to obtain a weighted feature map;

[0193] The second sub-feature map is weighted based on the weighted feature map to obtain an output feature map of the gating unit.

[0194] In this embodiment, the computation can be simplified and lightweight through feature map splitting, and the gate mechanism can be realized through activation function processing to improve the feature representation capability and thus enhance the image enhancement effect.

[0195] In some embodiments, the basic blocks include: an encoding block and a decoding block;

[0196] The processing module 902 is further configured to:

[0197] Encoding the initial feature map based on the encoding block to obtain an intermediate feature map;

[0198] Based on the decoding block, the intermediate feature map is decoded to obtain the target feature map.

[0199] In this embodiment, through the encoding and decoding process, the target feature map can be efficiently obtained based on the initial feature map.

[0200] In some embodiments, there are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions;

[0201] There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution;

[0202] The processing module 902 is further configured to:

[0203] For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution;

[0204] Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution;

[0205] The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

[0206] In this embodiment, by fusing feature maps of different resolutions, the feature representation capability can be improved, thereby enhancing the image enhancement effect.

[0207] In some embodiments, the acquisition module 903 is further configured to:

[0208] Convolving the target feature map to obtain a difference image;

[0209] The original image and the difference image are added to obtain the target image.

[0210] In this embodiment, the target image is obtained based on the difference image and the original image, which can be obtained simply and efficiently.

[0211] Figure 10 It is a schematic diagram according to the sixth embodiment of the present disclosure. This embodiment provides an image enhancement model training device, wherein the image enhancement model includes a basic block. The device 1000 includes: an extraction module 1001, a processing module 1002, an acquisition module 1003, a construction module 1004 and an adjustment module 1005.

[0212] The extraction module 1001 is used to extract features from image samples to obtain an initial feature map; the processing module 1002 is used to process the initial feature map according to the basic block to obtain a target feature map; the acquisition module 1003 is used to obtain a predicted image according to the target feature map; the construction module 1004 is used to construct a loss function according to the predicted image and the real image corresponding to the image sample; the adjustment module 1005 is used to adjust the model parameters of the basic block according to the loss function; wherein the basic block includes: an attention network and a feedforward network, and the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gating unit.

[0213] In this embodiment, the attention network is a residual network composed of attention layers and convolutional layers. It can introduce the attention mechanism of the Transformer network into convolutional neural networks (CNNs), thereby improving the network's feature extraction capabilities and enhancing the effectiveness of image enhancement models. The FFN is a residual network composed of gated units, which can simplify the network structure, improve representation capabilities while reducing weight, and enhance the effectiveness of image enhancement models. As a result, an efficient and lightweight image enhancement model can be obtained, enabling efficient and lightweight image enhancement.

[0214] In some embodiments, the attention layer is a channel attention layer; and / or,

[0215] The convolutional layer includes: a depth-wise separable convolutional layer; and / or,

[0216] The gate control unit is used for:

[0217] Splitting the input feature map of the gate unit into a first sub-feature map and a second sub-feature map;

[0218] Using a preset activation function, the first sub-feature map is activated to obtain a weighted feature map;

[0219] The second sub-feature map is weighted based on the weighted feature map to obtain an output feature map of the gating unit.

[0220] In this embodiment, a channel attention layer is used to focus on important channels in the feature map, improving feature representation capabilities and thereby enhancing image enhancement. A depthwise separable convolutional layer can reduce computational complexity and improve processing efficiency. Feature map splitting simplifies computation and achieves lightweighting. Activation function processing implements a gating mechanism, improving feature representation capabilities and thereby enhancing image enhancement.

[0221] In some embodiments, the basic blocks include: an encoding block and a decoding block;

[0222] The processing module 1002 is further configured to:

[0223] Encoding the initial feature map based on the encoding block to obtain an intermediate feature map;

[0224] Based on the decoding block, the intermediate feature map is decoded to obtain the target feature map.

[0225] In this embodiment, through the encoding and decoding process, the target feature map can be efficiently obtained based on the initial feature map.

[0226] In some embodiments, there are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions;

[0227] There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution;

[0228] The processing module 1002 is further configured to:

[0229] For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution;

[0230] Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution;

[0231] The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

[0232] In this embodiment, by fusing feature maps of different resolutions, the feature representation capability can be improved, thereby improving the image enhancement model and image enhancement effect.

[0233] In some embodiments, the acquisition module 1003 is further configured to:

[0234] Convolving the target feature map to obtain a difference image;

[0235] The image samples and the difference image are added to obtain the predicted image.

[0236] In this embodiment, the predicted image is obtained based on the difference image and the image sample, which can easily and efficiently obtain the predicted image.

[0237] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0238] It can be understood that the terms “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0239] It is understandable that unless there is any special limitation on the order of the steps in the process, it means that the temporal relationship between these steps is not limited.

[0240] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0241] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0242] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0243] like Figure 11 As shown, electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. RAM 1103 may also store various programs and data required for the operation of electronic device 1100. Computing unit 1101, ROM 1102, and RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to bus 1104.

[0244] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0245] The computing unit 1101 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the image enhancement method or the image enhancement model training method. For example, in some embodiments, the image enhancement method or the image enhancement model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image enhancement method or the image enhancement model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the image enhancement method or the image enhancement model training method in any other appropriate manner (for example, by means of firmware).

[0246] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0247] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable task processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0248] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0249] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0250] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0251] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical servers and VPS services ("Virtual Private Servers" or "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0252] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0253] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image enhancement method, comprising: Perform feature extraction on the original image to obtain an initial feature map; Based on the basic block, processing the initial feature map to obtain a target feature map; Acquire a target image based on the target feature map; The basic block includes: an attention network and a feedforward network, wherein the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gated unit; The basic block is functionally divided into encoding function and decoding function, and accordingly, the basic block is divided into encoding block and decoding block; The processing of the initial feature map based on the basic block to obtain a target feature map includes: Encoding the initial feature map based on the encoding block to obtain an intermediate feature map; Decoding the intermediate feature map based on the decoding block to obtain the target feature map; The attention layer is a channel attention layer. After the input feature map of the attention network is processed by the channel attention layer and the convolution layer respectively, the feature map output by the channel attention layer is added to the feature map output by the convolution layer to obtain the output feature map of the attention network. The output feature map is input into the feedforward network as the input feature map of the feedforward network. The convolution layer includes: a common convolution layer, a depth-separable convolution layer and a common convolution layer connected in sequence; The shortcut path of the feedforward network is a direct path; the residual path of the feedforward network includes: a depth-separable convolutional layer, a gating unit and an ordinary convolutional layer connected in sequence, the gating unit is a simplified gated attention unit, and the input feature map of the simplified gated attention unit is divided into two feature maps along the channel direction. One feature map is subjected to an activation function to obtain an activated feature map, and then is element-wise multiplied with the other feature map to obtain the output feature map of the simplified gated attention unit.

2. The method according to claim 1, wherein There are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions; There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution; The decoding block based on the intermediate feature map is decoded to obtain the target feature map, comprising: For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution; Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution; The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

3. The method according to claim 1, wherein The acquiring of a target image based on the target feature map includes: Convolving the target feature map to obtain a difference image; The original image and the difference image are added to obtain the target image.

4. A method for training an image enhancement model, wherein the image enhancement model includes a basic block, the method comprising: Perform feature extraction on the image sample to obtain an initial feature map; Based on the basic block, processing the initial feature map to obtain a target feature map; Based on the target feature map, obtaining a predicted image; Constructing a loss function based on the predicted image and the real image corresponding to the image sample; Adjusting model parameters of the basic block based on the loss function; The basic block includes: an attention network and a feedforward network, wherein the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gated unit; The basic block is functionally divided into encoding function and decoding function, and accordingly, the basic block is divided into encoding block and decoding block; The processing of the initial feature map based on the basic block to obtain a target feature map includes: Encoding the initial feature map based on the encoding block to obtain an intermediate feature map; Decoding the intermediate feature map based on the decoding block to obtain the target feature map; The attention layer is a channel attention layer. After the input feature map of the attention network is processed by the channel attention layer and the convolution layer respectively, the feature map output by the channel attention layer is added to the feature map output by the convolution layer to obtain the output feature map of the attention network. The output feature map is input into the feedforward network as the input feature map of the feedforward network. The convolution layer includes: a common convolution layer, a depth-separable convolution layer and a common convolution layer connected in sequence; The shortcut path of the feedforward network is a direct path; the residual path of the feedforward network includes: a depth-separable convolutional layer, a gating unit and an ordinary convolutional layer connected in sequence, the gating unit is a simplified gated attention unit, and the input feature map of the simplified gated attention unit is divided into two feature maps along the channel direction. One feature map is subjected to an activation function to obtain an activated feature map, and then is element-wise multiplied with the other feature map to obtain the output feature map of the simplified gated attention unit.

5. The method according to claim 4, wherein There are multiple encoding blocks and multiple decoding blocks, and each encoding block and each decoding block corresponds to one of multiple resolutions; There are multiple intermediate feature maps, and each intermediate feature map corresponds to a resolution; The decoding block based on the intermediate feature map is decoded to obtain the target feature map, comprising: For a current resolution among the multiple resolutions, fusing the intermediate feature map of the current resolution and the upsampled feature map to obtain a fused feature map; the upsampled feature map is obtained by upsampling the decoded feature map of the previous resolution; Decoding the fused feature map using a decoding block corresponding to the current resolution to obtain a decoded feature map of the current resolution; The decoded feature map when the current resolution is the initial resolution of the initial feature map is used as the target feature map.

6. The method according to claim 4, wherein: The step of obtaining a predicted image based on the target feature map includes: Convolving the target feature map to obtain a difference image; The image samples and the difference image are added to obtain the predicted image.

7. An image enhancement device comprising: An extraction module is used to extract features from the original image to obtain an initial feature map; a processing module, configured to process the initial feature map according to the basic block to obtain a target feature map; An acquisition module, configured to acquire a target image according to the target feature map; The basic block includes: an attention network and a feedforward network, wherein the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gated unit; The basic block is functionally divided into encoding function and decoding function, and accordingly, the basic block is divided into encoding block and decoding block; The processing module is further configured to: Encoding the initial feature map based on the encoding block to obtain an intermediate feature map; Decoding the intermediate feature map based on the decoding block to obtain the target feature map; The attention layer is a channel attention layer. After the input feature map of the attention network is processed by the channel attention layer and the convolution layer respectively, the feature map output by the channel attention layer is added to the feature map output by the convolution layer to obtain the output feature map of the attention network. The output feature map is input into the feedforward network as the input feature map of the feedforward network. The convolution layer includes: a common convolution layer, a depth-separable convolution layer and a common convolution layer connected in sequence; The shortcut path of the feedforward network is a direct path; the residual path of the feedforward network includes: a depth-separable convolutional layer, a gating unit and an ordinary convolutional layer connected in sequence, the gating unit is a simplified gated attention unit, and the input feature map of the simplified gated attention unit is divided into two feature maps along the channel direction. One feature map is subjected to an activation function to obtain an activated feature map, and then is element-wise multiplied with the other feature map to obtain the output feature map of the simplified gated attention unit.

8. A device for training an image enhancement model, wherein the image enhancement model includes a basic block, and the device comprises: An extraction module is used to extract features from image samples to obtain an initial feature map; a processing module, configured to process the initial feature map according to the basic block to obtain a target feature map; An acquisition module, configured to acquire a predicted image based on the target feature map; A construction module, configured to construct a loss function based on the predicted image and the real image corresponding to the image sample; An adjustment module, configured to adjust the model parameters of the basic block according to the loss function; The basic block includes: an attention network and a feedforward network, wherein the attention network is a residual network composed of an attention layer and a convolutional layer, and the feedforward network is a residual network composed of a gated unit; The basic block is functionally divided into encoding function and decoding function, and accordingly, the basic block is divided into encoding block and decoding block; The processing module is further configured to: Encoding the initial feature map based on the encoding block to obtain an intermediate feature map; Decoding the intermediate feature map based on the decoding block to obtain the target feature map; The attention layer is a channel attention layer. After the input feature map of the attention network is processed by the channel attention layer and the convolution layer respectively, the feature map output by the channel attention layer is added to the feature map output by the convolution layer to obtain the output feature map of the attention network. The output feature map is input into the feedforward network as the input feature map of the feedforward network. The convolution layer includes: a common convolution layer, a depth-separable convolution layer and a common convolution layer connected in sequence; The shortcut path of the feedforward network is a direct path; the residual path of the feedforward network includes: a depth-separable convolutional layer, a gating unit and an ordinary convolutional layer connected in sequence, the gating unit is a simplified gated attention unit, and the input feature map of the simplified gated attention unit is divided into two feature maps along the channel direction. One feature map is subjected to an activation function to obtain an activated feature map, and then is element-wise multiplied with the other feature map to obtain the output feature map of the simplified gated attention unit.

9. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images