Multi-task image enhancement method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of image processing, and discloses a multi-task image enhancement method, device and equipment and a storage medium, an end-to-end network architecture is adopted to construct a multi-task image enhancement model, and the network architecture comprises a shallow layer feature extraction module and a deep layer feature extraction module; symmetrical encoder, frequency regulator and decoder structures are adopted for the deep feature extraction module, and an encoder, a frequency regulator and a decoder are designed respectively; designing a loss function; and obtaining a data set, training the multi-task image enhancement model based on the data set and the loss function, and using the obtained multi-task image enhancement model to perform image enhancement on the to-be-enhanced image. According to the scheme, various image enhancement requirements can be processed at the same time, the universality and the effect of image enhancement are improved, important features can be extracted more effectively and the image enhancement result can be optimized more effectively through the processing and attention mechanism of the frequency regulator, and the characterization capability and the generalization capability of a multi-task image enhancement model are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a multi-task image enhancement method, device, equipment and storage medium. Background Art

[0002] Image enhancement technology plays an important role in assisted driving systems. It greatly enhances the recognition and judgment capabilities of driving assistance systems in complex environments by improving the quality of images captured by cameras, such as the clarity, contrast, and details of targets such as roads, vehicles, and pedestrians. Traditional single-task image enhancement methods improve image quality by introducing various image priors, such as dark channel priors, super Laplace priors, and gradient priors. However, these prior-based image enhancement methods have limited capabilities and often generate poor outputs with artifacts.

[0003] Most existing image enhancement algorithms focus on performance optimization in a single task or specific scenario, but in actual application environments, they often need to cope with diverse tasks and complex environmental conditions. This method comprehensively optimizes various different goals in the driving environment by simultaneously performing multiple enhancement tasks, such as rain removal, fog removal, snow removal, noise removal, and blur removal. In actual applications, this method can cope with various lighting conditions, weather conditions, and road scenes to ensure that drivers receive accurate and clear driving information. Summary of the invention

[0004] Based on this, it is necessary to propose a multi-task image enhancement method, device, equipment and storage medium to address the above problems.

[0005] To achieve the above objectives, the present application provides a first aspect of a multi-task image enhancement method, the method comprising: A multi-task image enhancement model is constructed using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Designing a loss function for the multi-task image enhancement model; Acquire a data set, and train the multi-task image enhancement model based on the data set and the loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

[0006] In some implementations, the shallow feature extraction module specifically includes: The three-channel input image is subjected to a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, and 1×1, and the output new features are residually connected to the input image.

[0007] In some implementations, a symmetrical encoder, frequency regulator, and decoder structure is used for the deep feature extraction module, and the encoder, frequency regulator, and decoder are designed separately, specifically including: Use a 3x3 convolutional layer to downsample the input feature map, reduce the image size and increase the number of channels to extract features; The deep feature extraction module is connected to 8 feature extraction modules, each of which includes three parts. The first part is high-level feature acquisition, which is used to extract high-level features and perform downsampling operations to constitute the encoder. The second part is a frequency regulator unit, which is composed of four identical frequency regulators. The third part is used to reconstruct the image and constitute a decoder through upsampling.

[0008] In some implementations, the design of the encoder and the decoder is performed through three branches, wherein the first branch includes: Perform 1×1 convolution on the input feature map to increase nonlinear transformation and reduce the number of parameters; Performing a dimension conversion operation to adapt the feature map to the input requirements of the multi-layer perceptron MLP; The feature map after the dimension conversion operation is sent to the multi-layer perceptron MLP for network layer processing, and then converted back to the original dimension; The feature map converted back to the original dimension is processed by the Sigmoid activation function.

[0009] In some implementations, the design of the encoder and the decoder is performed through three branches, wherein the second branch includes: Performing a 1×1 convolution on the input feature map and dynamically adjusting the weight of the convolution kernel to adapt to different feature distributions of the input data; Perform adaptive average pooling operation to determine feature representation; Input the feature representation into a 1×1 convolutional layer to generate an attention map; Attention weights are calculated by applying softmax, and the attention weights are element-wise multiplied by a set of learnable parameters and summed to determine the features associated with the input.

[0010] In some implementations, the design of the encoder and the decoder is performed through three branches, wherein the third branch includes: Performing depth convolution with kernel sizes of 1×1 and 3×3 on the input feature map to determine enhanced spatial features; Splitting the enhanced spatial features into two tensors and sending them to a dual-branch structure for processing, wherein the dual-branch structure uses two different filters and filters available features through a comprehensive receptive field, wherein the upper branch in the dual-branch structure includes a 1×1 convolution layer, and the lower branch in the dual-branch structure includes a 1×1 convolution layer and a 3×3 deep convolution layer, the upper branch filters fine-grained noise factors through the receptive field of the 1×1 convolution, and the lower branch filters coarse-grained noise factors by expanding the receptive field; After the dual-branch structure is processed, the dual-branch structure is connected, and an activation function is applied to unify feature differences to determine an output feature map.

[0011] In some embodiments, the designing the frequency regulator includes: Performing global average pooling, 1x1 convolution, and batch normalization on the features extracted by the encoder to obtain a low-frequency filter; The low-frequency filter weights are normalized using the softmax function to obtain the normalized low-frequency filter weights; Segmenting the input features in the channel dimension, and multiplying the segmentation result by the low-frequency filter weight to determine the low-frequency features; Subtracting the input feature from the low-frequency feature to determine a high-frequency feature; The low-frequency features and the high-frequency features are fused, and a global average pooling operation is performed on the fused features, and the fused features are converted into high-dimensional features through a convolution layer; Decompose the high-dimensional features into high-frequency attention feature vectors and low-frequency attention feature vectors through convolution operations; Connect the high-frequency and low-frequency attention feature vectors and pass them through the softmax function to obtain the normalized attention weights; The attention weight is divided into two parts, and multiplied with the corresponding high-frequency attention feature and low-frequency attention feature respectively to obtain a low-frequency map and a high-frequency map; The low-frequency image and the high-frequency image are added and integrated through a convolutional layer to obtain a final feature representation, and the final feature representation is sent to the decoder.

[0012] To achieve the above-mentioned purpose, the second aspect of the present application provides a multi-task image enhancement device, the device comprising: A model building module, used to build a multi-task image enhancement model using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A structure design module, used for adopting a symmetrical encoder, frequency regulator and decoder structure for the deep feature extraction module, and designing the encoder, frequency regulator and decoder respectively; A loss function module, used to design a loss function of the multi-task image enhancement model; A training module, used for acquiring a data set, and training the multi-task image enhancement model based on the data set and the loss function; The image enhancement module is used to perform image enhancement on the image to be enhanced using a trained multi-task image enhancement model.

[0013] To achieve the above-mentioned object, the third aspect of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps: A multi-task image enhancement model is constructed using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Designing a loss function for the multi-task image enhancement model; Acquire a data set, and train the multi-task image enhancement model based on the data set and the loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

[0014] To achieve the above-mentioned object, the fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps: A multi-task image enhancement model is constructed using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Designing a loss function for the multi-task image enhancement model; Acquire a data set, and train the multi-task image enhancement model based on the data set and the loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

[0015] The embodiments of the present invention have the following beneficial effects: The present invention constructs a multi-task image enhancement model through an end-to-end deep learning network architecture, because through the end-to-end design, the model can directly generate an enhanced image from the input image, without the need to design a specific pre-processing or post-processing module for each task separately. This architecture can handle different types of image degradation problems within the same framework, and achieve unified processing of multiple image enhancement tasks. Compared with traditional single-task image enhancement methods, it can simultaneously handle multiple image enhancement requirements, such as rain removal, fog removal, snow removal, denoising, deblurring and other tasks, thereby improving the versatility and effect of image enhancement. A symmetrical encoder-frequency regulator-decoder structure is designed, and through the processing of the frequency regulator and the attention mechanism, important features can be extracted more effectively and image enhancement results can be optimized, thereby enhancing the representation and generalization capabilities of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] in: Figure 1 is a schematic flow chart of a multi-task image enhancement method in one embodiment; Figure 2 is an overall framework diagram of a multi-task image enhancement method in one embodiment; Figure 3 An overall framework diagram of an encoder-frequency regulator-decoder in one embodiment; Figure 4 is a structural diagram of a multi-task image enhancement device in one embodiment; Figure 5 is a schematic diagram of the structure of a computer device in one embodiment; Figure 6 FIG. 4 is a schematic diagram of the structure of a computer-readable storage medium in one embodiment. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the embodiment of the present application, a multi-task image enhancement method is provided. Figure 1 , Figure 1 1 is a flow chart of a multi-task image enhancement method in an embodiment. The multi-task image enhancement method includes steps S1 to S5.

[0020] Step S1, using an end-to-end network architecture to build a multi-task image enhancement model, the network architecture includes a shallow feature extraction module and a deep feature extraction module; In some implementations, the shallow feature extraction module specifically includes: The three-channel input image is subjected to a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, and 1×1, and the output new features are residually connected with the input image.

[0021] In some embodiments, Figure 2 As shown, the input image size of the network is 256×256, the number of channels is three, and the shallow feature extraction module is used for the input image. , after a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, 1×1 and residual connections, the new features of the output are connected to the input image. This process can be expressed as:

[0022] in conv 3 represents a convolution operation with a convolution kernel size of 3×3. conv 1 indicates a convolution operation with a kernel size of 1×1. O It represents the features obtained after the shallow feature extraction stage, and then it is input into the first convolution layer in the deep feature module, that is, O is the subsequent input 3×3 convolution layer. Before the deep feature extraction module adopts the symmetrical encoder-frequency regulator-decoder structure, the number of channels is first expanded through the 3×3 convolution layer.

[0023] It should be noted that the advantage of the 3×3 convolution kernel lies in the local receptive field. The 3×3 convolution kernel can capture local features in each layer of the network, and compared with larger convolution kernels (such as 5×5 or 7×7), the 3×3 convolution kernel has smaller computational complexity and fewer parameters, which helps to reduce computational complexity and prevent overfitting.

[0024] The advantage of the 1×1 convolution kernel lies in dimensionality reduction and dimensionality increase. The 1×1 convolution kernel is mainly used for information integration between channels. It can change the depth (number of channels) of the feature map to achieve the purpose of dimensionality reduction or dimensionality increase.

[0025] The advantage of using a combination is that by stacking multiple 3×3 convolutional layers, compared to a single convolution layer with a larger convolution kernel.

[0026] Step S2, adopting a symmetrical encoder, frequency regulator and decoder structure for the deep feature extraction module, and designing the encoder, frequency regulator and decoder respectively; Specifically, the deep feature extraction module adopts a symmetrical encoder-frequency regulator-decoder structure.

[0027] In some implementations, the deep feature extraction module adopts a symmetrical encoder, frequency regulator, and decoder structure, and designs the encoder, frequency regulator, and decoder separately, specifically including: Use a 3x3 convolutional layer to downsample the input feature map, reduce the image size and increase the number of channels to extract features; The deep feature extraction module connects 8 feature extraction modules. Each deep feature extraction module consists of three parts. The first part is high-level feature acquisition, which is used to extract high-level features and perform downsampling operations to form an encoder. The second part is the frequency regulator unit, which consists of four identical frequency regulators. The third part is used to reconstruct the image and constitute a decoder through upsampling.

[0028] Specifically, the deep feature extraction module adopts a symmetrical encoder-frequency regulator-decoder structure. First, the number of channels is expanded through a 3×3 convolutional layer, and then 8 feature extraction modules are connected. The feature extraction module is divided into three parts. The first part is used to obtain high-level features and reduce the image size and expand the number of channels, that is, downsampling operations, which together constitute the encoder; the second part is composed of four identical frequency regulators, which can extract important features and constitute the encoder; the third part is used to reconstruct the image and perform upsampling, that is, expanding the image size and reducing the number of channels, to constitute the decoder.

[0029] In some implementations, in designing the encoder and the decoder, the design is performed through three branches, wherein the first branch includes: Perform 1×1 convolution on the input feature map to increase nonlinear transformation and reduce the number of parameters; Perform dimension conversion operations to make the feature map adapt to the input requirements of the multi-layer perceptron MLP; The feature map after the dimension conversion operation is sent to the multi-layer perceptron MLP for network layer processing, and then converted back to the original dimension; The feature map converted back to the original dimension is processed by the Sigmoid activation function.

[0030] In some implementations, in designing the encoder and the decoder, the design is performed through three branches, wherein the second branch includes: Perform 1×1 convolution on the input feature map and dynamically adjust the weight of the convolution kernel to adapt to the different feature distributions of the input data; Perform adaptive average pooling operation to determine feature representation; Input the feature representation into a 1×1 convolutional layer to generate an attention map; The attention weights are calculated by applying softmax, which is element-wise multiplied by a set of learnable parameters and summed to determine the features associated with the input.

[0031] In some implementations, in designing the encoder and the decoder, the design is performed through three branches, wherein the third branch includes: Perform deep convolution with kernel sizes of 1×1 and 3×3 on the input feature map to determine the enhanced spatial features; The enhanced spatial features are split into two tensors and sent to the dual-branch structure for processing. The dual-branch structure uses two different filters and filters the available features through the comprehensive receptive field. The upper branch in the dual-branch structure contains a 1×1 convolution layer, and the lower branch in the dual-branch structure contains a 1×1 convolution layer and a 3×3 deep convolution layer. The upper branch filters fine-grained noise factors through the receptive field of the 1×1 convolution, and the lower branch filters coarse-grained noise factors by expanding the receptive field. After the dual-branch structure is processed, the dual-branch structure is connected, and an activation function is applied to unify the feature differences to determine the output feature map.

[0032] In some embodiments, a frequency regulator is designed, comprising: Perform global average pooling, 1×1 convolution, and batch normalization on the features extracted by the encoder to obtain low-frequency filters; The low-frequency filter weights are normalized using the softmax function to obtain the normalized low-frequency filter weights; Segment the input features in the channel dimension, multiply the segmentation result with the low-frequency filter weight to determine the low-frequency features; Subtract the input features from the low-frequency features to determine the high-frequency features; The low-frequency features and high-frequency features are fused, and the fused features are globally averaged and pooled. The fused features are converted into high-dimensional features through the convolution layer. Decompose the high-dimensional features into high-frequency attention feature vectors and low-frequency attention feature vectors through convolution operations; Connect the high-frequency and low-frequency attention feature vectors and pass them through the softmax function to obtain the normalized attention weights; The attention weight is divided into two parts, which are multiplied with the corresponding high-frequency attention features and low-frequency attention features respectively to obtain the low-frequency map and high-frequency map; The low-frequency image and the high-frequency image are added and integrated through the convolution layer to obtain the final feature representation, which is then fed into the decoder.

[0033] Specifically, designing encoders and decoders includes: like Figure 3 As shown in the figure, the encoder and decoder are designed to consist of three branches. The first branch first performs a 1×1 convolution on the input feature map to increase the nonlinear transformation and reduce the number of parameters, and then performs a dimension conversion operation to adapt it to the input requirements of the multi-layer perceptron (MLP). The feature map after dimension conversion is sent to the multi-layer perceptron (MLP), processed by the network layer, and finally converted back to the original dimension. Finally, the converted and processed feature map is processed by the Sigmoid activation function. The second branch first performs the same processing as the first branch, and then dynamically adjusts the weight of the convolution kernel to adapt to the different feature distributions of the input data, thereby improving the representation and generalization capabilities of the model. In order to further remove the "noise factors" in the global features, the third branch uses the spatial attention module to accurately filter out the noise information, thereby improving the robustness and expressiveness of the model.

[0034] Furthermore, for the second branch of the encoder and decoder modules, this part first performs a Perform an adaptive average pooling operation, and then input the pooled feature representation into a 1×1 convolutional layer to generate an attention map Next, softmax is applied to compute the attention weights, which are element-wise multiplied by a set of learnable parameters and summed to obtain the features associated with the input. . This process can be expressed as:

[0035]

[0036] Where conv1 represents a convolution operation with a convolution kernel size of 1×1. represents the average pooling operation, represents the activation function, represents the learnable parameters, Represents the number of attention groups.

[0037] Furthermore, for the third branch of designing the encoder and decoder, enhanced spatial features are first obtained by using deep convolutions with kernel sizes of 1×1 and 3×3. The features are then split into two tensors and sent to a dual-branch structure, which uses two different filters to filter the available features through a comprehensive receptive field. The upper branch in the dual-branch structure contains only one 1×1 convolution layer, and the lower branch contains a 1×1 convolution layer and a 3×3 deep convolution layer. The upper branch filters fine-grained "noise factors" through the receptive field of 1×1 convolution, while the lower branch filters coarse-grained "noise factors" by expanding the receptive field. After the two branches are processed, they are connected together and an activation function is applied to unify the tiny feature differences to obtain. . This process can be expressed as:

[0038]

[0039]

[0040]

[0041] Among them, conv3 represents the convolution operation with a convolution kernel size of 3×3, and conv1 represents the convolution operation with a convolution kernel size of 3×3. represents a 3×3 depth convolution operation, ReLU represents the connection operation, ReLU represents the activation function, and b represents the bias parameter of the network.

[0042] Building a frequency regulator involves: like Figure 3 As shown in Figure 1, the module first performs a series of processing on the features extracted by the encoder: global average pooling, 1×1 convolution, and batch normalization to obtain low-frequency filters. The weights of these filters are normalized by the softmax function to obtain At the same time, the input features are split in the channel dimension, and the result is multiplied by the weight of the new feature map obtained by the filter to obtain the low-frequency feature Then, high-frequency features are obtained by subtracting low-frequency features from input features. Next, the frequency modifier selects the frequency components so that the model can focus on the important low-frequency and high-frequency information. This process first fuses the low-frequency features with the high-frequency features and then performs a global average pooling operation. Subsequently, the convolutional layer converts them into high-dimensional features. These high-dimensional features are further decomposed to generate high-frequency and low-frequency attention feature vectors through convolution operations. Then, the high-frequency and low-frequency attention feature vectors are connected and passed through the softmax function to obtain their normalized attention weights. Finally, these attention weights are divided into two parts and multiplied with the corresponding features to obtain low-frequency images and high-frequency images. These images are added and integrated through the convolution layer to generate the final feature representation , sent to the decoder. This process can be expressed as:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Among them, conv3 represents the convolution operation with a convolution kernel size of 3×3, and conv1 represents the convolution operation with a convolution kernel size of 3×3. Indicates a connection operation. represents the activation function, GAP represents the global pooling operation, represents batch normalization, R is the input feature, is the result of splitting the input features along the channel dimension.

[0049] Step S3, designing a loss function of a multi-task image enhancement model; Specifically, the loss function is calculated by summing the square of the difference and a small positive number, and then taking the square root. The difference is the difference between the original value in the image and the pixel value in the enhanced image. A small positive number is used to avoid division by zero errors. This process can be expressed as:

[0050] Among them, Y represents the pixel value of the enhanced image, and X represents the pixel value in the original image. represents a small positive number, preferably, The value range is [10 -8 ,10 -6 ].

[0051] Step S4, obtaining a data set, and training a multi-task image enhancement model based on the data set and the loss function; Step S5, performing image enhancement on the image to be enhanced using the trained multi-task image enhancement model; The technical solution of this embodiment is adopted, by designing an end-to-end deep network architecture, the network architecture includes a shallow feature extraction module and a deep feature extraction module using a symmetrical encoder, frequency regulator and decoder structure; and the loss function of the multi-task image enhancement model is designed to achieve unified processing of multiple image enhancement tasks, such as rain removal, fog removal, snow removal, denoising, deblurring, etc., thereby improving the effect and versatility of image enhancement. Through supervised training and learning, the method can learn the mapping relationship of multiple image enhancement tasks from a data set containing original images and corresponding enhanced images, and then realize efficient processing and optimization of different image enhancement tasks in practical applications. Through the frequency regulator and attention mechanism, important features are effectively extracted and the image enhancement results are optimized. The attention mechanism effectively filters noise information and improves the robustness of the model.

[0052] In the embodiment of the present application, a multi-task image enhancement device is provided. Figure 4 , Figure 4 2 is a structural diagram of a multi-task image enhancement device in an embodiment, wherein the multi-task image enhancement device includes a model building module 201, a structure design module 202, a loss function module 203, a training module 204, and an image enhancement module 205.

[0053] The model building module 201 is configured to build a multi-task image enhancement model using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A structure design module 202 is configured to adopt a symmetrical encoder, frequency regulator and decoder structure for the deep feature extraction module, and to design the encoder, frequency regulator and decoder respectively; A loss function module 203, configured to be used for designing a loss function of a multi-task image enhancement model; A training module 204 is configured to obtain a data set and train a multi-task image enhancement model based on the data set and a loss function; The image enhancement module 205 is configured to perform image enhancement on the image to be enhanced using a trained multi-task image enhancement model.

[0054] In some embodiments, the model building module 201 is further configured to perform a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, and 1×1 on the three-channel input image, and perform a residual connection between the output new features and the input image.

[0055] In some embodiments, the structure design module 202 is further configured to use a 3x3 convolutional layer to downsample the input feature map, reduce the image size and increase the number of channels, and extract features; The deep feature extraction module connects 8 feature extraction modules. Each deep feature extraction module consists of three parts. The first part is high-level feature acquisition, which is used to extract high-level features and perform downsampling operations to form an encoder. The second part is the frequency regulator unit, which consists of four identical frequency regulators. The third part is used to reconstruct the image and constitute a decoder through upsampling.

[0056] In some embodiments, the structure design module 202 is further configured to perform a 1×1 convolution on the input feature map to increase nonlinear transformation and reduce the number of parameters; Perform dimension conversion operations to make the feature map adapt to the input requirements of the multi-layer perceptron MLP; The feature map after the dimension conversion operation is sent to the multi-layer perceptron MLP for network layer processing, and then converted back to the original dimension; The feature map converted back to the original dimension is processed by the Sigmoid activation function.

[0057] In some embodiments, the structure design module 202 is further configured to perform 1×1 convolution on the input feature map and dynamically adjust the weight of the convolution kernel to adapt to different feature distributions of the input data; Perform adaptive average pooling operation to determine feature representation; Input the feature representation into a 1×1 convolutional layer to generate an attention map; The attention weights are calculated by applying softmax, which is element-wise multiplied by a set of learnable parameters and summed to determine the features associated with the input.

[0058] In some embodiments, the structure design module 202 is further configured to perform a depthwise convolution with kernel sizes of 1×1 and 3×3 on the input feature map to determine enhanced spatial features; The enhanced spatial features are split into two tensors and sent to the dual-branch structure for processing. The dual-branch structure uses two different filters and filters the available features through the comprehensive receptive field. The upper branch in the dual-branch structure contains a 1×1 convolution layer, and the lower branch in the dual-branch structure contains a 1×1 convolution layer and a 3×3 deep convolution layer. The upper branch filters fine-grained noise factors through the receptive field of the 1×1 convolution, and the lower branch filters coarse-grained noise factors by expanding the receptive field. After the dual-branch structure is processed, the dual-branch structure is connected, and an activation function is applied to unify the feature differences to determine the output feature map.

[0059] In some embodiments, the structure design module 202 is further configured to perform global average pooling, 1x1 convolution, and batch normalization on the features extracted by the encoder to obtain a low-frequency filter; The low-frequency filter weights are normalized using the softmax function to obtain the normalized low-frequency filter weights; Segment the input features in the channel dimension, multiply the segmentation result with the low-frequency filter weight to determine the low-frequency features; Subtract the input features from the low-frequency features to determine the high-frequency features; The low-frequency features and high-frequency features are fused, and the fused features are globally averaged and pooled. The fused features are converted into high-dimensional features through the convolution layer. Decompose the high-dimensional features into high-frequency attention feature vectors and low-frequency attention feature vectors through convolution operations; Connect the high-frequency and low-frequency attention feature vectors and pass them through the softmax function to obtain the normalized attention weights; The attention weight is divided into two parts, which are multiplied with the corresponding high-frequency attention features and low-frequency attention features respectively to obtain the low-frequency map and high-frequency map; The low-frequency image and the high-frequency image are added and integrated through the convolution layer to obtain the final feature representation, which is then fed into the decoder.

[0060] For other details about how each module in the multi-task image enhancement device implements the above technical solution, please refer to the description of the atrial fibrillation detection method under high-field strength magnetic resonance provided above, which will not be repeated here.

[0061] In an embodiment of the present application, a computer device is provided. Figure 5 , Figure 5 30 is a schematic diagram of the structure of a computer device in an embodiment. The device includes a memory 301 and a processor 302. The memory 301 stores a computer program. When the computer program is executed by the processor 302, the processor 302 executes the following steps: An end-to-end network architecture is used to build a multi-task image enhancement model. The network architecture includes a shallow feature extraction module and a deep feature extraction module. A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Design loss functions for multi-task image enhancement models; Obtain a dataset and train a multi-task image enhancement model based on the dataset and loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

[0062] Among them, the processor 302 can also be called a CPU (Central Processing Unit), and the processor 302 may be an integrated circuit chip with signal processing capabilities; the processor 302 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 302 can also be any conventional processor, etc.

[0063] In an embodiment of the present application, a computer readable storage medium is provided. Figure 6 , Figure 6 The structure diagram of a computer-readable storage medium in an embodiment is a schematic diagram, on which a readable computer program 401 is stored; wherein the computer program 401 may be stored in the above storage medium in the form of a software product, and includes a number of instructions for causing a computer device (which may be a personal computer, a server machine, or a network device, etc.) or a processor to perform the following steps: An end-to-end network architecture is used to build a multi-task image enhancement model. The network architecture includes a shallow feature extraction module and a deep feature extraction module. A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Design loss functions for multi-task image enhancement models; Obtain a dataset and train a multi-task image enhancement model based on the dataset and loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

[0064] The aforementioned storage media include: U disk, mobile hard disk, magnetic disk or CD, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program code, or terminal devices such as computers, service machines, mobile phones, and tablets.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0066] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-task image enhancement method, characterized in that: The method comprises: A multi-task image enhancement model is constructed using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A symmetrical encoder, frequency regulator and decoder structure is adopted for the deep feature extraction module, and the encoder, frequency regulator and decoder are designed respectively; Designing a loss function for the multi-task image enhancement model; Acquire a data set, and train the multi-task image enhancement model based on the data set and the loss function; Use the trained multi-task image enhancement model to perform image enhancement on the image to be enhanced.

2. The multi-task image enhancement method according to claim 1, characterized in that: The shallow feature extraction module specifically includes: The three-channel input image is subjected to a series of convolutions with kernel sizes of 3×3, 1×1, 3×3, and 1×1, and the output new features are residually connected to the input image.

3. The multi-task image enhancement method according to claim 2, characterized in that: The deep feature extraction module adopts a symmetrical encoder, frequency regulator and decoder structure, and designs the encoder, frequency regulator and decoder respectively, specifically including: Use a 3×3 convolutional layer to downsample the input feature map, reduce the image size and increase the number of channels to extract features; The deep feature extraction module is connected to 8 feature extraction modules, each of which includes three parts. The first part is high-level feature acquisition, which is used to extract high-level features and perform downsampling operations to constitute the encoder. The second part is a frequency regulator unit, which is composed of four identical frequency regulators. The third part is used to reconstruct the image and constitute a decoder through upsampling.

4. The multi-task image enhancement method according to claim 3, characterized in that: In designing the encoder and decoder, the design is carried out through three branches, wherein the first branch includes: Perform convolution with a kernel size of 1×1 on the input feature map to increase nonlinear transformation and reduce the number of parameters; Performing a dimension conversion operation on the feature map after the convolution process so that the feature map can adapt to the input requirements of the multi-layer perceptron MLP; The feature map after the dimension conversion operation is sent to the multi-layer perceptron MLP for network layer processing, and then converted back to the original dimension; The feature map converted back to the original dimension is processed by the Sigmoid activation function.

5. The multi-task image enhancement method according to claim 4, characterized in that: In the design of the encoder and the decoder, the design is performed through three branches, wherein the second branch includes: Performing a convolution with a kernel size of 1×1 on the input feature map, and dynamically adjusting the weight of the convolution kernel to adapt to different feature distributions of the input data; Perform adaptive average pooling operation to determine feature representation; Input the feature representation into a 1×1 convolutional layer to generate an attention map; Attention weights are calculated by applying softmax, and the attention weights are element-wise multiplied by a set of learnable parameters and summed to determine the features associated with the input.

6. The multi-task image enhancement method according to claim 5, characterized in that: In the design of the encoder and the decoder, the design is performed through three branches, wherein the third branch includes: Performing depth convolution with kernel sizes of 1×1 and 3×3 on the input feature map to determine enhanced spatial features; Splitting the enhanced spatial features into two tensors and sending them to a dual-branch structure for processing, wherein the dual-branch structure uses two different filters and filters available features through a comprehensive receptive field, wherein an upper branch in the dual-branch structure includes a 1×1 convolution layer, and a lower branch in the dual-branch structure includes a 1×1 convolution layer and a 3×3 deep convolution layer, wherein the upper branch filters fine-grained noise factors through the receptive field of the 1×1 convolution layer, and the lower branch filters coarse-grained noise factors by expanding the receptive field; After the dual-branch structure is processed, the dual-branch structure is connected, and an activation function is applied to unify feature differences to determine an output feature map.

7. The multi-task image enhancement method according to claim 6, characterized in that: Designing the frequency regulator includes: Performing global average pooling, 1×1 convolution, and batch normalization on the features extracted by the encoder to obtain a low-frequency filter; The low-frequency filter weights are normalized using the softmax function to obtain the normalized low-frequency filter weights; Segmenting the input features in the channel dimension, and multiplying the segmentation result by the low-frequency filter weight to determine the low-frequency features; Subtracting the input feature from the low-frequency feature to determine a high-frequency feature; The low-frequency features and the high-frequency features are fused, and a global average pooling operation is performed on the fused features, and the fused features are converted into high-dimensional features through a convolution layer; Decompose the high-dimensional features into high-frequency attention feature vectors and low-frequency attention feature vectors through convolution operations; Connecting the high-frequency attention feature vector and the low-frequency attention feature vector, and obtaining a normalized attention weight through the softmax function; The normalized attention weight is divided into two parts, and multiplied with the corresponding high-frequency attention feature and low-frequency attention feature respectively to obtain a low-frequency map and a high-frequency map; The low-frequency image and the high-frequency image are added and integrated through a convolutional layer to obtain a final feature representation, and the final feature representation is sent to the decoder.

8. A multi-task image enhancement device, characterized in that: The device comprises: A model building module, used to build a multi-task image enhancement model using an end-to-end network architecture, wherein the network architecture includes a shallow feature extraction module and a deep feature extraction module; A structure design module, used for adopting a symmetrical encoder, frequency regulator and decoder structure for the deep feature extraction module, and designing the encoder, frequency regulator and decoder respectively; A loss function module, used to design a loss function of the multi-task image enhancement model; A training module, used for acquiring a data set, and training the multi-task image enhancement model based on the data set and the loss function; The image enhancement module is used to perform image enhancement on the image to be enhanced using a trained multi-task image enhancement model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images