Image enhancement method

Abstract

The invention provides an image enhancement method, and relates to the technical field of image processing. The method comprises the steps of obtaining a feature map of a to-be-processed road scene image; a multi-frequency-domain feature coding module in the image enhancement network is adopted to carry out feature coding on the feature map, and multi-frequency-domain fusion features of the road scene image to be processed are obtained; a region modeling module in the image enhancement network is adopted to perform region modeling based on the multi-frequency-domain fusion features, and region fusion features of a to-be-enhanced region and other regions in the to-be-processed road scene image are obtained; a feature extraction module in the image enhancement network is adopted to perform feature extraction on the region fusion feature to obtain a target fusion feature of the to-be-processed road scene image; the target fusion features comprise local detail texture features and global context features; and decoding the target fusion feature by adopting a reconstruction decoding module in the image enhancement network to obtain an enhanced target road scene image. Therefore, the image processing precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image enhancement method. Background Art

[0002] In the video surveillance detection scene of highways, it is a common technical method to use cameras to capture images for vehicle and license plate detection, identification and tracking. The quality of the image is crucial to the detection effect. In real daily scenes, we often encounter severe weather such as fog, rain and snow, which have a significant impact on the imaging effect. When the atmosphere contains fog or raindrops, the incident light will be scattered, resulting in a decrease in the quality of the captured image, and the predetermined target cannot be captured, which will bring obstacles to the detection and identification of vehicles and license plates. Therefore, it is very important and practical to enhance the images taken in rainy and foggy weather to remove the rain and fog effects.

[0003] Existing defogging and deraining methods mainly include traditional image processing methods and deep learning-based image enhancement methods. Traditional defogging methods include methods based on dark channel prior algorithms and haze density estimation, which use the dark channel prior knowledge of the image and the atmospheric scattering model to separate the image color features and pixel density to remove haze. Traditional deraining algorithms use modeling based on the features of rain streaks to achieve deraining. Defogging algorithms based on deep learning use convolutional neural networks to automatically estimate the atmospheric illumination and transmittance in the image to remove the haze effect in the picture. Generative adversarial networks are also used to make the model learn the image features when there is no haze and when there is haze, to model and finally achieve the defogging effect. Deraining algorithms based on deep learning use time series methods to analyze the image features of rain through a video frame number of images to achieve the deraining effect. In actual scenes, fog and rain are often produced together, and the rain and fog effects will be more significant on highways due to the speeding of cars. A single method cannot achieve the simultaneous removal of rain and fog effects, resulting in poor image processing accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide an image enhancement method to solve the problems of poor image processing accuracy in the prior art.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides an image enhancement method, which is applied to an image enhancement network, and the method includes:

[0007] Obtaining a feature map of a road scene image to be processed;

[0008] Use the multi-frequency domain feature encoding module in the image enhancement network to perform feature encoding on the feature map to obtain the multi-frequency domain fusion features of the road scene image to be processed;

[0009] Use the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion features to obtain the region fusion features of the region to be enhanced and other regions in the road scene image to be processed;

[0010] Use the feature extraction module in the image enhancement network to perform feature extraction on the region fusion features to obtain the target fusion features of the road scene image to be processed; the target fusion features include: local detail texture features and global context features;

[0011] Use the reconstruction decoding module in the image enhancement network to decode the target fusion features to obtain the enhanced target road scene image.

[0012] Optionally, the obtaining of the feature map of the road scene image to be processed includes:

[0013] Perform image chunking on the road scene image to be processed to obtain multiple image patches;

[0014] Perform embedding processing on the multiple image patches to obtain the preliminary embedding features of the multiple image patches;

[0015] Perform linear mapping on the preliminary embedding features of the multiple image patches to obtain the target embedding features of the multiple image patches;

[0016] Obtain the feature map of the road scene image to be processed according to the target embedding features of the multiple image patches.

[0017] Optionally, the multi-frequency domain feature encoding module includes: a frequency domain transformation guidance module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module; the using of the multi-frequency domain feature encoding module in the image enhancement network to perform feature encoding on the feature map to obtain the multi-frequency domain fusion features of the road scene image to be processed includes:

[0018] Use the frequency domain transformation guidance module to perform frequency domain separation on the feature map to obtain the low-frequency features and high-frequency features of the road scene image to be processed;

[0019] Use the low-frequency encoding module to encode the low-frequency features to obtain low-frequency encoded features;

[0020] Use the high-frequency encoding module to encode the high-frequency features to obtain high-frequency encoded features;

[0021] The first fusion module is used to fuse the low-frequency encoded features and the high-frequency encoded features to obtain the multi-frequency domain fusion features.

[0022] Optionally, the low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer. Using the low-frequency encoding module to encode the low-frequency features to obtain low-frequency encoded features includes:

[0023] The average pooling layer is used to perform average pooling on the low-frequency features to obtain average pooling features;

[0024] The global self-attention layer is used to perform global self-attention calculation on the average pooling features to obtain global self-attention features;

[0025] The upsampling layer is used to perform upsampling on the global self-attention features to obtain the low-frequency encoded features.

[0026] Optionally, the high-frequency encoding module includes: a window partitioning layer, a local window attention layer, and a connection layer; using the high-frequency encoding module to encode the high-frequency features to obtain high-frequency encoded features includes:

[0027] The window partitioning layer is used to partition the high-frequency features to obtain a plurality of non-overlapping window features;

[0028] The local window attention layer is used to perform attention calculation on the plurality of non-overlapping window features respectively to obtain a plurality of window attention features;

[0029] The connection layer is used to splice the plurality of window attention features to obtain the high-frequency encoded features.

[0030] Optionally, the region modeling module includes: a global partitioning module and a second fusion module; using the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion features to obtain the region fusion features of the region to be enhanced and other regions in the road scene image to be processed includes:

[0031] The global partitioning module is used to perform partitioning calculation on the multi-frequency domain fusion features to obtain the feature weights of the region to be enhanced;

[0032] The second fusion module is used to fuse the features of the region to be enhanced and the features of other regions according to the feature weights of the region to be enhanced to obtain the region fusion features.

[0033] Optionally, the global partitioning module includes: a global distribution attention layer, a multi-head attention layer, and a first partitioning layer; the step of using the global partitioning module to perform partitioning calculation on the multi-frequency domain fusion features to obtain the feature weights of the region to be enhanced includes:

[0034] Using the global distribution attention layer to perform global attention calculation on the multi-frequency domain fusion features to obtain the density distribution probability of atmospheric particles in the road scene to be processed;

[0035] Using the multi-head attention layer to perform multi-head modeling on the multi-frequency domain fusion features respectively to obtain the multi-head features of the road scene to be processed;

[0036] Using the first partitioning layer to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed to obtain the feature weights of the region to be enhanced.

[0037] Optionally, the step of using the first partitioning layer to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed to obtain the feature weights of the region to be enhanced includes:

[0038] Obtaining a comprehensive feature according to the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed;

[0039] Performing non-linear fitting according to the comprehensive feature to obtain the feature weights of the region to be enhanced.

[0040] Optionally, the feature extraction module includes: a local extraction branch, a global extraction branch, and a third fusion module; the step of using the feature extraction module in the image enhancement network to perform feature extraction on the region fusion features to obtain the target fusion features of the road scene image to be processed includes:

[0041] Using the local extraction branch to perform local feature extraction on the region fusion features to obtain the local detail texture features;

[0042] Using the global extraction branch to perform global feature extraction on the region fusion features to obtain the global context features;

[0043] Using the third fusion module to fuse the local detail texture features and the global context features to obtain the target fusion features.

[0044] Optionally, the local extraction branch is a local convolutional branch; the step of using the local extraction branch to perform local feature extraction on the region fusion features to obtain the local detail texture features includes:

[0045] Perform a convolution operation on the region fusion feature using the local convolution branch to obtain the local detailed texture feature;

[0046] The global extraction branch is a global transmission network branch; performing global feature extraction on the region fusion feature using the global extraction branch to obtain the global context feature, including:

[0047] Perform a frequency domain conversion on the region fusion feature using the global transmission network branch, and perform an attention calculation on the converted frequency domain feature to obtain the global context feature.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] The present application provides an image enhancement method, which obtains a feature map of an image of a road scene to be processed; uses a multi-frequency domain feature encoding module in an image enhancement network to perform feature encoding on the feature map to obtain a multi-frequency domain fusion feature of the image of the road scene to be processed; uses a region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion feature to obtain a region fusion feature of the region to be enhanced and other regions in the image of the road scene to be processed; uses a feature extraction module in the image enhancement network to perform feature extraction on the region fusion feature to obtain a target fusion feature of the image of the road scene to be processed; the target fusion feature includes: a local detailed texture feature and a global context feature; uses a reconstruction decoding module in the image enhancement network to decode the target fusion feature to obtain an enhanced target road scene image. Thus, the image enhancement network can dynamically process different types of weather images, enhance key regions, while retaining details of other regions, improving the accuracy of image processing. Description of the Drawings

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 It is a schematic structural diagram of an image enhancement network provided by an embodiment of the present application;

[0052] Figure 2 It is a schematic structural diagram of a multi-frequency domain feature encoding module provided by an embodiment of the present application;

[0053] Figure 3 It is a schematic structural diagram of a region modeling module provided by an embodiment of the present application;

[0054] Figure 4 Structural schematic diagram of a feature extraction module provided by an embodiment of the present application;

[0055] Figure 5 Flow schematic diagram of an image enhancement method provided by the present application;

[0056] Figure 6 Flow schematic diagram of a method for obtaining a feature map of an image of a road scene to be processed provided by an embodiment of the present application;

[0057] Figure 7 Flow schematic diagram of a method for obtaining multi-frequency domain fusion features of an image of a road scene to be processed provided by an embodiment of the present application;

[0058] Figure 8 Flow schematic diagram of a method for obtaining low-frequency encoded features provided by an embodiment of the present application;

[0059] Figure 9 Flow schematic diagram of a method for obtaining high-frequency encoded features provided by an embodiment of the present application;

[0060] Figure 10 Flow schematic diagram of a method for obtaining regional fusion features of an image of a road scene to be processed provided by an embodiment of the present application;

[0061] Figure 11 Flow schematic diagram of a method for obtaining features of a region to be enhanced and features of other regions provided by an embodiment of the present application;

[0062] Figure 12 Flow schematic diagram of another method for obtaining features of a region to be enhanced and features of other regions provided by an embodiment of the present application;

[0063] Figure 13 Flow schematic diagram of a method for obtaining target fusion features of an image of a road scene to be processed provided by an embodiment of the present application;

[0064] Figure 14 Flow schematic diagram of a method for feature extraction provided by an embodiment of the present application;

[0065] Figure 15 Structural schematic diagram of a modified CNN branch provided by an embodiment of the present application;

[0066] Figure 16 Schematic diagram of an image enhancement device provided by an embodiment of the present application;

[0067] Figure 17 Schematic diagram of an electronic device provided by an embodiment of the present application.

[0068] Icons: 1601 - Acquisition Module, 1602 - Encoding Module, 1603 - Modeling Module, 1604 - Extraction Module, 1605 - Decoding Module, 1701 - Processor, 1702 - Storage Medium. Detailed Implementation Manner

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. Generally, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0071] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0072] In addition, terms such as "first" and "second" are used only for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0073] It should be noted that, without conflict, the features in the embodiments of the present invention can be combined with each other.

[0074] Before describing an image enhancement method provided in an embodiment of the present application, an image enhancement network applied to the image enhancement method provided in the embodiment of the present application will be described first. Figure 1 This is a schematic structural diagram of an image enhancement network provided in an embodiment of the present application. As Figure 1 shown, the image enhancement network includes: a multi-frequency domain feature encoding module, a region modeling module, a feature extraction module, and a reconstruction decoding module.

[0075] The entire image enhancement network adopts an end-to-end joint optimization strategy and can dynamically process different types of pictures through a global attention module and an adaptive feature mixing layer. Exemplarily, before executing this method, the image enhancement network is trained until a preset training cut-off condition is met.

[0076] Figure 2 This is a schematic structural diagram of a multi-frequency domain feature encoding module provided in an embodiment of the present application. As Figure 2As shown in the figure, the multi-frequency domain feature encoding module includes: a frequency domain transformation guiding module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module. The low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer. The high-frequency encoding module includes: a window partitioning layer, a local window attention layer, and a connection layer.

[0077] Figure 3 This is a schematic structural diagram of a region modeling module provided by an embodiment of the present application. As Figure 3 shown, the region modeling module includes: a global partitioning module and a second fusion module. The global partitioning module includes: a global distribution attention layer and a first partitioning layer, or the global partitioning module includes: a multi-head attention layer and a second partitioning layer.

[0078] Figure 4 This is a schematic structural diagram of a feature extraction module provided by an embodiment of the present application. As Figure 4 shown, the feature extraction module includes: a local extraction branch, a global extraction branch, and a third fusion module.

[0079] The following explains an image enhancement method provided by the present application through specific examples. Figure 5 This is a schematic flowchart of an image enhancement method provided by the present application, which is applied to an image enhancement network. The execution subject of this method can be an electronic device, and this electronic device can be a device with computing and processing functions, such as a desktop computer, a laptop computer, etc. As Figure 5 shown, this method includes:

[0080] S101. Obtain the feature map of the road scene image to be processed.

[0081] Exemplarily, the road scene image to be processed is a road rain, snow, haze image.

[0082] S102. Use the multi-frequency domain feature encoding module in the image enhancement network to perform feature encoding on the feature map to obtain the multi-frequency domain fusion feature of the road scene image to be processed.

[0083] Traditional image processing methods usually regard an image as a homogeneous information space, while the present application believes that an image is a complex information system composed of multiple frequency levels. Therefore, an intelligent multi-frequency domain feature encoding module is designed to process different spectra of the image.

[0084] S103. Use the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion feature to obtain the region fusion feature of the region to be enhanced and other regions in the road scene image to be processed.

[0085] Traditional image dehazing algorithms are often limited to local areas and difficult to capture the atmospheric particle distribution of the entire image. However, the global attention distribution mechanism proposed in this application breaks through this limitation. The key innovation lies in highlighting the area to be enhanced in the road scene image to be processed. For example, in highway scenes, haze may show non-uniform distribution. The method of this application can automatically identify and enhance the image area most severely affected by haze while retaining the detail information of other areas.

[0086] S104. Use the feature extraction module in the image enhancement network to extract features from the region fusion features to obtain the target fusion features of the road scene image to be processed.

[0087] Among them, the target fusion features include local detail texture features and global context features.

[0088] By separately extracting local detail texture and global context, important information such as local texture and global structure can be focused on in complex weather images.

[0089] S105. Use the reconstruction decoding module in the image enhancement network to decode the target fusion features to obtain the enhanced target road scene image.

[0090] The reconstruction decoding module maps the target fusion features extracted after multi-stage processing back to a high-quality image. It restores the comprehensive information in the high-dimensional feature space to the pixel space and outputs a clear RGB image. It smoothly combines global and local features to ensure the retention of detail texture and the overall structure of the image. By gradually upsampling, artifacts or blurring problems are avoided to ensure that the output image is natural and realistic.

[0091] Exemplarily, during training, the reconstruction decoding module is jointly trained with the entire network to directly minimize the reconstruction loss between the input image and the road scene image to be processed.

[0092] Thus, through the image enhancement network, different types of weather images can be dynamically processed, key areas can be enhanced, while the details of other areas are retained, improving the accuracy of image processing.

[0093] In summary, in this embodiment, a feature map of the road scene image to be processed is obtained; the multi-frequency domain feature encoding module in the image enhancement network is used to perform feature encoding on the feature map to obtain the multi-frequency domain fusion features of the road scene image to be processed; the region modeling module in the image enhancement network is used to perform region modeling based on the multi-frequency domain fusion features to obtain the region fusion features of the region to be enhanced and other regions in the road scene image to be processed; the feature extraction module in the image enhancement network is used to perform feature extraction on the region fusion features to obtain the target fusion features of the road scene image to be processed; the target fusion features include: local detail texture features and global context features; the reconstruction decoding module in the image enhancement network is used to decode the target fusion features to obtain the enhanced target road scene image. Thus, the image enhancement network can dynamically process different types of weather images, enhance key regions, while retaining the details of other regions, improving the accuracy of image processing.

[0094] Based on the above Figure 5 corresponding embodiment, the embodiment of the present application further provides a method for obtaining a feature map of a road scene image to be processed. Figure 6 The following is a schematic flowchart of a method for obtaining a feature map of a road scene image to be processed provided by the embodiment of the present application. As Figure 6 shown, obtaining the feature map of the road scene image to be processed in S101 includes:

[0095] S201. Perform image chunking on the road scene image to be processed to obtain a plurality of image patches.

[0096] The road scene image to be processed is divided into N patches, each patch having a size of p×p. For example, the specific chunking method is: PatchEmbed(R) = {P1, P2, …, P N}.

[0097] S202. Perform embedding processing on the plurality of image patches to obtain preliminary embedding features of the plurality of image patches.

[0098] Exemplarily, perform embedding processing on the plurality of image patches to obtain preliminary embedding features of the plurality of image patches.

[0099] S203. Perform linear mapping on the preliminary embedding features of the plurality of image patches to obtain target embedding features of the plurality of image patches.

[0100] Exemplarily, perform linear mapping on the preliminary embedding features of each image patch. The specific linear mapping formula is shown as formula (1) below:

[0101] Z i = Flatten(P i )·We +b e (1)

[0102] Among them, Z i is the embedded feature of each image patch.

[0103] S204. Obtain the feature map of the road scene image to be processed according to the target embedded features of multiple image patches.

[0104] Through the feature aggregation mechanism, reconstruct the target embedded features of multiple image patches into a complete feature map to obtain the feature map of the road scene image to be processed.

[0105] In summary, in this embodiment, the road scene image to be processed is segmented into multiple image patches; the multiple image patches are embedded to obtain the preliminary embedded features of the multiple image patches; the preliminary embedded features of the multiple image patches are linearly mapped to obtain the target embedded features of the multiple image patches; according to the target embedded features of the multiple image patches, the feature map of the road scene image to be processed is obtained. Thus, the feature map of the road scene image to be processed is accurately obtained.

[0106] In the above Figure 5 Based on the corresponding embodiment, the embodiment of the present application further provides a method for obtaining the multi-frequency domain fusion features of the road scene image to be processed. Figure 7 It is a schematic flowchart of a method for obtaining the multi-frequency domain fusion features of the road scene image to be processed provided by the embodiment of the present application. As Figure 7 shown, the multi-frequency domain feature encoding module includes: a frequency domain transformation guidance module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module; using the multi-frequency domain feature encoding module in the image enhancement network in S102 to perform feature encoding on the feature map to obtain the multi-frequency domain fusion features of the road scene image to be processed, including:

[0107] S301. Use the frequency domain transformation guidance module to perform frequency domain separation on the feature map to obtain the low-frequency features and high-frequency features of the road scene image to be processed.

[0108] Exemplarily, set a preset high-frequency band and a preset low-frequency band. Perform frequency domain separation on the feature maps of the preset high-frequency band and the preset low-frequency band, and use the Fourier transform to extract the frequency domain information to obtain the low-frequency features and high-frequency features of the road scene image to be processed. The specific formula of the Fourier transform is shown as formula (2) below:

[0109]

[0110] Among them, F(u, v) is the frequency domain feature.

[0111] S302. Use the low-frequency encoding module to encode the low-frequency features to obtain low-frequency encoded features.

[0112] Exemplarily, the low-frequency encoded features reflect the overall structure and macroscopic features of the image.

[0113] S303. Use the high-frequency encoding module to encode the high-frequency features to obtain high-frequency encoded features.

[0114] Exemplarily, the high-frequency encoded features capture the detailed texture and edge information.

[0115] For example, in the complex weather image of a highway, the low-frequency encoded features include the road and vehicle contours, while the high-frequency encoded features include the fine textures of raindrops and fog particles.

[0116] S304. Use the first fusion module to fuse the low-frequency encoded features and the high-frequency encoded features to obtain multi-frequency domain fusion features.

[0117] The fused multi-frequency domain fusion features contain both low-frequency encoded features and high-frequency encoded features, highlighting the overall structure and detailed texture.

[0118] Exemplarily, the feature fusion formula is as shown in formula (3) below:

[0119] F M (u, v) = LWAF H (u, v) + (1 - LWA)F L (u, v) (3)

[0120] Where, F M (u, v) is the multi-frequency domain fusion feature, LWA is the attention of the high-frequency encoded features, F H (u, v) is the high-frequency encoded feature, F L (u, v) is the low-frequency encoded feature.

[0121] In summary, in this embodiment, the multi-frequency domain feature encoding module includes: a frequency domain transformation guiding module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module; using the frequency domain transformation guiding module to perform frequency domain separation on the feature map to obtain the low-frequency features and high-frequency features of the road scene image to be processed; using the low-frequency encoding module to encode the low-frequency features to obtain low-frequency encoded features; using the high-frequency encoding module to encode the high-frequency features to obtain high-frequency encoded features; using the first fusion module to fuse the low-frequency encoded features and the high-frequency encoded features to obtain multi-frequency domain fusion features. Thus, through the processing of the low-frequency encoding module and the high-frequency encoding module, the overall structure and detailed texture are highlighted.

[0122] In the above Figure 7Based on the corresponding embodiments, the embodiments of the present application further provide a method for obtaining low-frequency encoded features. Figure 8 It is a schematic flowchart of a method for obtaining low-frequency encoded features provided by the embodiments of the present application. As Figure 8 shown, the low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer. Encoding the low-frequency features using the low-frequency encoding module in S302 to obtain low-frequency encoded features includes:

[0123] S401. Perform average pooling processing on the low-frequency features using the average pooling layer to obtain average pooling features.

[0124] Exemplarily, in order to avoid a large computational overhead at a lower layer due to a relatively large resolution of the feature map, an average pooling layer is used before the global self-attention layer operation to reduce the spatial scale.

[0125] S402. Perform global self-attention calculation on the average pooling features using the global self-attention layer to obtain global self-attention features.

[0126] S403. Perform upsampling processing on the global self-attention features using the upsampling layer to obtain low-frequency encoded features.

[0127] Exemplarily, an upsampling layer is used after the global self-attention layer operation to restore the original spatial dimension. This design greatly reduces the computational overhead and focuses the attention operation on embedding global information.

[0128] The processing formula of the low-frequency encoding module is specifically shown in the following formula (4):

[0129] Y l =Upsample(Attention(AvePooling(Z i ))) (4)

[0130] In summary, in this embodiment, the low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer. Perform average pooling processing on the low-frequency features using the average pooling layer to obtain average pooling features; perform global self-attention calculation on the average pooling features using the global self-attention layer to obtain global self-attention features; perform upsampling processing on the global self-attention features using the upsampling layer to obtain low-frequency encoded features. Thus, the low-frequency encoded features are accurately obtained.

[0131] On the basis of the above Figure 7 corresponding embodiments, the embodiments of the present application further provide a method for obtaining high-frequency encoded features. Figure 9 It is a schematic flowchart of a method for obtaining high-frequency encoded features provided by the embodiments of the present application. As Figure 9As shown, the high-frequency encoding module includes: a window partitioning layer, a local window attention layer, and a connection layer; in S303, the high-frequency encoding module is used to encode high-frequency features to obtain high-frequency encoded features, including:

[0132] S501. Use the window partitioning layer to partition the high-frequency features to obtain multiple non-overlapping window features.

[0133] Exemplarily, the high-frequency features are evenly partitioned into multiple non-overlapping window features.

[0134] S502. Use the local window attention layer to perform attention calculation on multiple non-overlapping window features respectively to obtain multiple window attention features.

[0135] S503. Use the connection layer to splice multiple window attention features to obtain high-frequency encoded features.

[0136] The specific attention calculation and splicing formulas for high-frequency encoded features are shown in formula (5) below:

[0137]

[0138] where N w is the number of windows, Q i , K i , V i are the query, key, and value of the i-th window respectively, and Concat represents splicing the calculation results of all windows.

[0139] In summary, in this embodiment, the high-frequency encoding module includes: a window partitioning layer, a local window attention layer, and a connection layer; use the window partitioning layer to partition the high-frequency features to obtain multiple non-overlapping window features; use the local window attention layer to perform attention calculation on multiple non-overlapping window features respectively to obtain multiple window attention features; use the connection layer to splice multiple window attention features to obtain high-frequency encoded features. Thus, high-frequency encoded features are accurately obtained.

[0140] Based on the above Figure 5 corresponding embodiment, the embodiment of the present application also provides a method for obtaining the region fusion features of the road scene image to be processed. Figure 10 It is a schematic flowchart of a method for obtaining the region fusion features of the road scene image to be processed provided by the embodiment of the present application. As Figure 10 shown, the region modeling module includes: a global partitioning module and a second fusion module; in S103, use the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion features to obtain the region fusion features of the region to be enhanced and other regions in the road scene image to be processed, including:

[0141] S601. Use the global partitioning module to perform partitioning calculations on the multi-frequency domain fusion features to obtain the feature weights of the area to be enhanced.

[0142] To highlight the area to be enhanced, the image is divided into the area to be enhanced and other areas. Correspondingly, the features of the area to be enhanced and the features of other areas are obtained.

[0143] Use the adaptive weight to dynamically adjust the weight of the features of the area to be enhanced.

[0144] S602. Use the second fusion module to fuse the features of the area to be enhanced and the features of other areas according to the feature weights of the area to be enhanced to obtain the region fusion features.

[0145] Exemplarily, the fusion formula is as shown in formula (6) below:

[0146] F enhanced (x, y) = W(x, y) · F key (x, y) + (1 - W(x, y)) · F base (x, y) (6)

[0147] Where, F enhanced (x, y) is the region fusion feature, F key (x, y) is the feature of the area to be enhanced. F base (x, y) is the feature of other areas.

[0148] In summary, in this embodiment, the region modeling module includes: a global partitioning module and a second fusion module; use the global partitioning module to perform partitioning calculations on the multi-frequency domain fusion features to obtain the feature weights of the area to be enhanced; use the second fusion module to fuse the features of the area to be enhanced and the features of other areas according to the feature weights of the area to be enhanced to obtain the region fusion features. Thus, the area to be enhanced is highlighted.

[0149] Based on the above Figure 10 corresponding embodiment, the embodiment of the present application also provides a method for obtaining the feature weights of the area to be enhanced. Figure 11 It is a schematic flowchart of a method for obtaining the feature weights of the area to be enhanced provided by the embodiment of the present application. As Figure 11 shown, the global partitioning module includes: a global distribution attention layer, a multi-head attention layer, and a first partitioning layer; in S601, using the global partitioning module to perform partitioning calculations on the multi-frequency domain fusion features to obtain the feature weights of the area to be enhanced includes:

[0150] S701. Use the global distribution attention layer to perform global attention calculations on the multi-frequency domain fusion features to obtain the density distribution probability of atmospheric particles in the road scene to be processed.

[0151] The weight matrix is calculated through the global attention distribution to provide the global relationship between pixels. After extracting features through the global attention mechanism, the probability density map of atmospheric particles is estimated. The formula for the probability density map of atmospheric particles is shown in the following formula (7):

[0152]

[0153] Among them, P(x, y) is the probability density of atmospheric particles, μ and σ are the mean and variance of the features, used to capture the global distribution characteristics of atmospheric particles, and Z is the normalization constant.

[0154] The probability density map of atmospheric particles can characterize which regions are more affected by haze, rain, snow, and which regions are less affected.

[0155] S702. Use the multi-head attention layer to perform multi-head modeling on the multi-frequency domain fusion features respectively to obtain the multi-head features of the road scene to be processed.

[0156] Exemplarily, use the multi-head attention layer to perform spatial modeling and spectral modeling on the multi-frequency domain fusion features respectively to obtain the spatial structure features and spectral information features of the road scene to be processed.

[0157] Exemplarily, use the multi-head attention mechanism (Multi-Head Attention, MHA) to dynamically learn the spatial-spectral correlation. Through the multi-head mechanism, the model can model the input features from different perspectives (for example, the spectral domain and the spatial domain).

[0158] The formula of the multi-head attention mechanism can be shown in the following formula (8):

[0159] MHA(Q, K, V) = Concat(Head1, Head2,..., Head h )W O (8)

[0160] Among them, each head of the multi-head attention independently learns the spatial or spectral correlation, and the calculation method of each head is shown in the following formula (9):

[0161]

[0162] Among them, Q i , K i , V i are the query, key, and value of the i-th head, is the dimension of each head.

[0163] S703. Use the first partition layer to perform partition calculations on the probability of the density distribution of atmospheric particles and the multi-head features in the road scene to be processed to obtain the feature weights of the area to be enhanced.

[0164] The density distribution probability of atmospheric particles can characterize the degree to which the area is affected by haze, rain, and snow. The multi-head features (spatial structure features and spectral information features) can also characterize the degree to which the area is affected by haze, rain, and snow. The feature weights of the area to be enhanced obtained from the density distribution probability of atmospheric particles and the multi-head features can also characterize the degree to which the area is affected by haze, rain, and snow.

[0165] In summary, in this embodiment, the global partitioning module includes: a global distribution attention layer and a first partitioning layer; the global distribution attention layer is used to perform global attention calculation on the multi-frequency domain fusion features to obtain the density distribution probability of atmospheric particles in the road scene to be processed; the multi-head attention layer is used to perform multi-head modeling on the multi-frequency domain fusion features respectively to obtain the multi-head features of the road scene to be processed; the first partitioning layer is used to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed to obtain the feature weights of the area to be enhanced. Thus, partitioning is achieved through the density distribution method.

[0166] Based on the above Figure 11 corresponding embodiment, the embodiment of the present application further provides a method for obtaining the feature weights of the area to be enhanced according to the density distribution probability and the multi-head features. Figure 12 It is a schematic flowchart of a method for obtaining the feature weights of the area to be enhanced according to the density distribution probability and the multi-head features provided by the embodiment of the present application. As Figure 12 shown, in S703, using the first partitioning layer to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed to obtain the feature weights of the area to be enhanced includes:

[0167] S801. Obtain comprehensive features according to the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed.

[0168] Perform global pooling calculation on the density distribution probability of atmospheric particles to obtain the probability density feature. The specific calculation method is shown in the following formula (10):

[0169] F prab = GAP(P9x,y)) (10)

[0170] where F prab is the probability density feature.

[0171] Exemplarily, taking the multi-head features as the spatial structure features and the spectral information features as an example, the fusion calculation method of the spatial structure features, the spectral information features, and the probability density feature is shown in the following formula (11):

[0172] F input = [F spatial , F spectral , F prab(11)

[0173] Among them, F input is the fused comprehensive feature, and F spatial is the spatial structure feature, and F spectral is the spectral information feature.

[0174] S802. Perform non-linear fitting based on the comprehensive feature to obtain the feature weight of the area to be enhanced.

[0175] The importance of the area is reflected by the fused comprehensive feature, and the GLP based on gated MLP is used for correlation modeling, so that the enhancement degree of different areas can be adaptively adjusted, that is, the feature weight of the area to be enhanced is adaptively adjusted.

[0176] Calculate the adaptive enhancement weight through GLU, and the specific calculation formula is shown in the following formula (12):

[0177] H = (F input W1 + b1) ⊙ σ(F input W2 + b2)

[0178] W(x, y) = Simgoid(W3H + b3) (12)

[0179] W1, W2, and W3 are learnable weight matrices, σ is the Sigmoid activation function, which serves as a gating mechanism, ⊙ represents element-wise multiplication, and b1, b2, and b3 are constants.

[0180] The GLP multi-layer perceptron is used to fit the non-linear relationship between the comprehensive feature and the weight, and the Sigmoid function is used to normalize the weight to [0, 1].

[0181] In summary, in this embodiment, the comprehensive feature is obtained according to the density distribution probability and multi-head features of atmospheric particles in the road scene to be processed; the non-linear fitting is performed according to the comprehensive feature to obtain the feature weight of the area to be enhanced. Thus, partitioning is achieved through the multi-head processing method.

[0182] Based on the above Figure 5 corresponding embodiment, the embodiment of the present application also provides a method for obtaining the target fusion feature of the road scene image to be processed. Figure 13 It is a schematic flowchart of a method for obtaining the target fusion feature of the road scene image to be processed provided by the embodiment of the present application. As Figure 13 shown, the feature extraction module includes: a local extraction branch, a global extraction branch, and a third fusion module; in S104, the feature extraction module in the image enhancement network is used to extract features from the region fusion feature to obtain the target fusion feature of the road scene image to be processed, including:

[0183] S901. Use the local extraction branch to perform local feature extraction on the region fusion feature to obtain the local detailed texture feature.

[0184] S902. Use the global extraction branch to perform global feature extraction on the region fusion feature to obtain the global context feature.

[0185] S903. Use the third fusion module to fuse the local detailed texture feature and the global context feature to obtain the target fusion feature.

[0186] The fused target fusion feature contains both local detailed texture information and global context relationships. Through this mechanism, the restoration quality and global consistency in the atmospheric removal task are improved.

[0187] Exemplarily, perform weighted calculation on the local detailed texture feature, the global context feature, the weight of the local detailed texture feature, and the weight of the global context feature to obtain the target fusion feature.

[0188] In summary, in this embodiment, the feature extraction module includes: a local extraction branch, a global extraction branch, and a third fusion module; use the local extraction branch to perform local feature extraction on the region fusion feature to obtain the local detailed texture feature; use the global extraction branch to perform global feature extraction on the region fusion feature to obtain the global context feature; use the third fusion module to fuse the local detailed texture feature and the global context feature to obtain the target fusion feature. Thus, the target fusion feature including local detailed texture and global context is obtained.

[0189] Based on the above Figure 13 corresponding embodiment, the embodiment of the present application further provides a method for feature extraction. Figure 14 It is a schematic flowchart of a method for feature extraction provided by the embodiment of the present application. As Figure 14 shown, the local extraction branch is a local convolutional branch, and the local convolutional branch uses a traditional convolutional neural network (CNN).

[0190] In S901, using the local extraction branch to perform local feature extraction on the region fusion feature to obtain the local detailed texture feature includes:

[0191] S1001. Use the local convolutional branch to perform a convolution operation on the region fusion feature to obtain the local detailed texture feature.

[0192] Local convolutional branch (Local CNN Branch): Use convolution operations to capture fine-grained texture and edge information, emphasizing the features of local regions in the spatial domain.

[0193] In order to facilitate the fusion of local convolution branches and global transmission network branches, the features of local convolution branches and global transmission network branches are downsampled and upsampled for alignment, the features of CNN are projected onto the dimension of transformer, and semantic alignment loss is performed on the projected features to achieve dual-branch feature fusion.

[0194] The local branch is based on the traditional convolutional neural network, which is suitable for capturing high-frequency details and texture features. Assume that the input feature is F in , the local branch is calculated as follows:

[0195]

[0196] Among them, F local is the convolution output feature, W(i,j) is the convolution kernel weight matrix, with a size of (2k+1)×(2k+1), and F in (x+i,y+j) is the local neighborhood pixel value.

[0197] The convolution operation can capture the fine-grained information of the local area, and by stacking multiple layers of convolution, richer local features can be extracted.

[0198] In order to make it easier for the CNN branch to learn how to extract high-quality semantic information from the Transformer branch and maintain the same output features as the Transformer, several convolutional blocks in the CNN branch are improved so that the convolution module simulates the structure of the transformer block as much as possible to better learn the semantic information of the transformer branch.

[0199] Figure 15 This is a schematic diagram of the structure of the modified CNN branch provided in the embodiment of the present application. Figure 15 As shown, the modified CNN branch includes: convolutional attention block, first additive normalization block, FFN feedforward network, and second additive normalization block.

[0200] After the input feature passes through the convolutional attention block, it is added to the input feature in the first addition normalization block and normalized. Then the feature passes through the FFN feedforward network and is added to itself and normalized in the second addition normalization block. The process can be expressed as: f = Norm(x + ConvAttentionx, y = Norm(f + FFN(f)).

[0201] Due to the semantic gap between Transformer and CNN, simply calculating the similarity between several learnable vectors and each pixel and then enhancing the pixels based on the similarity map and the learnable vectors is insufficient to capture rich context information. To better align the semantic information of Transformer, we expand the learnable vectors into learnable kernels. On the one hand, this converts the similarity calculation between pixels and learnable vectors into the similarity calculation between pixel blocks with learnable kernels. On the other hand, the convolutional operation with learnable kernels preserves more local spatial information to a certain extent. The operation of convolutional attention can be summarized as follows:

[0202]

[0203] where \(X\in\mathbb{R}\) C×H×W , \(K\in\mathbb{R}\) C×N×k×k represent the query and key values of the input feature map, \(C\), \(H\), \(W\) represent the channels, height, and width of the feature map, \(N\) is the number of learnable parameters, \(k\) represents the kernel size of the learnable parameters, and \(\theta\) represents grouped double normalization, that is, applying softmax in the dimension of \(H\times W\) and applying grouped L2 norm in the dimension of \(N\).

[0204] Considering the significant difference in the decoding space between CNN convolution and Transformer, directly aligning the decoded features with the output logits value can only achieve limited improvement. Therefore, the stage2 and stage4 features of the local branch CNN are cascaded and input into the point convolution for dimension expansion to achieve alignment. Then the high-dimensional features are passed to the Transformer decoder, and finally, the alignment loss with the original output of the Transformer decoder is calculated using the new output features and logits value of the Transformer decoder.

[0205] The global extraction branch is the global transmission network branch; the global transmission network branch uses the Transformer network.

[0206] In S902, the global extraction branch is used to globally extract the features of the region fusion features to obtain the global context features, including:

[0207] S1002. The global transmission network branch is used to perform frequency domain conversion on the region fusion features and calculate the attention on the converted frequency domain features to obtain the global context features.

[0208] The Global Transformer Branch: Based on frequency domain modeling of the global context of the image, long-term dependencies are extracted through the attention mechanism to enhance the understanding of the global distribution features.

[0209] The global transmission network branch models the global context using the Transformer framework, with the input feature F in being represented as First, the input feature is subjected to a frequency-domain transformation, denoted as the discrete Fourier transform. In the frequency domain, the global context is modeled through an attention mechanism.

[0210] The frequency-domain feature F freq is reshaped into a sequence: X = Reshape(F freq ), where N = H · W.

[0211] The attention is calculated as:

[0212]

[0213] where Q = XW Q , K = XW K , V = XW V , and W Q , W K , W V are the projection matrices for the query, key, and value.

[0214] The global feature is further processed through a multi-layer perceptron (MLP) to output the attention result: F global = MLP(Attention(Q, K, V)).

[0215] In the Transformer network of the global transmission network branch, we apply a dynamic learning mechanism of adaptive feature mixing to further enhance the expression ability of features at different levels. By dynamically learning the importance weights of features at each layer and combining the multi-head attention mechanism, non-linear feature fusion is achieved.

[0216] First, the features of each layer extracted from the Transformer network are serialized and dimension-reduced. The features of each layer are where H and W are the height and width of the feature map respectively, and C l is the number of channels. To unify the feature dimensions, we project each layer of features to the same dimension d through a dimension reduction operation: F' l = F l · W l , where

[0217] After projection, a set of features is obtained: l = 1, 2, …, L.

[0218] Next, each layer of features is flattened into a sequence form:

[0219] And splice all hierarchical features to form a sequence:

[0220] For the serialized feature X, extract the feature description of each layer through global pooling:

[0221] Splice the feature descriptions of all layers to form a description matrix:

[0222] Use the MLP weight generation network to generate importance weights for the features of each layer:

[0223]

[0224] where: MLP(Z l ) = σ2(W2·σ1(W1·Z l + v1)+ b2), W1, W2 and b1, b2 are the parameters of the MLP, and σ1, σ2 are activation functions. Use Softmax to ensure weight normalization.

[0225] In the dynamic fusion process, model the non-linear relationship between features through multi-head self-attention. The outputs of multiple heads are spliced and then linearly transformed: X attn = Concat(Head1, Head2, …)W O

[0226] where is the output projection matrix.

[0227] According to the learned weight w′, weight the original features: Finally, obtain the global context feature and the weight of the global context feature.

[0228] In summary, in this embodiment, the local extraction branch is the local convolution branch; the local convolution branch is used to perform a convolution operation on the region fusion feature to obtain the local detailed texture feature; the global transmission network branch is used to perform a frequency domain conversion on the region fusion feature and calculate the attention on the converted frequency domain feature to obtain the global context feature. Thus, the local detailed texture feature and the global context feature are accurately obtained.

[0229] Therefore, the self-attention mechanism is adopted to extract global context features and establish long-range dependencies to retain the overall image information. Combining the multi-frequency domain fusion features, the region attention mechanism is used to adaptively adjust the weights of key regions to highlight the regions to be enhanced while suppressing irrelevant backgrounds. Finally, through the adaptive masking mechanism based on multi-frequency domain features, the detail contrast is enhanced within the enhanced regions while maintaining the original features of the non-enhanced regions to achieve natural enhancement of the overall image.

[0230] The following describes an image enhancement device, equipment, storage medium, etc. provided by the present application for execution. For the specific implementation process and technical effects, please refer to the above, and will not be elaborated below.

[0231] Figure 16 The following is a schematic diagram of an image enhancement device provided by an embodiment of the present application. As Figure 16 shown, the device includes:

[0232] An acquisition module 1601, configured to acquire a feature map of a road scene image to be processed.

[0233] An encoding module 1602, configured to perform feature encoding on the feature map by using a multi-frequency domain feature encoding module in an image enhancement network to obtain multi-frequency domain fusion features of the road scene image to be processed.

[0234] A modeling module 1603, configured to perform region modeling based on the multi-frequency domain fusion features by using a region modeling module in an image enhancement network to obtain region fusion features of the regions to be enhanced and other regions in the road scene image to be processed.

[0235] An extraction module 1604, configured to perform feature extraction on the region fusion features by using a feature extraction module in an image enhancement network to obtain target fusion features of the road scene image to be processed; the target fusion features include: local detail texture features and global context features.

[0236] A decoding module 1605, configured to perform decoding on the target fusion features by using a reconstruction decoding module in an image enhancement network to obtain an enhanced target road scene image.

[0237] Further, the acquisition module 1601 is specifically configured to perform image segmentation on the road scene image to be processed to obtain a plurality of image patches; perform embedding processing on the plurality of image patches to obtain preliminary embedding features of the plurality of image patches; perform linear mapping on the preliminary embedding features of the plurality of image patches to obtain target embedding features of the plurality of image patches; and obtain a feature map of the road scene image to be processed according to the target embedding features of the plurality of image patches.

[0238] Further, the encoding module 1602 is specifically configured that the multi-frequency domain feature encoding module includes: a frequency domain transformation guiding module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module; the frequency domain transformation guiding module is used to perform frequency domain separation on the feature map to obtain the low-frequency feature and the high-frequency feature of the road scene image to be processed; the low-frequency encoding module is used to encode the low-frequency feature to obtain a low-frequency encoded feature; the high-frequency encoding module is used to encode the high-frequency feature to obtain a high-frequency encoded feature; the first fusion module is used to fuse the low-frequency encoded feature and the high-frequency encoded feature to obtain a multi-frequency domain fusion feature.

[0239] Further, the encoding module 1602 is specifically configured that the low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer; the average pooling layer is used to perform average pooling processing on the low-frequency feature to obtain an average pooling feature; the global self-attention layer is used to perform global self-attention calculation on the average pooling feature to obtain a global self-attention feature; the upsampling layer is used to perform upsampling processing on the global self-attention feature to obtain a low-frequency encoded feature.

[0240] Further, the encoding module 1602 is specifically configured that the high-frequency encoding module includes: a window partitioning layer, a local window attention layer, and a connection layer; the window partitioning layer is used to partition the high-frequency feature to obtain a plurality of non-overlapping window features; the local window attention layer is used to perform attention calculation on each of the plurality of non-overlapping window features to obtain a plurality of window attention features; the connection layer is used to splice the plurality of window attention features to obtain a high-frequency encoded feature.

[0241] Further, the modeling module 1603 is specifically configured that the region modeling module includes: a global partitioning module and a second fusion module; the global partitioning module is used to perform partitioning calculation on the multi-frequency domain fusion feature to obtain the feature weight of the region to be enhanced; the second fusion module is used to fuse the feature of the region to be enhanced and the features of other regions according to the feature weight of the region to be enhanced to obtain a region fusion feature.

[0242] Further, the modeling module 1603 is specifically configured that the global partitioning module includes: a global distribution attention layer, a multi-head attention layer, and a first partitioning layer; the global distribution attention layer is used to perform global attention calculation on the multi-frequency domain fusion feature to obtain the density distribution probability of atmospheric particles in the road scene to be processed; the multi-head attention layer is used to perform multi-head modeling on the multi-frequency domain fusion feature respectively to obtain the multi-head features of the road scene to be processed; the first partitioning layer is used to perform partitioning calculation on the density distribution probability of atmospheric particles in the road scene to be processed and the multi-head features to obtain the feature weight of the region to be enhanced.

[0243] Further, the modeling module 1603 is specifically configured to obtain a comprehensive feature according to the density distribution probability and multi-head features of atmospheric particles in the road scene to be processed; perform non-linear fitting on the comprehensive feature to obtain the feature weight of the area to be enhanced.

[0244] Further, the extraction module 1604 is specifically configured to include: a local extraction branch, a global extraction branch, and a third fusion module; use the local extraction branch to perform local feature extraction on the region fusion feature to obtain local detailed texture features; use the global extraction branch to perform global feature extraction on the region fusion feature to obtain global context features; use the third fusion module to fuse the local detailed texture features and the global context features to obtain the target fusion feature.

[0245] Further, the extraction module 1604 is specifically configured to have the local extraction branch as a local convolutional branch; use the local convolutional branch to perform a convolution operation on the region fusion feature to obtain local detailed texture features; the global extraction branch is a global transmission network branch; use the global transmission network branch to perform frequency domain conversion on the region fusion feature and calculate the attention on the converted frequency domain feature to obtain global context features.

[0246] Figure 17 It is a schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device can be a device with computing and processing functions.

[0247] The electronic device includes: a processor 1701 and a storage medium 1702. The processor 1701 and the storage medium 1702 are connected through a bus.

[0248] The storage medium 1702 is used to store a program, and the processor 1701 calls the program stored in the storage medium 1702 to execute the above method embodiment. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0249] Optionally, the present invention further provides a storage medium including a program, which is used to execute the above method embodiment when executed by a processor. In several embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other forms.

[0250] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0251] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a hardware plus software functional unit.

[0252] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a storage medium. The above-mentioned software functional unit stored in a storage medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute some steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk, or an optical disc that can store program codes.

Claims

1. An image enhancement method, characterized in that, Applied to an image enhancement network, the method includes: Obtain a feature map of the road scene image to be processed; Use the multi-frequency domain feature encoding module in the image enhancement network to perform feature encoding on the feature map to obtain the multi-frequency domain fusion feature of the road scene image to be processed; Use the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion feature to obtain the region fusion feature of the region to be enhanced and other regions in the road scene image to be processed; Use the feature extraction module in the image enhancement network to perform feature extraction on the region fusion feature to obtain the target fusion feature of the road scene image to be processed; the target fusion feature includes: local detail texture features and global context features; Use the reconstruction decoding module in the image enhancement network to decode the target fusion feature to obtain the enhanced target road scene image.

2. The method according to claim 1, wherein The obtaining of the feature map of the road scene image to be processed includes: Perform image block division on the road scene image to be processed to obtain a plurality of image patches; Perform embedding processing on the plurality of image patches to obtain the preliminary embedding features of the plurality of image patches; Perform linear mapping on the preliminary embedding features of the plurality of image patches to obtain the target embedding features of the plurality of image patches; Obtain the feature map of the road scene image to be processed according to the target embedding features of the plurality of image patches.

3. The method according to claim 1, wherein The multi-frequency domain feature encoding module includes: a frequency domain transformation guiding module, a low-frequency encoding module, a high-frequency encoding module, and a first fusion module; the using of the multi-frequency domain feature encoding module in the image enhancement network to perform feature encoding on the feature map to obtain the multi-frequency domain fusion feature of the road scene image to be processed includes: Use the frequency domain transformation guiding module to perform frequency domain separation on the feature map to obtain the low-frequency feature and high-frequency feature of the road scene image to be processed; Use the low-frequency encoding module to encode the low-frequency feature to obtain a low-frequency encoded feature; Use the high-frequency encoding module to encode the high-frequency feature to obtain a high-frequency encoded feature; Use the first fusion module to fuse the low-frequency encoded feature and the high-frequency encoded feature to obtain the multi-frequency domain fusion feature.

4. The method according to claim 3, wherein The low-frequency encoding module includes: an average pooling layer, a global self-attention layer, and an upsampling layer; the using of the low-frequency encoding module to encode the low-frequency feature to obtain a low-frequency encoded feature includes: Use the average pooling layer to perform average pooling on the low-frequency feature to obtain an average pooling feature; Use the global self-attention layer to perform global self-attention calculation on the average pooling feature to obtain a global self-attention feature; Use the upsampling layer to perform upsampling on the global self-attention feature to obtain the low-frequency encoded feature.

5. The method according to claim 3, wherein The high-frequency encoding module includes: a window division layer, a local window attention layer, and a connection layer; the using of the high-frequency encoding module to encode the high-frequency feature to obtain a high-frequency encoded feature includes: Use the window division layer to divide the high-frequency feature to obtain a plurality of non-overlapping window features; Use the local window attention layer to calculate the attention for each of the multiple non-overlapping window features respectively, obtaining multiple window attention features; Use the concatenation layer to concatenate the multiple window attention features, obtaining the high-frequency encoded features.

6. The method according to claim 1, wherein The region modeling module includes: a global partitioning module and a second fusion module; Use the region modeling module in the image enhancement network to perform region modeling based on the multi-frequency domain fusion features, obtaining the region fusion features of the region to be enhanced and other regions in the road scene image to be processed, including: Use the global partitioning module to perform partitioning calculation on the multi-frequency domain fusion features, obtaining the feature weights of the region to be enhanced; Use the second fusion module to fuse the features of the region to be enhanced and the features of other regions according to the feature weights of the region to be enhanced, obtaining the region fusion features.

7. The method according to claim 6, characterized in that, The global partitioning module includes: a global distribution attention layer, a multi-head attention layer, and a first partitioning layer; Use the global partitioning module to perform partitioning calculation on the multi-frequency domain fusion features, obtaining the feature weights of the region to be enhanced, including: Use the global distribution attention layer to perform global attention calculation on the multi-frequency domain fusion features, obtaining the density distribution probability of atmospheric particles in the road scene to be processed; Use the multi-head attention layer to perform multi-head modeling on the multi-frequency domain fusion features respectively, obtaining the multi-head features of the road scene to be processed; Use the first partitioning layer to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed, obtaining the feature weights of the region to be enhanced.

8. The method according to claim 7, wherein The use of the first partitioning layer to perform partitioning calculation on the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed, obtaining the feature weights of the region to be enhanced, includes: Obtain a comprehensive feature according to the density distribution probability of atmospheric particles and the multi-head features in the road scene to be processed; Perform non-linear fitting according to the comprehensive feature, obtaining the feature weights of the region to be enhanced.

9. The method according to claim 1, characterized in that, The feature extraction module includes: a local extraction branch, a global extraction branch, and a third fusion module; Use the feature extraction module in the image enhancement network to perform feature extraction on the region fusion features, obtaining the target fusion features of the road scene image to be processed, including: Use the local extraction branch to perform local feature extraction on the region fusion features, obtaining the local detailed texture features; Use the global extraction branch to perform global feature extraction on the region fusion features, obtaining the global context features; Use the third fusion module to fuse the local detailed texture features and the global context features, obtaining the target fusion features.

10. The method according to claim 9, wherein The local extraction branch is a local convolutional branch; The use of the local extraction branch to perform local feature extraction on the region fusion features, obtaining the local detailed texture features, includes: Use the local convolutional branch to perform a convolutional operation on the region fusion features, obtaining the local detailed texture features; The global extraction branch is a global transmission network branch; the step of using the global extraction branch to perform global feature extraction on the region fusion feature to obtain the global context feature includes: Using the global transmission network branch to perform frequency domain conversion on the region fusion feature, and performing attention calculation on the converted frequency domain feature to obtain the global context feature.

Images