Image processing method, device, electronic device and storage medium

Abstract

The present invention discloses an image processing method, device, electronic device, and storage medium. The method includes: obtaining an original image to be processed, downsampling the original image, and performing first feature extraction to obtain shallow features; inputting the shallow features into a multi-layer codec structure for codec processing to obtain first features; upsampling the first features and performing convolution processing to obtain a first image; performing a wavelet transform on the first image to obtain low-frequency components and high-frequency components; extracting low-frequency content from the low-frequency components to obtain a low-frequency output; enhancing high-frequency details from the high-frequency components to obtain a high-frequency output; and fusing the low-frequency output and the high-frequency output to obtain a target image. The embodiments of the present invention effectively improve the robust generalization capability of cross-resolution moiré removal tasks by combining a multi-layer codec structure with a neighboring cross-attention mechanism, and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art

[0002] The following methods exist for moiré removal in the existing art: Traditional moiré removal methods: Traditional pre-processing moiré removal methods improve image quality by improving hardware architecture. Nishioka et al. added a low-pass filter before the camera, but this design easily leads to oversmoothing and is costly. Therefore, post-processing moiré removal methods have been proposed. Sidorov et al. used Fourier spectral models to identify moiré. Yang and Liu et al. used low-rank sparse matrix factorization to remove moiré. However, these post-processing methods are typically based on specific physical models and have difficulty extracting complex moiré features. Deep learning-based moiré removal methods: Deep learning-based moiré removal methods have been widely studied in recent years. Sun et al. proposed the convolutional neural network (DMCNN) and created the first moiré dataset, TIP2018. He et al. proposed the feature-based MopNet for moiré removal. Yue et al. proposed AMNet, which utilizes addition and multiplication modules to improve image quality. However, these methods ignore the frequency domain complexity of moiré. In response, Liu et al. proposed WDNet, which operates in the wavelet domain and has two branches and a direction-aware module. Zheng et al. learned frequency-domain priors for texture removal and tone mapping. However, while these methods perform well at low resolutions, they struggle with high-resolution moiré removal. Consequently, He et al. proposed FHDe2Net for HD image moiré removal, and Yu et al. proposed ESDNet for ultra-HD images. However, these methods are all designed for a specific resolution and operate in a single domain or stage, which can further degrade image quality. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image processing method, apparatus, electronic device, and storage medium, which can efficiently perform image de-moiré processing.

[0004] In one aspect, an embodiment of the present invention provides an image processing method, comprising:

[0005] Obtaining the original image to be processed, downsampling the original image, and performing first feature extraction to obtain shallow features;

[0006] Input the shallow features into the multi-layer encoding and decoding structure for encoding and decoding to obtain the first features;

[0007] The multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through the neighboring cross attention mechanism;

[0008] Upsampling the first feature and performing convolution processing to obtain a first image;

[0009] Performing wavelet transform on the first image to obtain a low-frequency component and a high-frequency component;

[0010] According to the low-frequency component, a low-frequency output is obtained by extracting the low-frequency content; and according to the high-frequency component, a high-frequency output is obtained by enhancing the high-frequency details;

[0011] The low-frequency output and the high-frequency output are fused to obtain the target image.

[0012] Optionally, the original image is downsampled and feature extracted to obtain shallow features, including:

[0013] The original image is input into the sub-pixel downsampling layer for multiple downsampling, and then the convolution layer is used for feature extraction to obtain shallow features.

[0014] Optionally, the shallow features are input into a multi-layer encoding and decoding structure for encoding and decoding to obtain the first features, including:

[0015] Input shallow features into a multi-layer encoder for multiple encoding processes to obtain several encoding results; wherein the input data of each encoder in the multi-layer encoder includes the output result of the previous encoder, and the encoding result includes the output results of the encoders in each layer;

[0016] Each encoding result is transmitted through the neighboring cross attention mechanism, and the second feature of each encoding result is extracted at different scales. Then, the results of the second feature extraction of each encoding result are subjected to feature interaction to obtain the input features corresponding to each decoder.

[0017] The input feature is input into a multi-layer decoder for multiple decoding processes to obtain a first feature; wherein the input data of each decoder in the multi-layer decoder includes the input feature corresponding to each decoder and the output result of the previous decoder.

[0018] Optionally, the encoding process of each encoder and the decoding process of each decoder include:

[0019] Perform multi-scale downsampling on the input data, and then perform cascade processing on the downsampling results of each scale; the cascade processing includes residual dense convolution processing and residual void convolution processing;

[0020] The results of each cascade processing are hierarchically fused to obtain the output result.

[0021] Optionally, performing a wavelet transform on the first image to obtain a low-frequency component and a high-frequency component includes:

[0022] The first image is decomposed into low-frequency components and high-frequency components by two-dimensional Haar discrete wavelet transform.

[0023] Optionally, obtaining a low-frequency output by extracting low-frequency content according to the low-frequency component includes:

[0024] The low-frequency component is input into the low-frequency content extraction network, and color loss is introduced to obtain the low-frequency output;

[0025] Among them, the low-frequency content extraction network includes multiple layers of sequentially connected convolutional layers and activation functions, and the low-frequency content extraction network is a U-Net encoder-decoder architecture.

[0026] Optionally, according to the high-frequency component, high-frequency detail enhancement is performed to obtain a high-frequency output, including:

[0027] The high-frequency component is input into the high-frequency detail enhancement network, and detail loss is introduced to obtain high-frequency output;

[0028] Among them, the high-frequency detail enhancement network includes multiple layers of sequentially connected convolutional layers and activation functions.

[0029] In another aspect, an embodiment of the present invention provides an image processing apparatus, comprising:

[0030] The first module is used to obtain the original image to be processed, downsample the original image, and perform first feature extraction to obtain shallow features;

[0031] The second module is used to input the shallow features into the multi-layer encoding and decoding structure for encoding and decoding to obtain the first features;

[0032] The multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through the neighboring cross attention mechanism;

[0033] A third module is used to upsample the first feature and perform convolution processing to obtain a first image;

[0034] A fourth module is used to perform wavelet transform on the first image to obtain a low-frequency component and a high-frequency component;

[0035] The fifth module is configured to obtain a low-frequency output by extracting low-frequency content based on the low-frequency component; and to obtain a high-frequency output by enhancing high-frequency details based on the high-frequency component;

[0036] The sixth module is used to fuse the low-frequency output and the high-frequency output to obtain the target image.

[0037] In another aspect, an embodiment of the present invention provides an electronic device including a processor and a memory;

[0038] The memory is used to store programs;

[0039] The processor executes the program to implement the above method.

[0040] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0041] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0042] The embodiment of the present invention first obtains the original image to be processed, downsamples the original image, and extracts the first feature to obtain the shallow feature; the shallow feature is input into a multi-layer codec structure for codec processing to obtain the first feature; wherein the multi-layer codec structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through a neighboring cross-attention mechanism; the first feature is upsampled and convolution is performed to obtain the first image; the first image is subjected to wavelet transform to obtain low-frequency components and high-frequency components; based on the low-frequency components, a low-frequency output is obtained by extracting low-frequency content; and based on the high-frequency components, a high-frequency output is obtained by enhancing high-frequency details; the low-frequency output and the high-frequency output are fused to obtain the target image. The embodiment of the present invention effectively improves the robust generalization capability in cross-resolution de-moiré tasks by combining a multi-layer codec structure and a neighboring cross-attention mechanism. The embodiment of the present invention can efficiently perform de-moiré processing on images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic flow chart of an image processing method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the overall architecture principle of image processing provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the hierarchical structure of an encoder and a decoder provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structural principle of the cascaded RDB and RDiB modules provided in an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structural principle of the neighboring cross attention mechanism provided by an embodiment of the present invention;

[0049] FIG6( a ) is an example diagram of an output image of the MMDS stage provided by an embodiment of the present invention;

[0050] FIG6( b ) is an example diagram of an image of an LL component provided by an embodiment of the present invention;

[0051] FIG6( c ) is an example image diagram of the LH component provided by an embodiment of the present invention;

[0052] FIG6( d ) is an image example of the HL component provided by an embodiment of the present invention;

[0053] FIG6( e ) is an example image diagram of the HH component provided by an embodiment of the present invention;

[0054] FIG7( a ) is an example image of a low-frequency sub-image provided by an embodiment of the present invention;

[0055] FIG7( b ) is an example image of a high-frequency sub-image provided by an embodiment of the present invention;

[0056] FIG7( c ) is an example diagram of an image of a low-frequency output of an FSRS according to an embodiment of the present invention;

[0057] FIG7( d ) is an example diagram of an image of a high-frequency output of an FSRS according to an embodiment of the present invention;

[0058] Figure 8 A schematic structural diagram of an image processing device provided by an embodiment of the present invention;

[0059] Figure 9 A schematic diagram of the framework of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] On the one hand, if Figure 1 As shown, an embodiment of the present invention provides an image processing method, including:

[0062] S100, obtaining an original image to be processed, downsampling the original image, and performing a first feature extraction to obtain shallow features;

[0063] It should be noted that in some embodiments, downsampling the original image and performing feature extraction to obtain shallow features can include: inputting the original image into a sub-pixel downsampling layer for multiple downsampling, and then using a convolutional layer to perform feature extraction to obtain shallow features.

[0064] S200, inputting the shallow feature into the multi-layer encoding and decoding structure for encoding and decoding processing to obtain the first feature;

[0065] The multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through the neighboring cross attention mechanism;

[0066] It should be noted that, in some embodiments, step S200 may include: inputting shallow features into a multi-layer encoder for multiple encoding processes to obtain several encoding results; wherein, the input data of each encoder in the multi-layer encoder includes the output result of the previous encoder, and the encoding result includes the output results of the encoders of each layer; transmitting each encoding result through a neighboring cross-attention mechanism, and performing second feature extraction of different scales on each encoding result; then performing feature interaction on the results of the second feature extraction of each encoding result to obtain the input features corresponding to each decoder; inputting the input features into the multi-layer decoder for multiple decoding processes to obtain the first features; wherein, the input data of each decoder in the multi-layer decoder includes the input features corresponding to each decoder and the output result of the previous decoder.

[0067] In some embodiments, the encoding processing of each encoder and the decoding processing of each decoder may include: multi-scale downsampling of the input data, and then cascading processing of the downsampling results of each scale; wherein the cascade processing includes residual dense convolution processing and residual void convolution processing; and hierarchically fusing the results of each cascade processing to obtain the output result.

[0068] S300, upsampling the first feature and performing convolution processing to obtain a first image;

[0069] In some specific embodiments, the first feature may be input into a sub-pixel upsampling layer for multiple downsampling, and then convolution processing may be performed using a convolution layer to obtain a first image.

[0070] S400, performing wavelet transform on the first image to obtain low-frequency components and high-frequency components;

[0071] It should be noted that, in some embodiments, step S400 may include: decomposing the first image into low-frequency components and high-frequency components by using a two-dimensional Haar discrete wavelet transform.

[0072] S500, extracting low-frequency content based on the low-frequency component to obtain a low-frequency output; and enhancing high-frequency details based on the high-frequency component to obtain a high-frequency output;

[0073] It should be noted that in some embodiments, a low-frequency output is obtained by extracting low-frequency content based on the low-frequency component, including: inputting the low-frequency component into a low-frequency content extraction network, and introducing color loss to obtain a low-frequency output; wherein the low-frequency content extraction network includes multiple layers of sequentially connected convolutional layers and activation functions, and the low-frequency content extraction network is a U-Net codec architecture.

[0074] In some embodiments, high-frequency output is obtained by high-frequency detail enhancement based on the high-frequency component, including: inputting the high-frequency component into a high-frequency detail enhancement network, and introducing detail loss to obtain high-frequency output; wherein the high-frequency detail enhancement network includes multiple layers of convolutional layers and activation functions connected in sequence.

[0075] S600: Fusing the low-frequency output and the high-frequency output to obtain a target image.

[0076] To illustrate the technical principles of the embodiments of the present invention in detail, the present invention will be further described below with reference to the accompanying drawings and some specific embodiments. It should be understood that the following is an explanation of the present invention and cannot be regarded as a limitation of the present invention.

[0077] First, it should be noted that, in order to facilitate the understanding of the technical solution, the professional technologies that may be involved in the specific embodiments of the present invention are explained:

[0078] Attention Mechanism: Attention mechanisms are widely used in the field of deep learning. Hu et al. proposed the channel attention mechanism SENet. Woo et al. designed a channel and spatial attention mechanism CBAM. Li et al. proposed a global-to-local instantaneous representation attention mechanism GLTR. At the same time, the self-attention mechanism and the visual transformer (ViT) have shown great value in the field of deep learning. In general, an effective attention mechanism can guide the network to pay more attention to the required features. However, existing attention mechanisms pay little attention to the interactive features of different layers, which are often very important in multi-scale applications.

[0079] Application of Wavelet Transform in Deep Learning: Wavelet transform is a powerful analysis tool in the time and frequency domains. The discrete wavelet transform (DWT) can deconstruct images into low-frequency and high-frequency components. In recent years, wavelet transform technology has been widely used in deep learning, for example in image compression, image enhancement, and quality assessment. Moiré patterns span a large frequency domain and can be severely blended with the image content, while also compromising high-frequency details. Wavelet transforms can deconstruct such complex signals and process them in different frequency subbands.

[0080] In some specific embodiments, Figure 2 As shown, the method of this embodiment of the present invention consists of two stages: a multi-level multi-scale demoiré stage (MMDS) and a frequency domain separation and reconstruction stage (FSRS). First, shallow image features are extracted through sub-pixel convolution and 3×3 convolution layers. These features are then fed into a three-level codec structure, with each codec level operating at three scales. Furthermore, this embodiment of the present invention designs a novel plug-and-play neighboring cross-attention mechanism to transfer and fuse features from different levels. The output of the MMDS is then fed into the FSRS, which extracts the low-frequency and high-frequency components of the image through wavelet transform. These components are then fed into the LCEN and HDEN, respectively. Finally, the image is converted and output using the inverse wavelet transform (IDWT). Color loss, detail loss, and content loss are applied to the low-frequency output, high-frequency output, and output image, respectively, to supervise the learning of information related to the target task. The multi-scale demoiré stage (MMDS) and the frequency domain separation and reconstruction stage (FSRS) together constitute a dual-domain multi-level multi-scale demoiré system (DMMNet) based on deep learning.

[0081] The details of each part of DMMNet are introduced in the following sections.

[0082] 1. Multi-level and multi-scale moiré removal stage:

[0083] Moiré patterns are widely distributed in the spatial domain and are heavily integrated with image content. Therefore, a multi-level, multi-scale de-moiré stage (MMDS) is proposed. To expand the receptive field and reduce computational complexity, a sub-pixel convolutional layer is first introduced to downsample the image by a factor of two, quadrupling the number of channels compared to the original image. Convolutional layers are then used to extract shallow features. These shallow features are then fed into a multi-level, multi-scale encoder-decoder architecture: The encoder outputs have two destinations: 1) All encoder outputs are fed through neighboring cross-attention and then into the decoder; 2) The outputs of the first two encoder layers are downsampled and fed into the next encoder layer.

[0084] The decoder output also has two destinations: 1) The output of the first decoder layer serves as the final image output. 2) The remaining decoder layers are upsampled to the previous decoder layer. The following describes the encoder-decoder architecture and neighboring crisscross attention mechanism for this stage.

[0085] 1.1 Each codec level:

[0086] The structure of each codec layer is as follows Figure 3 As shown, the input features of each encoder are first downsampled to three scales: 1, 2, and 4 via bilinear interpolation.

[0087] Then, the embodiment of the present invention designs a cascade of residual dense convolution (RDB) and residual dilated convolution (RDiB) modules to extract features. Figure 4 As shown in the figure, the residual dense convolution consists of three layers of convolutional layers and the Leaky-ReLU activation function, which is used to concatenate the features obtained by convolution and perform a weighted residual connection with the original input; the residual dilated convolution consists of five layers of dilated convolution and the ReLU activation function, where the dilated convolution ratio is set to (1, 2, 3, 2, 1). Finally, the output of the residual dense convolution is added to the output of the residual dilated convolution to obtain the output of the cascade model.

[0088] The output of the cascaded model is then fed into a fusion layer consisting of a fully connected layer, a convolutional module, and a sigmoid activation function. Finally, the features at the three scales are summed to produce the output of each encoder-decoder layer.

[0089] 1.2 Neighboring Cross Attention Mechanism

[0090] Jump connections are often used in feature extraction, but this mechanism will transmit dissimilar semantic information and increase the amount of computation. Therefore, the embodiment of the present invention proposes a neighboring cross attention mechanism, such as Figure 5 As shown, it contains three branches and two steps: top branch, middle branch and bottom branch, as well as feature extraction step and feature interaction step.

[0091] Feature extraction step: This step first uses a 2D transposed convolution to upsample the feature maps of the middle and bottom branches to the size of the top feature. Subsequently, convolutional layers and fully connected layers are used to extract shallow features, and RDiB is used to extract deep features.

[0092] Feature interaction step: To better integrate and transmit features at different levels, the output of RDiB is fed into the cross-gating unit. The output of RDiB and the cross-gating unit are then multiplied together and passed through the Dropout layer to obtain the interactive feature output, which is then sent to the decoder for further processing.

[0093] 2. Frequency domain separation and reconstruction stage - FSRS

[0094] Moiré is widely distributed in the frequency domain and damages the high-frequency details of the image and causes color degradation. In addition, the output of MMDS still has a small amount of moiré residual, as shown in Figure 6(a). To solve these problems, the embodiment of the present invention proposes a frequency domain separation and reconstruction stage (FSRS). Figure 2 As shown, the output of MMDS is first decomposed into low-frequency and high-frequency components through a two-dimensional Haar discrete wavelet transform, as shown in Figures 6(b), 6(c), 6(d) and 6(e), which represent the low-frequency component (LL component) and various high-frequency components (LH component, HL component and HH component) respectively, where L and H represent low-pass and high-pass filters, respectively. The low-pass (L) filter captures the contours and surfaces of the image, while the high-pass (H) filter focuses on detail information such as edges and fine textures. The low-frequency component LL contains the overall surface of the image, while the components decomposed by the high-pass filter, namely LH, HL, and HH, contain finer details. Considering that the frequency domain sub-image contains different low-frequency and high-frequency feature information, the embodiment of the present invention designs two lightweight networks to extract them respectively.

[0095] Low-Frequency Content Extraction Network (LCEN): For low-frequency components, a U-Net-like codec architecture consisting of six convolutional layers is designed. Considering that the decomposed low-frequency sub-image pixels contain a large number of negative values, Leaky-ReLU is used as the activation function of the convolutional layer. Furthermore, the low-frequency sub-image contains relatively little texture detail (as shown in Figure 7(a)) but rich color information. Therefore, the embodiments of the present invention introduce color loss to better reconstruct color information.

[0096] High-Frequency Detail Enhancement Network (HDEN): High-frequency components contain less content and color information, but richer high-frequency texture details. Therefore, the present invention designs a high-frequency detail enhancement network to preserve the texture details of the image. The high-frequency sub-image contains finer details (as shown in Figure 7(b)). Considering that downsampling will lose the original image content, no downsampling operation is performed. Instead, four convolutional layers and the Leaky-ReLU activation function are used, and detail loss is introduced to enhance the network performance.

[0097] The (low-frequency and high-frequency) outputs of FSRS are shown in Figure 7(c) and Figure 7(d). Through the low-frequency content extraction network and the high-frequency detail enhancement network, the color information of the low-frequency components and the texture details of the high-frequency components can be well improved.

[0098] 3. Loss function:

[0099] In order to better utilize the different outputs of the network, the embodiment of the present invention designs three loss functions: content loss, color loss, and detail loss. Content loss is defined as:

[0100] L con =L1+L p (0.1)

[0101] Among them, L1 is the pixel level loss, L p is the perceptual loss. The color loss is defined as:

[0102] L color =-(I gl log(I l )+(1-I gl )log(1-I l )) (0.2)

[0103] Among them, I l is the low-frequency component output of FSRS, I gl is its true value. The detail loss is defined as:

[0104]

[0105] Among them, I h is the high frequency component output of FSRS, I gh is its true value.

[0106] Therefore the total loss function is defined as:

[0107] L total =λ con L con +λ color L color +λ detail L detail (0.4)

[0108] where λ con ,λ color ,λ detail As the relevant hyperparameter, set λ con =0.8,λ color =λ con =0.2.

[0109] In summary, the present invention innovatively proposes a dual-domain, multi-level, and multi-scale de-moiré system DMMNet based on deep learning, which uses publicly available low-resolution, high-resolution, and ultra-clear moiré images as data sets to improve the performance of the model system in cross-resolution de-moiré tasks. During the experiment, it was found that using only a single-stage, single-domain processing technology would cause slight moiré residues and color deviations in the model, while losing high-frequency details of the image. Considering that the wavelet transform can effectively deconstruct the low-frequency and high-frequency components of the image, the present invention proposes a dual-domain, two-stage processing architecture, which combines a multi-level, multi-scale codec model and a neighboring cross-attention mechanism to effectively improve the robust generalization ability of the model system in cross-resolution de-moiré tasks. The present invention uses publicly available low-resolution, high-resolution, and ultra-clear moiré images as data sets to improve the performance of the model system in cross-resolution de-moiré tasks. Experimental comparisons show that the PSNR performance of the embodiment of the present invention on a high-resolution dataset is 0.29dB higher than that of the current state-of-the-art method (top 1 PSNR is 24.88dB), reaching 25.17dB. Simultaneously, on a low-resolution dataset, the PSNR performance reaches the top 2, and the SSIM metric exceeds the top 1, reaching 0.9224. This demonstrates the effectiveness of our approach. Furthermore, no current method achieves good results across all resolutions, including low-resolution and high-resolution tasks. The DMMNet of the embodiment of the present invention is able to outperform current state-of-the-art methods in this cross-resolution scenario. The embodiment of the present invention can be applied in the following areas: 1. High-level vision tasks with moiré (when moiré is present in an image, high-level tasks such as object detection, image segmentation, and face recognition can also be applied to the method or DMMNet of the present invention). 2. The de-moiré technique can be applied to any scenario involving screen capture, and can also be extended to capturing high-frequency texture images and printing. 3. The cross-resolution solution strategy can be extended to other low-level vision tasks.

[0110] On the other hand, Figure 8As shown, an embodiment of the present invention provides an image processing device 700, including: a first module 710, used to obtain an original image to be processed, downsample the original image, and perform first feature extraction to obtain shallow features; a second module 720, used to input the shallow features into a multi-layer encoding and decoding structure for encoding and decoding processing to obtain first features; wherein the multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through a neighboring cross-attention mechanism; a third module 730, used to upsample the first features and perform convolution processing to obtain a first image; a fourth module 740, used to perform wavelet transform on the first image to obtain low-frequency components and high-frequency components; a fifth module 750, used to obtain low-frequency output by extracting low-frequency content based on the low-frequency components; and to obtain high-frequency output by enhancing high-frequency details based on the high-frequency components; a sixth module 760, used to fuse the low-frequency output and the high-frequency output to obtain a target image.

[0111] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0112] like Figure 9 As shown, another aspect of the embodiment of the present invention further provides an electronic device 800, including a processor 810 and a memory 820;

[0113] The memory 820 is used to store programs;

[0114] The processor 810 executes the program to implement the above method.

[0115] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0117] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0119] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0120] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art will be able to implement the present invention as set forth in the claims using ordinary skill without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0121] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0122] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution apparatus, device, or apparatus (e.g., a computer-based apparatus, a device including a processor, or other apparatus that can fetch instructions from and execute instructions on an instruction execution apparatus, device, or apparatus). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution apparatus, device, or apparatus.

[0123] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0124] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0125] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0126] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0127] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. An image processing method, characterized in that: include: Acquire an original image to be processed, downsample the original image, and perform first feature extraction to obtain shallow features; Inputting the shallow feature into a multi-layer encoding and decoding structure for encoding and decoding to obtain a first feature; The multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through a neighboring cross attention mechanism; The step of inputting the shallow feature into a multi-layer encoding and decoding structure for encoding and decoding to obtain the first feature includes: Inputting the shallow features into the multiple layers of encoders for multiple encoding processes to obtain a plurality of encoding results; wherein the input data of each encoder in the multiple layers of encoders includes the output result of the previous encoder, and the encoding results include the output results of the encoders in each layer; Transmitting each of the encoding results through the neighboring cross attention mechanism, performing second feature extraction of different scales on each of the encoding results; then performing feature interaction on the results of the second feature extraction of each of the encoding results to obtain input features corresponding to each of the decoders; Inputting the input feature into the multiple layers of decoders for multiple decoding processes to obtain a first feature; wherein the input data of each decoder in the multiple layers of decoders includes the input feature corresponding to each decoder and the output result of the previous decoder; The encoding process of each encoder and the decoding process of each decoder include: Perform multi-scale downsampling on the input data, and then perform cascade processing on the downsampling results of each scale respectively; wherein the cascade processing includes residual dense convolution processing and residual void convolution processing; Performing hierarchical fusion on the results of each cascade process to obtain an output result; Upsampling the first feature and performing convolution processing to obtain a first image; Performing wavelet transform on the first image to obtain a low-frequency component and a high-frequency component; The step of performing wavelet transform on the first image to obtain a low-frequency component and a high-frequency component includes: Decomposing the first image into the low-frequency component and the high-frequency component by two-dimensional Haar discrete wavelet transform; According to the low-frequency component, a low-frequency output is obtained by extracting low-frequency content; and according to the high-frequency component, a high-frequency output is obtained by enhancing high-frequency details; The step of extracting low-frequency content based on the low-frequency component to obtain a low-frequency output includes: Inputting the low-frequency component into a low-frequency content extraction network and introducing color loss to obtain a low-frequency output; The low-frequency content extraction network includes multiple layers of sequentially connected convolutional layers and activation functions, and the low-frequency content extraction network is a U-Net codec architecture; Obtaining a high-frequency output by enhancing high-frequency details according to the high-frequency component includes: Inputting the high-frequency component into a high-frequency detail enhancement network and introducing detail loss to obtain a high-frequency output; The high-frequency detail enhancement network includes multiple layers of sequentially connected convolutional layers and activation functions; The low-frequency output and the high-frequency output are fused to obtain a target image.

2. An image processing method according to claim 1, characterized in that: The downsampling of the original image and feature extraction to obtain shallow features include: The original image is input into the sub-pixel downsampling layer for multiple downsampling, and then the convolution layer is used to extract features to obtain shallow features.

3. An image processing device, characterized in that: include: The first module is used to obtain an original image to be processed, downsample the original image, and perform first feature extraction to obtain shallow features; The second module is used to input the shallow features into the multi-layer encoding and decoding structure for encoding and decoding to obtain the first features; The multi-layer encoding and decoding structure includes multiple layers of encoders and decoders with the same number of layers, and the encoders and decoders transmit data through a neighboring cross attention mechanism; The step of inputting the shallow feature into a multi-layer encoding and decoding structure for encoding and decoding to obtain the first feature includes: Inputting the shallow features into the multiple layers of encoders for multiple encoding processes to obtain a plurality of encoding results; wherein the input data of each encoder in the multiple layers of encoders includes the output result of the previous encoder, and the encoding results include the output results of the encoders in each layer; Transmitting each of the encoding results through the neighboring cross attention mechanism, performing second feature extraction of different scales on each of the encoding results; then performing feature interaction on the results of the second feature extraction of each of the encoding results to obtain input features corresponding to each of the decoders; Inputting the input feature into the multiple layers of decoders for multiple decoding processes to obtain a first feature; wherein the input data of each decoder in the multiple layers of decoders includes the input feature corresponding to each decoder and the output result of the previous decoder; The encoding process of each encoder and the decoding process of each decoder include: Perform multi-scale downsampling on the input data, and then perform cascade processing on the downsampling results of each scale respectively; wherein the cascade processing includes residual dense convolution processing and residual void convolution processing; Performing hierarchical fusion on the results of each cascade process to obtain an output result; a third module, configured to upsample the first feature and perform convolution processing to obtain a first image; A fourth module is configured to perform wavelet transform on the first image to obtain a low-frequency component and a high-frequency component; The step of performing wavelet transform on the first image to obtain a low-frequency component and a high-frequency component includes: Decomposing the first image into the low-frequency component and the high-frequency component by two-dimensional Haar discrete wavelet transform; A fifth module is configured to obtain a low-frequency output by extracting low-frequency content according to the low-frequency component; and to obtain a high-frequency output by enhancing high-frequency details according to the high-frequency component; The step of extracting low-frequency content based on the low-frequency component to obtain a low-frequency output includes: Inputting the low-frequency component into a low-frequency content extraction network and introducing color loss to obtain a low-frequency output; The low-frequency content extraction network includes multiple layers of sequentially connected convolutional layers and activation functions, and the low-frequency content extraction network is a U-Net codec architecture; Obtaining a high-frequency output by enhancing high-frequency details according to the high-frequency component includes: Inputting the high-frequency component into a high-frequency detail enhancement network and introducing detail loss to obtain a high-frequency output; The high-frequency detail enhancement network includes multiple layers of sequentially connected convolutional layers and activation functions; The sixth module is used to perform fusion processing on the low-frequency output and the high-frequency output to obtain a target image.

4. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 2.

Images