Image moire removing method and system based on channel attention and space attention

Abstract

The invention provides an image moire removing method and system based on channel attention and space attention, and the method comprises the steps: carrying out the standardization processing of a feature image of a to-be-processed image, and determining the feature image after the standardization processing; performing spatial information extraction processing on the standardized feature image to determine spatial information; performing channel attention feature extraction processing on the spatial information to determine channel attention features; carrying out nonlinear transformation on the channel attention features and the standardized feature images, and fusing the channel attention features and the standardized feature images by adopting a gating mechanism to determine fusion features; and carrying out moire removal processing on the to-be-processed image containing the fusion features, and determining an image subjected to moire removal. According to the moire removal method and device, the feature channel weight is dynamically adjusted in combination with channel attention and space information, key information is highlighted, the repair area is accurately positioned, waste of computing resources is reduced, and efficient and high-quality moire removal is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image demoiring method and system based on channel attention and spatial attention. Background Technology

[0002] Image inpainting is a complex and challenging field that aims to remove various imperfections from images, such as moiré patterns, scratches, and stains, to restore the original appearance and detailed features. Accurate extraction and restoration of detailed features are crucial in this process. In moiré removal, from a channel perspective, the feature information of different color channels has varying importance for image understanding and inpainting. For example, the green channel is generally resistant to moiré patterns, while the red and blue channels are more susceptible. Therefore, there is complementarity between channels, and the rational utilization of inter-channel information can enhance the model's ability to understand and represent image features. From a spatial perspective, the texture and structural details of an image often depend on the spatial information of local regions. Complex textures such as moiré patterns often exhibit local periodic or directional structures, and this structural information is crucial for accurate inpainting and removal. Simultaneously, spatial information also helps maintain the overall consistency and coherence of the image during the inpainting process.

[0003] The importance of channel attention mechanisms in image inpainting is increasingly evident. Channel attention mechanisms can automatically learn the importance of features in different channels and dynamically adjust the weight distribution between channels, thereby enhancing the model's ability to focus on key features. However, relying solely on channel attention mechanisms has limitations. The texture characteristics of moiré patterns often exhibit localized directional patterns, which require spatial information for accurate identification and removal. Therefore, introducing spatial attention mechanisms can compensate for the shortcomings of channel attention mechanisms, allowing for more accurate localization and restoration of detailed features in images by perceiving spatial information.

[0004] In patent "CN113723516B; A method, apparatus, device and medium for moiré pattern removal based on neural networks", a moiré pattern removal model is constructed using a pre-defined self-attention enhanced convolutional structure component moiré pattern removal layer, a first tone mapping layer, a second tone mapping layer, and a domain adaptive network. This model possesses both an attention mechanism and domain adaptive network attributes, focusing on useful features, adjusting network attention points, reducing computational load and parameter count, and improving accuracy. However, it only employs a channel attention mechanism and does not incorporate a spatial attention mechanism, thus failing to compensate for the shortcomings of the channel attention mechanism.

[0005] Existing attention mechanisms often face high computational costs in practical applications, with computational complexity quadratically increased. This is highly disadvantageous for high-resolution image inpainting tasks and scenarios requiring deployment on edge devices. Moreover, some existing methods fail to adequately balance the combination of channel attention and spatial attention, making it difficult to fully leverage the advantages of both.

[0006] Therefore, designing an efficient and effective attention mechanism that can fully utilize the complementarity of information between channels, capture spatial distribution features, and maintain low computational complexity has become an urgent problem to be solved in the field of image restoration. Summary of the Invention

[0007] To address one of the shortcomings of existing technologies, the purpose of this application is to provide an image demoiring method and system based on channel attention and spatial attention.

[0008] A first aspect of this application provides an image demoiring method based on channel attention and spatial attention, comprising:

[0009] The feature images of the image to be processed are standardized to determine the standardized feature images.

[0010] The standardized feature image is subjected to spatial information extraction processing to determine the spatial information.

[0011] The spatial information is subjected to channel attention feature extraction processing to determine the channel attention features;

[0012] The channel attention features and the standardized feature image are subjected to nonlinear transformation and fused using a gating mechanism to determine the fused features;

[0013] The image to be processed, which contains the fusion features, is subjected to demoiring processing to determine the image after demoiring.

[0014] Optionally, the step of performing spatial feature extraction processing on the standardized feature image to determine spatial information includes:

[0015] The standardized feature image is convolved using a preset convolution kernel to determine the first spatial information;

[0016] The standardized feature image is subjected to dilation convolution processing using a preset first dilation rate and the preset convolution kernel to determine the second spatial information;

[0017] The normalized feature image is subjected to dilation convolution processing using a preset second dilation rate and the preset convolution kernel to determine the third spatial information;

[0018] The first spatial information, the second spatial information, and the third spatial information are fused to determine the spatial information.

[0019] The spatial information and the original features of the standardized feature image are residually concatenated to determine new features of the standardized feature image.

[0020] Optionally, the step of performing channel attention feature extraction processing on the spatial information to determine the channel attention features includes:

[0021] The spatial information is subjected to spatial-channel projection to determine the query matrix, key matrix, and value matrix;

[0022] The channel attention weights are determined based on the query matrix and the key matrix.

[0023] The channel attention features are determined by performing a matrix multiplication operation between the channel attention weights and the value matrix.

[0024] Optionally, determining the channel attention weights based on the query matrix and the key matrix includes:

[0025] Based on the query matrix and the key matrix, determine the channel attention weight matrix based on the query and the key;

[0026] The channel attention weight matrix based on query and key is normalized to determine the channel attention weights.

[0027] Optionally, the step of performing a nonlinear transformation on the channel attention features and the standardized feature image, and then fusing them using a gating mechanism to determine the fused features, includes:

[0028] Element-wise addition is performed on the channel attention features and the standardized feature image to generate the first fused feature;

[0029] Spatial-channel projection is performed on the first fused feature to determine the first feature image and the second feature image;

[0030] The first feature image is activated using a preset nonlinear function to determine the gating weights;

[0031] The fused features are determined by performing element-wise multiplication on the gating weights and the second feature image.

[0032] Optionally, the step of spatial-channel projection of the first fused feature to determine the first feature image and the second feature image includes:

[0033] The first fused feature is channel-projected using a preset first convolution, and the first fused feature after channel projection is aggregated using a preset second convolution. The channel projection and aggregation operations are repeated twice to determine the first feature image and the second feature image.

[0034] Optionally, the step of performing demoiring processing on the image to be processed containing the fusion features to determine the demoired image includes:

[0035] The fused features are fused with the original image features of the image to be processed to determine new image features of the image to be processed;

[0036] Based on the repair information in the fusion features of the new image features of the image to be processed, the moiré region in the image to be processed is corrected, and the corrected features are determined.

[0037] The modified features are then subjected to dimensionality reduction processing to determine the dimensionality-reduced features.

[0038] The dimensionality-reduced features are mapped to the image space using a preset mapping function to generate a preliminary demoiré image.

[0039] The image with preliminarily demoired textures is post-processed to determine the image after demoired textures.

[0040] A second aspect of this application provides an image demoiring system based on channel attention and spatial attention, comprising:

[0041] The standardization processing module is used to standardize the feature images of the image to be processed and determine the standardized feature images.

[0042] The spatial information extraction module is used to extract spatial information from the standardized feature image to determine spatial information.

[0043] The channel attention feature extraction module is used to perform channel attention feature extraction processing on the spatial information to determine the channel attention features;

[0044] The fusion module is used to perform nonlinear transformation on the channel attention features and the standardized feature image, and to fuse them using a gating mechanism to determine the fused features;

[0045] The image demoiré module is used to perform demoiré processing on the image to be processed containing the fusion features, and to determine the image after demoiré processing.

[0046] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0047] A fourth aspect of this application provides an electronic device comprising:

[0048] A memory on which computer programs are stored;

[0049] A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0050] This application proposes an image moiré removal method based on channel attention and spatial attention. It introduces a spatial awareness mechanism to extract spatial information, concentrates computational resources to repair key areas, reduces computation in non-key areas, and improves the quality and efficiency of image repair. It also introduces a channel attention mechanism to dynamically adjust channel weights, improves feature representation capabilities, and combines the channel attention mechanism and spatial awareness mechanism to dynamically adjust channel weights, highlight key information, accurately locate repair areas, and achieve efficient and high-quality removal of moiré patterns from images.

[0051] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0052] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This is a flowchart illustrating an image demoiring method based on channel attention and spatial attention, according to an exemplary embodiment.

[0054] Figure 2 This is a schematic diagram illustrating a process for acquiring channel attention features according to an exemplary embodiment.

[0055] Figure 3 This is a schematic diagram illustrating a process for acquiring fused features using a gating mechanism according to an exemplary embodiment.

[0056] Figure 4 This is a schematic diagram illustrating the structure of an image demoiring system based on channel attention and spatial attention, according to an exemplary embodiment. Detailed Implementation

[0057] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0058] In existing technologies, using channel attention mechanisms alone to perform image demoiring has limitations. Moiré patterns exhibit localized directional textures, requiring spatial information for accurate identification and removal. However, existing attention mechanisms are computationally expensive, with a quadratic computational complexity. Furthermore, existing methods fail to adequately balance the combination of channel attention and spatial attention, making it difficult to leverage the advantages of both. To address these issues, this application provides an image demoiring method based on channel attention and spatial attention to resolve the aforementioned problems.

[0059] Figure 1 This is a flowchart illustrating an image demoiring method based on channel attention and spatial attention, according to an exemplary embodiment.

[0060] Reference Figure 1 As shown, in one embodiment of this application, an image demoiring method based on channel attention and spatial attention includes steps S11 to S15.

[0061] S11, standardize the feature image of the image to be processed, and determine the standardized feature image.

[0062] Specifically, the feature image of the image to be processed is represented as follows: Where h represents the height of the feature image, w represents the width of the feature image, and c represents the number of channels in the feature image.

[0063] S12 performs spatial information extraction processing on the standardized feature image to determine the spatial information.

[0064] Specifically, a preset residual dilated dense block (RDDB) can be used to extract spatial information with the maximum receptive field from the standardized feature image.

[0065] S13, perform channel attention feature extraction processing on spatial information to determine channel attention features.

[0066] S14 performs a nonlinear transformation on the channel attention features and the standardized feature image, and uses a gating mechanism to fuse them to determine the fused features.

[0067] S15, perform demoiring processing on the feature map of the image to be processed containing fusion features, and determine the image after demoiring.

[0068] Specifically, the feature map of the image to be processed, containing fused features, is represented as follows:

[0069] In the embodiments described above, the feature images of the image to be processed are standardized to unify the range of data values, thereby improving the stability and convergence speed of subsequent calculations. Spatial information extraction processing is performed on the standardized feature images, introducing a spatial awareness mechanism to focus on key regions where moiré patterns are concentrated in the image to be processed, and to concentrate computational resources to repair key regions, thereby improving the quality and efficiency of image restoration and reducing the waste of computational resources. Channel attention feature extraction processing is performed on the spatial information, introducing a channel attention mechanism to learn the importance of different channels, dynamically adjust channel weights, enhance feature expression capabilities, improve image restoration accuracy, and improve the overall image restoration quality. Nonlinear transformation is performed on the channel attention features and the standardized feature images, and a gating mechanism is used for fusion to combine the spatial awareness mechanism and the channel attention mechanism to obtain fused features. Then, the fused features are subjected to demoiré processing to achieve efficient and high-quality moiré removal.

[0070] To standardize the range of data values, in some specific embodiments of this application, for S11, the feature image of the image to be processed is standardized to determine the standardized feature image. This can be done by:

[0071] The feature images of the image to be processed can be standardized using normalization, Z-fractional standardization, or wavelet transform domain standardization to determine the standardized feature images.

[0072] For example, a normalization process is used to standardize the feature images of the image to be processed:

[0073] Scaling the pixel values ​​in a feature image to a preset range can be achieved through a simple linear transformation, calculated as follows:

[0074] X norm =(XX) min ) / (X max -X min )

[0075] Where X represents the original pixel value, X min X represents the minimum pixel value in the feature image. max X represents the maximum pixel value in the feature image. norm These are the normalized pixel values.

[0076] For example, Z-score normalization is used to normalize the feature image of the image to be processed:

[0077] Determine the zero mean and unit standard deviation of the pixel values ​​in the feature image, and then perform standardization based on the zero mean and unit standard deviation of the pixel values ​​in the feature image. The calculation formula is as follows:

[0078] X_standard=(X-μ) / σ

[0079] Where μ represents the zero mean of the pixel values ​​in the feature image, σ represents the standard deviation of the pixel values ​​in the feature image, and X_standard represents the standardized pixel values.

[0080] For example, the feature image of the image to be processed is standardized using a wavelet transform domain standardization method:

[0081] Wavelet transform is performed on the feature image to decompose it into sub-bands of different frequency bands. Then, normalization or Z-fractional standardization is performed on each sub-band. Finally, inverse wavelet transform is performed to obtain the standardized feature image.

[0082] In some specific embodiments of this application, there is an image to be processed with a size of 256×256×3, and its pixel value range is [0,255]:

[0083] First, the image to be processed is converted into a feature image, for example, a feature map of size 64×64×64 is obtained through a feature extraction network; then the feature map is normalized by dividing the pixel value of each channel by the maximum pixel value of that channel, and scaling the pixel value range of each channel to [0,1] to obtain a normalized feature image, which prepares for subsequent steps such as spatial information extraction and channel attention feature extraction.

[0084] The embodiments described above in this application standardize the feature images of the image to be processed to ensure that the numerical range of the input data is consistent, thereby improving the stability and convergence speed of subsequent calculations.

[0085] To introduce a spatial perception mechanism, in some specific embodiments of this application, for S12, the spatial information is extracted from the standardized feature image to determine the spatial information, which can be done using S121 to S125.

[0086] Specifically, the standardized feature image can be input into a preset Residual Dilated Dense Block (RDDB) for spatial information extraction to determine spatial information. The data format of the standardized feature image matches the input requirements of the preset Residual Dilated Dense Block (RDDB), including the number of channels and size.

[0087] If the data format of the standardized feature image does not match the preset input requirements of the Residual Dilated Dense Block (RDDB), zero-padding or cropping operations can be performed on the standardized feature image to meet the preset input channel number and size requirements of the Residual Dilated Dense Block (RDDB).

[0088] S121, a preset convolution kernel is used to perform convolution processing on the standardized feature image to determine the first spatial information.

[0089] Specifically, the preset convolution kernel is a 3×3 convolution kernel.

[0090] An activation function ReLU is connected after the pre-defined convolution kernel in this step to enhance the nonlinear representation capability of the pre-defined residual dilated dense block (RDDB) and improve the discriminative power of the first spatial information.

[0091] S122, using a preset first dilation rate and a preset convolution kernel, perform dilation convolution processing on the standardized feature image to determine the second spatial information.

[0092] S123, using a preset second dilation rate and a preset convolution kernel to perform dilation convolution processing on the standardized feature image to determine the third spatial information.

[0093] This application expands the receptive field and acquires spatial information at different scales by performing dilation convolution operations through steps S122 to S123, setting different dilation rates. The preset first dilation rate and the preset second dilation rate can be set to 1, 2, 4, etc. The 3×3 convolution kernel samples the feature image at a larger interval, thereby capturing a wider range of spatial context information. After each dilation convolution operation, the ReLU activation function is also used to introduce nonlinear characteristics, enhancing the nonlinear representation capability of the preset residual dilated dense block (RDDB).

[0094] S124, perform fusion processing on the first spatial information, the second spatial information, and the third spatial information to determine the spatial information.

[0095] Specifically, the fusion process can employ element-level addition or concatenation operations to fully utilize the multi-scale spatial information extracted by different convolution operations, integrate spatial features under different receptive fields, and form a richer and more comprehensive spatial information representation.

[0096] S125, perform residual connection between spatial information and the original features of the standardized feature image to determine new features of the standardized feature image.

[0097] Specifically, the new features of the standardized feature image are used as the output of the preset Residual Dilated Dense Block (RDDB) to update and enhance the original features of the image to be processed, while retaining the original features and introducing new spatial feature representations.

[0098] The design of residual connections helps to preserve the original feature information while introducing new spatial feature representations, thereby updating and enhancing features. This residual learning approach can alleviate the gradient vanishing problem during network training and improve the training efficiency and performance of the network.

[0099] In the above embodiments of this application, residual dilated dense blocks are used to extract spatial information from the standardized feature images. A spatial awareness mechanism is introduced, dilated convolution is used to increase the receptive field while maintaining a balance between the number of parameters and the amount of computation. Dense connections are used to fuse features from different layers, making full use of shallow representation information.

[0100] Figure 2 This is a schematic diagram illustrating a process for acquiring channel attention features according to an exemplary embodiment.

[0101] exist Figure 2 In this context, SCP stands for convolution operation, which includes both 1×1 convolution operations and 3×3 depthwise convolution operations.

[0102] Reference Figure 2 The diagram shows the introduction of a channel attention mechanism. For S13, channel attention features are extracted from spatial information to determine the channel attention features, which can be achieved using S131 to S133.

[0103] S131, Perform spatial-channel projection on the spatial information to determine the query matrix, key matrix, and value matrix.

[0104] Specifically, a 1×1 convolution is used to project the spatial information into a new channel space, thereby better capturing the feature patterns and interactions between channels. Then, a 3×3 depthwise convolution is used to aggregate the spatial information mapped to this new channel space. Through this space-channel projection using three 1×1 and 3×3 depthwise convolutions, the spatial information is mapped to the query Q, key K, and value V space, thus determining the query matrix. Key matrix and value matrix

[0105] S132, Determine the channel attention weights based on the query matrix and the key matrix.

[0106] Specifically, based on the query matrix and the key matrix, a channel attention weight matrix based on the query and key is determined.

[0107] The calculation formula is as follows:

[0108]

[0109] Among them, Attention(Q ' ,K ′ ) represents the channel attention weight, Q ′ Represents the query matrix, K ′ V represents the bond matrix. ′ Represents a value matrix.

[0110] The channel attention weight matrix based on query and key is normalized to determine the channel attention weights.

[0111] Specifically, the normalization process can employ the softmax function to normalize the query and key-based channel attention weight matrix into a probability distribution, which serves as the channel attention weight, representing the importance of each channel.

[0112] S133, perform matrix multiplication on the channel attention weights and the value matrix to determine the channel attention features.

[0113] Specifically, the channel attention weights are multiplied by the value matrix to obtain the channel attention features.

[0114] The embodiments described above introduce a channel attention mechanism in image restoration, which automatically learns the importance of different channels, dynamically adjusts the channel weights, learns and emphasizes the features of channels that are less affected by interference, enhances the understanding of image colors and the ability to express features, improves the restoration accuracy of complex moiré textures, and improves the quality and accuracy of image restoration, thereby solving the problems of insufficient differentiation of different channel features and limited restoration effects in traditional methods.

[0115] Figure 3 This is a schematic diagram illustrating a process for acquiring fused features using a gating mechanism according to an exemplary embodiment.

[0116] exist Figure 3 In this context, SCP stands for convolution operation, which includes both 1×1 convolution operations and 3×3 depthwise convolution operations.

[0117] Reference Figure 3 As shown, in order to combine spatial perception and channel attention, in some specific embodiments of this application, for S14, the channel attention features and the standardized feature image are subjected to nonlinear transformation and fused using a gating mechanism to determine the fused features, which can be implemented as S141 to S144.

[0118] S141, perform element-wise addition on the channel attention features and the standardized feature image to generate the first fused feature.

[0119] Specifically, the first fusion feature is represented as F ′ .

[0120] S142, Spatial-channel projection is performed on the first fused feature to determine the first feature image and the second feature image.

[0121] Specifically, a preset first convolution is used to perform channel projection on the first fused feature, and a preset second convolution is used to perform aggregation operation on the first fused feature after channel projection. The channel projection and aggregation operations are repeated twice to determine the first feature image and the second feature image.

[0122] The first convolution can be a 1×1 convolution, and the second convolution can be a 3×3 depth convolution.

[0123] A 1×1 convolution is used to perform channel projection on the first fused feature, mapping the first fused feature to another new channel space. Then, a 3×3 depth convolution is used to aggregate the first fused feature mapped to another new channel space. After two 1×1 convolutions and a 3×3 depth convolution, the space-channel projection is used to obtain the first feature image and the second feature image.

[0124] S143, the first feature image is activated using a preset nonlinear function to determine the gating weights.

[0125] Specifically, the preset nonlinear function can be the sigmoid function.

[0126] S144 performs element-wise multiplication on the gating weights and the second feature image to determine the fused features.

[0127] Specifically, during the element-wise multiplication operation between the gating weights and the second feature image, important information relevant to the current repair task is selected and identified as the fusion features.

[0128] The embodiments described above in this application, through nonlinear transformation and gating mechanisms, combined with channel attention features and the original features of the standardized feature image, further enhance feature representation capabilities and achieve effective fusion of spatial and channel information.

[0129] To achieve image demoiring processing, in some specific embodiments of this application, for S15, the image to be processed containing fusion features is subjected to demoiring processing to determine the image after demoiring, including: S151 to S155.

[0130] S151, the fusion features are fused with the original image features of the image to be processed to determine new image features of the image to be processed.

[0131] Specifically, by fusing the fusion features with the original image features of the image to be processed, the original details and structural information of the image to be processed are preserved, while the repair information in the fusion features is used to correct the moiré pattern region.

[0132] S152, Based on the repair information in the fusion features of the new image features of the image to be processed, the moiré region in the image to be processed is corrected, and the corrected features are determined.

[0133] S153, perform dimensionality reduction on the corrected features to determine the dimensionality-reduced features.

[0134] Specifically, by performing dimensionality reduction on the corrected features, computational complexity is reduced and redundant information is removed, while key image features and details are preserved. Dimensionality reduction can be achieved through feature selection or feature transformation techniques.

[0135] S154 uses a preset mapping function to map the dimensionality-reduced features to the image space, generating a preliminary demoiré image.

[0136] Specifically, the preset mapping function is represented as a mapping function from features to an image. When designing this mapping function, it is necessary to ensure that it can effectively convert information in the feature space into clear visual information in the image space, while minimizing distortion as much as possible.

[0137] S155, perform post-processing on the initially demoired image to determine the image after demoiring.

[0138] Specifically, post-processing may include, but is not limited to, color correction, contrast adjustment, and sharpening operations to enhance the visual effect of the image. The image after moiré removal can achieve the best state in terms of color fidelity, detail clarity, and overall visual quality. The image after moiré removal has no obvious moiré interference visually, and the image quality meets the predetermined standards and requirements.

[0139] The embodiments described above combine channel attention and spatial awareness, focusing on areas where moiré patterns are concentrated. The channel attention mechanism highlights channel features less affected by moiré patterns, while the spatial awareness mechanism accurately locates the texture structure of the moiré patterns. Through multi-dimensional feature extraction and enhancement capabilities, the inpainting network can more effectively remove moiré patterns and restore the original details and colors of the image.

[0140] This application provides an image demoiring method based on channel attention and spatial attention. It combines channel attention and spatial perception, leveraging the complementary information between channels and the advantages of spatial distribution characteristics to enhance feature representation capabilities. This allows for a more comprehensive and accurate understanding and processing of complex textures and color distortions in images, achieving efficient restoration. It solves the problem that a single attention mechanism cannot fully capture image features, promotes the development of image restoration technology, and has significant application effects in moiré removal tasks.

[0141] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0142] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0143] The image demoiring method based on channel attention and spatial attention provided in this application has achieved excellent removal results on mainstream moiré datasets. The following explanation is based on common experimental metrics for image inpainting, such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS).

[0144] Experimental results show that the image demoiring method based on channel attention and spatial attention provided in this application achieves a peak signal-to-noise ratio (PSNR) of 25.26, a structural similarity (SSIM) of 0.8518, and a learned perceptual image patch similarity (LPIPS) of 0.1253 on the FHDMi dataset.

[0145] Experimental results using other methods, such as ESDNet, on the FHDMi dataset show a peak signal-to-noise ratio (PSNR) of 24.50, a structural similarity (SSIM) of 0.8351, and a learning-aware patch similarity (LPIPS) of 0.1354.

[0146] The experimental results of the MopNet method on the FHDMi dataset are as follows: peak signal-to-noise ratio (PSNR) of 22.76, structural similarity (SSIM) of 0.7958, and learned perceptual patch similarity (LPIPS) of 0.1794.

[0147] The experimental results above show that neither the ESDNet method nor the MopNet method performs as well as the solution proposed in this patent.

[0148] In addition, the image demoiring method based on channel attention and spatial attention provided in this application also has significant advantages in experimental results on the UHDM dataset. Its experimental metrics are: peak signal-to-noise ratio (PSNR) of 22.22, structural similarity (SSIM) of 0.8018, and learned perceptual image patch similarity (LPIPS) of 0.2434.

[0149] The experimental results of ESNet on the UHDM dataset are as follows: Peak Signal-to-Noise Ratio (PSNR) of 22.12, SSIM of 0.7956, and LPIPS of 0.2551. The experimental results of MopNet on the UHDM dataset are as follows: Peak Signal-to-Noise Ratio (PSNR) of 19.49, SSIM of 0.7572, and LPIPS of 0.3857.

[0150] Therefore, the image demoiring method based on channel attention and spatial attention proposed in this application has good applicability in moiré removal.

[0151] Figure 4 This is a schematic diagram illustrating the structure of an image demoiring system based on channel attention and spatial attention, according to an exemplary embodiment.

[0152] Reference Figure 4 As shown in one embodiment of this application, an image demoiring system 100 based on channel attention and spatial attention is provided, including: a normalization processing module 110, a spatial information extraction module 120, a channel attention feature extraction module 130, a fusion module 140, and an image demoiring module 150.

[0153] The standardization processing module 110 is used to standardize the feature image of the image to be processed and determine the feature image after standardization.

[0154] The spatial information extraction module 120 is used to extract spatial information from the standardized feature image to determine spatial information.

[0155] The channel attention feature extraction module 130 is used to perform channel attention feature extraction processing on spatial information to determine channel attention features;

[0156] The fusion module 140 is used to perform nonlinear transformation on the channel attention features and the standardized feature image, and to fuse them using a gating mechanism to determine the fused features;

[0157] Image demoiré module 150 is used to perform demoiré processing on the image to be processed containing fusion features and to determine the image after demoiré processing.

[0158] In the embodiments described above, the standardization processing module 110 standardizes the feature images of the image to be processed to unify the data value range and improve the stability and convergence speed of subsequent calculations; the spatial information extraction module 120 performs spatial information extraction processing on the standardized feature images, introduces a spatial perception mechanism, focuses on key areas where moiré patterns are concentrated in the image to be processed, and concentrates computing resources to repair key areas, thereby improving the quality and efficiency of image repair and reducing the waste of computing resources; the channel attention feature extraction module 130 performs channel attention feature extraction processing on the spatial information, introduces a channel attention mechanism, learns the importance of different channels, dynamically adjusts channel weights, enhances feature expression capabilities, improves image repair accuracy, and improves the overall image repair quality; the fusion module 140 performs nonlinear transformation on the channel attention features and the standardized feature images, and uses a gating mechanism to fuse them, combining spatial perception and channel attention mechanisms to obtain fused features; and the image demoiré module 150 performs demoiré processing on the fused features to achieve efficient and high-quality moiré removal.

[0159] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0160] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0161] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0162] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0163] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0164] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0165] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0166] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An image demoiring method based on channel attention and spatial attention, characterized in that, include: The feature images of the image to be processed are standardized to determine the standardized feature images. The standardized feature image is subjected to spatial information extraction processing to determine the spatial information. The spatial information is subjected to channel attention feature extraction processing to determine the channel attention features; The channel attention features and the standardized feature image are subjected to nonlinear transformation and fused using a gating mechanism to determine the fused features; The image to be processed, which contains the fusion features, is subjected to demoiring processing to determine the image after demoiring.

2. The image demoiring method based on channel attention and spatial attention according to claim 1, characterized in that, The step of performing spatial feature extraction processing on the standardized feature image to determine spatial information includes: The standardized feature image is convolved using a preset convolution kernel to determine the first spatial information; The standardized feature image is subjected to dilation convolution processing using a preset first dilation rate and the preset convolution kernel to determine the second spatial information; The normalized feature image is subjected to dilation convolution processing using a preset second dilation rate and the preset convolution kernel to determine the third spatial information; The first spatial information, the second spatial information, and the third spatial information are fused to determine the spatial information. The spatial information and the original features of the standardized feature image are residually concatenated to determine new features of the standardized feature image.

3. The image demoiring method based on channel attention and spatial attention according to claim 1, characterized in that, The process of extracting channel attention features from the spatial information to determine the channel attention features includes: The spatial information is subjected to spatial-channel projection to determine the query matrix, key matrix, and value matrix; The channel attention weights are determined based on the query matrix and the key matrix. The channel attention features are determined by performing a matrix multiplication operation between the channel attention weights and the value matrix.

4. The image demoiring method based on channel attention and spatial attention according to claim 3, characterized in that, The step of determining the channel attention weights based on the query matrix and the key matrix includes: Based on the query matrix and the key matrix, determine the channel attention weight matrix based on the query and the key; The channel attention weight matrix based on query and key is normalized to determine the channel attention weights.

5. The image demoiring method based on channel attention and spatial attention according to claim 1, characterized in that, The process of performing a nonlinear transformation on the channel attention features and the standardized feature image, and then fusing them using a gating mechanism to determine the fused features, includes: Element-wise addition is performed on the channel attention features and the standardized feature image to generate the first fused feature; Spatial-channel projection is performed on the first fused feature to determine the first feature image and the second feature image; The first feature image is activated using a preset nonlinear function to determine the gating weights; The fused features are determined by performing element-wise multiplication on the gating weights and the second feature image.

6. The image demoiring method based on channel attention and spatial attention according to claim 5, characterized in that, The step of spatial-channel projection of the first fused feature to determine the first feature image and the second feature image includes: The first fused feature is channel-projected using a preset first convolution, and the first fused feature after channel projection is aggregated using a preset second convolution. The channel projection and aggregation operations are repeated twice to determine the first feature image and the second feature image.

7. The image demoiring method based on channel attention and spatial attention according to claim 1, characterized in that, The step of performing demoiring processing on the image to be processed containing the fusion features, and determining the image after demoiring, includes: The fused features are fused with the original image features of the image to be processed to determine new image features of the image to be processed; Based on the repair information in the fusion features of the new image features of the image to be processed, the moiré region in the image to be processed is corrected, and the corrected features are determined. The modified features are then subjected to dimensionality reduction processing to determine the dimensionality-reduced features. The dimensionality-reduced features are mapped to the image space using a preset mapping function to generate a preliminary demoiré image. The image with preliminarily demoired textures is post-processed to determine the image after demoired textures.

8. An image demoiring system based on channel attention and spatial attention, characterized in that, include: The standardization processing module is used to standardize the feature images of the image to be processed and determine the standardized feature images. The spatial information extraction module is used to extract spatial information from the standardized feature image to determine spatial information. The channel attention feature extraction module is used to perform channel attention feature extraction processing on the spatial information to determine the channel attention features; The fusion module is used to perform nonlinear transformation on the channel attention features and the standardized feature image, and to fuse them using a gating mechanism to determine the fused features; The image demoiré module is used to perform demoiré processing on the image to be processed containing the fusion features, and to determine the image after demoiré processing.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images