Image feature extraction, image noise reduction method and related device

Abstract

The application discloses an image feature extraction method and an image noise reduction method and related devices, and belongs to the technical field of image processing. The application can strengthen the connection between the local area and the global information of the features from the progressive angle of the large local receptive field and the global receptive field, so that the feature extraction method based on the self-attention mechanism can still capture the long-distance pixel dependence while reducing the calculation complexity, can well remove the image noise in the image noise reduction task based on the pixel level, and can maximize the retention of the detail information of the image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image feature extraction and image noise reduction method and related apparatus. Background Technology

[0002] High-resolution images preserve rich detail, providing a clearer and more intuitive visual experience. Applying high-resolution images to computer vision tasks such as object detection and segmentation yields better recognition results. However, during image acquisition and transmission, factors such as the inherent characteristics of the image sensor and the transmission medium introduce significant noise, compromising the image's detail. Noisy images generally perform poorly in computer vision tasks. Therefore, effectively removing noise while preserving as much detail as possible to obtain high-quality images is crucial. Summary of the Invention

[0003] This application provides an image feature extraction and image noise reduction method and related apparatus, which can solve the problems of related technologies. The technical solution is as follows:

[0004] On the one hand, an image feature extraction method is provided, the method comprising:

[0005] The three-dimensional matrix of the target image from which features are to be extracted is divided into multiple first window matrices, each of which corresponds to multiple pixels;

[0006] Based on the plurality of first window matrices, a self-attention mechanism is used to determine the in-window feature matrix, wherein the in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the plurality of first window matrices;

[0007] The feature matrix within the window is divided to obtain multiple second window matrices, each second window matrix corresponding to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other;

[0008] Based on the plurality of second window matrices, a self-attention mechanism is used to determine the inter-window feature matrix, wherein the inter-window feature matrix indicates the pixel correlation corresponding to different window matrices in the plurality of second window matrices;

[0009] The feature matrix of the target image is determined based on the three-dimensional matrix of the target image and the inter-window feature matrix.

[0010] On the other hand, an image denoising method is provided, the method comprising:

[0011] The target image to be denoised is input into a trained image denoising model, which includes a feature extraction module employing a self-attention mechanism.

[0012] The feature extraction module extracts features from the input matrix to obtain the output matrix, where the input matrix is ​​the matrix determined based on the target image and input to the feature extraction module.

[0013] The feature extraction process of the feature extraction module includes: dividing the input matrix into multiple first window matrices, each first window matrix corresponding to multiple pixels; determining an intra-window feature matrix based on the multiple first window matrices using a self-attention mechanism, the intra-window feature matrix indicating the pixel correlation corresponding to the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each second window matrix corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining an inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism, the inter-window feature matrix indicating the pixel correlation corresponding to different window matrices among the multiple second window matrices; and determining the output matrix based on the input matrix and the inter-window feature matrix.

[0014] The denoised target image output by the image denoising model is determined based on the output matrix.

[0015] On the other hand, an image feature extraction apparatus is provided, the apparatus comprising:

[0016] The first partitioning module is used to partition the three-dimensional matrix of the target image to be extracted into multiple first window matrices, each first window matrix corresponding to multiple pixels;

[0017] The in-window feature determination module is used to determine the in-window feature matrix based on the plurality of first window matrices using a self-attention mechanism. The in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the plurality of first window matrices.

[0018] The second partitioning module is used to partition the feature matrix within the window to obtain multiple second window matrices. Each second window matrix corresponds to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other.

[0019] The inter-window feature determination module is used to determine the inter-window feature matrix based on the plurality of second window matrices using a self-attention mechanism. The inter-window feature matrix indicates the pixel correlations corresponding to different window matrices in the plurality of second window matrices.

[0020] The image feature determination module is used to determine the feature matrix of the target image based on the three-dimensional matrix of the target image and the inter-window feature matrix.

[0021] On the other hand, an image noise reduction apparatus is provided, the apparatus comprising:

[0022] An image input module is used to input the target image to be denoised into a trained image denoising model, the image denoising model including a feature extraction module employing a self-attention mechanism;

[0023] The feature extraction module is used to extract features from the input matrix to obtain an output matrix, wherein the input matrix refers to the matrix input to the feature extraction module based on the target image;

[0024] The feature extraction process of the feature extraction module includes: dividing the input matrix into multiple first window matrices, each first window matrix corresponding to multiple pixels; determining an intra-window feature matrix based on the multiple first window matrices using a self-attention mechanism, the intra-window feature matrix indicating the pixel correlation corresponding to the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each second window matrix corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining an inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism, the inter-window feature matrix indicating the pixel correlation corresponding to different window matrices among the multiple second window matrices; and determining the output matrix based on the input matrix and the inter-window feature matrix.

[0025] An image output module is used to determine the denoised target image output by the image denoising model based on the output matrix.

[0026] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for executing the computer program stored in the memory to implement the steps of the method described in the first or second aspect above.

[0027] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of the method described in the first or second aspect above.

[0028] On the other hand, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the steps of the method described in the first or second aspect above.

[0029] The technical solution provided in this application can bring at least the following beneficial effects:

[0030] In this embodiment, multiple first window matrices are obtained by dividing the three-dimensional matrix of the target image, thereby determining the similarity of pixels within each first window matrix. This allows for the acquisition of a larger local receptive field, i.e., obtaining long-distance pixel dependencies, while also simplifying computational complexity. Furthermore, multiple second window matrices are obtained by dividing the feature matrix within each window, thereby determining the similarity of pixels between different second window matrices. This allows for a certain degree of acquisition of the global receptive field of the image, enabling sufficient information exchange between windows and further obtaining even longer-distance pixel dependencies. In other words, the method provided in this embodiment, in addition to reducing computational complexity, strengthens the connection between feature local regions and global information from the progressive perspective of larger local and global receptive fields. This ensures that the self-attention mechanism feature extraction method can still capture long-distance pixel dependencies, effectively removing image noise in pixel-level image denoising tasks while preserving image detail information to the maximum extent. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram illustrating the determination of a two-dimensional matrix QKV according to an embodiment of this application;

[0033] Figure 2 This is a schematic diagram illustrating the principle of a self-attention mechanism provided in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram illustrating the application of a self-attention mechanism to image feature extraction, as provided in an embodiment of this application.

[0035] Figure 4 This is a flowchart of an image feature extraction method provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram illustrating the determination of a feature matrix within a window, provided in an embodiment of this application.

[0037] Figure 6 This is a schematic diagram illustrating the determination of the inter-window feature matrix provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of an improved feature extraction module employing an attention mechanism, provided in an embodiment of this application.

[0039] Figure 8 This is a flowchart of an image noise reduction method provided in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of the structure of an image denoising model provided in an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the structure of an upsampling module and an downsampling module provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of the structure of an upsampling module provided in an embodiment of this application;

[0043] Figure 12 This is a schematic diagram illustrating the training process of an image denoising model provided in an embodiment of this application;

[0044] Figure 13 This is a schematic diagram of the structure of an image feature extraction device provided in an embodiment of this application;

[0045] Figure 14 This is a schematic diagram of the structure of an image noise reduction device provided in an embodiment of this application;

[0046] Figure 15 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0047] Figure 16 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0049] With the development of sensor and digital image processing technologies, acquiring high-resolution images has become more convenient and diverse, through methods such as smartphones, digital cameras, vehicle cameras, and surveillance cameras. High-resolution images can preserve rich details, providing a clearer and more intuitive visual presentation. Applying high-resolution images to computer vision tasks such as object detection and segmentation can achieve better recognition results. However, during image acquisition and transmission, factors such as the inherent characteristics of the image sensor, the transmission medium, and the recording equipment can introduce a significant amount of noise into the image, compromising its details. Noisy images often perform poorly in computer vision tasks. Therefore, effectively removing noise while preserving as much detail as possible to obtain high-quality images is crucial.

[0050] Current image denoising techniques mainly include traditional image denoising methods and deep learning-based image denoising methods. Traditional image denoising methods can be divided into three categories based on the signal domain: the first category is spatial domain methods, which mainly process pixels in the image spatial domain; the second category is frequency domain methods, which mainly process pixels in the image frequency domain; and the third category is transform domain methods, which mainly correct image coefficients in the transform domain and then perform inverse transform to obtain the final processed spatial domain image.

[0051] With the rapid development of processor computing power and deep learning theory, deep learning-based image denoising methods have become a hot topic as a novel image denoising technique. Compared with traditional image denoising methods, deep learning-based methods offer advantages such as clearer images and shorter processing times. Although deep learning-based image denoising methods can achieve better results, they still have some shortcomings. For example, feature extraction methods in deep learning networks struggle to achieve a good balance between capturing a large receptive field (the area mapped by the elements of the output matrix of each layer in the input matrix of that layer) and simplifying computational complexity. Since image denoising is a pixel-level visual task, it generally requires high semantic detail. This necessitates forming dense semantic context information in the image to obtain long-distance pixel dependencies, thereby achieving better denoising results. A crucial aspect of achieving this is relying on the feature extraction mechanism in the deep learning network to capture a sufficiently large receptive field. However, in the field of deep learning vision, popular image feature extraction methods, such as convolution and self-attention mechanisms, often introduce a large number of parameters and computational overhead while capturing a large receptive field.

[0052] Convolution computation utilizes a convolution kernel, i.e., a weight matrix, which progressively "scans" the input matrix. As the convolution kernel "slides," it calculates the product of the weight matrix and the scanned data matrix, then summarizes the results into a single output pixel. The convolution kernel repeats this process at all its traversed locations until the input matrix is ​​transformed into a new feature matrix. Convolutional kernels offer advantages such as parameter sharing and sparse connectivity. Convolutional neural networks built with kernels possess powerful feature learning and representation capabilities compared to traditional machine learning methods, achieving significant results in computer vision. However, a drawback is the limited receptive field, making it unable to effectively capture long-distance pixel dependencies. A larger receptive field can be achieved by increasing the kernel size or stacking small kernels. However, this introduces a large number of parameters, increasing network complexity and hindering deployment on mobile devices with limited memory and computing power for real-time image denoising. Alternatively, dilated convolution can be used to increase the receptive field without pooling loss, allowing each convolution's output to contain a wider range of image information. However, if the dilation coefficient is not set appropriately, it can cause a grid effect in the output features.

[0053] The Transformer model was initially applied in natural language processing, effectively addressing the limitations of RNN (Recurrent Neural Network) models in terms of limited memory length and inability to be parallelized. Recently, it has been pioneered for cross-domain applications in computer vision tasks with promising results. The core of the Transformer model is its self-attention mechanism, which excels at capturing the internal correlations of data or features. Compared to convolutional kernels, which obtain a local receptive field, the self-attention mechanism establishes global pixel dependencies by calculating the similarity between any two pixels, thus obtaining a global receptive field with fewer parameters. Therefore, neural network denoising methods based on self-attention often effectively remove noise while preserving image details to the maximum extent. However, a drawback is that the receptive field is the entire image size. Especially for large input matrices, the computational complexity increases quadratically with spatial resolution, and there is significant information redundancy. This makes it difficult to deploy on mobile devices with limited memory and computing power for high-resolution real-time image denoising and other visual tasks.

[0054] Since self-attention mechanisms have a significant advantage over convolutional kernels in acquiring large receptive fields of images, this application proposes an improved feature extraction method employing a self-attention mechanism. This method can acquire a larger image receptive field and capture long-distance dependencies between pixels, while significantly simplifying its computational complexity. Furthermore, using the improved feature extraction module employing a self-attention mechanism as a submodule, a lightweight, high-resolution image denoising network model based on an encoder and decoder structure is reconstructed.

[0055] To facilitate understanding, before providing a detailed explanation of the methods provided in the embodiments of this application, we will first introduce the self-attention mechanism and image feature extraction using the self-attention mechanism.

[0056] 1. Introduction to the self-attention mechanism

[0057] The core idea of ​​the self-attention mechanism is an addressing process, which mainly consists of the following three steps.

[0058] (1) Assume that the two-dimensional input matrix fed into the self-attention mechanism is X. The input matrix X is transformed linearly to obtain three different two-dimensional matrices Q, K and V.

[0059] Please refer to the following: Figure 1 Input matrix X and matrix W Q Multiply to obtain matrix Q, input matrix X and matrix W. K Multiply to obtain matrix K, input matrix X and matrix W. V Multiplying them yields matrix V. Where matrix W is... Q W K W V These are all learnable parameters, mainly used to improve the model's fitting ability.

[0060] (2) Calculate the similarity between matrices Q and K to obtain the similarity weight matrix. The similarity operation is usually performed using the dot product operation. Then, use the softmax function (normalization exponent) to normalize the similarity weight matrix to obtain the normalized matrix, which can represent the weight distribution of matrix V.

[0061] The dot product of matrix Q and matrix K can be understood as matrix Q multiplied by the transpose of matrix K. Step (2) can be represented by the following formula (1):

[0062]

[0063] In the above formula (1), S is the normalized matrix, and d k The number of columns in the input matrix X, divided by The reason is that the value after the dot product operation is large, causing the gradient to become very small after softmax. Therefore, dividing by... To perform scaling.

[0064] (3) Multiply the normalized matrix with matrix V to obtain the output matrix of the self-attention mechanism.

[0065] The multiplication of the normalized matrix with matrix V can be understood as a weighted sum of the weights and values ​​of matrix V. The above calculation process can be performed using the following formula (2) or... Figure 2 To express.

[0066]

[0067] Self-attention mechanisms can handle sequential data well and can achieve full dependency of sequential data, but the computational cost of self-attention mechanisms is very high.

[0068] 2. Introduction to Image Feature Extraction Using Self-Attention Mechanism

[0069] The essence of self-attention is to calculate the similarity between any two pixels in an image, thereby obtaining long-distance pixel dependencies, that is, the similarity between two pixels that are far apart. Applying it to image denoising can effectively remove noise while better preserving image details and highlighting edge information.

[0070] Since language data in natural language processing is sequential, while image data processed in computer vision is three-dimensional, a dimensionality reshaping method is needed to transform the three-dimensional matrix of the image into a two-dimensional matrix, thereby applying the self-attention mechanism to image feature extraction. Please refer to [reference needed]. Figure 3 The implementation process can be mainly divided into the following steps.

[0071] (1) Assume that the three-dimensional matrix of the image from which features are to be extracted is X H×W×C The 3D matrix has a height of H, a width of W, and a depth of C. Reshaping the 3D matrix yields a 2D matrix X. (HW)×C The two-dimensional matrix X after dimension reshaping (HW)×C respectively with two-dimensional matrix W Q C×C W K C×C W V C×C Performing the multiplication operation yields the corresponding two-dimensional matrix Q. (HW)×C K (HW)×C V (HW)×C .

[0072] Dimensional reshaping of the 3D matrix refers to concatenating the elements corresponding to all pixels in the 3D matrix to obtain a 2D matrix. The number of rows in this 2D matrix is ​​H×W, the number of pixels in the image, and the number of columns is the depth C. This depth C can also be referred to as the number of channels C.

[0073] During the dimensional reshaping process, the 3D matrix is ​​concatenated row by row. Specifically, the element value corresponding to the first pixel in the first row of the 3D matrix is ​​concatenated to the first row of the 2D matrix, the element value corresponding to the second pixel in the first row of the 3D matrix is ​​concatenated to the second row of the 2D matrix, and so on. After concatenating the element values ​​corresponding to the other pixels in the first row of the 3D matrix, the element values ​​corresponding to the pixels in the second row of the 3D matrix are then concatenated, until the element values ​​corresponding to all pixels are concatenated. Alternatively, the 3D matrix can also be concatenated column by column.

[0074] The above examples illustrate how to join rows or columns. In practical applications, other methods can also be used. The following section will use row-by-row joining as an example.

[0075] Wherein, the two-dimensional matrix W Q C×C W K C×C W V C×C All of these are predetermined parameter matrices, and the aforementioned two-dimensional matrix Q is determined. (HW)×C K (HW)×C V (HW)×C The method can be expressed by the following formula:

[0076]

[0077] According to the above formula (3), Q is generated. (HW)×C The computational cost is (HW) × C × C. Similarly, generating K... (HW)×C and V (HW)×C The computational cost for each case is (HW)×C×C, so the total computational cost for this process is:

[0078] Ω1=(HW×C×C)×3=3HWC 2 (4)

[0079] (2) Perform pairwise similarity calculation on the pixels in the image, i.e., dot product operation.

[0080] First, consider the two-dimensional matrix K. (HW)×C Transpose the matrix to obtain a two-dimensional matrix K. C×(HW) Then the two-dimensional matrix Q (HW)×C With two-dimensional matrix K C×(HW)Perform matrix multiplication, that is, calculate the similarity between any two pairs of pixels, to obtain the pixel similarity weight matrix A. (HW)×(HW) In other words, the pixel similarity weight matrix A (HW)×(HW) Each element in each row represents the similarity between one pixel and all other pixels, calculated using the following formula:

[0081] Q (HW)×C ×K C×(HW) =A (HW)×(HW) (5)

[0082] Next, the pixel similarity weight matrix A (HW)×(HW) Divide by d k =C. Then divide by Each row of the resulting matrix is ​​then subjected to softmax normalization to obtain the pixel similarity probability matrix M. (HW)×(HW) Pixel similarity probability matrix M (HW)×(HW) The values ​​in the i-th row represent the similarity probabilities of the i-th pixel with all pixels in sequence (including the i-th pixel itself), where i ∈ [1, H*W]. Dividing by is ignored here. Including the computational cost of softmax, the total computational cost of this process is:

[0083] Ω2=HW×C×HW=(HW) 2 C (6)

[0084] (3) The pixel similarity probability matrix M (HW)×(HW) With two-dimensional matrix V (HW)×C Perform matrix multiplication to obtain a two-dimensional matrix O. (HW)×C Finally, for the two-dimensional matrix O... (HW)×C Dimensional reshaping is performed to obtain the final output matrix F. H×W×C .

[0085] Wherein, the pixel similarity probability matrix M (HW)×(HW) With two-dimensional matrix V (HW)×C Matrix multiplication can be represented by the following formula:

[0086] M (HW)×(HW) ×V (HW)×C =O (HW)×C (7)

[0087] This process involves weighted summation of pixel similarity probability values ​​and corresponding pixel values ​​to obtain the output pixel value for each position. The computational complexity is as follows:

[0088] Ω3 = HW × HW × C = (HW) 2 C (8)

[0089] It should be noted that the dimension reshaping in step (3) is the reverse process of the dimension reshaping in step (1). That is, if the dimension reshaping in step (1) is performed row by row, step (3) is also the reverse process of step (1) row by row. If the dimension reshaping in step (1) is performed column by column, step (3) is also the reverse process of step (1) column by column.

[0090] The above three steps describe the computational process of applying the self-attention mechanism to image feature extraction, with a total computational cost of 3 HWC. 2 +2(HW) 2 C, the calculation formula is as follows:

[0091] ΩS=Ω1+Ω2+Ω3=3HWC 2 +(HW) 2 C+(HW) 2 C = 3HWC 2 +2(HW) 2 C (9)

[0092] As can be seen from the above process, the self-attention mechanism extracts features from the global pixels (i.e., HW pixels) of the image, obtaining a global receptive field, which is beneficial for capturing long-distance pixel dependencies. When applied to feature extraction of noisy images, it can achieve better denoising effects and preserve image details as much as possible. However, as shown in formula (9), the computational complexity of the self-attention mechanism increases quadratically with spatial resolution (H*W). For feature extraction of large-sized high-resolution images, the computational load is huge; and repeated calculation of similarity between any two pixels also causes a lot of information redundancy. Therefore, the self-attention mechanism cannot maintain a good balance between obtaining a large receptive field and simplifying computational complexity.

[0093] The image feature extraction method and image noise reduction method provided in the embodiments of this application will be described in detail below. The execution subject of the embodiments of this application can be a computer device, which can be a terminal or a server.

[0094] The terminal can be any electronic product that can interact with the user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as PC (Personal Computer), mobile phone, smartphone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), tablet computer, smart car system, smart TV, smart speaker, etc.

[0095] A server can be a standalone server, a server cluster or distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, or a cloud computing service center.

[0096] Those skilled in the art should understand that the above-described terminals and servers are merely examples. Other existing or future terminals or servers that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0097] It should be noted that the application scenarios and execution entities described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of execution entities, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0098] Please refer to Figure 4 , Figure 4 This is a flowchart of an image feature extraction method provided in an embodiment of this application. The method includes the following steps.

[0099] Step 401: Divide the three-dimensional matrix of the target image from which features are to be extracted to obtain multiple first window matrices, each first window matrix corresponding to multiple pixels.

[0100] In some embodiments, the three-dimensional matrix of the target image is divided according to a first partitioning size to obtain multiple first window matrices. The three-dimensional matrix of the target image includes three dimensions: height, width, and depth. The first partitioning size includes partitioning dimensions in the height direction and width direction. After dividing the three-dimensional matrix of the target image according to the first partitioning size, each resulting first window matrix is ​​also a three-dimensional matrix, and each first window matrix also includes three dimensions: height, width, and depth.

[0101] In some embodiments, the first division dimension includes the same division dimension in the height direction as in the width direction. In practical applications, the division dimensions in the height direction and the division dimensions in the width direction of the three-dimensional matrix may also be different.

[0102] Furthermore, the height of the three-dimensional matrix can be a multiple of the division dimensions in the height direction, and the width of the three-dimensional matrix can be a multiple of the division dimensions in the width direction. Of course, in some other embodiments, the height of the three-dimensional matrix may not be a multiple of the division dimensions in the height direction, and / or the width of the three-dimensional matrix may not be a multiple of the division dimensions in the width direction; this application does not limit this.

[0103] When the height and width of the 3D matrix are both multiples of the height-direction division size, each first window matrix obtained by dividing the 3D matrix of the target image according to the first division size has the same size. When the height and / or width of the 3D matrix are not multiples of the height-direction division size, after dividing the 3D matrix of the target image according to the first division size, some first window matrices may be smaller. In this case, these smaller first window matrices are padded with zeros to ensure that each padded first window matrix has the same size, facilitating subsequent calculations.

[0104] For example, please refer to Figure 5 Assume the three-dimensional matrix of the target image is X. H×W×C The 3D matrix has a height of H, a width of W, and a depth of C. The first partitioning dimension is (S, S). After partitioning the 3D matrix according to the first partitioning dimension, each resulting first window matrix has a size of S×S×C. The number of these multiple first window matrices is (HW / S). 2 In other words, after dividing the three-dimensional matrix according to the first partitioning size, we obtain (HW / S). 2 ) first window matrices R S×S×C .

[0105] Step 402: Based on the multiple first window matrices, a self-attention mechanism is used to determine the in-window feature matrix. The in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the multiple first window matrices.

[0106] In some embodiments, the in-window feature matrix can be determined by the following steps (1)-(5).

[0107] (1) For each first window matrix, the first window matrix is ​​divided into multiple block matrices. Each block matrix corresponds to L pixels, where L is an integer greater than 1 and less than the number of pixels corresponding to the first window matrix.

[0108] In other words, each first window matrix can be divided into multiple block matrices, and the number of pixels corresponding to each block matrix is ​​less than the number of pixels corresponding to each first window matrix. Based on the multiple block matrices corresponding to each first window matrix, the feature matrix corresponding to each first window matrix is ​​determined according to the following steps (2)-(4), thereby obtaining the feature matrix corresponding to each first window matrix.

[0109] In some embodiments, each first window matrix is ​​divided according to a second partitioning size to obtain multiple block matrices corresponding to each first window matrix, wherein the second partitioning size is smaller than the first partitioning size.

[0110] The second partitioning dimension includes a height partitioning dimension and a width partitioning dimension. The height partitioning dimension included in the second partitioning dimension is smaller than the height partitioning dimension included in the first partitioning dimension, and the width partitioning dimension included in the second partitioning dimension is smaller than the width partitioning dimension included in the first partitioning dimension. After dividing each first window matrix using the second partitioning dimension, each block matrix obtained is also a three-dimensional matrix, and each block matrix also includes three dimensions: height, width, and depth.

[0111] In some embodiments, the second division dimension includes the same division dimension in the height direction as the division dimension in the width direction. In practical applications, the second division dimension may include different division dimensions in the height direction and the width direction.

[0112] Furthermore, the height of the first window matrix can be a multiple of the height dimension included in the second partitioning size, and the width of the first window matrix can be a multiple of the width dimension included in the second partitioning size. Of course, in other embodiments, the height of the first window matrix may not be a multiple of the height dimension included in the second partitioning size, and / or, the width of the first window matrix may not be a multiple of the width dimension included in the second partitioning size.

[0113] When the height of the first window matrix is ​​a multiple of the height dimension included in the second partitioning size, and the width of the first window matrix is ​​a multiple of the width dimension included in the second partitioning size, each block matrix obtained by dividing the first window matrix according to the second partitioning size has the same size. When the height of the first window matrix is ​​not a multiple of the height dimension included in the second partitioning size, and / or the width of the first window matrix is ​​not a multiple of the width dimension included in the second partitioning size, after dividing the first window matrix according to the second partitioning size, some block matrices may have smaller sizes. In this case, these block matrices can be padded with zeros to ensure that each padded block matrix has the same size, facilitating subsequent calculations.

[0114] For example, please continue to refer to Figure 5 Assuming the second partitioning size is (P, P), the three-dimensional matrix is ​​partitioned according to the first partitioning size (S, S) to obtain (HW / S). 2 ) window matrices R S×S×C Then, for each window matrix R S×S×C According to the second partition size (P, P), the window matrix R is... S×S×C Divide into (S) 2 / P 2 ) block matrix U P×P×C . Figure 5 The window matrix R in the upper left corner S×S×C Let's take an example to illustrate this.

[0115] (2) Reorganize the elements at the same position in the multiple block matrices to obtain L reorganized matrices.

[0116] Since the multiple block matrices are of the same size, the elements at the same positions in these block matrices can be recombined into one matrix, resulting in L recombined matrices. Each recombined matrix is ​​also a three-dimensional matrix, and each recombined matrix includes three dimensions: height, width, and depth.

[0117] For example, please continue to refer to Figure 5 , Figure 5 The window matrix R in the upper left corner S×S×C After partitioning, multiple block matrices U are obtained. P ×P×C , to the multiple block matrices U P×P×C Rearranging elements at the same position in the matrix yields L rearranged matrices, where L equals P. 2 The recombination matrix can be denoted as B. (S / P)×(S / P)×C Similarly, after performing the same operation on all first window matrices, a total of (HW / S) is obtained. 2 )×P 2 A recombination matrix.

[0118] (3) Use a self-attention mechanism to determine the feature matrices corresponding to the L recombination matrices respectively.

[0119] In some embodiments, it can be in accordance with Figure 3 The self-attention mechanism shown determines the feature matrix corresponding to each of the L recombination matrices. That is, each recombination matrix is ​​used as... Figure 3 The input matrix of the self-attention mechanism shown is processed by... Figure 3 After processing using the method shown, Figure 3 The final output matrix of the self-attention mechanism shown is used as the feature matrix corresponding to each recombination matrix.

[0120] (4) Restore the element positions of the feature matrices corresponding to the L recombination matrices to obtain the feature matrix corresponding to the first window matrix.

[0121] After determining the feature matrices corresponding to the L recombination matrices, the element positions of the feature matrices corresponding to the L recombination matrices are restored according to the inverse process of recombination to obtain the feature matrices corresponding to the first window matrix.

[0122] (5) Combine the feature matrices corresponding to the multiple first window matrices to obtain the in-window feature matrix.

[0123] Following the same method described above, after processing the other first window matrices in the plurality of first window matrices, the feature matrix corresponding to each first window matrix in the plurality of first window matrices can be obtained. Then, the feature matrices corresponding to the plurality of first window matrices are combined according to the reverse process of the division in step 401 to obtain the in-window feature matrix.

[0124] For example, please continue to refer to Figure 5 After determining the feature matrix corresponding to each first window matrix through the self-attention mechanism, the feature matrices corresponding to multiple first window matrices are combined to obtain the in-window feature matrix Y. H×W×C The feature matrix within this window is also a three-dimensional matrix, and the number of pixels corresponding to the feature matrix within this window is the same as the number of pixels corresponding to the three-dimensional matrix of the target image, and they correspond one-to-one.

[0125] by Figure 5 For example, combining the above formula (9), the computational workload of steps 401-402 is as follows:

[0126]

[0127] Comparing formulas (9) and (10) above, it can be seen that the processing steps 401-402 significantly reduce the computational load of pixel feature extraction, and its computational complexity no longer increases quadratically with spatial resolution. After dividing the three-dimensional matrix of the target image into multiple larger first window matrices through step 401, feature extraction is performed on the same window matrix within these first window matrices through step 402. This retains the advantage of the self-attention mechanism in acquiring a large receptive field of the image and strengthens the connection between features within the window. Since adjacent pixels have a certain similarity, they do not need to perform similarity calculations. Therefore, when extracting features from the first window matrix through step 402, the first window matrix is ​​divided into multiple block matrices, and the elements at the same position in these multiple block matrices are recombined. This ensures that each recombined matrix includes elements from block matrices at different positions. That is, the same recombined matrix corresponds to multiple non-adjacent pixels. In this way, when extracting features from the recombined matrix through the self-attention mechanism, the similarity between non-adjacent pixels can be determined, avoiding the determination of the similarity between adjacent pixels, which greatly reduces the redundant information of the self-attention mechanism. In other words, by combining steps 401 and 402, the size of the input matrix fed into the self-attention mechanism can be greatly reduced (i.e., from H×W×C to (S / P)×(S / P)×C) while obtaining a large receptive field of the image, thus greatly reducing the computational load of the self-attention mechanism.

[0128] Since steps 401-402 above mainly determine the similarity of pixels within a window, the aim is to further obtain a large receptive field of the image and simplify computational complexity while obtaining long-distance pixel dependencies. However, performing feature extraction solely according to the above process limits the connections between pixels to their respective windows, failing to determine the connections between pixels across windows. Therefore, to further strengthen the pixel connections between windows, the inter-window feature matrix is ​​determined through steps 403-404.

[0129] Step 403: Divide the feature matrix within the window to obtain multiple second window matrices. Each second window matrix corresponds to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other.

[0130] In some embodiments, the feature matrix within the window is divided according to a first partitioning size to obtain multiple second window matrices.

[0131] Since the feature matrix within the window is also a three-dimensional matrix, including three dimensions: height, width, and depth, and the first partitioning dimension includes the partitioning dimension in the height direction and the partitioning dimension in the width direction, after partitioning the feature matrix within the window using the first partitioning dimension, each second window matrix obtained is also a three-dimensional matrix, and each second window matrix also includes three dimensions: height, width, and depth.

[0132] In some embodiments, the height of the feature matrix within the window can be a multiple of the height-direction division size included in the first division size, and the width of the feature matrix within the window can be a multiple of the width-direction division size included in the first division size. Of course, in other embodiments, the height of the feature matrix within the window may not be a multiple of the height-direction division size included in the first division size, and / or, the width of the feature matrix within the window may not be a multiple of the width-direction division size included in the first division size; this application does not limit this.

[0133] When the height of the feature matrix within the window is a multiple of the height-direction division size included in the first division size, and the width of the feature matrix within the window is a multiple of the width-direction division size included in the first division size, each second window matrix obtained by dividing the feature matrix within the window according to the first division size has the same size. When the height of the feature matrix within the window is not a multiple of the height-direction division size included in the first division size, and / or the width of the feature matrix within the window is not a multiple of the width-direction division size included in the first division size, after dividing the feature matrix within the window according to the first division size, there may be some second window matrices with smaller sizes. In this case, these second window matrices can be padded with zeros to ensure that each padded second window matrix has the same size, facilitating subsequent calculations.

[0134] Since the number of pixels corresponding to the feature matrix within the window is the same as the number of pixels corresponding to the three-dimensional matrix of the target image, and they correspond one-to-one, if zero padding is not required after dividing the three-dimensional matrix of the target image according to the first division size, then zero padding is also not required after dividing the feature matrix within the window according to the first division size. Conversely, if zero padding is required after dividing the three-dimensional matrix of the target image according to the first division size, then zero padding is also required after dividing the feature matrix within the window according to the first division size.

[0135] Since the number of pixels corresponding to the feature matrix within the window is the same as the number of pixels corresponding to the three-dimensional matrix of the target image, and they correspond one-to-one, the number of multiple first window matrices obtained by dividing the three-dimensional matrix of the target image according to the same division method is the same as the number of multiple second window matrices obtained by dividing the feature matrix within the window, and the pixels of the first window matrix and the second window matrix at the same position correspond.

[0136] For example, please refer to Figure 6 The feature matrix within the window is Y H×W×CThe first partitioning size is (S, S). After partitioning the feature matrix within the window according to the first partitioning size, the size of each second window matrix is ​​S×S×C, and the number of these multiple second window matrices is (HW / S). 2 In other words, after dividing the feature matrix within the window according to the first division size, (HW / S) is obtained. 2 ) second window matrix R S×S×C .

[0137] Step 404: Based on the multiple second window matrices, a self-attention mechanism is used to determine the inter-window feature matrix, which indicates the pixel correlation corresponding to different window matrices in the multiple second window matrices.

[0138] In some embodiments, the inter-window feature matrix can be determined by the following steps (1)-(3).

[0139] (1) Reorganize the elements at the same position in the multiple second window matrices to obtain N reorganized matrices, where N is the number of the multiple pixels mentioned above and N is an integer greater than 1.

[0140] Since the multiple second window matrices are of the same size, elements at the same position in these matrices can be recombined into a single matrix, resulting in N recombined matrices. Each recombined matrix is ​​also a three-dimensional matrix, and each recombined matrix includes three dimensions: height, width, and depth.

[0141] For example, please continue to refer to Figure 6 The resulting multiple window matrices R S×S×C Rearranging elements at the same position in the matrix yields N rearrangement matrices, where N equals S. 2 The recombination matrix can be denoted as B. (H / S)×(W / S)×C .

[0142] (2) The feature matrices corresponding to the N recombination matrices are determined by using a self-attention mechanism.

[0143] In some embodiments, it can be in accordance with Figure 3 The self-attention mechanism shown determines the feature matrix corresponding to each of the N recombination matrices. That is, each recombination matrix is ​​used as... Figure 3 The input matrix of the self-attention mechanism shown is processed by... Figure 3 After processing using the method shown, Figure 3 The final output matrix of the self-attention mechanism shown is used as the feature matrix corresponding to each recombination matrix.

[0144] (3) Restore the element positions of the feature matrices corresponding to the N recombination matrices to obtain the feature matrix corresponding to each second window matrix.

[0145] After determining the feature matrices corresponding to the N recombination matrices, the element positions of the feature matrices corresponding to the N recombination matrices are restored according to the inverse process of recombination to obtain the feature matrix corresponding to each second window matrix.

[0146] (4) Combine the feature matrices corresponding to the multiple second window matrices to obtain the inter-window feature matrix.

[0147] After determining the feature matrix corresponding to each of the multiple second window matrices, the feature matrices corresponding to the multiple second window matrices are combined according to the reverse process of division in step 403 to obtain the inter-window feature matrix.

[0148] For example, please continue to refer to Figure 6 After determining the feature matrix corresponding to each second window matrix through the self-attention mechanism, the feature matrices corresponding to multiple second window matrices are combined to obtain the inter-window feature matrix E. H×W×C The inter-window feature matrix is ​​also a three-dimensional matrix, and the number of pixels corresponding to the inter-window feature matrix is ​​the same as the number of pixels corresponding to the three-dimensional matrix of the target image, and they correspond one-to-one.

[0149] by Figure 6 For example, combining the above formula (9), the computational workload of steps 403-404 is as follows:

[0150]

[0151] Comparing formulas (9) and (11) above, it can be seen that the processing steps 403-404 significantly reduce the computational load of pixel feature extraction. Furthermore, in step 403, dividing the feature matrix within the window according to the same partitioning size as the three-dimensional matrix of the target image can reduce redundant information while fully determining the similarity of pixels between windows. This avoids the problem of redundant information caused by repeatedly determining the similarity of pixels within the window when the partitioning size of the feature matrix within the window is smaller than the partitioning size of the three-dimensional matrix of the target image. It also avoids the problem of some inter-window pixel similarities not being determined when the partitioning size of the feature matrix within the window is larger than the partitioning size of the three-dimensional matrix of the target image.

[0152] Determining the similarity of pixels between windows through steps 403-404 above can capture the global receptive field to a certain extent, allowing pixels to obtain dependencies over longer distances. Specifically, the larger the receptive field, the greater the distance of dependencies that pixels can obtain; conversely, the smaller the receptive field, the smaller the distance of dependencies that pixels can obtain.

[0153] Step 405: Determine the feature matrix of the target image based on the three-dimensional matrix of the target image and the inter-window feature matrix.

[0154] In some embodiments, the feature matrix of the target image is obtained by adding the three-dimensional matrix of the target image to the inter-window feature matrix.

[0155] The processes described in steps 401-402 can be implemented using a feature extraction module based on secondary block division, the processes described in steps 403-404 can be implemented using a feature extraction module based on primary block division, and the process described in step 405 can be implemented using a matrix addition module. For ease of description, the feature extraction module based on secondary block division is referred to as module A, and the feature extraction module based on primary block division is referred to as module B. Modules A, B, and the matrix addition module are cascaded to obtain module E. Module A is a feature extraction module based on information within a window, which obtains a larger local receptive field of the image, allowing for sufficient communication of contextual information within the local area, and enabling pixels to obtain longer-distance dependencies. Module B is a feature extraction module based on information between windows, which to some extent obtains a global receptive field of the image, allowing for sufficient communication of information between windows, and enabling pixels to obtain even longer-distance dependencies. Module E, aiming to obtain rich semantic context information and reduce computational complexity, leverages the strengths of modules A and B, complementing their weaknesses. It concatenates modules A and B, using the output of module A as the input of module B, and adds the 3D matrix of the target image to the output matrix of module B. This results in the final improved feature extraction module employing a self-attention mechanism. Figure 7 As shown. The total computational cost of the entire process is the sum of the computational costs of modules A and B, calculated using the following formula:

[0156] ΩE=ΩA+ΩB=6HWC 2 +2 (S / P) 2 HWC+2(HW / S) 2 C (12)

[0157] Assuming H = 1024, W = 1024, C = 16, S = 64, and P = 32, the method provided in the embodiments of this application is similar to... Figure 3 The difference in computational complexity between the methods shown is:

[0158]

[0159] TFLOPs (Tera floating-point operations) can be used to measure the complexity of algorithms and models. 1 TFLOPs represents one trillion (10^12) floating-point operations.

[0160] For feature extraction of high-resolution images, the difference in computational complexity between the two methods demonstrates that the improved feature extraction method in this application significantly simplifies computational complexity. In addition to reducing computational complexity, the E module strengthens the connection between local feature regions and global information from a progressive perspective of larger local and global receptive fields, enabling the improved self-attention mechanism feature extraction method to still capture long-distance pixel dependencies. Applying it to pixel-level image denoising tasks effectively removes image noise while preserving image detail information to the maximum extent.

[0161] In this embodiment, the three-dimensional matrix of the target image is divided into multiple first window matrices to determine the similarity of pixels within each first window matrix. This allows for the acquisition of a larger local receptive field, i.e., long-distance pixel dependencies, while also simplifying computational complexity. Furthermore, by dividing the feature matrix within a window into multiple second window matrices, the similarity of pixels between different second window matrices is determined. This allows for a certain degree of acquisition of the global receptive field of the image, enabling sufficient information exchange between windows and further obtaining even longer-distance pixel dependencies. In other words, the method provided in this embodiment, in addition to reducing computational complexity, strengthens the connection between local feature regions and global information from a progressive perspective of larger local and global receptive fields, ensuring that the self-attention mechanism feature extraction method can still capture long-distance pixel dependencies.

[0162] Please refer to Figure 8 , Figure 8 This is a flowchart of an image noise reduction method provided in an embodiment of this application. The method includes the following steps.

[0163] Step 801: Input the target image to be denoised into the trained image denoising model, which includes a feature extraction module employing a self-attention mechanism.

[0164] The target image to be denoised is the same as the above. Figure 4 In the embodiments, the target images for which features are to be extracted can be the same image or different images, and this application does not limit this.

[0165] The image denoising model includes a feature extraction module employing a self-attention mechanism, which can be an improved version of the self-attention mechanism feature extraction module described in this application embodiment. The structure of the image denoising model is as follows: Figure 9As shown, this image denoising model mainly includes an encoder unit, a cross-layer connection unit, and a decoder unit. The encoder unit is responsible for extracting features from the target image. The cross-layer connection unit is responsible for concatenating the shallow output features of the encoder unit and the corresponding deep output features of the decoder unit in the channel dimension to achieve multi-scale fusion. The decoder unit is responsible for decoding the encoded feature map output by the encoder.

[0166] The encoder unit includes multiple feature extraction modules with different partitioning parameters, and the decoder unit includes multiple feature extraction modules with different partitioning parameters. These partitioning parameters include the size used to partition the input matrix and the size used to partition the feature matrix within the window. In other words, the multiple feature extraction modules in the encoder unit use different sizes to partition the input matrix, and the multiple feature extraction modules in the decoder unit use different sizes to partition the input matrix and the feature matrix within the window.

[0167] For example, the encoder unit includes an upsampling module, an E-module 1, an downsampling module 1, an E-module 2, a downsampling module 2, and an E-module 3. The structures of the upsampling module and the downsampling module are as follows: Figure 10 As shown, both modules consist of cascaded convolutional layers with a kernel size of 3*3 and a kernel count of 16, and activation layers. These activation layers can be ReLU activation layers or activation layers of other functions. The difference lies in the convolutional kernels in the downsampling module, which operate with a stride of 2, meaning the height and width of the output features are halved after the input features pass through the downsampling module. Conversely, the convolutional kernels in the dimensionality-upgrading module operate with a stride of 1, meaning the height and width of the output features remain unchanged after the input features pass through the dimensionality-upgrading module. Module E is the improved feature extraction module using a self-attention mechanism proposed in this embodiment.

[0168] The decoder unit sequentially includes upsampling module 1, dimensionality reduction module 1, E module 4, upsampling module 2, dimensionality reduction module 2, E module 5, and dimensionality reduction module 3. Each upsampling module uses bilinear interpolation, such as... Figure 11 As shown. After the input features are processed by the upsampling module, the height and width of the output features are doubled; the dimensionality reduction module is as follows. Figure 10 As shown, dimensionality reduction module 1 and dimensionality reduction module 2 are both composed of convolutional layers with a kernel size of 1*1 and a number of 16 kernels, and activation layers cascaded together; dimensionality reduction module 3 is composed of convolutional layers with a kernel size of 1*1 and a number of 4 kernels, and activation layers cascaded together; module E is the improved feature extraction module using a self-attention mechanism proposed in the embodiments of this application.

[0169] In the image denoising model described above, the first partition size of the E module includes the same partition size in both the height and width directions, and the second partition size includes the same partition size in both the height and width directions. The first partition size is denoted as (S, S), and the second partition size is denoted as (P, P). The sizes of S and P in different E modules of this image denoising model are shown in Table 1 below.

[0170] Table 1

[0171] E module S P Module E1 64 32 Module E2 32 16 Module E3 16 8 Module E4 32 16 Module E5 64 32

[0172] The above Figures 9-11 The structure shown is an example, and other structures can be used in practical applications. Similarly, the dimensions in Table 1 above are an example, and other dimensions can be used in practical applications. This application does not limit the scope of the embodiments.

[0173] The image denoising model provided in this application embodiment may include at least one feature extraction module employing a self-attention mechanism. The following description will take one of the feature extraction modules as an example.

[0174] Step 802: The feature extraction module extracts features from the input matrix to obtain the output matrix. The input matrix refers to the matrix that is input to the feature extraction module based on the target image.

[0175] The feature extraction process of this feature extraction module includes: dividing the input matrix into multiple first window matrices, each corresponding to multiple pixels; determining intra-window feature matrices using a self-attention mechanism based on the multiple first window matrices, the intra-window feature matrix indicating the pixel correlation of the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining inter-window feature matrices using a self-attention mechanism based on the multiple second window matrices, the inter-window feature matrix indicating the pixel correlation of different window matrices among the multiple second window matrices; and determining the output matrix based on the input matrix and the inter-window feature matrix.

[0176] In some embodiments, determining the in-window feature matrix based on the plurality of first window matrices using a self-attention mechanism includes: for each first window matrix, dividing the first window matrix into a plurality of block matrices, each block matrix corresponding to G pixels, where G is an integer greater than 1 and less than the number of pixels corresponding to the first window matrix; recombining the elements at the same position in the plurality of block matrices to obtain G recombined matrices; determining the feature matrix corresponding to each of the G recombined matrices using a self-attention mechanism; restoring the element positions of the feature matrices corresponding to the G recombined matrices to obtain the feature matrix corresponding to the first window matrix; and combining the feature matrices corresponding to the plurality of first window matrices to obtain the in-window feature matrix.

[0177] In some embodiments, determining the inter-window feature matrix based on the plurality of second window matrices using a self-attention mechanism includes: recombining elements at the same position in the plurality of second window matrices to obtain T recombined matrices, where T is the number of pixels corresponding to the second window matrices and T is an integer greater than 1; determining the feature matrices corresponding to the T recombined matrices respectively using a self-attention mechanism; restoring the element positions of the feature matrices corresponding to the T recombined matrices to obtain the feature matrix corresponding to each second window matrix; and combining the feature matrices corresponding to the plurality of second window matrices to obtain the inter-window feature matrix.

[0178] In some embodiments, determining the output matrix based on the input matrix and the inter-window feature matrix includes: adding the input matrix to the inter-window feature matrix to obtain the output matrix.

[0179] It should be noted that the process of extracting features from the input matrix to obtain the output matrix using this feature extraction module is the same as described above. Figure 4 The process in the embodiment is similar; for detailed implementation details, please refer to the above. Figure 4 The relevant descriptions in the embodiments. Additionally, since the first and second division dimensions used in the embodiments of this application may differ from those described above... Figure 4 The embodiments use different sizes, and the number of pixels corresponding to each block matrix obtained in this embodiment is different from that described above. Figure 4 The number of pixels corresponding to each block matrix obtained in the embodiments may be different. The number of pixels corresponding to the second window matrix obtained in this embodiment may be different from the above. Figure 4 The number of pixels corresponding to the second window matrix obtained in the embodiments is different. Therefore, in this embodiment, G is used to represent the number of pixels corresponding to the block matrix, and T is used to represent the number of pixels corresponding to the second window matrix.

[0180] Step 803: Determine the denoised target image output by the image denoising model based on the output matrix.

[0181] The output matrix is ​​processed by other modules to finally obtain the denoised target image output by the image denoising model.

[0182] The aforementioned image denoising model refers to a trained image denoising model. In this embodiment, the image denoising model to be trained can also be trained to obtain a trained image denoising model. For example, a training dataset is obtained, which includes multiple sets of training samples. Each set of training samples includes a noisy sample image and a corresponding noise-free sample image. Based on these multiple sets of training samples, the image denoising model to be trained is trained to obtain a trained image denoising model.

[0183] In some embodiments, multiple frames of images of the same scene can be captured continuously using the same imaging device. The pixel values ​​at the same location in these multiple frames are then weighted and averaged to obtain a noise-free sample image. Then, one frame is randomly selected from these multiple frames as a noisy sample image, thus obtaining a set of training samples. Multiple sets of training samples can be determined in the same way.

[0184] In other embodiments, to enhance the richness of the training samples, after obtaining multiple sets of training samples using the above method, data augmentation can be performed on these multiple sets of training samples to increase the number of training samples. For example, operations such as rotation, mirroring, flipping, and cropping can be performed on each set of training samples to obtain more sets of training samples.

[0185] To improve the training speed of the model and adapt to the input image size of the image denoising model, the multiple sets of training samples can be cropped. For example, the multiple sets of training samples can be cropped into 512*512 image blocks. It should be noted that the above 512*512 size is just an example. In practical applications, the cropping size can be determined according to the computing power of the computer device.

[0186] In some embodiments, the process of training the image denoising model to be trained based on the multiple sets of training samples includes: selecting the i-th batch of training samples from the multiple sets of training samples; inputting the noisy sample images from the i-th batch of training samples into the (i-1)-th batch of updated image denoising models to obtain the predicted sample images output by the image denoising models; determining the loss value of the i-th batch based on the noise-free sample images and the corresponding predicted sample images from the i-th batch of training samples; performing backpropagation based on the loss value of the i-th batch to update the parameters of the (i-1)-th batch of updated image denoising models to obtain the i-th batch of updated image denoising models; if the i-th batch of updated image denoising models has not converged, then setting i = i + 1, and returning to selecting the i-th batch of training samples from the multiple sets of training samples; if the i-th batch of updated image denoising models has converged, then determining the i-th batch of updated image denoising models as trained image denoising models.

[0187] It should be noted that the training samples in the i-th batch are a subset of the training samples in the multiple training sets, and the training samples in different batches are different. When i=1, the image denoising model updated in the (i-1)-th batch is the image denoising model to be trained.

[0188] Furthermore, there are several ways to determine whether the image denoising model updated in the i-th batch has converged. For example, one could determine whether the difference between the loss value of the i-th batch and the loss value of the (i-1)-th batch is less than a minimum threshold. If the difference is less than the minimum threshold, the image denoising model updated in the i-th batch is considered converged; otherwise, it is considered non-converged. Alternatively, one could determine the number of updates to the image denoising model updated in the i-th batch. If the number of updates reaches a threshold, the image denoising model updated in the i-th batch is considered converged; otherwise, it is considered non-converged.

[0189] The aforementioned minimum threshold and number of attempts threshold are preset, and can be adjusted according to different needs under different circumstances.

[0190] In this embodiment, the loss value of the i-th batch can be determined using the L1 norm loss (L1_Loss) as the loss function. The calculation formula for this loss function is as follows:

[0191]

[0192] In the above formula, L1_Loss refers to the loss value of the i-th batch, h(x j ) refers to the predicted sample image corresponding to the j-th training sample in the i-th batch of training samples, y jIt refers to the noise-free sample image in the j-th training sample, and m is the number of training samples in the i-th batch.

[0193] In this embodiment, a batch of training samples is combined to determine a loss value, and the parameters of the image denoising model are adjusted using this loss value. Instead of determining a loss value for a set of training samples, this approach can improve the fit of the model training.

[0194] For example, 500 pairs of noisy and noise-free images, each with a size of 1080×1920, were captured using a camera. These 500 pairs of images were rotated, mirrored, and flipped to obtain 2500 pairs of 1080×1920 images. These 2500 pairs were then cropped, and the images were divided into blocks of 512×512 pixels with an interval of (200, 200), resulting in 60,000 pairs of 512×512 images, which were used as the training dataset. Each batch of training samples consisted of 16 pairs of 512×512 images. The noisy images were processed by the image denoising model to output a predicted image. This predicted image was then compared with the corresponding noise-free image to calculate a loss, which was then used for backpropagation to update the model parameters. The image denoising model was trained 500 times on the training dataset, meaning the entire training dataset was traversed 500 times. The model parameters were then saved, and the final model was identified as the trained image denoising model. The training process is as follows: Figure 12 As shown.

[0195] In this embodiment, multiple first window matrices are obtained by dividing the three-dimensional matrix of the target image, thereby determining the similarity of pixels within each first window matrix. This allows for the acquisition of a larger local receptive field, i.e., obtaining long-distance pixel dependencies, while also simplifying computational complexity. Furthermore, multiple second window matrices are obtained by dividing the feature matrix within each window, thereby determining the similarity of pixels between different second window matrices. This allows for a certain degree of acquisition of the global receptive field of the image, enabling sufficient information exchange between windows and further obtaining even longer-distance pixel dependencies. In other words, the method provided in this embodiment, in addition to reducing computational complexity, strengthens the connection between feature local regions and global information from the progressive perspective of larger local and global receptive fields. This ensures that the self-attention mechanism feature extraction method can still capture long-distance pixel dependencies, effectively removing image noise in pixel-level image denoising tasks while preserving image detail information to the maximum extent.

[0196] Figure 13 This is a schematic diagram of an image feature extraction device provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both. Please refer to... Figure 13The device includes: a first segmentation module 1301, an in-window feature determination module 1302, a second segmentation module 1303, an inter-window feature determination module 1304, and an image feature determination module 1305.

[0197] The first partitioning module 1301 is used to partition the three-dimensional matrix of the target image to be extracted into multiple first window matrices, each first window matrix corresponding to multiple pixels.

[0198] The in-window feature determination module 1302 is used to determine the in-window feature matrix based on the plurality of first window matrices using a self-attention mechanism. The in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the plurality of first window matrices.

[0199] The second partitioning module 1303 is used to partition the feature matrix within the window to obtain multiple second window matrices. Each second window matrix corresponds to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other.

[0200] The inter-window feature determination module 1304 is used to determine the inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism. The inter-window feature matrix indicates the pixel correlation corresponding to different window matrices in the multiple second window matrices.

[0201] The image feature determination module 1305 is used to determine the feature matrix of the target image based on the three-dimensional matrix of the target image and the inter-window feature matrix.

[0202] Optionally, the window feature determination module 1302 is specifically used for:

[0203] For each first window matrix, the first window matrix is ​​divided into multiple block matrices. Each block matrix corresponds to L pixels, where L is an integer greater than 1 and less than the number of pixels.

[0204] Rearrange the elements at the same position in multiple block matrices to obtain L rearranged matrices;

[0205] A self-attention mechanism is used to determine the feature matrices corresponding to the L recombination matrices;

[0206] The feature matrix corresponding to the first window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the L recombination matrices.

[0207] The feature matrices corresponding to multiple first window matrices are combined to obtain the in-window feature matrix.

[0208] Optionally, the window feature determination module 1304 is specifically used for:

[0209] Recombining elements at the same position in multiple second window matrices yields N recombination matrices, where N is the number of pixels and is an integer greater than 1.

[0210] A self-attention mechanism is used to determine the feature matrices corresponding to the N recombination matrices;

[0211] The feature matrix corresponding to each second window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the N recombination matrices.

[0212] The feature matrices corresponding to multiple second window matrices are combined to obtain the inter-window feature matrix.

[0213] Optionally, the image feature determination module 1305 is specifically used for:

[0214] The feature matrix of the target image is obtained by adding the three-dimensional matrix of the target image to the inter-window feature matrix.

[0215] In this embodiment, the three-dimensional matrix of the target image is divided into multiple first window matrices to determine the similarity of pixels within each first window matrix. This allows for the acquisition of a larger local receptive field, i.e., long-distance pixel dependencies, while also simplifying computational complexity. Furthermore, by dividing the feature matrix within a window into multiple second window matrices, the similarity of pixels between different second window matrices is determined. This allows for a certain degree of acquisition of the global receptive field of the image, enabling sufficient information exchange between windows and further obtaining even longer-distance pixel dependencies. In other words, the method provided in this embodiment, in addition to reducing computational complexity, strengthens the connection between local feature regions and global information from a progressive perspective of larger local and global receptive fields, ensuring that the self-attention mechanism feature extraction method can still capture long-distance pixel dependencies.

[0216] Figure 14 This is a schematic diagram of an image noise reduction device provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both. Please refer to... Figure 14 The device includes: an image input module 1401, a feature extraction module 1402, and an image output module 1403.

[0217] The image input module 1401 is used to input the target image to be denoised into the trained image denoising model. The image denoising model includes a feature extraction module that employs a self-attention mechanism.

[0218] The feature extraction module 1402 is used to extract features from the input matrix to obtain the output matrix. The input matrix refers to the matrix that is input to the feature extraction module based on the target image.

[0219] The feature extraction process of the feature extraction module includes: dividing the input matrix into multiple first window matrices, each corresponding to multiple pixels; determining the intra-window feature matrix using a self-attention mechanism based on the multiple first window matrices, the intra-window feature matrix indicating the pixel correlation corresponding to the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining the inter-window feature matrix using a self-attention mechanism based on the multiple second window matrices, the inter-window feature matrix indicating the pixel correlation corresponding to different window matrices among the multiple second window matrices; and determining the output matrix based on the input matrix and the inter-window feature matrix.

[0220] Image output module 1403 is used to determine the denoised target image output by the image denoising model based on the output matrix.

[0221] Optionally, the feature extraction module 1402 is specifically used for:

[0222] For each first window matrix, the first window matrix is ​​divided into multiple block matrices, each block matrix corresponding to G pixels, where G is an integer greater than 1 and less than the number of pixels;

[0223] Rearrange the elements at the same position in multiple block matrices to obtain G recombination matrices;

[0224] A self-attention mechanism is used to determine the feature matrices corresponding to the G recombination matrices;

[0225] The feature matrix corresponding to the first window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the G recombination matrices.

[0226] The feature matrices corresponding to multiple first window matrices are combined to obtain the in-window feature matrix.

[0227] Optionally, the feature extraction module 1402 is specifically used for:

[0228] Recombining elements at the same position in multiple second window matrices yields T recombination matrices, where T is the number of pixels and T is an integer greater than 1.

[0229] The feature matrices corresponding to the T recombination matrices are determined by using a self-attention mechanism;

[0230] The feature matrix corresponding to each second window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the T recombination matrices.

[0231] The feature matrices corresponding to multiple second window matrices are combined to obtain the inter-window feature matrix.

[0232] Optionally, the feature extraction module 1402 is specifically used to: add the input matrix to the inter-window feature matrix to obtain the output matrix.

[0233] Optionally, the device further includes:

[0234] The dataset acquisition module is used to acquire the training dataset, which includes multiple sets of training samples. Each set of training samples includes a noisy sample image and a corresponding noise-free sample image.

[0235] The model training module is used to train the image denoising model to be trained based on multiple sets of training samples, so as to obtain the trained image denoising model.

[0236] Optionally, the image denoising model includes an encoder unit, a cross-layer connection unit, and a decoder unit; the encoder unit includes multiple feature extraction modules with different partitioning parameters, and the decoder unit includes multiple feature extraction modules with different partitioning parameters, including the size for partitioning the input matrix and the size for partitioning the feature matrix within the window.

[0237] In this embodiment, multiple first window matrices are obtained by dividing the three-dimensional matrix of the target image, thereby determining the similarity of pixels within each first window matrix. This allows for the acquisition of a larger local receptive field, i.e., obtaining long-distance pixel dependencies, while also simplifying computational complexity. Furthermore, multiple second window matrices are obtained by dividing the feature matrix within each window, thereby determining the similarity of pixels between different second window matrices. This allows for a certain degree of acquisition of the global receptive field of the image, enabling sufficient information exchange between windows and further obtaining even longer-distance pixel dependencies. In other words, the method provided in this embodiment, in addition to reducing computational complexity, strengthens the connection between feature local regions and global information from the progressive perspective of larger local and global receptive fields. This ensures that the self-attention mechanism feature extraction method can still capture long-distance pixel dependencies, effectively removing image noise in pixel-level image denoising tasks while preserving image detail information to the maximum extent.

[0238] It should be noted that the devices provided in the above embodiments are only illustrated by the division of the above functional modules when implementing the corresponding functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the devices and methods provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0239] Figure 15This is a structural block diagram of a terminal 1500 provided in an embodiment of this application. The terminal 1500 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1500 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0240] Typically, terminal 1500 includes a processor 1501 and a memory 1502.

[0241] Processor 1501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0242] Memory 1502 includes one or more computer-readable storage media that are non-transitory. Memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1502 are used to store at least one instruction for execution by processor 1501 to implement the method provided in the method embodiments of this application.

[0243] In some embodiments, the terminal 1500 further includes a peripheral device interface 1503 and at least one peripheral device. The processor 1501, memory 1502, and peripheral device interface 1503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1503 via a bus, signal line, or circuit board. The peripheral device includes at least one of the following: a radio frequency circuit 1504, a touch display screen 1505, a camera 1506, an audio circuit 1507, a positioning component 1508, and a power supply 1509.

[0244] Peripheral interface 1503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1501 and memory 1502. In some embodiments, processor 1501, memory 1502 and peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1501, memory 1502 and peripheral interface 1503 can be implemented on separate chips or circuit boards.

[0245] The radio frequency (RF) circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1504 communicates with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1504 also includes circuitry related to NFC (Near Field Communication).

[0246] Display screen 1505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1501 for processing. In this case, display screen 1505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1505, which serves as the front panel of terminal 1500; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 1500 or in a folded design; in still other embodiments, display screen 1505 may be a flexible display screen, disposed on a curved or folded surface of terminal 1500. Furthermore, display screen 1505 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0247] The camera assembly 1506 is used to acquire images or videos. Optionally, the camera assembly 1506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1506 also includes a flash. The flash is a single-color temperature flash, or it can be a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, used for light compensation at different color temperatures.

[0248] The audio circuit 1507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1501 for processing, or input to the radio frequency circuit 1504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1500. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1507 also includes a headphone jack.

[0249] The positioning component 1508 is used to locate the current geographic location of the terminal 1500 in order to enable navigation or LBS (Location Based Service). The positioning component 1508 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0250] Power supply 1509 is used to power the various components in terminal 1500. Power supply 1509 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0251] Those skilled in the art will understand that Figure 15 The structure shown does not constitute a limitation on terminal 1500 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0252] Figure 16This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including random access memory (RAM) 1602 and read-only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the CPU 1601. The server 1600 also includes a basic input / output system (I / O system) 1606 that facilitates information transfer between various devices within the computer, and a mass storage device 1607 for storing the operating system 1613, application programs 1614, and other program modules 1615.

[0253] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609 for user input, such as a mouse or keyboard. Both the display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include the input / output controller 1610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0254] Mass storage device 1607 is connected to central processing unit 1601 via a mass storage controller (not shown) connected to system bus 1605. Mass storage device 1607 and its associated computer-readable media provide non-volatile storage for server 1600. That is, mass storage device 1607 may include computer-readable media (not shown) such as hard disk or CD-ROM drive.

[0255] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1604 and mass storage device 1607 described above can be collectively referred to as memory.

[0256] According to various embodiments of this application, server 1600 can also connect to and operate on a remote computer on a network, such as the Internet. That is, server 1600 can connect to network 1612 via network interface unit 1611 connected to system bus 1605, or it can use network interface unit 1611 to connect to other types of networks or remote computer systems (not shown). The aforementioned memory also includes one or more programs stored in the memory and configured to be executed by the CPU.

[0257] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the methods described in the above embodiments. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device. It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application may be a non-volatile storage medium; in other words, it may be a non-transient storage medium.

[0258] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the computer-readable storage medium described above. That is, in some embodiments, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the method described above.

[0259] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0260] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0261] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image feature extraction method, characterized in that, The method includes: The three-dimensional matrix of the target image from which features are to be extracted is divided into multiple first window matrices, each of which corresponds to multiple pixels; Based on the plurality of first window matrices, a self-attention mechanism is used to determine the in-window feature matrix, wherein the in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the plurality of first window matrices; The feature matrix within the window is divided to obtain multiple second window matrices, each second window matrix corresponding to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other; Based on the plurality of second window matrices, a self-attention mechanism is used to determine the inter-window feature matrix, wherein the inter-window feature matrix indicates the pixel correlation corresponding to different window matrices in the plurality of second window matrices; The feature matrix of the target image is obtained by adding the three-dimensional matrix of the target image and the inter-window feature matrix.

2. The method as described in claim 1, characterized in that, The step of determining the in-window feature matrix based on the plurality of first window matrices using a self-attention mechanism includes: For each first window matrix, the first window matrix is ​​divided into multiple block matrices, each block matrix corresponding to L pixels, where L is an integer greater than 1 and less than the number of pixels. The elements at the same position in the multiple block matrices are reorganized to obtain L reorganized matrices; The self-attention mechanism is used to determine the feature matrices corresponding to the L recombination matrices respectively; The feature matrix corresponding to the first window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the L recombination matrices. The feature matrices corresponding to the multiple first window matrices are combined to obtain the in-window feature matrix.

3. The method as described in claim 1 or 2, characterized in that, The step of determining the inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism includes: The elements at the same position in the plurality of second window matrices are recombined to obtain N recombined matrices, where N is the number of the plurality of pixels and N is an integer greater than 1; The self-attention mechanism is used to determine the feature matrices corresponding to the N recombination matrices respectively; The feature matrix corresponding to each second window matrix is ​​obtained by restoring the element positions of the feature matrices corresponding to the N recombination matrices; The feature matrices corresponding to the multiple second window matrices are combined to obtain the inter-window feature matrix.

4. An image denoising method, characterized in that, The method includes: The target image to be denoised is input into a trained image denoising model, which includes a feature extraction module employing a self-attention mechanism. The feature extraction module extracts features from the input matrix to obtain the output matrix, where the input matrix is ​​the matrix determined based on the target image and input to the feature extraction module. The feature extraction process of the feature extraction module includes: dividing the input matrix into multiple first window matrices, each first window matrix corresponding to multiple pixels; determining an intra-window feature matrix based on the multiple first window matrices using a self-attention mechanism, the intra-window feature matrix indicating the pixel correlation corresponding to the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each second window matrix corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining an inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism, the inter-window feature matrix indicating the pixel correlation corresponding to different window matrices among the multiple second window matrices; and adding the input matrix and the inter-window feature matrix to obtain the output matrix. The denoised target image output by the image denoising model is determined based on the output matrix.

5. The method as described in claim 4, characterized in that, The image denoising model includes an encoder unit, a cross-layer connection unit, and a decoder unit; The encoder unit includes multiple feature extraction modules with different partitioning parameters, and the decoder unit includes multiple feature extraction modules with different partitioning parameters. The partitioning parameters include the size for partitioning the input matrix and the size for partitioning the in-window feature matrix.

6. An image feature extraction device, characterized in that, The device includes: The first partitioning module is used to partition the three-dimensional matrix of the target image to be extracted into multiple first window matrices, each first window matrix corresponding to multiple pixels; The in-window feature determination module is used to determine the in-window feature matrix based on the plurality of first window matrices using a self-attention mechanism. The in-window feature matrix indicates the pixel correlation corresponding to the same window matrix among the plurality of first window matrices. The second partitioning module is used to partition the feature matrix within the window to obtain multiple second window matrices. Each second window matrix corresponds to multiple pixels, and the pixels of the first window matrix and the second window matrix at the same position correspond to each other. The inter-window feature determination module is used to determine the inter-window feature matrix based on the plurality of second window matrices using a self-attention mechanism. The inter-window feature matrix indicates the pixel correlations corresponding to different window matrices in the plurality of second window matrices. The image feature determination module is used to add the three-dimensional matrix of the target image and the inter-window feature matrix to obtain the feature matrix of the target image.

7. An image noise reduction device, characterized in that, The device includes: An image input module is used to input the target image to be denoised into a trained image denoising model, the image denoising model including a feature extraction module employing a self-attention mechanism; The feature extraction module is used to extract features from the input matrix to obtain an output matrix, wherein the input matrix refers to the matrix determined based on the target image and input to the feature extraction module; The feature extraction process of the feature extraction module includes: dividing the input matrix into multiple first window matrices, each first window matrix corresponding to multiple pixels; determining an intra-window feature matrix based on the multiple first window matrices using a self-attention mechanism, the intra-window feature matrix indicating the pixel correlation corresponding to the same window matrix among the multiple first window matrices; dividing the intra-window feature matrix into multiple second window matrices, each second window matrix corresponding to multiple pixels, with pixels in the first window matrix and second window matrix at the same position corresponding; determining an inter-window feature matrix based on the multiple second window matrices using a self-attention mechanism, the inter-window feature matrix indicating the pixel correlation corresponding to different window matrices among the multiple second window matrices; and adding the input matrix and the inter-window feature matrix to obtain the output matrix. An image output module is used to determine the denoised target image output by the image denoising model based on the output matrix.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program includes instructions that, when executed on a computer, cause the computer to perform the steps of the method according to any one of claims 1-5.

Images